Pith. sign in

REVIEW 5 major objections 6 minor 62 references

Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approach

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A sparse mixture of projectors, with separate experts and routers per modality, lowers word error rates for LLM-based audio-visual speech recognition, especially for 1B and 3B LLMs.

desk verdict A useful AVSR engineering paper with a clear architecture and consistent gains, but the central claim that sparse routing (rather than extra parameters) explains DEDR's advantage is not established. read the letter →

arxiv 2505.14336 v2 pith:OG7YBAYQ submitted 2025-05-20 eess.AS cs.CVcs.MMcs.SD

Umberto Cappellazzo , Minsu Kim , Stavros Petridis , Daniele Falavigna , Alessio Brutti This is my paper · ORCID
classification eess.AScs.CVcs.MMcs.SD
keywords audio-visualspeechrecognitionlargelanguagemodelsmixtureofexpertssparseroutingprojectorscalingworderrorrateparameter-efficientfine-tuningnoiserobustness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Llama-SMoP, a way to attach audio and video encoders to a small large language model (1B and 3B parameters) for audio-visual speech recognition by replacing the usual linear projector with a sparse mixture of small expert projectors. The central claim is that scaling the projector, rather than the LLM itself, improves recognition accuracy, and the paper reports lower word error rates than single-projector and feature-fusion baselines across three Llama sizes on the LRS3 benchmark. Among three routing configurations, the one with separate experts and separate routers per modality (DEDR) performs best, with gains concentrated in the smaller models and in noisy audio conditions. If correct, this gives a cost-conscious way to deploy LLM-based AVSR with much smaller backbones.

What carries the argument

The central object is the Sparse Mixture of Projectors (SMoP) module, a Top-K sparsely-gated mixture-of-experts layer that replaces the single linear projector between frozen audio/video encoders and a frozen, LoRA-tuned LLM. For a token $x$, a linear router $R(x)=\mathrm{Top\text{-}K}(\mathrm{Softmax}(xW),K)$ selects the top $K$ of $N$ two-layer MLP experts, and the output is $z=\sum_i R(x)_i E_i(x)$. The three variants differ in whether the router and the expert pool are shared across modalities; DEDR uses disjoint routers and disjoint experts per modality, which the paper's ablations identify as the best configuration. The module is trained with load-balancing and router z-losses to prevent expert collapse.

What would settle it

Train a dense projector with the same total parameter count as the SMoP module on the same LRS3 setting and compare word error rates; if the dense version matches or beats SMoP, the claim that sparse routing is responsible collapses. A second check is to measure activated parameters, FLOPs, and latency per utterance for SMoP versus the dense projector; if inference cost is not lower, the efficiency claim fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that an audio-visual LLM's bottleneck is the linear projector, not just the LLM's size: replacing that projector with a sparsely-gated mixture of two-layer MLPs, and routing audio and video tokens through modality-specific experts, yields consistent word-error-rate improvements over the single-projector baseline and over intermediate-feature fusion baselines. The DEDR variant uses a separate router and separate pool of experts for each modality; the router selects Top-K=2 experts per token, and the output is the gated, weighted sum of the selected experts' outputs. The paper also shows that the benefit grows as the audio encoder and LLM become smaller, and that the router assigns tokens to experts almost evenly, with no need for more than four experts per pool.

Load-bearing premise

The paper assumes that the word-error-rate gains come from sparse routing rather than from simply having more projector parameters or from searching over configurations, and that the added inference cost is negligible; it provides no matched-parameter dense baseline or direct overhead measurement.

Editorial extensions

If this is right

  • DEDR SMoP improves AVSR word error rates across 1B, 3B, and 8B Llama backbones, with the largest relative gains at 1B and 3B.
  • SMoP also helps the single-modality ASR and VSR tasks, with a nearly 4-point WER reduction on VSR with Llama 3.2-1B.
  • SMoP remains effective under babble noise at low signal-to-noise ratios, where audio-only baselines degrade sharply.
  • Adding SMoP to audio while using DCI or MM-Fuser on video can beat the single-projector baseline, showing the module composes with other scaling methods.
  • Increasing the number of experts beyond four per pool does not improve performance, and the router activates experts roughly evenly.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If DEDR's gains hold, the same sparse-projector recipe could transfer to other frozen-encoder-plus-LLM multimodal systems, since the module is model-agnostic.
  • The pattern of larger gains at smaller LLM and encoder sizes suggests that in resource-constrained settings, parameter budget may be better spent on the projection layer than on LLM size.
  • A matched-parameter dense projector comparison would isolate whether sparse routing, rather than parameter count, drives the gains; the paper does not include that comparison.
  • Monitoring expert activation under different noise conditions could reveal whether the router learns noise-conditional specialization, which the paper does not analyze.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes Llama-SMoP, a sparse mixture-of-experts (MoE) projector for LLM-based audio-visual speech recognition (AVSR). It replaces the standard linear projector between frozen audio/video encoders and a LoRA-tuned LLM with a Top-K routed pool of two-layer MLP experts. Three router/expert configurations are compared: JEJR (shared experts, shared router), DEDR (modality-specific experts and routers), and JEDR (shared experts, modality-specific routers). On LRS3 AVSR, DEDR is reported to give the best WER across Llama 3.2-1B, Llama 3.2-3B, and Llama 3.1-8B, outperforming Llama-AVSR, DCI, and MM-Fuser. The paper also reports noise robustness, ASR/VSR experiments, expert-count ablations, and expert activation frequencies, and claims that the SMoP module adds negligible inference overhead.

Significance. If the central claim holds, replacing the projector with a sparsely-gated MoE module is a simple, model-agnostic way to improve LLM-based AVSR with smaller LLMs, which is practically relevant for resource-constrained deployment. The paper has useful breadth: it evaluates three LLM sizes and several Whisper encoders, includes a noise-robustness table, and provides ablations on expert count and activation balance. However, the attribution of the WER gains to sparse routing is not yet established, because no matched-parameter dense baseline or all-experts-active control is reported, no efficiency metrics are measured, and the winning configuration is selected on the same LRS3 benchmark used for reporting.

major comments (5)
  1. [§3.1, §3.2, Table 1] The central claim that sparse routing causes the WER gains is not established because no matched-parameter dense baseline is included. DEDR uses six expert MLPs (two pools of three) versus four shared experts in JEJR/JEDR, so DEDR has 1.5× the projector parameters of the shared-pool variants while the per-token activation count is the same (Top-2 per token). I do not find the concern that DEDR activates more experts per token than JEJR to be correct as described; both use Top-2 routing. The valid issue is total capacity: a dense two-layer MLP projector with matched total and activated parameters, plus an all-experts-active control, is needed to separate routing effects from capacity effects. In addition, the winning DEDR configuration is selected after comparing variants on LRS3 itself, so selection on the test benchmark is a live confound; configuration selection should be done on a validation split.
  2. [Abstract, §4] The abstract and conclusion state that SMoP incurs 'negligible additional parameter activation and computational overhead' and 'minimal additional inference overhead', but no FLOPs, per-token activated parameter counts, latency, or throughput measurements are reported anywhere in the paper. DEDR also doubles the number of routers compared with JEJR. Without efficiency measurements, the efficiency claim is unsupported; please add concrete measurements for each variant and baseline.
  3. [Table 1, Table 2, Figs. 3–4] No error bars or significance tests are provided. All results appear to be single runs, and several reported gaps are small: for example, in Table 1 the 3B row shows 2.29 for SMoP-3DEDR versus 2.60 for DCI, and the 8B row shows 0.96 versus 1.09 for Llama-AVSR. Run-to-run variation in fine-tuning could be of this size. Please report results over multiple seeds with standard deviations, and use a significance test or clear confidence intervals before claiming superiority.
  4. [§3.2, Table 1] The text says that SMoP 'can be effectively combined' with DCI and MM-Fuser, but the hybrid rows in Table 1 are inconsistent with this claim. For Llama 3.2-3B, SMoP-4-V + DCI-A gives 3.05 and SMoP-4-V + MM-Fuser-A gives 2.51, both worse than the Llama-AVSR baseline of 2.80. For Llama 3.1-8B, the hybrids give 2.79 and 1.86 versus the baseline 1.09. The claim needs to be corrected or explained, e.g., by analyzing why the combination only helps at the smallest scale.
  5. [§3.4, Fig. 3] Section 3.4 says the ASR and VSR experiments use 'a single modality-specific router and a single pool of experts', whereas the abstract and contributions say DEDR achieves superior performance on ASR, VSR, and AVSR. Please clarify whether the ASR/VSR experiments are the DEDR design restricted to one modality or a different SMoP variant; if the latter, the abstract overstates the ASR/VSR evidence. In addition, Figure 3 uses 'Llama SMoP' without defining whether this is SMoP-4, DEDR, or another variant.
minor comments (6)
  1. [After Abstract] The provenance statement 'Only non-Meta authors conducted any of the dataset preprocessing...' is not part of the scientific content and reads as a compliance notice. It should be moved to the acknowledgements section and integrated grammatically.
  2. [Figure 1 caption] The caption contains 'and represent whether the module is trained or kept frozen' but the symbols that should follow are missing from the rendered text; please restore them.
  3. [§3.5, Fig. 5] Please define the abbreviations in the Figure 5 legend (e.g., 'SMoP-4, ASR') in the caption, and state whether the activation frequencies are measured on a training or test split.
  4. [Table 2] The naming in Table 2 ('SMoP-4 A' and 'SMoP-3DEDR A-V') is inconsistent with Table 1 ('SMoP-4JEJR' and 'SMoP-3DEDR'); standardize the notation.
  5. [Eqs. (1)–(3)] After Top-K masking, the router weights are not renormalized in Eqs. (1)–(3). If renormalization is performed, state it; if not, clarify that the selected expert outputs are combined with weights that do not sum to one, since this affects the interpretation of the gating.
  6. [Fig. 4] The ASR WER scale (around 4) and the VSR WER scale (around 40) differ by an order of magnitude; consider using separate y-axes or a table for readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are empirical benchmark comparisons with no derivation-level reduction to inputs.

full rationale

Llama-SMoP is an empirical systems paper: the reported WER numbers in Tables 1 and 2 and Figures 3-5 are direct measurements on LRS3, not quantities derived from the model equations or from a fitted parameter renamed as a prediction. Equation (1)-(3) define the SMoP computation, and Equation (4)-(5) define the training loss; none of these equations presuppose the conclusion that DEDR outperforms JEJR or JEDR. The choice of DEDR after comparing the three configurations on LRS3 is a model-selection risk (the same test set is used for selection and reporting), but selection among empirically evaluated variants is not circular: the reported winner is not forced by construction, and a different LLM size could in principle favor a different configuration. The self-citations [33], [37], [55] are baselines and prior method references, not load-bearing uniqueness theorems or unverified premises on which the argument depends; [33] is a legitimate external baseline that the paper re-implements or adopts. The unsupported claim of 'negligible additional parameter activation and computational overhead' is a lack-of-evidence / experimental-control concern (no matched dense baseline, no FLOPs measurement), not a circularity: absence of attribution does not mean the result was assumed in the input. No quoted passage exhibits a quantity being defined in terms of the target result, a fitted value being reported as a prediction, or a cited theorem by the same authors that forbids alternatives. Therefore the circularity score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim depends on standard MoE machinery, on the adequacy of frozen Whisper and AV-HuBERT encoders with LoRA-tuned Llama, and on a set of architecture hyperparameters (top-2 routing, expert counts, compression rate, loss weights) that are set by hand or by ablation. No new physical entities are introduced; the SMoP module is an architectural component.

free parameters (5)
  • top-K experts = 2
    Number of experts activated per token; set to 2 for all configurations, affects routing sparsity.
  • expert pool sizes = DEDR: 3 per modality (6 total); JEJR/JEDR: 4 shared
    Total experts per pool chosen via ablation; DEDR results use 3 experts per modality, which is not shown to be optimal for AVSR specifically.
  • token compression rate = 3
    Number of consecutive tokens stacked along the hidden dimension for audio and video, reducing sequence length.
  • auxiliary loss weights = alpha_b=0.01, alpha_z=0.001
    Weights for load-balancing and router z-loss, taken from prior MoE work and set by hand.
  • learning rates = 1e-3 (ASR/AVSR), 5e-4 (VSR)
    Optimization hyperparameters; no sensitivity analysis reported.
assumptions (4)
  • standard math Top-K softmax gating as defined in Eq. (1)-(3) is a valid way to combine expert outputs.
    Standard sparsely-gated MoE formulation from Shazeer et al. [35].
  • domain assumption Frozen Whisper and AV-HuBERT encoders plus LoRA-tuned Llama provide a sufficient base for comparing projector designs.
    Only the projector and LoRA are trained, so encoder and LLM quality are assumed not to be the bottleneck; this is not tested.
  • domain assumption LRS3 test-set WER is a reliable measure and reported differences are meaningful without variance estimates.
    The paper reports single numbers with no error bars or significance tests.
  • ad hoc to paper The three SMoP router/expert configurations span a representative design space for multimodal projection.
    The variants are introduced by the authors without a formal argument that they cover the relevant design space.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approach." pith.science (2026). https://pith.science/paper/OG7YBAYQ

@misc{pith2026250514336,
  author       = {Pith},
  title        = {Pith review of: Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approach},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OG7YBAYQ}},
  note         = {Machine review of arXiv:2505.14336}
}
read the original abstract

Audio-Visual Speech Recognition (AVSR) enhances robustness in noisy environments by integrating visual cues. While recent advances integrate Large Language Models (LLMs) into AVSR, their high computational cost hinders deployment in resource-constrained settings. To address this, we propose Llama-SMoP, an efficient Multimodal LLM that employs a Sparse Mixture of Projectors (SMoP) module to scale model capacity without increasing inference costs. By incorporating sparsely-gated mixture-of-experts (MoE) projectors, Llama-SMoP enables the use of smaller LLMs while maintaining strong performance. We explore three SMoP configurations and show that Llama-SMoP DEDR (Disjoint-Experts, Disjoint-Routers), which uses modality-specific routers and experts, achieves superior performance on ASR, VSR, and AVSR tasks. Ablation studies confirm its effectiveness in expert activation, scalability, and noise robustness.

Figures

Figures reproduced from arXiv: 2505.14336 by the authors.

Figure 1
Figure 1. Illustration of the overall framework of the proposed Llama-SMoP model, where audio and video tokens are embed￾ded using a sparsely-gated mixture-of-experts scheme. and represent whether the module is trained or kept frozen. a significant challenge: they require a large number of param￾eters. Recent works, such as Llama-AVSR [33] and Llama￾MTSK [34], have shown that larger LLMs generally achieve better speech recogn… view at source ↗
Figure 2
Figure 2. Detailed illustration of the three proposed SMoP configurations. (a) Joint-Experts, Joint-Router (JEJR) uses one multimodal router and one pool of expert for embedding audio-visual representations. (b) Disjoint-Experts, Disjoint-Routers (DEDR) uses modality￾specific routers and experts for embedding modality-specific representations. (c) Joint-Experts, Disjoint-Routers (JEDR) uses modality￾specific routers and one s… view at source ↗
Figure 3
Figure 3. (Left). ASR results for Llama-SMoP using different-size Whisper models with Llama 3.2-1B. (Middle). ASR results for Llama-SMoP using different-size Whisper models with Llama 3.2-3B. (Right). VSR results for Llama-SMoP with Llama 3.2-1B/3.2-3B. 3.59 3.58 3.71 4.2 39.71 41.52 41.57 41.78 Number of Experts WER (ASR) WER (VSR) 0 1 2 3 4 5 0 10 20 30 40 50 4 8 12 16 ASR VSR [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Proportion of tokens assigned to each expert, either as first or second choice. both DCI and MM-Fuser degrade performance in both configu￾rations. We attribute this to the inherently higher WER in VSR, which may exacerbate the ambiguity of visual speech (i.e., lip move…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 48 canonical work pages

  1. [1]

    Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approach

    Introduction Automated speech recognition technologies have made signif- icant progress and are widely employed in various real-world applications [1]. In particular, Auditory Speech Recognition (ASR) technology [2, 3], which uses audio as its input modal- ity, is the most widely used and recognized by users. How- ever, in real-world scenarios, audio can ...

    work page Pith review arXiv 2025

  2. [2]

    This is crucial in resource-constrained LLM-based A VSR systems, as we aim to improve performance despite using smaller-scale LLMs and pre-trained encoders

    Llama-SMoP We propose Llama-SMoP, an MLLM employing sparsely-gated mixture-of-experts [35, 36] to increase model capacity without a proportional increase in computational cost. This is crucial in resource-constrained LLM-based A VSR systems, as we aim to improve performance despite using smaller-scale LLMs and pre-trained encoders. Llama-SMoP computes aud...

    work page

  3. [3]

    Transcribe{task prompt}to text

    Experiments and Results 3.1. Implementation Details Datasets. We train and evaluate Llama-SMoP on LRS3 [51], the largest publicly available dataset for A VSR. LRS3 contains 433hours of transcribed English video clips from TED talks. Pre-Processing. We follow [11, 33] for the pre-processing of the dataset. For the video modality, we crop the mouth re- gion...

    work page

  4. [4]

    Its key innovation is replacing the lin- ear projector with a Top-K sparse MoE module

    Conclusion We present Llama-SMoP, an MLLM optimized for improved audio-visual processing. Its key innovation is replacing the lin- ear projector with a Top-K sparse MoE module. This approach allows for more efficient processing of multimodal audio-visual tokens, and we investigate three SMoP designs based on varying router and expert configurations. Llama...

    work page 2020

  5. [5]

    Google usm: Scaling automatic speech recog- nition beyond 100 languages,

    Y . Zhanget al., “Google usm: Scaling automatic speech recog- nition beyond 100 languages,”arXiv preprint arXiv:2303.01037, 2023

    arXiv 2023

  6. [6]

    Deep speech 2: End-to-end speech recognition in english and mandarin,

    D. Amodeiet al., “Deep speech 2: End-to-end speech recognition in english and mandarin,” inICML. PMLR, 2016, pp. 173–182

    work page 2016

  7. [7]

    End-to-end speech recognition: A sur- vey,

    R. Prabhavalkaret al., “End-to-end speech recognition: A sur- vey,”TASLP, 2023

    work page 2023

  8. [8]

    Audio-visual speech modeling for continuous speech recognition,

    S. Dupontet al., “Audio-visual speech modeling for continuous speech recognition,”IEEE transactions on multimedia, 2000

    work page 2000

Show all 62 references
  1. [9]

    Investigation of speech separation as a front- end for noise robust speech recognition,

    A. Narayananet al., “Investigation of speech separation as a front- end for noise robust speech recognition,”TASLP, 2014

    2014

  2. [10]

    Audio-visual speech recognition using deep learn- ing,

    K. Nodaet al., “Audio-visual speech recognition using deep learn- ing,”Applied intelligence, vol. 42, pp. 722–737, 2015

    2015

  3. [11]

    Deep audio-visual speech recognition,

    T. Afouraset al., “Deep audio-visual speech recognition,”IEEE TPAMI, vol. 44, no. 12, pp. 8717–8727, 2018

    2018

  4. [12]

    Audio-visual speech recognition with a hybrid ctc/attention architecture,

    S. Petridiset al., “Audio-visual speech recognition with a hybrid ctc/attention architecture,” inSLT, 2018

    2018

  5. [13]

    End-to-end audio-visual speech recognition with conformers,

    P. Maet al., “End-to-end audio-visual speech recognition with conformers,” inICASSP, 2021

    2021

  6. [14]

    Visual context-driven audio feature enhancement for robust end-to-end audio-visual speech recognition,

    J. Honget al., “Visual context-driven audio feature enhancement for robust end-to-end audio-visual speech recognition,” inInter- speech, 2022

    2022

  7. [15]

    Auto-avsr: Audio-visual speech recognition with automatic labels,

    P. Maet al., “Auto-avsr: Audio-visual speech recognition with automatic labels,” inICASSP, 2023

    2023

  8. [16]

    Whisper-flamingo: Integrating visual features into whisper for audio-visual speech recognition and translation,

    A. Rouditchenkoet al., “Whisper-flamingo: Integrating visual features into whisper for audio-visual speech recognition and translation,” inInterspeech, 2024

    2024

  9. [17]

    A survey on self-supervised learning: Algorithms, applications, and future trends,

    J. Guiet al., “A survey on self-supervised learning: Algorithms, applications, and future trends,”TPAMI, 2024

    2024

  10. [18]

    Learning audio-visual speech representation by masked multimodal cluster prediction,

    B. Shiet al., “Learning audio-visual speech representation by masked multimodal cluster prediction,” inICLR, 2022

    2022

  11. [19]

    Jointly learning visual and auditory speech representations from raw data,

    A. Haliassoset al., “Jointly learning visual and auditory speech representations from raw data,” inICLR, 2023

    2023

  12. [20]

    u-hubert: Unified mixed-modal speech pretraining and zero-shot transfer to unlabeled modality,

    W.-N. Hsu and B. Shi, “u-hubert: Unified mixed-modal speech pretraining and zero-shot transfer to unlabeled modality,” NeurIPS, vol. 35, pp. 21 157–21 170, 2022

    2022

  13. [21]

    Braven: Improving self-supervised pre- training for visual and auditory speech recognition,

    A. Haliassoset al., “Braven: Improving self-supervised pre- training for visual and auditory speech recognition,” inICASSP, 2024

    2024

  14. [22]

    Unified speech recognition: A single model for auditory, visual, and audiovisual inputs,

    ——, “Unified speech recognition: A single model for auditory, visual, and audiovisual inputs,” inNeurIPS, 2024

    2024

  15. [23]

    Gpt-4 technical report,

    J. Achiamet al., “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  16. [24]

    Llama: Open and efficient foundation lan- guage models,

    H. Touvronet al., “Llama: Open and efficient foundation lan- guage models,”arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  17. [25]

    Improved baselines with visual instruction tuning,

    H. Liuet al., “Improved baselines with visual instruction tuning,” inCVPR, 2024

    2024

  18. [26]

    On generative spoken language modeling from raw audio,

    K. Lakhotiaet al., “On generative spoken language modeling from raw audio,”Transactions of the Association for Computa- tional Linguistics, vol. 9, pp. 1336–1354, 2021

    2021

  19. [27]

    Audiogpt: Understanding and generating speech, music, sound, and talking head,

    R. Huanget al., “Audiogpt: Understanding and generating speech, music, sound, and talking head,” inAAAI, 2024

    2024

  20. [28]

    Let’s go real talk: Spoken dialogue model for face- to-face conversation,

    S. Parket al., “Let’s go real talk: Spoken dialogue model for face- to-face conversation,” inACL, 2024

    2024

  21. [29]

    Developing instruction-following speech language model without speech instruction-tuning data,

    K. Luet al., “Developing instruction-following speech language model without speech instruction-tuning data,” inICASSP, 2025

    2025

  22. [30]

    Ssr: Alignment-aware modality connector for speech language models,

    W. Tanet al., “Ssr: Alignment-aware modality connector for speech language models,”arXiv preprint arXiv:2410.00168, 2024

    2024 arXiv

  23. [31]

    It’s never too late: Fusing acoustic information into large language models for automatic speech recognition,

    C. Chenet al., “It’s never too late: Fusing acoustic information into large language models for automatic speech recognition,” in ICLR, 2024

    2024

  24. [32]

    Large language models are efficient learners of noise-robust speech recognition,

    Y . Huet al., “Large language models are efficient learners of noise-robust speech recognition,” inICLR, 2024

    2024

  25. [33]

    An embarrassingly simple approach for llm with strong asr capacity,

    Z. Maet al., “An embarrassingly simple approach for llm with strong asr capacity,”arXiv preprint arXiv:2402.08846, 2024

    2024 arXiv

  26. [34]

    Connecting speech encoder and large language model for asr,

    W. Yuet al., “Connecting speech encoder and large language model for asr,” inICASSP, 2024

    2024

  27. [35]

    Prompting large language models with speech recognition abilities,

    Y . Fathullahet al., “Prompting large language models with speech recognition abilities,” inICASSP, 2024

    2024

  28. [36]

    Where visual speech meets language: Vsp-llm framework for efficient and context-aware visual speech process- ing,

    J. Yeoet al., “Where visual speech meets language: Vsp-llm framework for efficient and context-aware visual speech process- ing,” inFindings of the EMNLP, 2024, pp. 11 391–11 406

    2024

  29. [37]

    Large language models are strong audio- visual speech recognition learners,

    U. Cappellazzoet al., “Large language models are strong audio- visual speech recognition learners,” inICASSP, 2025

    2025

  30. [38]

    Adaptive audio-visual speech recognition via matryoshka-based multimodal llms,

    U. Cappellazzo, M. Kim, and S. Petridis, “Adaptive audio-visual speech recognition via matryoshka-based multimodal llms,”arXiv preprint arXiv:2503.06362, 2025

    2025 arXiv

  31. [39]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,

    N. Shazeeret al., “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” inICLR, 2016

    2016

  32. [40]

    Gshard: Scaling giant models with condi- tional computation and automatic sharding,

    D. Lepikhinet al., “Gshard: Scaling giant models with condi- tional computation and automatic sharding,” inICLR, 2021

    2021

  33. [41]

    Efficient fine-tuning of audio spectrogram transformers via soft mixture of adapters,

    U. Cappellazzoet al., “Efficient fine-tuning of audio spectrogram transformers via soft mixture of adapters,” inInterspeech, 2024

    2024

  34. [42]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model,

    A. Liuet al., “Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model,”arXiv preprint arXiv:2405.04434, 2024

    2024 arXiv

  35. [43]

    Mixture of a million experts,

    X. O. He, “Mixture of a million experts,”arXiv preprint arXiv:2407.04153, 2024

    2024 arXiv

  36. [44]

    Olmoe: Open mixture-of-experts lan- guage models,

    N. Muennighoffet al., “Olmoe: Open mixture-of-experts lan- guage models,” inICLR, 2025

    2025

  37. [45]

    Cumo: Scaling multimodal llm with co-upcycled mixture-of-experts,

    J. Liet al., “Cumo: Scaling multimodal llm with co-upcycled mixture-of-experts,” inNeurIPS, 2024

    2024

  38. [46]

    Chartmoe: Mixture of expert connector for advanced chart understanding,

    Z. Xuet al., “Chartmoe: Mixture of expert connector for advanced chart understanding,” inICLR, 2025

    2025

  39. [47]

    Dense connector for mllms,

    H. Yaoet al., “Dense connector for mllms,” inNeurIPS, 2024

    2024

  40. [48]

    Visual instruction tuning,

    H. Liuet al., “Visual instruction tuning,” inNeurIPS, 2023

    2023

  41. [49]

    Vila: On pre-training for visual language models,

    J. Linet al., “Vila: On pre-training for visual language models,” inCVPR, 2024

    2024

  42. [50]

    Multimodal autoregressive pre-training of large vi- sion encoders,

    E. Finiet al., “Multimodal autoregressive pre-training of large vi- sion encoders,”arXiv preprint arXiv:2411.14402, 2024

    2024 arXiv

  43. [51]

    Meteor: Mamba-based traversal of rationale for large language and vision models,

    B. Leeet al., “Meteor: Mamba-based traversal of rationale for large language and vision models,” inNeurIPS, 2024

    2024

  44. [52]

    Lora: Low-rank adaptation of large language mod- els,

    E. Huet al., “Lora: Low-rank adaptation of large language mod- els,” inICLR, 2021

    2021

  45. [53]

    St-moe: Designing stable and transferable sparse expert models,

    B. Zophet al., “St-moe: Designing stable and transferable sparse expert models,”arXiv preprint arXiv:2202.08906, 2022

    2022 arXiv

  46. [54]

    Mmfuser: Multimodal multi-layer feature fuser for fine-grained vision-language understanding,

    Y . Caoet al., “Mmfuser: Multimodal multi-layer feature fuser for fine-grained vision-language understanding,”arXiv preprint arXiv:2410.11829, 2024

    2024 arXiv

  47. [55]

    Lrs3-ted: a large-scale dataset for visual speech recognition,

    T. Afouraset al., “Lrs3-ted: a large-scale dataset for visual speech recognition,”arXiv preprint arXiv:1809.00496, 2018

    2018 arXiv

  48. [56]

    Robust speech recognition via large-scale weak supervision,

    A. Radfordet al., “Robust speech recognition via large-scale weak supervision,” inICML, 2023

    2023

  49. [57]

    The llama 3 herd of models,

    A. Dubeyet al., “The llama 3 herd of models,”arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  50. [58]

    Towards a unified view of parameter-efficient transfer learning,

    J. Heet al., “Towards a unified view of parameter-efficient transfer learning,” inICLR, 2022

    2022

  51. [59]

    Parameter-efficient transfer learning of au- dio spectrogram transformers,

    U. Cappellazzoet al., “Parameter-efficient transfer learning of au- dio spectrogram transformers,” inIEEE MLSP, 2024

    2024

  52. [60]

    Llama-omni: Seamless speech interaction with large language models,

    Q. Fanget al., “Llama-omni: Seamless speech interaction with large language models,”arXiv preprint arXiv:2409.06666, 2024

    2024 arXiv

  53. [61]

    Mixtures of experts for audio-visual learning,

    Y . Cheng, Y . Li, J. Heet al., “Mixtures of experts for audio-visual learning,”NeurIPS, vol. 37, pp. 219–243, 2024

    2024

  54. [62]

    Mohave: Mixture of hierarchical audio- visual experts for robust speech recognition,

    S. Kimet al., “Mohave: Mixture of hierarchical audio- visual experts for robust speech recognition,”arXiv preprint arXiv:2502.10447, 2025

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.