Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

EvoMoE: Expert Evolution in Mixture of Experts for Multimodal Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read EvoMoE evolves four experts from a single trainable FFN and routes each token by modality, outperforming MoE-LLaVA with fewer activated parameters.

desk verdict EvoMoE is a plausible incremental MoE-tuning recipe with consistent but small gains; the central evolution formula is too ambiguous to reproduce as written, and the parameter-efficiency claims are overstated in places. read the letter →

arxiv 2505.23830 v1 pith:PJTD3OZK submitted 2025-05-28 cs.CL

Linglin Jing , Yuting Gao , Zhigang Wang , Wang Lan , Yiwen Tang , Wenhai Wang , Kaipeng Zhang , Qingpei Guo This is my paper · ORCID
classification cs.CL
keywords MixtureofExpertsMultimodalLargeLanguageModelsExpertDiversityToken-awareRoutingHypernetworkMoE-tuningSparseActivationVision-LanguageInstructionTuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that two persistent failure modes of mixture-of-experts tuning for multimodal LLMs—experts that end up interchangeable, and a router that ignores whether tokens come from images or text—can be fixed by two changes. Instead of cloning one feed-forward network into several experts, the authors evolve extra experts from a single trainable expert by mixing its current weights with its gradient update at a random rate $\beta$. They also replace the shared linear router with a Dynamic Token-aware Router whose weights are generated per token by two modality-specific hypernetworks. On top of the MoE-LLaVA baseline, EvoMoE reports average gains of roughly 0.9 to 1.4 percentage points across seven benchmarks for five backbone LLMs from 0.5B to 7B while activating only a top-1 expert rather than top-2. If the recipe holds, sparse multimodal models can be tuned to be more accurate and cheaper to run than the replication-based MoE baseline.

What carries the argument

The central mechanism is the expert-evolution update $\theta_n \leftarrow \beta \theta_1 + (1-\beta) \nabla \theta_1$, where $\theta_1$ is the single trainable FFN from Stage I, $\nabla \theta_1$ is described as the gradient update for that expert, and $\beta$ is an evolution rate randomly sampled each training step from three ranges (0.9–0.99, 0.8–0.89, 0.7–0.79) to produce experts 2, 3, and 4. This operation is what is claimed to turn one expert's trajectory into functionally diverse experts. The second load-bearing component is the Dynamic Token-aware Router (DTR): two hypernetworks $H_V$ and $H_T$, each a two-layer MLP, generate the weights of down- and up-projection layers separately for visual and text tokens, and a final MLP $\phi$ converts the projected token into expert probabilities. DTR is what makes routing depend on modality and per-token content rather than a fixed linear map.

What would settle it

Train the same MoE-LLaVA baseline but replace expert evolution with a plain exponential moving average of the single expert's weights (or EMA plus dropout), keeping the DTR unchanged; if benchmark scores match EvoMoE's within noise, the evolutionary-diversity story is not what drives the gains. Alternatively, measure a functional diversity metric, such as pairwise token-routing overlap or output cosine distance between the four experts after training; if evolved experts are as similar to expert 1 as replicated experts are, the uniformity problem is not actually solved.

Watch

Extended reading notes

Core claim

EvoMoE's central claim is that expert uniformity and router rigidity, not the MoE structure itself, are what hold MoE-tuned multimodal LLMs back. As evidence, the authors show that shuffling the router weights of MoE-LLaVA leaves average benchmark performance essentially unchanged, meaning replicated-initialized experts converge to near-identical functions. Their remedy is a two-part construction: expert evolution, which derives $N$ experts by taking a single trained feed-forward network $\theta_1$ and applying $\theta_n \leftarrow \beta \theta_1 + (1-\beta) \nabla \theta_1$ with $\beta$ drawn from different ranges for different experts; and the Dynamic Token-aware Router (DTR), which uses two small hypernetworks, one for visual tokens and one for text tokens, to generate down- and up-projection weights per token before a final MLP picks the top-1 expert. Across the pipeline, only expert 1 and the DTR are ever trained, while the evolved experts stay frozen, so the method adds sparse capacity without training extra experts. On Qwen2-0.5B, StableLM-1.6B, Qwen-1.8B, Phi-2.7B, and OpenChat-7B, EvoMoE outperforms MoE-LLaVA on average while activating fewer parameters.

Load-bearing premise

The entire expert-evolution recipe rests on equation (1) being a well-defined way to convert one expert's training signal into new valid expert weights, but the paper never specifies what $\nabla \theta_1$ means: if it is the raw gradient it has the wrong shape to be a weight vector, and if it is the post-update parameter vector the method reduces to exponential averaging of one expert's own trajectory, which is not shown to produce functional diversity.

Editorial extensions

If this is right

  • EvoMoE's reported results imply that sparse multimodal LLMs can beat both their dense counterparts and the MoE-LLaVA baseline while activating only one expert per token, lowering inference cost for the same or better accuracy.
  • The DTR ablation, which replaces the linear router with modality-specific hypernetwork routing, gives a concrete recipe: separating visual and text token routing helps more than any shared-router variant, so future MoE MLLMs should route by modality first.
  • The shuffled-router experiment implies that replicated initialization in MoE-tuning produces redundant experts; any method that provably increases expert diversity, whether by evolution or otherwise, should recover similar gains, making diversity a testable design target.
  • The authors' negative results—unfreezing all experts in Stage III, adding shared experts, and removing the first MoE layer all hurt—suggest that the freeze-evolved-experts plus top-1 protocol is load-bearing rather than incidental.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If $\nabla \theta_1$ is the raw gradient, equation (1) cannot directly produce valid network weights because the gradient has the wrong shape, so the actual implementation likely uses the post-optimizer parameter vector; under that reading the method is a form of weight averaging, and the claimed 'evolution' may be better described as randomized exponential moving average. This is our inference fr
  • A testable extension of the paper's logic: the same evolution-plus-modality-router recipe should transfer to dense-to-sparse conversion of purely text LLMs, where the modality distinction in DTR would collapse to token-position or token-type differences; the paper does not test this.
  • The authors report that randomly sampling $\beta$ across ranges works better than any single fixed $\beta$; our inference is that what matters is creating experts whose routing distributions differ, so an even simpler diversity-promoting prior, such as different learning rates per expert or different dropout masks, might match EvoMoE without the evolution formula.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes EvoMoE, a mixture-of-experts tuning framework for multimodal large language models, with two main components: an expert evolution strategy that produces multiple FFN experts from a single trainable expert via Eq. (1), and a Dynamic Token-aware Router (DTR) that uses hypernetworks to generate token-specific routing weights. The method is evaluated on five LLM backbones (0.5B to 7B) across seven multimodal benchmarks, reporting consistent average gains over the MoE-LLaVA baseline, alongside ablations for each component, expert diversity strategies, top-k choices, and expert counts.

Significance. If the empirical results hold, the paper offers a simple and apparently effective recipe for improving sparse MLLMs: the two proposed mechanisms each contribute measurable gains, and the improvements are consistent across five model sizes. The paper also includes useful ablations (noise, dropout, contrastive loss, local loss, router variants) and supplementary training details. However, the central expert-evolution equation is not precisely defined, the activated-parameter claims are internally contradictory, and some ablation numbers differ between the main text and the supplement. These issues prevent the reader from verifying the method's core mechanism and its headline efficiency claim, so the current version is not ready for publication.

major comments (4)
  1. [Section 3.2, Eq. (1)] The symbol ∇θ1 is never defined precisely. If it denotes the raw gradient vector, then θn is a sum of parameters and gradients of incompatible scales and is not a valid FFN weight vector; if it denotes a gradient-based update of θ1 (e.g., θ1 − η∇θ), then each θn is at most a single step from θ1, and the claimed functional diversity of experts is unverified. The paper also does not state whether Eq. (1) is applied once at the end of Stage II or iteratively at each training step; the text in §3.2 and §4.1 says β is randomly assigned 'at each training step,' which, combined with Eq. (1), would produce experts that change every step with no accumulation rule. Because this formula is the sole mechanism for generating experts 2–N, the method is not reproducible as written.
  2. [Section 4.2 and Tables 1, 8] The paper repeatedly claims that EvoMoE achieves better performance 'with fewer activated parameters' (Section 4.2 and contribution list). Table 1 contradicts this for the 0.5B model: EvoMoE activates 0.7B parameters versus MoE-LLaVA's 0.6B. For Phi-2.7B, Table 1 reports 3.0B activated parameters for EvoMoE, while supplementary Table 8 reports 4.5B for the same configuration, which would be more than the MoE-LLaVA baseline's 3.6B. These discrepancies directly undermine the paper's central efficiency claim and must be reconciled.
  3. [Section 4.3, Tables 3 and 15] The ablation of the evolution value β reports different numbers for the same conditions in the main paper and the supplement. For example, 'Expert 2' with β=0.9 has VQAv2 76.4 and MME 1305.7 in Table 3, but Table 15 reports VQAv2 76.8 and MME 1290.7 for β=0.9; similarly, 'Expert 3'/β=0.8 has MME 1297.3 in Table 3 but 1277.3 in Table 15. If these are the same experiment, the discrepancy is a data-integrity issue; if they are different settings, the paper should say so explicitly.
  4. [Section 3.3, Eqs. (5)–(8)] The DTR module cannot be reconstructed from the text because the hypernetwork architecture is unspecified: the layer widths, output dimensions of Θ_up and Θ_down, and the exact parameter count of the router are not reported (Table 8 gives only a single aggregate 'Router' value). Please provide these details or release code, since DTR is one of the two core contributions.
minor comments (5)
  1. [Section 4.3] The first sentence contains a typo: 'peform' should be 'perform'.
  2. [Table 2] The abbreviation 'M-T[25]' in the table header is not defined in the caption; please spell out 'MoE-tuning' and clarify the reference.
  3. [Supplementary Table 9] In the Qwen-1.8B row, the MME value for the MoE baseline is printed as '12881', which appears to be missing a decimal point and should likely be '1288.1'.
  4. [Section 4.1 and Supplementary Table 7] The main text states a batch size of 4 with gradient accumulation 2, while Table 7 lists global batch sizes of 256, 64, and 64 for the three stages; please clarify the relationship between local batch size, gradient accumulation, and the reported global batch size.
  5. [Table 1 caption] The caption notes that MME values are divided by 20 for the AVG computation, but the table displays raw MME values; please state this explicitly in the main text near the table to avoid confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: EvoMoE's central claims are empirical benchmark measurements of a proposed training recipe, not derivations that reduce to their own inputs.

full rationale

EvoMoE is an empirical methods paper; its central claims are benchmark measurements of a training recipe, not derivations. The two proposed components, expert evolution (Eq. 1) and DTR (Eqs. 5-8), are defined operationally and their value is established by comparisons against MoE-LLaVA and ablations on held-out benchmarks such as MME, MMBench, TextVQA, POPE, VQAv2, GQA, and SQA. No reported benchmark number is the same as a fitted parameter by construction: the evolution rate β is ablated and then randomized over selected ranges, but the reported average scores are evaluation outputs, not the objective being optimized. The baseline comparison MoE-LLaVA is an external method, not a self-citation. The paper does inherit the MoE-LLaVA pretraining and data pipeline, but that is reuse of an external baseline, not circular reasoning. The genuine weaknesses are non-circular: Eq. (1) never defines ∇θ1 precisely (raw gradient vs. parameter update), making the expert-evolution mechanism ambiguous and hard to reproduce, and Table 8's 4.5B activated parameter count for Phi-2.7B contradicts Table 1's 3.0B. These are correctness and reproducibility issues to be weighed separately; they do not make the central empirical claim equivalent to its inputs. Therefore no circularity is present.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central method depends on a set of hand-chosen hyperparameters (beta ranges, alpha, expert count, top-k) and on the unstated premise that EMA-style copies of one FFN become functionally diverse. No new physical or architectural entities are introduced; the DTR is a composition of known hypernetwork and MoE routing ideas.

free parameters (4)
  • Expert evolution rate beta (sampling ranges) = [0.9-0.99], [0.8-0.89], [0.7-0.79]
    Three ranges are used to generate experts 2-4, selected after ablations in Table 15. This is hand-tuned hyperparameter search on the same benchmarks reported in the paper.
  • Auxiliary loss weight alpha = 0.001
    Set manually in Section 3.4 to weight the load-balance auxiliary loss; follows the MoE-LLaVA setting.
  • Number of experts and top-k = 4 experts, top-1
    Chosen empirically: Table 11 shows top-1 beats top-2 and Table 12 shows 4 experts beat 2 for this framework.
  • DTR hypernetwork architecture (layer sizes not reported) = unspecified
    The paper does not report the hidden sizes or output dimension of hypernetworks H_V and H_T, nor the generated down/up-sampling layer widths, although these determine router capacity.
assumptions (3)
  • domain assumption MoE-LLaVA pretrained weights and stage-wise recipe are a valid starting point for EvoMoE.
    Section 3.1 builds EvoMoE on MoE-LLaVA pretraining and the same data splits; if that base is flawed, the comparisons inherit its issues.
  • domain assumption Single-run benchmark differences of about 1% are meaningful.
    No seeds, error bars, or significance tests are reported; the paper treats the observed gaps as real improvements.
  • domain assumption The shuffle-router experiment isolates expert uniformity.
    Figure 1a and Table 10 interpret null results under shuffled routing as evidence of uniform experts, but router rigidity (uniform logits across tokens) would also produce a null result.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EvoMoE: Expert Evolution in Mixture of Experts for Multimodal Large Language Models." pith.science (2026). https://pith.science/paper/PJTD3OZK

@misc{pith2026250523830,
  author       = {Pith},
  title        = {Pith review of: EvoMoE: Expert Evolution in Mixture of Experts for Multimodal Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PJTD3OZK}},
  note         = {Machine review of arXiv:2505.23830}
}
read the original abstract

Recent advancements have shown that the Mixture of Experts (MoE) approach significantly enhances the capacity of large language models (LLMs) and improves performance on downstream tasks. Building on these promising results, multi-modal large language models (MLLMs) have increasingly adopted MoE techniques. However, existing multi-modal MoE tuning methods typically face two key challenges: expert uniformity and router rigidity. Expert uniformity occurs because MoE experts are often initialized by simply replicating the FFN parameters from LLMs, leading to homogenized expert functions and weakening the intended diversification of the MoE architecture. Meanwhile, router rigidity stems from the prevalent use of static linear routers for expert selection, which fail to distinguish between visual and textual tokens, resulting in similar expert distributions for image and text. To address these limitations, we propose EvoMoE, an innovative MoE tuning framework. EvoMoE introduces a meticulously designed expert initialization strategy that progressively evolves multiple robust experts from a single trainable expert, a process termed expert evolution that specifically targets severe expert homogenization. Furthermore, we introduce the Dynamic Token-aware Router (DTR), a novel routing mechanism that allocates input tokens to appropriate experts based on their modality and intrinsic token values. This dynamic routing is facilitated by hypernetworks, which dynamically generate routing weights tailored for each individual token. Extensive experiments demonstrate that EvoMoE significantly outperforms other sparse MLLMs across a variety of multi-modal benchmarks, including MME, MMBench, TextVQA, and POPE. Our results highlight the effectiveness of EvoMoE in enhancing the performance of MLLMs by addressing the critical issues of expert uniformity and router rigidity.

Figures

Figures reproduced from arXiv: 2505.23830 by the authors.

Figure 1
Figure 1. Two key challenges in MoE-tuning. (a) Expert Uniformity: Randomly shuffling the router during inference results in negligible performance degradation, suggesting uniformity among experts derived from replicated initialization. (b) Router Rigidity: Kernel density estimation (KDE) of the logits for image and text tokens reveals that the linear router generates input-insensitive selections, leading to static distributi… view at source ↗
Figure 2
Figure 2. The framework of EvoMoE. EvoMoE comprises three stages of instruction-tuning: (a) Warm-up: Begin training with multi-modal instruction data to familiarize the model with under￾standing capabilities, utilizing parameters initialized during the MoE-LLaVA [25] pretraining stage. (b) Expert Evolution: Train only FFN1, while evolving other experts from FFN1, and (c) Dynamic Token-aware Router: Use FFNs evolved in Stage I… view at source ↗
Figure 3
Figure 3. Dynamic Token-aware Router (DTR). Two input-guided hypernetworks dy￾namically generate network parameters for the up-sampling and down-sampling layers based on visual and text tokens. The final lin￾ear layer predicts probabilities and selects the top-k experts. In this module, only the hyper￾networks and the linear layer are trainable. We exclusively train θ1 of expert 1, allowing it to evolve into different experts… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Design analysis of DTR. (a) sin￾gle router; (b) modality-specific router; (c) modality-specific router with shared routing; (d) hyperNet with shared routing. Design analysis of DTR [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Distribution of input modalities across different experts (a) Previous methods exhibited almost uniform distributions across different inputs, leading to router rigidity. (b) EvoMoE dynami￾cally allocates input tokens to the most suitable experts based on their modalit…
Figure 6
Figure 6. Figure 6: Visual input examples. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. General Incomplete Multimodal Learning via Dynamic Quality Perception

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A unified multimodal learning framework models modality degradation as a continuous variable and uses a noise-aware quality estimator to adaptively weight fused representations under both intra- and inter-modality mis...

Reference graph

Works this paper leans on

54 extracted references · 16 canonical work pages · cited by 1 Pith paper

  1. [1]

    arXiv preprint arXiv:2303.08774 (2023)

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  2. [2]

    arXiv preprint arXiv:2309.16609 (2023)

    Bai, J., Bai, S., Chu, Y ., Cui, Z., Dang, K., Deng, X., Fan, Y ., Ge, W., Han, Y ., Huang, F., et al.: Qwen technical report. arXiv preprint arXiv:2309.16609 (2023)

    arXiv 2023

  3. [3]

    5-vl technical report

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al.: Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923 (2025)

    arXiv 2025

  4. [4]

    arXiv preprint arXiv:2402.17834 (2024)

    Bellagente, M., Tow, J., Mahan, D., Phung, D., Zhuravinskyi, M., Adithyan, R., Baicoianu, J., Brooks, B., Cooper, N., Datta, A., et al.: Stable lm 2 1.6 b technical report. arXiv preprint arXiv:2402.17834 (2024)

    arXiv 2024

  5. [5]

    ACM transactions on intelligent systems and technology 15(3), 1–45 (2024)

    Chang, Y ., Wang, X., Wang, J., Wu, Y ., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y ., et al.: A survey on evaluation of large language models. ACM transactions on intelligent systems and technology 15(3), 1–45 (2024)

    work page 2024

  6. [6]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Chen, J., Guo, L., Sun, J., Shao, S., Yuan, Z., Lin, L., Zhang, D.: Eve: efficient vision-language pre-training with masked prediction and modality-aware moe. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 38, pp. 1110–1119 (2024)

    work page 2024

  7. [7]

    arXiv preprint arXiv:2401.16160 (2024)

    Chen, S., Jie, Z., Ma, L.: Llava-mole: Sparse mixture of lora experts for mitigating data conflicts in instruction finetuning mllms. arXiv preprint arXiv:2401.16160 (2024)

    arXiv 2024

  8. [8]

    In: International conference on machine learning

    Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International conference on machine learning. pp. 1597–1607. PmLR (2020)

    2020

Show all 54 references
  1. [9]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al.: Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern reco...

    2024

  2. [10]

    Chu, X., Qiao, L., Lin, X., Xu, S., Yang, Y ., Hu, Y ., Wei, F., Zhang, X., Zhang, B., Wei, X., Shen, C.: Mobilevlm : A fast, strong and open vision language assistant for mobile devices (2023), https://arxiv.org/abs/2312.16886

    2023 arXiv

  3. [11]

    Chu, X., Qiao, L., Zhang, X., Xu, S., Wei, F., Yang, Y ., Sun, X., Hu, Y ., Lin, X., Zhang, B., Shen, C.: Mobilevlm v2: Faster and stronger baseline for vision language model (2024), https://arxiv.org/abs/2402.03766

    2024 arXiv

  4. [12]

    ArXiv abs/2306.13394 (2023), https://api.semanticscholar.org/CorpusID: 259243928

    Fu, C., Chen, P., Shen, Y ., Qin, Y ., Zhang, M., Lin, X., Qiu, Z., Lin, W., Yang, J., Zheng, X., Li, K., Sun, X., Ji, R.: Mme: A comprehensive evaluation benchmark for multimodal large language models. ArXiv abs/2306.13394 (2023), https://api.semanticscholar.org/CorpusID: 259243928

    2023 arXiv

  5. [13]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Goyal, Y ., Khot, T., Summers-Stay, D., Batra, D., Parikh, D.: Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 6904–6913 (2017)

    2017

  6. [14]

    arXiv preprint arXiv:2407.21783 (2024)

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al.: The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

  7. [15]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Hudson, D.A., Manning, C.D.: Gqa: A new dataset for real-world visual reasoning and compositional question answering. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 6700–6709 (2019)

    2019

  8. [16]

    arXiv preprint arXiv:2410.21276 (2024) 10

    Hurst, A., Lerer, A., Goucher, A.P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al.: Gpt-4o system card. arXiv preprint arXiv:2410.21276 (2024) 10

    2024 arXiv

  9. [17]

    Microsoft Research Blog 1(3), 3 (2023)

    Javaheripi, M., Bubeck, S., Abdin, M., Aneja, J., Bubeck, S., Mendes, C.C.T., Chen, W., Del Giorno, A., Eldan, R., Gopi, S., et al.: Phi-2: The surprising power of small language models. Microsoft Research Blog 1(3), 3 (2023)

    2023

  10. [18]

    IEEE Network (2024)

    Kan, K.B., Mun, H., Cao, G., Lee, Y .: Mobile-llama: Instruction fine-tuning open-source llm for network analysis in 5g networks. IEEE Network (2024)

    2024

  11. [19]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Lai, X., Tian, Z., Chen, Y ., Li, Y ., Yuan, Y ., Liu, S., Jia, J.: Lisa: Reasoning segmentation via large language model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9579–9589 (2024)

    2024

  12. [20]

    In: European Conference on Computer Vision

    Lee, B.K., Park, B., Won Kim, C., Man Ro, Y .: Moai: Mixture of all intelligence for large language and vision models. In: European Conference on Computer Vision. pp. 273–302. Springer (2024)

    2024

  13. [21]

    arXiv preprint arXiv:2501.08313 (2025)

    Li, A., Gong, B., Yang, B., Shan, B., Liu, C., Zhu, C., Zhang, C., Guo, C., Chen, D., Li, D., et al.: Minimax-01: Scaling foundation models with lightning attention. arXiv preprint arXiv:2501.08313 (2025)

    2025 arXiv

  14. [22]

    arXiv preprint arXiv:2306.05425 (2023)

    Li, B., Zhang, Y ., Chen, L., Wang, J., Pu, F., Yang, J., Li, C., Liu, Z.: Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306.05425 (2023)

    2023 arXiv

  15. [23]

    Li, Y ., Zhang, Y ., Wang, C., Zhong, Z., Chen, Y ., Chu, R., Liu, S., Jia, J.: Mini-gemini: Mining the potential of multi-modality vision language models (2024), https://arxiv.org/abs/ 2403.18814

    2024 arXiv

  16. [24]

    arXiv preprint arXiv:2305.10355 (2023)

    Li, Y ., Du, Y ., Zhou, K., Wang, J., Zhao, W.X., Wen, J.R.: Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355 (2023)

    2023 arXiv

  17. [25]

    arXiv preprint arXiv:2401.15947 (2024)

    Lin, B., Tang, Z., Ye, Y ., Cui, J., Zhu, B., Jin, P., Huang, J., Zhang, J., Pang, Y ., Ning, M., et al.: Moe-llava: Mixture of experts for large vision-language models. arXiv preprint arXiv:2401.15947 (2024)

    2024 arXiv

  18. [26]

    arXiv preprint arXiv:2412.19437 (2024)

    Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al.: Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 (2024)

    2024 arXiv

  19. [27]

    CoRR (2023)

    Liu, F., Lin, K., Li, L., Wang, J., Yacoob, Y ., Wang, L.: Aligning large multi-modal model with robust instruction tuning. CoRR (2023)

    2023

  20. [28]

    Liu, H., Li, C., Li, Y ., Lee, Y .J.: Improved baselines with visual instruction tuning (2023)

    2023

  21. [29]

    arXiv preprint arXiv:2412.14219 (2024)

    Liu, J., Tang, P., Wang, W., Ren, Y ., Hou, X., Heng, P.A., Guo, M., Li, C.: A survey on inference optimization techniques for mixture of experts models. arXiv preprint arXiv:2412.14219 (2024)

    2024 arXiv

  22. [30]

    Liu, Y ., Duan, H., Zhang, Y ., Li, B., Zhang, S., Zhao, W., Yuan, Y ., Wang, J., He, C., Liu, Z., et al.: Mmbench: Is your multi-modal model an all-around player? In: European conference on computer vision. pp. 216–233. Springer (2024)

    2024

  23. [31]

    arXiv preprint arXiv:2403.05525 (2024)

    Lu, H., Liu, W., Zhang, B., Wang, B., Dong, K., Liu, B., Sun, J., Ren, T., Li, Z., Yang, H., et al.: Deepseek-vl: towards real-world vision-language understanding. arXiv preprint arXiv:2403.05525 (2024)

    2024 arXiv

  24. [32]

    Advances in Neural Information Processing Systems 35, 2507–2521 (2022)

    Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.W., Zhu, S.C., Tafjord, O., Clark, P., Kalyan, A.: Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems 35, 2507–2521 (2022)

    2022

  25. [33]

    Advances in Neural Information Processing Systems 35, 9564–9576 (2022)

    Mustafa, B., Riquelme, C., Puigcerver, J., Jenatton, R., Houlsby, N.: Multimodal contrastive learning with limoe: the language-image mixture of experts. Advances in Neural Information Processing Systems 35, 9564–9576 (2022)

    2022

  26. [34]

    arXiv preprint arXiv:2307.06435 (2023) 11

    Naveed, H., Khan, A.U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., Mian, A.: A comprehensive overview of large language models. arXiv preprint arXiv:2307.06435 (2023) 11

    2023 arXiv

  27. [35]

    In: International conference on machine learning

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021)

    2021

  28. [36]

    In: International conference on machine learning

    Rajbhandari, S., Li, C., Yao, Z., Zhang, M., Aminabadi, R.Y ., Awan, A.A., Rasley, J., He, Y .: Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale. In: International conference on machine learning. pp. 18332–18346. PMLR (2022)

    2022

  29. [37]

    arXiv preprint arXiv:2501.04322 (2025)

    Rang, M., Bi, Z., Liu, C., Tang, Y ., Han, K., Wang, Y .: Eve: Efficient multimodal vision language models with elastic visual experts. arXiv preprint arXiv:2501.04322 (2025)

    2025 arXiv

  30. [38]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Singh, A., Natarajan, V ., Shah, M., Jiang, Y ., Chen, X., Batra, D., Parikh, D., Rohrbach, M.: Towards vqa models that can read. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 8317–8326 (2019)

    2019

  31. [39]

    arXiv preprint arXiv:2403.05530 (2024)

    Team, G., Georgiev, P., Lei, V .I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al.: Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 (2024)

    2024 arXiv

  32. [40]

    arXiv preprint arXiv:2504.07491 (2025)

    Team, K., Du, A., Yin, B., Xing, B., Qu, B., Wang, B., Chen, C., Zhang, C., Du, C., Wei, C., et al.: Kimi-vl technical report. arXiv preprint arXiv:2504.07491 (2025)

    2025 arXiv

  33. [41]

    Nature medicine 29(8), 1930–1940 (2023)

    Thirunavukarasu, A.J., Ting, D.S.J., Elangovan, K., Gutierrez, L., Tan, T.F., Ting, D.S.W.: Large language models in medicine. Nature medicine 29(8), 1930–1940 (2023)

    2023

  34. [42]

    arXiv preprint arXiv:2309.11235 (2023)

    Wang, G., Cheng, S., Zhan, X., Li, X., Song, S., Liu, Y .: Openchat: Advancing open-source language models with mixed-quality data. arXiv preprint arXiv:2309.11235 (2023)

    2023 arXiv

  35. [43]

    arXiv preprint arXiv:2311.07574 (2023)

    Wang, J., Meng, L., Weng, Z., He, B., Wu, Z., Jiang, Y .G.: To see is to believe: Prompting gpt-4v for better visual instruction tuning. arXiv preprint arXiv:2311.07574 (2023)

    2023 arXiv

  36. [44]

    Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., Dong, G., Wei, H., Lin, H., Tang, J., Wang, J., Yang, J., Tu, J., Zhang, J., Ma, J., Yang, J., Xu, J., Zhou, J., Bai, J., He, J., Lin, J., Dang, K., Lu, K., Chen, K., Yang, K., Li, M....

    2024 arXiv

  37. [45]

    5 technical report

    Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al.: Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115 (2024)

    2024 arXiv

  38. [46]

    Yuan, Z., Li, Z., Huang, W., Ye, Y ., Sun, L.: Tinygpt-v: Efficient multimodal large language model via small backbones (2024), https://arxiv.org/abs/2312.16862

    2024 arXiv

  39. [47]

    In: 2023 USENIX Annual Technical Conference (USENIX ATC 23)

    Zhai, M., He, J., Ma, Z., Zong, Z., Zhang, R., Zhai, J.: {SmartMoE}: Efficiently training {Sparsely-Activated} models through combining offline and online parallelization. In: 2023 USENIX Annual Technical Conference (USENIX ATC 23). pp. 961–975 (2023)

    2023

  40. [48]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Zhai, X., Mustafa, B., Kolesnikov, A., Beyer, L.: Sigmoid loss for language image pre-training. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 11975– 11986 (2023)

    2023

  41. [49]

    arXiv preprint arXiv:2403.13447 (2024)

    Zhang, W., Lin, T., Liu, J., Shu, F., Li, H., Zhang, L., Wanggui, H., Zhou, H., Lv, Z., Jiang, H., et al.: Hyperllava: Dynamic visual and language expert tuning for multimodal large language models. arXiv preprint arXiv:2403.13447 (2024)

    2024 arXiv

  42. [50]

    arXiv preprint arXiv:2307.04087 (2023)

    Zhao, B., Wu, B., He, M., Huang, T.: Svit: Scaling up visual instruction tuning. arXiv preprint arXiv:2307.04087 (2023)

    2023 arXiv

  43. [51]

    arXiv preprint arXiv:2402.12656 (2024) 12

    Zhao, H., Qiu, Z., Wu, H., Wang, Z., He, Z., Fu, J.: Hypermoe: Towards better mixture of experts via transferring among experts. arXiv preprint arXiv:2402.12656 (2024) 12

    2024 arXiv

  44. [52]

    In: Proceedings of the 43rd IEEE/ACM International Conference on Computer-Aided Design

    Zhong, S., Liang, L., Wang, Y ., Wang, R., Huang, R., Li, M.: Adapmoe: Adaptive sensitivity- based expert gating and management for efficient moe inference. In: Proceedings of the 43rd IEEE/ACM International Conference on Computer-Aided Design. pp. 1–9 (2024)

    2024

  45. [53]

    In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing

    Zhu, T., Qu, X., Dong, D., Ruan, J., Tong, J., He, C., Cheng, Y .: Llama-moe: Building mixture- of-experts from llama with continual pre-training. In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. pp. 15913–15923 (2024)

    2024

  46. [54]

    Increasing Expert Diversity

    Zong, Z., Ma, B., Shen, D., Song, G., Shao, H., Jiang, D., Li, H., Liu, Y .: Mova: Adapting mixture of vision experts to multimodal context. arXiv preprint arXiv:2404.13046 (2024) 13 Supplementary Materials A Overview This document provides a list of supplemental materials to ...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.