Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders

T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that the distinct strengths of several pre-trained visual encoders can be distilled into a single student encoder inside a vision-language model, yielding multi-encoder benefits at single-encoder inference cost, as…

desk verdict Solid empirical integration—multi-teacher KD with MoLE works—but the CLIP-attention weighting has a real blind spot on OCR tasks, and the SOTA claim needs matched-data and error bars. read the letter →

arxiv 2501.01709 v3 pith:2USCKC2L submitted 2025-01-03 cs.CV cs.AI

Jiajun Cao , Yuan Zhang , Tao Huang , Ming Lu , Qizhe Zhang , Ruichuan An , Ningning MA , Shanghang Zhang This is my paper · ORCID
classification cs.CVcs.AI
keywords knowledgedistillationvision-languagemodelsvisualencodersmixture-of-expertsLoRACLIPattentionLLaVAmulti-teacher
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MoVE-KD sets out to prove that a vision-language model can inherit the combined strengths of several visual encoders—CLIP, EVA-02, ConvNeXt, and optionally SAM-L—without running any of them at inference time. The method distills their visual tokens into one student encoder initialized from CLIP, using per-teacher MLP adapters to align token spaces, a Mixture-of-LoRA-Experts module to keep conflicting teacher knowledge separate, and CLIP's [CLS] attention to reweight both image tokens and teachers during distillation. The paper reports consistent gains over the LLaVA and LLaVA-NeXT baselines across eight benchmarks, with the added MoLE parameters accounting for only about 0.3% of the total. If the claim holds, upgrading a VLM's perception reduces to swapping in one distilled encoder rather than adding a mixture of encoders.

What carries the argument

The load-bearing device is the attention-weighted distillation loss $L_{kd} = \sum_{i=1}^{m} W^{(tea)}_i \sum_{j=1}^{n} (W^{(tok)}_j + \frac{1}{n}) \mathrm{MSE}(V^{(t)}_{i,j}, V^{(s)}_j)$, where token weights come from CLIP's [CLS]-to-patch attention and teacher weights come from the average CLIP attention to each teacher's tokens. This loss tells the student which image regions and which teachers carry valuable knowledge for a given input. The Mixture-of-LoRA-Experts (MoLE) router, which selects one of several low-rank LoRA adapters inside the student's feed-forward layers, is what keeps each teacher's specialized knowledge from being destroyed by interference during joint distillation.

What would settle it

Build a test set whose answers depend on text or objects that CLIP's [CLS] attention scores low, and train MoVE-KD on it with both CLIP-attention token weights and uniform token weights; if uniform weighting matches or beats attention weighting on that set, the claim that CLIP attention reliably identifies the valuable tokens fails.

Watch

Extended reading notes

Core claim

The central claim is that multi-encoder visual knowledge can be transferred into a single visual tower through a three-part distillation procedure. Dedicated two-layer MLP adapters project each teacher's tokens into a common space; a Mixture-of-LoRA-Experts module inside the student's feed-forward layers routes each input to specialized low-rank experts so that conflicting teacher knowledge is separated rather than averaged; and the distillation loss is reweighted by CLIP's [CLS] attention, which up-weights informative visual tokens and gives each teacher a per-image importance score. The student is initialized from CLIP, and CLIP is kept as one teacher with a fixed high weight to anchor the student's own knowledge. The paper reports gains on eight benchmarks and shows that adding a fourth teacher (SAM-L) improves results further, supporting the scalability of the design.

Load-bearing premise

The method assumes that the attention CLIP's [CLS] token pays to image patches is a correct, fixed map of what is worth learning for every downstream vision-language task, so that reweighting tokens and teachers by this map always helps rather than hurting.

Editorial extensions

If this is right

  • A VLM with the MoVE-KD student encoder matches or exceeds the single-encoder and multi-encoder baselines on most of the eight benchmarks, so users get multi-encoder benefits without multi-encoder inference cost.
  • The v1.1 variant with SAM-L as a fourth teacher improves over v1.0 on several benchmarks, indicating the approach continues to gain from additional teachers rather than saturating immediately.
  • Unfreezing the vision encoder alone lowers performance (Table 5), so the reported gains come from the distillation modules themselves rather than from simply letting the encoder train.
  • The MoLE module adds about 0.3% of total parameters, so the parameter overhead of reconciling multiple teacher knowledge is small.
  • CLIP needs a fixed high teacher weight (0.8), so the student's inherited CLIP knowledge must be anchored during distillation to avoid forgetting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural test the paper does not run: replace the fixed CLIP [CLS] attention weights with a learned or task-conditioned weighting and measure TextVQA; if the text-reading loss disappears, the attention prior is the cause rather than the vision improvement.
  • The paper's conclusion that the projector becomes the bottleneck at larger LLM scales implies a testable consequence: MoVE-KD's gains should grow when the distilled student is paired with a stronger vision-language projector.
  • The MoLE router's expert choices can be analyzed post hoc; if experts specialize by teacher or by visual domain, the routing signal could be reused at inference to skip unused experts and save computation.
  • Because the weighting inherits CLIP's blind spots, tasks whose answers live in low-CLIP-attention regions (such as dense OCR) are the places where a multi-attention or ensemble-salience weighting would plausibly be more robust.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. MoVE-KD proposes to distill multiple pre-trained visual encoders (CLIP, EVA-02, ConvNeXt, and optionally SAM-L) into a single student visual encoder inside LLaVA and LLaVA-NeXT, preserving single-encoder inference efficiency. The student is initialized from CLIP and fine-tuned with the standard VLM loss plus a distillation loss (Eq. 2) from the teacher encoders. Per-teacher encoder adapters align the teacher feature spaces, a mixture-of-LoRA-experts (MoLE) router selects specialized student parameters per input, and the CLIP [CLS] cross-attention map is used to weight visual tokens (Eq. 3) and teacher contributions (Eq. 4) in the distillation loss. The paper reports results on VQAv2, GQA, TextVQA, VizWiz, POPE, SQA, MME, and MMB, showing improvements over LLaVA-1.5 and LLaVA-NeXT baselines on most benchmarks, with TextVQA degradations in some configurations.

Significance. If the empirical claims hold, the paper makes a useful practical contribution: it offers a way to obtain multi-encoder visual knowledge at near-single-encoder inference cost, with only a small parameter overhead from LoRA experts, and it provides an interesting template for multi-teacher distillation into VLMs. The paper also contains well-structured ablations and useful controls (MoLE without KD, freeze versus unfreeze), and the public code release supports reproducibility. However, the headline state-of-the-art claim is not yet established because the comparison with AM-RADIO is not data-matched, the results are single-run scores without error bars, and the central assumption that CLIP [CLS] attention is a fixed task-independent saliency map is plausible but unvalidated and is in tension with the paper's own TextVQA results.

major comments (5)
  1. [Sec. 4.2, Table 1] The paper's own results contradict the unqualified claim that the attention-based distillation strategy enhances performance: on LLaVA-NeXT-7B, MoVE-KD-v1.0 lowers TextVQA from 64.9 to 63.7, and on LLaVA-NeXT-13B from 67.1 to 65.8. The explanation that TextVQA questions are 'not related to vision' is inaccurate, since TextVQA is a visual OCR benchmark; a more consistent reading is that CLIP [CLS] attention underweights text regions, so Eq. 3 down-weights the tokens needed for reading. Because this weighting is the central novelty, the authors should either qualify the claim, analyze the failure mode, or compare Eq. 3 against uniform, text-guided, or learned token weighting on TextVQA and related OCR tasks.
  2. [Sec. 4.2, Table 1] The comparison with AM-RADIO is not matched: RADIO is pre-trained on DataComp-1B with 1.4B image-text pairs, while MoVE-KD uses only LCS-558K, and the paper acknowledges this as 'a bit unfair' but still concludes state-of-the-art performance. Without a same-data, same-compute baseline (for example, training RADIO or a comparable single-encoder distillation baseline on the same data budget), the SOTA claim and the claim that MoVE-KD overcomes RADIO's knowledge forgetting are not established. Please also report the actual training and inference costs, including FLOPs and added parameter counts.
  3. [Sec. 3.3, Eqs. (2)-(4); Sec. 5.1] The teacher and token weights are derived from the frozen CLIP [CLS] attention map, and this choice is justified only by qualitative visualization and an appeal to a 'human-like' foreground/background definition. No benchmark-based validation is provided for the assumption that one fixed saliency map is reliable across tasks. The paper reports (Sec. 5.1) that text-driven foreground weighting hurt, but it does not compare against uniform token weighting or a learned weighting, so the specific contribution of Eqs. 3-4 beyond MoLE and adapters is not isolated. Since Eq. 2 multiplies these weights into the loss, an unreliable prior can actively suppress useful tokens; the TextVQA drops in Table 1 are consistent with this risk. Please add ablations with alternative token/teacher weighting schemes and a per-task reliability analysis.
  4. [Tables 1, 2, 4, 5] All results are reported as single numbers without error bars or significance tests, and many reported differences are within 0.5-1.5 points (for example, LLaVA-1.5-7B GQA 62.0 vs 63.2, and MME 1510.7 vs 1524.5). Given that the central claims are quantitative (comprehensive improvements and state-of-the-art performance), the manuscript should report variance over multiple seeds or otherwise quantify uncertainty; otherwise the improvements may not be distinguishable from training noise.
  5. [Eq. (4), Sec. 3.3] The teacher weight in Eq. 4 is computed as a dot product between CLIP's [CLS] token V_cls and each teacher's tokens V_t^i. Because the teacher encoders have different output dimensionalities, this dot product is only defined if V_t denotes the adapter-projected teacher tokens in the student/CLIP space; the text and Figure 3 do not state this explicitly. Please specify the exact tensors used in Eq. 4 and confirm whether the teacher weight reflects CLIP-teacher agreement after projection rather than arbitrary adapter scaling.
minor comments (5)
  1. [Eq. (3)] The notation is inconsistent: W(V) in Eq. 3 is used for the key projection, not the value projection; please rename it W(K) and align with the accompanying text.
  2. [Eq. (2)] The constant 1/n added to W(tok) is not motivated; please explain its role or clarify whether it is absorbed by re-normalization.
  3. [Sec. 4.3, Table 1] The versions MoVE-KD-v1.0 and MoVE-KD-v1.1 are used in Table 1 but defined only later in Sec. 4.3; please define them at first use.
  4. [Table 1] Several baseline rows (RADIO) have missing entries for TextVQA, VizWiz, SQA, MME, and MMB, which makes the comparison incomplete; please fill in the available numbers or state explicitly that they are unavailable.
  5. [Throughout] There are several typos and formatting slips (for example, 'eperts', 'V iunca', 'attetion', and inconsistent capitalization of Softmax) that should be corrected in the revision.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MoVE-KD is an empirical distillation framework whose gains are measured on external benchmarks, and its self-referential CLIP-teacher/CLIP-initialization design is disclosed rather than disguised as a prediction.

full rationale

The paper does not derive benchmark results from fitted equations. Equations (2)-(4) define a training loss, not a predictor; the reported VQAv2, GQA, TextVQA, and other scores come from held-out evaluation after training on LLaVA's standard datasets. The closest element to circularity is that the student is initialized from CLIP, CLIP is also a teacher with a hand-set weight of 0.8, and both token and teacher weights are computed from CLIP's [CLS] attention. This makes part of the training a self-distillation that anchors the student to its own initialization, and the teacher weight in Eq. (4) is trivially high for the CLIP teacher. However, the paper discloses this explicitly: 'Note that our student is initialized by the pre-trained CLIP encoder, to prevent severe forgetting of its own knowledge, we involve CLIP as one teacher and set a relatively high fixed weight for it.' The central claim does not reduce to that anchor: the method's value is tested by whether adding EVA-02, ConvNeXt, and SAM-L teachers plus the MoLE structure improves external benchmarks, which is an empirical question answered by Tables 1 and 2. The self-citations (e.g., Refs. [17] and [51]) are motivational or contrastive and are not load-bearing for the main result. No uniqueness theorem is imported from the authors, and no known result is renamed as a new derivation. The unvalidated assumption that CLIP [CLS] attention defines task-independent foreground is a correctness risk, not a circularity, because the paper does not define its target metric in terms of that assumption.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on four domain assumptions about CLIP attention quality, adapter alignment, LoRA expert routing, and fixed-data multi-teacher distillation. There are five hand-chosen or fitted settings, most notably the CLIP teacher weight (0.8) selected on the evaluation suite. No new physical or conceptual entities are introduced; MoLE is an architectural module rather than an invented entity.

free parameters (5)
  • CLIP teacher fixed weight = 0.8
    Chosen by grid search over 0.6, 0.7, 0.8, 0.9 on the benchmark suite (Table 4); this directly tunes the main distillation balance.
  • KD loss weight lambda_kd = 0.5
    Set in Sec. 4.1; no sensitivity analysis is shown.
  • Number of MoLE experts = 3
    Set in Sec. 4.1; no sensitivity analysis is shown.
  • LoRA rank = 32
    Set in Sec. 4.1; no sensitivity analysis is shown.
  • Teacher composition = CLIP + EVA-02 + ConvNeXt for v1.0; add SAM-L for v1.1
    The set of teachers is chosen manually, motivated by Eagle, and it changes the reported results.
assumptions (4)
  • domain assumption CLIP [CLS] attention provides a fixed, human-aligned foreground and background definition useful for weighting tokens and teachers.
    Introduced in Sec. 3.3 and Sec. 5.1. If this assumption fails for a task, the distillation down-weights informative tokens or teachers; the TextVQA degradations are consistent with this risk.
  • domain assumption A two-layer MLP encoder adapter can align each teacher's token space to the student token space.
    Assumed in Sec. 3.2. No analysis of alignment quality is provided.
  • domain assumption LoRA experts with top-1 routing can retain each teacher's specialty without catastrophic forgetting.
    Assumed in Sec. 3.2 and supported only indirectly by the ablations in Tables 2 and 3.
  • domain assumption Distillation from multiple teachers is beneficial at a fixed data budget, with no additional data required.
    Stated in Sec. 4.1 as 'we do not introduce additional datasets'. The method relies on the existing LLaVA data being sufficient.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders." pith.science (2026). https://pith.science/paper/2USCKC2L

@misc{pith2026250101709,
  author       = {Pith},
  title        = {Pith review of: MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2USCKC2L}},
  note         = {Machine review of arXiv:2501.01709}
}
read the original abstract

Visual encoders are fundamental components in vision-language models (VLMs), each showcasing unique strengths derived from various pre-trained visual foundation models. To leverage the various capabilities of these encoders, recent studies incorporate multiple encoders within a single VLM, leading to a considerable increase in computational cost. In this paper, we present Mixture-of-Visual-Encoder Knowledge Distillation (MoVE-KD), a novel framework that distills the unique proficiencies of multiple vision encoders into a single, efficient encoder model. Specifically, to mitigate conflicts and retain the unique characteristics of each teacher encoder, we employ low-rank adaptation (LoRA) and mixture-of-experts (MoEs) to selectively activate specialized knowledge based on input features, enhancing both adaptability and efficiency. To regularize the KD process and enhance performance, we propose an attention-based distillation strategy that adaptively weighs the different encoders and emphasizes valuable visual tokens, reducing the burden of replicating comprehensive but distinct features from multiple teachers. Comprehensive experiments on popular VLMs, such as LLaVA and LLaVA-NeXT, validate the effectiveness of our method. Our code is available at: https://github.com/hey-cjj/MoVE-KD.

Figures

Figures reproduced from arXiv: 2501.01709 by the authors.

Figure 1
Figure 1. Comparison of LLaVA-1.5-7B [ [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Concentration of [CLS] attention. The left subfigure (a) is the input image, and the right subfigure (b) is the [CLS] attention visualization of the pre-trained CLIP, showing CLIP’s focus on valuable regions of the image. vision foundation models, its performance is constrained by conflicts arising from learning diverse and often competing characteristics within a shared backbone. Our method fine-tunes a base model … view at source ↗
Figure 3
Figure 3. The pipeline of MoVE-KD. MoVE-KD projects teacher encoders’ outputs using encoder adapters, assigns teacher weight and token weight based on CLIP’s [CLS] attention. To mitigate knowledge conflicts, we incorporates MoLE structure in the student encoder. which cannot inspire the potential of students due to the limited ability of the single teacher. In contrast, the AM￾RADIO [36] starts to utilize multiple vision expe… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The generation of token weight. We employ the atten￾tion map of the [CLS] token to measure the contribution of vision tokens to knowledge distillation. and W(tok) and W(tea) denote the token-level and teacher￾level weight vectors. Given the above motivation, we now ela…
Figure 5
Figure 5. Figure 5: The visualization of CLIP [CLS] attention and student [CLS] attention. fixed. When observing, humans tend to focus on dynamic, complex, semantically rich elements, while being less sen￾sitive to repetitive items (e.g., sky, water, and grasslands). Hence, we propose usi…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. GenRecal: Generation after Recalibration from Large to Small Vision-Language Models

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A learnable Recalibrator bridges different tokenizers so that small VLMs can distill knowledge from any large VLM, improving their benchmark scores.

Reference graph

Works this paper leans on

53 extracted references · 13 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv:2303.08774, 2023. 2

    arXiv 2023

  2. [2]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736,

  3. [3]

    Does combining parameter- efficient modules improve few-shot transfer accuracy?arXiv preprint arXiv:2402.15414, 2024

    Nader Asadi, Mahdi Beitollahi, Yasser Khalil, Yinchuan Li, Guojun Zhang, and Xi Chen. Does combining parameter- efficient modules improve few-shot transfer accuracy?arXiv preprint arXiv:2402.15414, 2024. 4

    arXiv 2024

  4. [4]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. 2, 5

    arXiv 2023

  5. [5]

    Qwen-VL: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL: A frontier large vision-language model with versatile abilities. arXiv:2308.12966, 2023. 2

    arXiv 2023

  6. [6]

    Deepseek LLM: Scaling open-source lan- guage models with longtermism

    Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek LLM: Scaling open-source lan- guage models with longtermism. arXiv:2401.02954, 2024. 2

    arXiv 2024

  7. [7]

    InternVL: Scaling up vision founda- tion models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. InternVL: Scaling up vision founda- tion models and aligning for generic visual-linguistic tasks. arXiv:2312.14238, 2023. 2

    arXiv 2023

  8. [8]

    Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices

    Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang, Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, et al. Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices. arXiv preprint arXiv:2312.16886, 2023. 5

    arXiv 2023

Show all 53 references
  1. [9]

    Vision transformers need registers

    Timoth ´ee Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In The Twelfth International Conference on Learning Representa- tions, 2024. 2, 8

    2024

  2. [10]

    Eva-02: A visual representation for neon genesis

    Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xin- long Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. Image and Vision Computing, 149:105171,

  3. [11]

    MME: A comprehensive evalu- ation benchmark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. MME: A comprehensive evalu- ation benchmark for multimodal large language models. arXiv:2306.13394, 2023. 5

    2023 arXiv

  4. [12]

    Dat- acomp: In search of the next generation of multimodal datasets

    Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Dat- acomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Sys...

    2024

  5. [13]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913...

    2017

  6. [14]

    Vizwiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608–3617,

  7. [15]

    Distilling the knowledge in a neural net- work

    Geoffrey Hinton. Distilling the knowledge in a neural net- work. arXiv preprint arXiv:1503.02531, 2015. 1, 2

    2015 arXiv

  8. [16]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 4

    2021 arXiv

  9. [17]

    Masked distillation with receptive tokens

    Tao Huang, Yuan Zhang, Shan You, Fei Wang, Chen Qian, Jian Cao, and Chang Xu. Masked distillation with receptive tokens. arXiv preprint arXiv:2205.14589, 2022. 4

    2022 arXiv

  10. [18]

    GQA: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 5

    2019

  11. [19]

    Adaptive mixtures of local experts.Neu- ral computation, 3(1):79–87, 1991

    Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts.Neu- ral computation, 3(1):79–87, 1991. 4

    1991

  12. [20]

    Lift3d foun- dation policy: Lifting 2d large-scale pretrained models for robust 3d robotic manipulation, 2024

    Yueru Jia, Jiaming Liu, Sixiang Chen, Chenyang Gu, Zhilue Wang, Longzan Luo, Lily Lee, Pengwei Wang, Zhongyuan Wang, Renrui Zhang, and Shanghang Zhang. Lift3d foun- dation policy: Lifting 2d large-scale pretrained models for robust 3d robotic manipulation, 2024. 1

    2024

  13. [21]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 4015–4026, 2023. 7

    2023

  14. [22]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv:2305.10355, 2023. 5

    2023 arXiv

  15. [23]

    Uni- moe: Scaling unified multimodal llms with mixture of ex- perts

    Yunxin Li, Shenyuan Jiang, Baotian Hu, Longyue Wang, Wanqi Zhong, Wenhan Luo, Lin Ma, and Min Zhang. Uni- moe: Scaling unified multimodal llms with mixture of ex- perts. arXiv preprint arXiv:2405.11273, 2024. 4

    2024 arXiv

  16. [24]

    Mini-gemini: Mining the potential of multi-modality vision language models

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv:2403.18814, 2024. 1, 2

    2024 arXiv

  17. [25]

    Moe-llava: Mixture of experts for large vision- language models

    Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Jinfa Huang, Junwu Zhang, Yatian Pang, Munan Ning, et al. Moe-llava: Mixture of experts for large vision- language models. arXiv preprint arXiv:2401.15947 , 2024. 4

    2024 arXiv

  18. [26]

    Draw-and-understand: Leveraging visual prompts to enable mllms to comprehend what you want

    Weifeng Lin, Xinyu Wei, Ruichuan An, Peng Gao, Bocheng Zou, Yulin Luo, Siyuan Huang, Shanghang Zhang, and Hongsheng Li. Draw-and-understand: Leveraging visual prompts to enable mllms to comprehend what you want. arXiv preprint arXiv:2403.20271, 2024. 2

    2024 arXiv

  19. [27]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv:2310.03744, 2023. 1, 2, 3, 5

    2023 arXiv

  20. [28]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 5

    2024

  21. [29]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 2024. 5

    2024

  22. [30]

    MMBench: Is your multi-modal model an all-around player? arXiv:2307.06281, 2023

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. MMBench: Is your multi-modal model an all-around player? arXiv:2307.06281, 2023. 5

    2023 arXiv

  23. [31]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feicht- enhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition , pages 11976–11986,

  24. [32]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems , 35:2507–2521,

  25. [33]

    Llm as dataset ana- lyst: Subpopulation structure discovery with large language model

    Yulin Luo, Ruichuan An, Bocheng Zou, Yiming Tang, Ji- aming Liu, and Shanghang Zhang. Llm as dataset ana- lyst: Subpopulation structure discovery with large language model. In European Conference on Computer Vision, pages 235–252. Springer, 2025. 2

    2025

  26. [34]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 2

    2023 arXiv

  27. [35]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  28. [36]

    Am-radio: Agglomerative vision foundation model reduce all domains into one

    Mike Ranzinger, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. Am-radio: Agglomerative vision foundation model reduce all domains into one. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12490–12500, 2024. 1, 3, 5, 6

    2024

  29. [37]

    When do we not need larger vision models? In European Conference on Computer Vision, pages 444–462

    Baifeng Shi, Ziyang Wu, Maolin Mao, Xin Wang, and Trevor Darrell. When do we not need larger vision models? In European Conference on Computer Vision, pages 444–462. Springer, 2025. 2

    2025

  30. [38]

    Eagle: Exploring the design space for multimodal llms with mixture of encoders

    Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, De-An Huang, Hongxu Yin, Karan Sapra, Yaser Yacoob, Humphrey Shi, et al. Eagle: Exploring the design space for multimodal llms with mixture of encoders. arXiv preprint arXiv:2408.15998, 2024. 1, 5

    2024 arXiv

  31. [39]

    Llava-mod: Making llava tiny via moe knowledge distillation

    Fangxun Shu, Yue Liao, Le Zhuo, Chenning Xu, Guanghao Zhang, Haonan Shi, Long Chen, Tao Zhong, Wanggui He, Siming Fu, et al. Llava-mod: Making llava tiny via moe knowledge distillation. arXiv preprint arXiv:2408.15881 ,

    arXiv

  32. [40]

    Towards VQA models that can read

    Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards VQA models that can read. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pages 8317–8326, 2019. 5

    2019

  33. [41]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Jo- han Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv:2312.11805, 2023. 2

    2023 arXiv

  34. [42]

    Cambrian- 1: A fully open, vision-centric exploration of multimodal llms

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian- 1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.16860, 2024. 1

    2024 arXiv

  35. [43]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv:2302.13971, 2023. 2

    2023 arXiv

  36. [44]

    Contrastive learning rivals masked image modeling in fine-tuning via feature distillation

    Yixuan Wei, Han Hu, Zhenda Xie, Zheng Zhang, Yue Cao, Jianmin Bao, Dong Chen, and Baining Guo. Contrastive learning rivals masked image modeling in fine-tuning via feature distillation. arXiv preprint arXiv:2205.14141, 2022. 2, 5

    2022 arXiv

  37. [45]

    Beyond full fine-tuning: Harnessing the power of lora for multi-task instruction tuning

    Chunlei Xin, Yaojie Lu, Hongyu Lin, Shuheng Zhou, Hui- jia Zhu, Weiqiang Wang, Zhongyi Liu, Xianpei Han, and Le Sun. Beyond full fine-tuning: Harnessing the power of lora for multi-task instruction tuning. In Proceedings of the 2024 Joint International Conference on Computatio...

    2024

  38. [46]

    One student knows all experts know: From sparse to dense

    Fuzhao Xue, Xiaoxin He, Xiaozhe Ren, Yuxuan Lou, and Yang You. One student knows all experts know: From sparse to dense. arXiv preprint arXiv:2201.10890, 2022. 3

    2022 arXiv

  39. [47]

    Baichuan 2: Open large-scale language models

    Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305, 2023. 2

    2023 arXiv

  40. [48]

    mplug-owl: Modularization empowers large language models with multimodality

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv:2304.14178,

    arXiv

  41. [49]

    Learn- ing from multiple teacher networks

    Shan You, Chang Xu, Chao Xu, and Dacheng Tao. Learn- ing from multiple teacher networks. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1285–1294, 2017. 3

    2017

  42. [50]

    Yi: Open foundation models by 01

    Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652, 2024. 2

    2024 arXiv

  43. [51]

    Sparsevlm: Vi- sual token sparsification for efficient vision-language model inference

    Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, et al. Sparsevlm: Vi- sual token sparsification for efficient vision-language model inference. arXiv preprint arXiv:2410.04417, 2024. 7

    arXiv

  44. [52]

    Beyond llava-hd: Diving into high-resolution large multimodal models

    Yi-Fan Zhang, Qingsong Wen, Chaoyou Fu, Xue Wang, Zhang Zhang, Liang Wang, and Rong Jin. Beyond llava-hd: Diving into high-resolution large multimodal models. arXiv preprint arXiv:2406.08487, 2024. 2

    2024 arXiv

  45. [53]

    Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023. 5, 6

    2023

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.