Pith. sign in

REVIEW 6 major objections 7 minor 1 cited by

MoSE: Skill-by-Skill Mixture-of-Experts Learning for Embodied Autonomous Machines

T0 review · 6 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A sub-3B mixture-of-experts vision-language model, routed by human-defined skills rather than token statistics, matches 8B-level specialists on autonomous-driving and robot-reasoning benchmarks.

desk verdict Skill-oriented MoE for embodied reasoning is a real idea with a striking robot result, but the missing router-pretraining ablation and thin CODA delta keep it from being SOTA as claimed. read the letter →

arxiv 2507.07818 v2 pith:4TUSP5EJ submitted 2025-07-10 cs.AI cs.CVcs.LG

Lu Xu , Jiaqian Yu , Xiongfeng Peng , Yiwei Chen , Weiming Li , Jaewook Yoo , Sunghyun Chunag , Dongwook Lee
show 2 more authors
Daehyun Ji Chao Zhang
This is my paper
classification cs.AIcs.CVcs.LG
keywords mixture-of-expertsvision-languagemodelautonomousdrivingroboticmanipulationskill-basedroutinghierarchicalreasoningembodiedAIparameterefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a small vision-language model (under 3B activated parameters) can outperform much larger embodied-reasoning specialists if its mixture-of-experts layers are guided by human-defined skills rather than left to learn routing from token statistics alone. It introduces MoSE, which labels training examples with hierarchical driving or manipulation skills, pretrains the router on those labels, and then fine-tunes the whole model in one forward pass spanning perception, prediction, and planning. On the CODA autonomous-driving reasoning benchmark MoSE scores 66.03, beating a same-backbone MoE baseline (64.33) and matching or exceeding 7B-8B specialists, and on Robo2VLM it scores 47.65 versus 33.57 for the MoE baseline. The authors' point is that careful skill-oriented routing is what makes small models data-efficient enough to specialize without large-scale training data.

What carries the argument

The load-bearing object is the skill-centric router: a standard MoE router is repurposed so that, at each hierarchical stage, it selects among experts labeled with skills defined by LLMs and human experts, such as detecting key objects, predicting behavior, and evaluating relevance. Before fine-tuning, the router is pretrained with a cross-entropy loss on the mean token $T_{\mathrm{Mean}}$ of the previous layer, using 2,000 rule- or GPT-annotated examples; afterward the full model is fine-tuned with the usual next-token loss while the vision encoder stays frozen. Even-numbered transformer FFNs are replaced by MoSE layers with one shared expert and four skill experts, and hierarchical layers are grouped into perception, prediction, and planning blocks so the model reasons step by step in a single forward pass.

What would settle it

Train the exact MoSE architecture on CODA and Robo2VLM with the router-pretraining loss (Eq. 2) removed, keeping all other settings identical; if scores stay at roughly 66.03 and 47.65, the skill labels are not doing the work, and the result reduces to an MoE capacity effect.

Watch

Extended reading notes

Core claim

The central discovery is that routing experts by predefined, hierarchically organized skills—not by learned token clusters—lets a sub-3B VLM reach scores competitive with 8B-level specialist models on corner-case driving reasoning and robot manipulation reasoning. The model assigns each layer group to a reasoning stage (perception, prediction, planning), routes tokens through skill-specific experts plus one shared expert, and produces a skill chain that doubles as an interpretable explanation. With 2,000 skill-annotated samples used only to pretrain the router, MoSE reaches 66.03 on CODA and 47.65 on Robo2VLM; the authors report that increasing the skill label count to 3,000 does not help, and that expert weights become more diverse than in a standard MoE.

Load-bearing premise

The load-bearing premise is that the 2,000 rule- and GPT-annotated skill labels used to pretrain the router are what produce the expert specialization and the reported gains; the paper never ablates MoSE with and without router pretraining, so if the same architecture trained with only the standard language loss reaches the same scores, the skill mechanism is not the cause.

Editorial extensions

If this is right

  • A sub-3B MoE VLM can match 7B-8B embodied-reasoning specialists on CODA and Robo2VLM, which would make high-level driving and manipulation reasoning feasible on cheaper hardware.
  • Because skill routing produces an explicit skill chain in one forward pass, users get an interpretable step-by-step reasoning trace without multi-round dialogue or extra computation.
  • MoSE's performance keeps climbing as training data grows, while the plain MoE baseline plateaus, suggesting skill routing extracts more value from embodied datasets.
  • The expert-group scaling variant adds capacity without retraining the router or collecting more skill annotations, so the method can grow with data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the skill labels are truly causal, the recipe should transfer to other domains with stable skill taxonomies—say, medical image reasoning or indoor navigation—provided someone defines the hierarchy; nothing in the method depends on driving or manipulation specifically.
  • The 2,000-versus-3,000 label plateau suggests annotation volume is not the bottleneck; what matters is the quality and coverage of the skill definitions, so the practical effort should go into skill taxonomy design rather than scaling annotation.
  • A direct test of the interpretability claim would be measuring whether the emitted skill chains align with human-annotated reasoning steps on CODA; the paper visualizes skill chains but does not quantify alignment.
  • The reported inference-speed advantage is stated without acceleration tricks, but the paper does not report end-to-end latency on robot hardware; measuring wall-clock time on an edge device would test the deployment claim.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 7 minor

Summary. The paper proposes Mixture-of-Skill-Experts (MoSE), a sparse MoE architecture for small (<3B parameter) vision-language models in autonomous driving and robot manipulation. The method defines task-specific hierarchical skills, annotates a small skill dataset (rule-based/GPT), pretrains the router with a supervised cross-entropy loss on the mean token of the previous layer, and then fine-tunes the MoSE blocks. Experiments on CODA and Robo2VLM report that MoSE outperforms a same-backbone MoE baseline and several larger specialist models, and an expert-group variant scales capacity without retraining the router.

Significance. The potential contribution is practical: if the results hold, MoSE gives a way to inject domain skill structure into small MoE VLMs, improving reasoning with fewer active parameters than 7B-8B specialists. The inclusion of a same-backbone MoE baseline is a strength, and the Robo2VLM improvement (47.65 vs 33.57) is large and encouraging; evaluation on held-out test sets with an external judge also speaks against circularity. However, the CODA gains over the MoE baseline are modest (66.03 vs 64.33 and 66.40 vs 64.33), no variance is reported, and the key mechanism is not isolated by an ablation. The paper's novelty is plausible but not yet established.

major comments (6)
  1. [Router Pretraining / Table 3] The central claim that the skill-label router pretraining of Eq. (2) is the cause of the reported gains is not tested. The paper never compares MoSE against MoSE without router pretraining or with shuffled/random skill labels. Table 3 only varies the skill-annotation budget from 2,000 to 3,000 samples, which cannot distinguish the semantic content of the labels from the extra pretraining step. Because MoSE also differs from the baseline in layer allocation ([4,5,5]), the always-on shared expert, and possibly the number of activated experts, the observed improvements could be a generic MoE/capacity effect. Please add these controls; they are the load-bearing experiment for the method.
  2. [Experimental Results / Tables 1-2] All performance numbers are single-run point estimates. The CODA gap between MoSE and the MoE baseline is small (66.03 vs 64.33; 66.40 vs 64.33 for the expert-group variant), so without seed variance or a significance test it is not clear that this difference is reliable. The Robo2VLM difference is large, but it is still a single run. Please report at least three seeds with means and standard deviations for the main comparisons, or explicitly state the number of seeds used.
  3. [Table 3 vs Table 1] The 2,000-label row of Table 3 is internally inconsistent with Table 1: Table 1 reports MoSE Regional=71.10 and Suggest.=68.90, while Table 3 reports Regional=68.9 and Suggest.=71.1 for the same 2,000-label setting, with the same overall Score=66.03. Please correct the inconsistent columns and state which table is authoritative; as printed, the skill-data-size conclusion in Table 3 cannot be verified.
  4. [Introduction] The Introduction claims 'Without any acceleration techniques, our model improves inference speed by at least 15.7%', but no latency, throughput, or wall-clock measurement, nor the comparison protocol, appears in the experimental section. Since efficiency is a central motivation, this quantitative claim must be either supported by an experiment or removed.
  5. [Contribution list / Table 1] The abstract and contribution list say MoSE 'outperforms' and achieves 'state-of-the-art' performance on AD corner-case reasoning, but Table 1 lists systems with higher scores than MoSE's 66.03, e.g., TSLVLM+GPT4V at 72.12 and RtoG(GPT-4o) at 71.39. Please qualify the claim to open-source specialist models without external GPT refinement, or justify the exclusion of these comparisons.
  6. [Expert Networks and Training] The training description is contradictory: the text says 'we conduct supervised fine-tuning on the entire model' and then says 'we only finetune the MoSE blocks while keeping the rest of the model frozen.' In addition, Eq. (2) uses the mean token of the previous layer as router input, but the paper does not state whether this mean token is also used at inference, which would make every token in a layer share the same skill expert. Please clarify the exact parameter updates and the token-level vs layer-level routing behavior, as both affect the comparison with the token-level MoE baseline.
minor comments (7)
  1. [Experimental Results] The text refers to 'Table 1 shows our results on the Robo2VLM dataset' when the Robo2VLM results are in Table 2; please correct the cross-reference.
  2. [Related Work] The method is named 'Med-MoE' in the Related Work but the corresponding reference is titled 'MoE-TinyMed'; please harmonize the name and the citation.
  3. [Throughout] The paper contains several typos and inconsistent notations: 'layesr', 'manopulation', 'Qwen2VL' vs 'Qwen2-VL', 'W.Exp Grp' in the Table 1 note, and 'Left:Moe' in Figure 8; please proofread.
  4. [Supplementary Material] The annotation prompts, the skill-chain definition, and additional expert-weight visualizations are deferred to supplementary material, but no supplementary document is included with the manuscript; please either include the appendix or remove the references.
  5. [Experimental Setup] The training hyperparameters for router pretraining and supervised fine-tuning (learning rate, epochs, batch size, optimizer, number of router-pretraining steps) are not reported; please add them to the experimental setup.
  6. [Skill Data Annotation] Please clarify whether the 2,000/3,000 skill-annotation samples are drawn exclusively from each dataset's training split, since annotation samples from the test split would compromise the evaluation.
  7. [Skill Chain / Figure 2] The 'skill chain' is described as supporting reasoning and interpretability, but no quantitative or human evaluation of interpretability is provided; please state whether it is a post-hoc visualization or an auxiliary output.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MoSE's performance claims are measured on external held-out benchmarks, and the skill-label router pretraining (Eq. 2) is an auxiliary objective that does not appear in the evaluation score.

full rationale

The derivation chain is: define and annotate skill labels on a training subset, pretrain routers with Eq. (2), then fine-tune with the language loss Eq. (3), and finally report CODA and Robo2VLM scores evaluated on official test sets by GPT-4o. The final scores are not computed from the skill labels or from the router-pretraining objective; they are functions of the model's generated text against held-out ground truth. The comparison to a same-backbone general-MoE baseline (Qwen2-VL MoE) provides an external control, and the cited prior work (MoE-LLaVA, Med-MoE, Qwen2-VL) is not self-citational. The main scientific limitation is that the paper never ablates router pretraining with zero or shuffled skill labels, so the causal contribution of the skill mechanism is not isolated; however, a missing control is a correctness/evidence concern, not circularity, because no reported quantity reduces by construction to the skill annotations. The use of GPT-4o both for skill annotation and for evaluation creates a potential evaluator-alignment confound, but it is not a formal reduction of the prediction to its inputs. Therefore no circular step is established.

Assumptions & free parameters 6 free parameters · 4 assumptions · 1 invented entities

The central mechanism depends on manually defined skill taxonomies, hand-selected architecture hyperparameters, and a small labeled annotation set. The only direct evidence that the skill labels matter is the MoSE versus MoE comparison; the ledger records the design choices that are not derived from first principles or from external benchmarks.

free parameters (6)
  • number of MoSE layers = 14
    Chosen to replace FFN layers in all even-numbered transformer layers of Qwen2-VL-2B; no sensitivity analysis is given.
  • layer-to-skill allocation = [4,5,5]
    Allocated to perception, prediction, and planning (and robot equivalents); the text says a uniform hierarchical distribution yields better performance, but no comparison table is shown.
  • router pretraining sample size = 2,000
    Chosen for both AD and robot tasks; Table 3 shows 3,000 yields no gain, but no smaller sizes are tested.
  • experts per MoSE layer = 4 skill experts + 1 shared expert
    Used for both pretraining and finetuning; no sweep is reported.
  • input image resolution = [800,450]
    Chosen to balance image quality and GPU memory; not part of the claimed method.
  • scaling-up expert groups = n=2 (image/text)
    Each skill expert is split into two modality-specific experts in the scaling variant; no comparison with other n values is given.
assumptions (4)
  • domain assumption Driving and manipulation reasoning can be decomposed into a small hierarchical skill taxonomy (perception, prediction, planning, and equivalent robot skills).
    Assumed in Skill Definition and shown in Fig. 3; if wrong, the routing labels do not correspond to useful competencies.
  • domain assumption The router can be pretrained from the mean token of the previous layer (Eq. 2) and this transfers to the full model.
    Introduced in Router Pretraining; no analysis of information loss from mean-pooling is provided.
  • domain assumption GPT-4o-based skill definition and GPT-4o-based evaluation produce consistent, unbiased quality measures.
    Skills are defined with GPT and refined by humans; CODA scores are GPT-4o judgments without human agreement statistics.
  • domain assumption The official CODA and Robo2VLM train/test splits are used as stated.
    The paper follows 'official training and testing settings' (Dataset Settings), but no code or processed data confirms the splits.
invented entities (1)
  • Skill chain
    purpose: A sequence of activated skills across MoSE layers used to guide generation and provide interpretability (Fig. 2c).
    The skill chain is visible only inside the model; no external prediction or test is derived from it, so it functions as an internal explanatory structure rather than an independently verified entity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MoSE: Skill-by-Skill Mixture-of-Experts Learning for Embodied Autonomous Machines." pith.science (2026). https://pith.science/paper/4TUSP5EJ

@misc{pith2026250707818,
  author       = {Pith},
  title        = {Pith review of: MoSE: Skill-by-Skill Mixture-of-Experts Learning for Embodied Autonomous Machines},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4TUSP5EJ}},
  note         = {Machine review of arXiv:2507.07818}
}
read the original abstract

To meet the growing demand for smarter, faster, and more efficient embodied AI solutions, we introduce a novel Mixture-of-Expert (MoE) method that significantly boosts reasoning and learning efficiency for embodied autonomous systems. General MoE models demand extensive training data and complex optimization, which limits their applicability in embodied AI such as autonomous driving (AD) and robotic manipulation. In this work, we propose a skill-oriented MoE called MoSE, which mimics the human learning and reasoning process skill-by-skill, step-by-step. We introduce a skill-oriented routing mechanism that begins with defining and annotating specific skills, enabling experts to identify the necessary competencies for various scenarios and reasoning tasks, thereby facilitating skill-by-skill learning. To better align with multi-step planning in human reasoning and in end-to-end driving models, we build a hierarchical skill dataset and pretrain the router to encourage the model to think step-by-step. Unlike other multi-round dialogues, MoSE integrates valuable auxiliary tasks (e.g. perception-prediction-planning for AD, and high-level and low-level planning for robots) in one single forward process without introducing any extra computational cost. With less than 3B sparsely activated parameters, our model effectively grows more diverse expertise and outperforms models on both AD corner-case reasoning tasks and robot reasoning tasks with less than 40% of the parameters.

Figures

Figures reproduced from arXiv: 2507.07818 by the authors.

Figure 1
Figure 1. Activated model size and performance of various [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overall model pipeline: (a) the step-by-step process of the model, where it progresses from perception to prediction [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The process of constructing skill data consists of [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Pipeline for multi-modal scaling up. This approach [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Visualization for reasoning results, where GPT Evaluation reflects the GPT model’s assessment of the generated [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Visualization of the Robot Manipulation reason [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Performance variations of MoE and MoSE after training on different data scales. MoSE demonstrates better perfor [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Visualization of expert weight distribution similar [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. D$^3$-MoE:Dual Disentangled Diffusion Mixture-of-Experts for Style-Controllable End-to-End Autonomous Driving

    cs.RO 2026-06 unverdicted novelty 6.0 of 10

    D³-MoE disentangles style and physical axes with diffusion and self-supervised MoE experts to produce style-controllable trajectories, reporting SOTA 88.2 PDMS on NAVSIM.

Reference graph

Works this paper leans on

44 extracted references · 14 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023

    arXiv 2023

  3. [3]

    Stable lm 2 1.6 b technical report

    Marco Bellagente, Jonathan Tow, Dakota Mahan, Duy Phung, Maksym Zhuravinskyi, Reshinth Adithyan, James Baicoianu, Ben Brooks, Nathan Cooper, Ashish Datta, et al. Stable lm 2 1.6 b technical report. arXiv preprint arXiv:2402.17834, 2024

    arXiv 2024

  4. [4]

    Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom

    Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. arXiv preprint arXiv:1903.11027, 2019

    arXiv 1903

  5. [5]

    Shikra: Unleashing multimodal llm's referential dialogue magic

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm's referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023 a

    arXiv 2023

  6. [6]

    Automated evaluation of large vision-language models on self-driving corner cases

    Kai Chen, Yanze Li, Wenhua Zhang, Yanxin Liu, Pengxiang Li, Ruiyuan Gao, Lanqing Hong, Meng Tian, Xinhai Zhao, Zhenguo Li, et al. Automated evaluation of large vision-language models on self-driving corner cases. arXiv preprint arXiv:2404.10595, 2024 a

    arXiv 2024

  7. [7]

    Driving with llms: Fusing object-level vector modality for explainable autonomous driving

    Long Chen, Oleg Sinavski, Jan H \"u nermann, Alice Karnsund, Andrew James Willmott, Danny Birch, Daniel Maund, and Jamie Shotton. Driving with llms: Fusing object-level vector modality for explainable autonomous driving. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 14093--14100. IEEE, 2024 b

    work page 2024

  8. [8]

    AdaMV-MoE : Adaptive multi-task vision mixture-of-experts

    Tianlong Chen, Xuxi Chen, Xianzhi Du, Abdullah Rashwan, Fan Yang, Huizhong Chen, Zhangyang Wang, and Yeqing Li. AdaMV-MoE : Adaptive multi-task vision mixture-of-experts. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 17346--17357, 2023 b

    work page 2023

Show all 44 references
  1. [9]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024 c

    2024 arXiv

  2. [10]

    Qwen-vl-max: A high-performance vision-language model, 2024

    Alibaba Cloud. Qwen-vl-max: A high-performance vision-language model, 2024

    2024

  3. [11]

    Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. Deepseekmoe: Towards ultimate expert specialization in mixture-of-ex...

    2024

  4. [12]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv...

    2010 arXiv

  5. [13]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23 0 (120): 0 1--39, 2022

    2022

  6. [14]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini Team Google: Petko Georgiev and 1135 other authors. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  7. [15]

    From regional to general: A vision-language model-based framework for corner cases comprehension in autonomous driving

    XU HAN, Yehua Huang, Xiaowen Chu, et al. From regional to general: A vision-language model-based framework for corner cases comprehension in autonomous driving. In ECCV 2024 Workshop on Multimodal Perception and Comprehension of Corner Cases in Autonomous Driving

    2024

  8. [16]

    Planning-oriented autonomous driving

    Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planning-oriented autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17853--17862, 2023

    2023

  9. [17]

    Drivemm: All-in-one large multimodal model for autonomous driving

    Zhijian Huang, Chengjian Feng, Feng Yan, Baihui Xiao, Zequn Jie, Yujie Zhong, Xiaodan Liang, and Lin Ma. Drivemm: All-in-one large multimodal model for autonomous driving. arXiv preprint arXiv:2412.07689, 2024 a

    2024 arXiv

  10. [18]

    Making large language models better planners with reasoning-decision alignment, 2024 b

    Zhijian Huang, Tao Tang, Shaoxiang Chen, Sihao Lin, Zequn Jie, Lin Ma, Guangrun Wang, and Xiaodan Liang. Making large language models better planners with reasoning-decision alignment, 2024 b

    2024

  11. [19]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne...

    2024

  12. [20]

    Moe-tinymed: Mixture of experts for tiny medical large vision-language models

    Songtao Jiang, Tuo Zheng, Yan Zhang, Yeying Jin, and Zuozhu Liu. Moe-tinymed: Mixture of experts for tiny medical large vision-language models. arXiv preprint arXiv:2404.10237, 2024 b

    2024 arXiv

  13. [21]

    Gshard: Scaling giant models with conditional computation and automatic sharding

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020

    2006 arXiv

  14. [22]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  15. [23]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730--19742. PMLR, 2023

    2023

  16. [24]

    Moe-llava: Mixture of experts for large vision-language models

    Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Jinfa Huang, Junwu Zhang, Yatian Pang, Munan Ning, et al. Moe-llava: Mixture of experts for large vision-language models. arXiv preprint arXiv:2401.15947, 2024 a

    2024 arXiv

  17. [25]

    Moma: Efficient early-fusion pre-training with mixture of modality-aware experts, 2024 b

    Xi Victoria Lin, Akshat Shrivastava, Liang Luo, Srinivasan Iyer, Mike Lewis, Gargi Ghosh, Luke Zettlemoyer, and Armen Aghajanyan. Moma: Efficient early-fusion pre-training with mixture of modality-aware experts, 2024 b

    2024

  18. [26]

    Improved baselines with visual instruction tuning, 2023 a

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023 a

    2023

  19. [27]

    Visual instruction tuning, 2023 b

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023 b

    2023

  20. [28]

    Llava-next: Improved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024

    2024

  21. [29]

    GPT -driver: Learning to drive with gpt

    Jiageng Mao, Yuxi Qian, Junjie Ye, Hang Zhao, and Yue Wang. GPT -driver: Learning to drive with gpt. arXiv preprint arXiv:2310.01415, 2023

    2023 arXiv

  22. [30]

    Gpt-4v system card

    OpenAI. Gpt-4v system card. OpenAI, 2023

    2023

  23. [31]

    Languagempc: Large language models as decision makers for autonomous driving

    Hao Sha, Yao Mu, Yuxuan Jiang, Li Chen, Chenfeng Xu, Ping Luo, Shengbo Eben Li, Masayoshi Tomizuka, Wei Zhan, and Mingyu Ding. Languagempc: Large language models as decision makers for autonomous driving. arXiv preprint arXiv:2310.03026, 2023

    arXiv

  24. [32]

    Lmdrive: Closed-loop end-to-end driving with large language models

    Hao Shao, Yuxuan Hu, Letian Wang, Guanglu Song, Steven L Waslander, Yu Liu, and Hongsheng Li. Lmdrive: Closed-loop end-to-end driving with large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15120--15130, 2024

    2024

  25. [33]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017

    2017 arXiv

  26. [34]

    Drivelm: Driving with graph visual question answering

    Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Jens Bei wenger, Ping Luo, Andreas Geiger, and Hongyang Li. Drivelm: Driving with graph visual question answering. arXiv preprint arXiv:2312.14150, 2023

    2023 arXiv

  27. [35]

    Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024 a

    2024 arXiv

  28. [36]

    Omnidrive: A holistic llm-agent framework for autonomous driving with 3d perception, reasoning and planning

    Shihao Wang, Zhiding Yu, Xiaohui Jiang, Shiyi Lan, Min Shi, Nadine Chang, Jan Kautz, Ying Li, and Jose M Alvarez. Omnidrive: A holistic llm-agent framework for autonomous driving with 3d perception, reasoning and planning. arXiv preprint arXiv:2405.01533, 2024 b

    2024 arXiv

  29. [37]

    On the road with gpt-4v (ision): Early explorations of visual-language model on autonomous driving

    Licheng Wen, Xuemeng Yang, Daocheng Fu, Xiaofeng Wang, Pinlong Cai, Xin Li, Tao Ma, Yingxuan Li, Linran Xu, Dengke Shang, et al. On the road with gpt-4v (ision): Early explorations of visual-language model on autonomous driving. arXiv preprint arXiv:2311.05332, 2023

    2023 arXiv

  30. [38]

    Two-stage lvlm system: 1st place solution for eccv 2024 corner case scene understanding challenge

    Ying Xue, Haiming Zhang, Yiyao Zhu, Wending Zhou, Shuguang Cui, and Zhen Li. Two-stage lvlm system: 1st place solution for eccv 2024 corner case scene understanding challenge. In ECCV 2024 Workshop on Multimodal Perception and Comprehension of Corner Cases in Autonomous Driving

    2024

  31. [39]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024

    2024 arXiv

  32. [40]

    Flex-moe: Modeling arbitrary modality combination via the flexible mixture-of-experts

    Sukwon Yun, Inyoung Choi, Jie Peng, Yangfan Wu, Jingxuan Bao, Qiyiwen Zhang, Jiayi Xin, Qi Long, and Tianlong Chen. Flex-moe: Modeling arbitrary modality combination via the flexible mixture-of-experts. NeurIPS, 2024

    2024

  33. [41]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975--11986, 2023

    2023

  34. [42]

    Minidrive: More efficient vision-language models with multi-level 2d features as text tokens for autonomous driving

    Enming Zhang, Xingyuan Dai, Yisheng Lv, and Qinghai Miao. Minidrive: More efficient vision-language models with multi-level 2d features as text tokens for autonomous driving. arXiv preprint arXiv:2409.07267, 2024 a

    2024 arXiv

  35. [43]

    Diversifying the expert knowledge for task-agnostic pruning in sparse mixture-of-experts, 2024 b

    Zeliang Zhang, Xiaodong Liu, Hao Cheng, Chenliang Xu, and Jianfeng Gao. Diversifying the expert knowledge for task-agnostic pruning in sparse mixture-of-experts, 2024 b

    2024

  36. [44]

    St-moe: Designing stable and transferable sparse expert models

    Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models. arxiv 2022. arXiv preprint arXiv:2202.08906

    2022 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.