Pith. sign in

REVIEW 2 major objections 4 minor 77 references

Automated multi-objective search converts a homogeneous vision-language model into a heterogeneous one that matches teacher accuracy while more than doubling decoding speed.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 00:52 UTC pith:VH7HI47I

load-bearing objection Solid engineering paper: first automated MIP search over per-layer attention mechanisms for VLMs, delivers a public 4B model that matches its teacher at 2.5 imes decode with <2% training cost. the 2 major comments →

arxiv 2607.09029 v1 pith:VH7HI47I submitted 2026-07-10 cs.CV

MOSAIC: Adaptive Inter-layer Composition for Efficient Heterogeneous Vision-Language Models

Yuncheng Yang , Feiyang Ye , Shixian Luo , Yinna Zhu , Lianlei Shan , Wangcai Zhao , Kuo Zhang , Yan Chen
show 2 more authors
Yong Wu Yan Xie
This is my paper
classification cs.CV
keywords vision-language modelsheterogeneous architecturesneural architecture searchmixed integer programmingknowledge distillationefficient attentionlinear attentionhardware-aware search
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Homogeneous vision-language models pay a quadratic cost for full attention and are hard to reshape for specific hardware. Prior hybrid designs mix efficient operators by hand with fixed ratios, which is suboptimal and not hardware-adaptive. MOSAIC treats every layer as a choice among linear, sparse, low-rank, and full attention plus FFN width options, scores candidates cheaply by local distillation, and solves a multi-objective mixed-integer program that maximizes language and vision capability under a measured latency budget. After the structure is chosen, a two-stage recovery—global off-policy distillation then dual-teacher on-policy distillation with a large oracle and the original teacher—restores performance. The resulting MOSAIC-4B matches its 4B teacher across image, video, and text benchmarks, delivers roughly 1.8× prefilling and 2.5× decoding speedups, and costs less than 2% of the original training budget. A sympathetic reader cares because the method turns expensive hand redesign into a constrained optimization problem that can be re-run for new latency targets.

Core claim

A homogeneous VLM can be automatically rewritten as an optimized heterogeneous stack by solving a multi-objective MIP over a unified per-layer search space of efficient attention and FFN variants; after two-stage dual-teacher distillation, the derived model matches the original teacher’s average performance while meeting strict hardware speedup targets at a tiny fraction of the original training cost.

What carries the argument

Multi-objective Mixed Integer Programming (MIP) over binary per-layer selections: each candidate is scored by relative capability (KL, PPL, LLM and VLM benchmarks) after blockwise local distillation, then the solver maximizes a Tchebycheff scalarization subject to an additive measured runtime budget.

Load-bearing premise

Scores obtained by swapping one candidate block into an otherwise teacher-driven residual stream remain predictive of how a fully heterogeneous stack will perform after global recovery, so the MIP solution is near-optimal for the finished model.

What would settle it

Run the full MOSAIC pipeline on Qwen3-VL-4B-Instruct under the stated 1.5× latency budget; if the recovered model falls more than a few points behind the teacher’s average on the same 19 image/video/text benchmarks, or fails to realize the claimed TTFT/TPOT speedups in the same vLLM-style measurement, the central claim is false.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper introduces MOSAIC, a hardware-aware method that converts a homogeneous VLM (Qwen3-VL-4B-Instruct) into a heterogeneous architecture by searching a per-layer space of linear (KDA/GDN), sparse (SWA), low-rank (MLA), and full (GQA) attention operators plus FFN width scalings. Selection is cast as a multi-objective MIP (Tchebycheff scalarization of KL, PPL, LLM and VLM scores subject to an additive profiled latency budget). After blockwise local distillation for scoring, a two-stage recovery (global off-policy KD from the 4B teacher, then dual-teacher on-policy distillation with a 235B oracle) yields MOSAIC-4B. Empirically, MOSAIC-4B matches the teacher’s average performance on 19 image/video/text benchmarks while delivering 1.76× prefilling and 2.54× decoding speed-ups at <2 % of the original training cost; systematic ablations of fixed-ratio baselines, search-space components, objectives and recovery stages are provided, and the model is released publicly.

Significance. If the empirical results hold under independent verification, the work is significant for efficient multimodal modeling. It is the first automated, hardware-constrained search over heterogeneous attention mechanisms for VLMs (rather than hand-crafted periodic interleaving or unimodal size-only NAS). The public MOSAIC-4B checkpoint, the open MIP formulation, the two-stage recovery recipe, and the extensive ablations (Tables 4–7, Figs. 3–4,7) supply a concrete, reproducible path from a strong homogeneous teacher to a faster student at modest extra cost. The discovered layer-wise patterns (SWA early, linear mid-network, MLA/GQA late) also offer transferable design intuitions. These contributions are of clear practical value for long-context and resource-constrained VLM deployment.

major comments (2)
  1. [Section 3.2, Eqs. (2),(4)–(6)] Section 3.2 (Eqs. 2, 4–6) and the BLD procedure: capability scores S^k_{l,i} are obtained by inserting a single candidate into an otherwise teacher residual stream. The MIP then treats these scores and the profiled runtimes as additive. While the final recovered MOSAIC-4B succeeds (Tables 1–3), the paper never directly measures how well the isolated scores rank full heterogeneous stacks after global recovery. Because the central efficiency claim rests on the MIP finding a near-optimal configuration under the latency budget, a short correlation study or a small set of random/full-stack re-evaluations of high- versus low-scoring architectures would strengthen (or bound) the search heuristic.
  2. [Section 4.2, Appendix B.4, Fig. 3] Section 4.2 / Appendix B.4 and Fig. 3: all latency numbers (TTFT, TPOT, block profiles) are measured on a single platform (H800) with a custom-extended vLLM backend that the authors themselves implemented for the heterogeneous operators. The hardware-aware claim is therefore demonstrated only for this stack. Reporting at least one additional backend or device class (or releasing the profiling harness) would make the “strict hardware latency constraints” claim more robust and would let readers assess portability of the discovered architectures.
minor comments (4)
  1. [Tables 1–3] Tables 1–3 report only point estimates; even a single-run standard deviation or bootstrap interval on the average Δ would help readers judge whether the −0.6 % / −0.8 % / −2.3 % gaps are stable.
  2. [Figure 4, Appendix C] Figure 4 and the architecture diagrams in Appendix C annotate FFN pruning with an asterisk, but the precise intermediate dimensions chosen by the MIP for each layer are never listed; a short table or supplementary file would aid reproducibility.
  3. [Section 3.1] The combinatorial size is written “35 36” (missing exponentiation symbol) in Section 3.1; likewise a few minor typos (“w.r.t.”, “prefilling” vs. “prefill”) appear throughout.
  4. [Section 3.3, Appendix B.2] The dual-teacher stage uses a 235B oracle; while total training cost is claimed <2 %, a one-sentence breakdown of GPU-hours spent on oracle generation versus student updates would clarify the practical resource envelope for readers who lack a 235B teacher.

Circularity Check

0 steps flagged

No significant circularity: empirical NAS + recovery pipeline evaluated on external held-out benchmarks; MIP scores and final metrics are not definitionally equivalent.

full rationale

The paper's derivation chain is a standard hardware-aware NAS pipeline followed by distillation recovery. Blockwise Local Distillation (Eq. 1) initializes candidates by MSE alignment to teacher activations; multi-capability scores S^k_{l,i} (Eq. 2) are obtained by single-block insertion into the teacher residual stream and measured on external data (Olmo3 PPL/KL, MMLU, SeedBench). The multi-objective MIP (Eqs. 5–11) simply maximises the additive sum of those measured scores subject to independently profiled runtimes; it does not fit free parameters that are later re-labelled as predictions, nor does any equation reduce a claimed performance number to a fitted input by construction. Final MOSAIC-4B results (Tables 1–3, Fig. 3) are obtained after full two-stage recovery and evaluated on a disjoint suite of 19 public benchmarks; ablations (Tables 4–7) further isolate the contribution of search versus fixed ratios and of each recovery stage. Self-citations are limited to operator definitions (KDA, GDN, MLA, SWA) and standard libraries; none supply a load-bearing uniqueness theorem or smuggled ansatz that forces the central claim. The method is therefore self-contained against external measurements and exhibits no circular reduction.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 1 invented entities

The central claim rests on a small set of free hyper-parameters (latency target, Tchebycheff weights, distillation coefficients) chosen by the authors, standard assumptions of residual-stream compatibility and additive runtime, and the invented search-and-recovery pipeline itself. No new physical entities are postulated; the method is engineering.

free parameters (4)
  • global latency Target (1.5× speedup)
    Hand-chosen constraint that defines the feasible set of the MIP; different targets produce different architectures (Fig. 7).
  • Tchebycheff preference weights w
    Set equal across four objectives; any other weighting would alter the selected architecture.
  • distillation coefficients α=0.1, β=0.5
    Trade-off parameters between hard CE and soft KL losses in the two recovery stages; chosen by the authors.
  • FFN scaling ratios {100 %, 84 %, …, 0 %}
    Discrete intermediate-dimension choices that define the FFN half of the search space.
axioms (3)
  • domain assumption Runtime of a heterogeneous stack is well-approximated by the sum of independently profiled block runtimes.
    Used to write the linear latency constraint (Eq. 6); never verified for non-additive interactions such as cache contention.
  • ad hoc to paper Blockwise local distillation with teacher inputs yields capability scores that remain predictive after global recovery.
    Core modelling assumption of Sec. 3.2 that justifies scoring candidates in isolation.
  • standard math Tchebycheff scalarisation recovers Pareto-optimal solutions of the multi-objective MIP.
    Invoked via reference [37]; standard multi-objective optimisation result.
invented entities (1)
  • MOSAIC search-and-recovery pipeline (BLD + multi-objective MIP + dual-teacher on-policy distillation) no independent evidence
    purpose: To automate the conversion of a homogeneous VLM into a hardware-constrained heterogeneous model while recovering accuracy.
    The entire method is a new engineering construct introduced by the paper; no independent external evidence exists outside the reported experiments.

pith-pipeline@v1.1.0-grok45 · 28176 in / 2691 out tokens · 37670 ms · 2026-07-13T00:52:20.354934+00:00 · methodology

0 comments
read the original abstract

Vision-Language Models (VLMs) have achieved success using homogeneous Transformers to process multimedia data. Recent studies show that heterogeneous structures interleaving efficient mechanisms, like linear attention, improve both performance and inference latency over homogeneous designs. However, these efforts rely on handcrafted static mixing patterns, which are sub-optimal and difficult to adapt to specific hardware. To bridge this gap, we propose Multi-Objective Search for Adaptive Inter-layer Composition (MOSAIC), a hardware-aware search method that automatically transforms homogeneous models into optimized heterogeneous architectures. MOSAIC integrates diverse efficiency mechanisms--including linear, sparse, and low-rank operators--into a unified search space. By formulating the selection as a multi-objective Mixed Integer Programming (MIP) problem, our method identifies optimal configurations that maximize downstream performance under strict hardware latency constraints. To mitigate performance degradation from structural transitions, we introduce a two-stage parameter recovery process: global off-policy distillation to stabilize internal representations, followed by a dual-teacher on-policy distillation leveraging a 235B oracle for knowledge expansion and the original 4B teacher for distributional stability. We validate MOSAIC through MOSAIC-4B, derived from Qwen3-VL-4B-Instruct. Results demonstrate that MOSAIC-4B matches the baseline's performance across multiple benchmarks while requiring less than 2% of the original training cost. Furthermore, it substantially improves inference efficiency, achieving 1.76x prefilling and 2.54x decoding speedups.

Figures

Figures reproduced from arXiv: 2607.09029 by Feiyang Ye, Kuo Zhang, Lianlei Shan, Shixian Luo, Wangcai Zhao, Yan Chen, Yan Xie, Yinna Zhu, Yong Wu, Yuncheng Yang.

Figure 1
Figure 1. Figure 1: Average performance on image understanding bench [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the MOSAIC method, which consists of two steps: (1) Heterogeneous Architecture Search, which identifies the [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Left: The time to first token (TTFT) of Qwen3-VL-4B [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The runtime reduction of layer-wise architectural re [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Hardware performance analysis of various attention variants. Left: Profiling of block-wise latency across different hardware stages, illustrating the specific time breakdown. Right: Comparison of scaling efficiency across prefill lengths from 1k to 96k (FFN cost excluded). purpose VLM captioning dataset commonly employed as multimodal pretraining data. Each block is trained for one epoch with the MSE loss.… view at source ↗
Figure 6
Figure 6. Figure 6: Comparison of end-to-end inference latency. MOSAIC [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: The runtime reduction of layer-wise architectural replacements in MOSAIC under varying latency constraint settings ( [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

77 extracted references · 38 linked inside Pith

  1. [1]

    Composer: A search framework for hybrid neural architecture design

    Bilge Acun, Prasoon Sinha, Newsha Ardalani, Sangmin Bae, Alicia Golden, Chien-Yu Lin, Meghana Madhyastha, Fei Sun, Neeraja J Yadwadkar, and Carole-Jean Wu. Composer: A search framework for hybrid neural architecture design. arXiv preprint arXiv:2510.00379, 2025. 3

    arXiv 2025

  2. [2]

    Llava-onevision-1.5: Fully open framework for democratized multimodal training

    Xiang An, Yin Xie, Kaicheng Yang, Wenkang Zhang, Xiuwei Zhao, Zheng Cheng, Yirui Wang, Songcen Xu, Changrui Chen, Didi Zhu, et al. Llava-onevision-1.5: Fully open framework for democratized multimodal training. arXiv preprint arXiv:2509.23661, 2025. 1, 2

    Pith/arXiv arXiv 2025

  3. [3]

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025. 8

    Pith/arXiv arXiv 2025

  4. [4]

    Long- former: The long-document transformer.arXiv preprint arXiv:2004.05150, 2020

    Iz Beltagy, Matthew E Peters, and Arman Cohan. Long- former: The long-document transformer.arXiv preprint arXiv:2004.05150, 2020. 10, 3

    Pith/arXiv arXiv 2004

  5. [5]

    Worldsense: A synthetic benchmark for grounded reasoning in large language models.arXiv preprint arXiv:2311.15930, 2023

    Youssef Benchekroun, Megi Dervishi, Mark Ibrahim, Jean- Baptiste Gaya, Xavier Martinet, Gr ´egoire Mialon, Thomas Scialom, Emmanuel Dupoux, Dieuwke Hupkes, and Pas- cal Vincent. Worldsense: A synthetic benchmark for grounded reasoning in large language models.arXiv preprint arXiv:2311.15930, 2023. 7

    Pith/arXiv arXiv 2023

  6. [6]

    Puzzle: Distillation-based nas for inference-optimized llms

    Akhiad Bercovich, Tomer Ronen, Talor Abramovich, et al. Puzzle: Distillation-based nas for inference-optimized llms. InICML, 2025. 3, 4

    2025

  7. [7]

    Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al.π 0: A vision-language- action flow model for general robot control.arXiv preprint arXiv:2410.24164, 2024. 1

    Pith/arXiv arXiv 2024

  8. [8]

    Once-for-all: Train one network and specialize it for efficient deployment

    Han Cai et al. Once-for-all: Train one network and specialize it for efficient deployment. InICLR, 2020. 3

    2020

  9. [9]

    Generating long sequences with sparse transformers.arXiv preprint arXiv:1904.10509, 2019

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers.arXiv preprint arXiv:1904.10509, 2019. 2

    Pith/arXiv arXiv 1904

  10. [10]

    Molmo2: Open weights and data for vision-language mod- els with video understanding and grounding.arXiv preprint arXiv:2601.10611, 2026

    Christopher Clark, Jieyu Zhang, Zixian Ma, Jae Sung Park, Mohammadreza Salehi, Rohun Tripathi, Sangho Lee, Zhongzheng Ren, Chris Dongjoo Kim, Yinuo Yang, et al. Molmo2: Open weights and data for vision-language mod- els with video understanding and grounding.arXiv preprint arXiv:2601.10611, 2026. 8

    Pith/arXiv arXiv 2026

  11. [11]

    Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457,

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457,

    Pith/arXiv arXiv

  12. [12]

    Train- ing verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Train- ing verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021. 7

    Pith/arXiv arXiv 2021

  13. [13]

    Deepseek-v2: A strong, economical, and ef- ficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434, 2024

    DeepSeek-AI. Deepseek-v2: A strong, economical, and ef- ficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434, 2024. 2, 1

    Pith/arXiv arXiv 2024

  14. [14]

    Nemotron-climb: Clustering-based it- erative data mixture bootstrapping for language model pre- training.arXiv preprint arXiv:2504.13161, 2025

    Shizhe Diao, Yu Yang, Yonggan Fu, Xin Dong, Dan Su, Markus Kliegl, Zijia Chen, Peter Belcak, Yoshi Suhara, Hongxu Yin, et al. Nemotron-climb: Clustering-based it- erative data mixture bootstrapping for language model pre- training.arXiv preprint arXiv:2504.13161, 2025. 2

    Pith/arXiv arXiv 2025

  15. [15]

    Layernas: Neural architecture search in polynomial complexity.arXiv preprint arXiv:2304.11517,

    Yicheng Fan, Dana Alon, Jingyue Shen, Daiyi Peng, Keshav Kumar, Yun Long, Xin Wang, Fotis Iliopoulos, Da-Cheng Juan, and Erik Vee. Layernas: Neural architecture search in polynomial complexity.arXiv preprint arXiv:2304.11517,

    Pith/arXiv arXiv

  16. [16]

    Mme: A comprehensive evaluation bench- mark for multimodal large language models.arXiv preprint arXiv:2306.13394, 2023

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation bench- mark for multimodal large language models.arXiv preprint arXiv:2306.13394, 2023. 7

    Pith/arXiv arXiv 2023

  17. [17]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in 10 video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in 10 video analysis. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 24108– 24118, 2025. 7

    2025

  18. [18]

    Blink: Multimodal large language models can see but not perceive

    Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. InEuropean Conference on Com- puter Vision, pages 148–166. Springer, 2024. 7

    2024

  19. [19]

    Mamba: Linear-time sequence mod- eling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence mod- eling with selective state spaces. InFirst conference on lan- guage modeling, 2024. 1, 2

    2024

  20. [20]

    Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, et al. Seed1. 5-vl technical report.arXiv preprint arXiv:2505.07062, 2025. 1

    Pith/arXiv arXiv 2025

  21. [21]

    Mea- suring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mea- suring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020. 7

    Pith/arXiv arXiv 2009

  22. [22]

    Step 3.5 flash: Open frontier-level intelligence with 11b active parameters.arXiv preprint arXiv:2602.10604, 2026

    Ailin Huang, Ang Li, Aobo Kong, Bin Wang, Binxing Jiao, Bo Dong, Bojun Wang, Boyu Chen, Brian Li, Buyun Ma, et al. Step 3.5 flash: Open frontier-level intelligence with 11b active parameters.arXiv preprint arXiv:2602.10604, 2026. 3, 7

    arXiv 2026

  23. [23]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. InProceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 6700–6709, 2019. 7

    2019

  24. [24]

    Python-MIP: collection of Python tools for the modeling and solution of mixed-integer linear programs.https://github.com/coin- or/ python-mip, 2023

    COIN-OR Foundation Inc. Python-MIP: collection of Python tools for the modeling and solution of mixed-integer linear programs.https://github.com/coin- or/ python-mip, 2023. Version 1.15.0. 5

    2023

  25. [25]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lam- ple, Lucile Saulnier, et al. Mistral 7b.arXiv preprint arXiv:2310.06825, 2023. 10, 3

    Pith/arXiv arXiv 2023

  26. [26]

    Fine-tuning vision-language-action models: Optimizing speed and suc- cess.arXiv preprint arXiv:2502.19645, 2025

    Moo Jin Kim, Chelsea Finn, and Percy Liang. Fine-tuning vision-language-action models: Optimizing speed and suc- cess.arXiv preprint arXiv:2502.19645, 2025. 1

    Pith/arXiv arXiv 2025

  27. [27]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. InPro- ceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. 7, 2

    2023

  28. [28]

    Jamba: Hybrid transformer-mamba language models

    Barak Lenz, Opher Lieber, Alan Arazi, Amir Bergman, Avshalom Manevich, Barak Peleg, Ben Aviram, Chen Al- magor, Clara Fridman, Dan Padnos, et al. Jamba: Hybrid transformer-mamba language models. InThe thirteenth in- ternational conference on learning representations, 2025. 1, 2, 3

    2025

  29. [29]

    Minimax-01: Scaling foundation models with lightning attention.arXiv preprint arXiv:2501.08313, 2025

    Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, et al. Minimax-01: Scaling foundation models with lightning attention.arXiv preprint arXiv:2501.08313, 2025. 1

    Pith/arXiv arXiv 2025

  30. [30]

    Seed-bench: Bench- marking multimodal large language models

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Bench- marking multimodal large language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 13299–13308, 2024. 7

    2024

  31. [31]

    Mvbench: A comprehensive multi-modal video understand- ing benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understand- ing benchmark. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195– 22206, 2024. 7

    2024

  32. [32]

    Matvlm: Hybrid mamba-transformer for efficient vision-language modeling

    Yingyue Li, Bencheng Liao, Wenyu Liu, and Xinggang Wang. Matvlm: Hybrid mamba-transformer for efficient vision-language modeling. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 20878– 20888, 2025. 1, 3

    2025

  33. [33]

    Recogdrive: A reinforced cognitive frame- work for end-to-end autonomous driving.arXiv preprint arXiv:2506.08052, 2025

    Yongkang Li, Kaixin Xiong, Xiangyu Guo, Fang Li, Sixu Yan, Gangwei Xu, Lijun Zhou, Long Chen, Haiyang Sun, Bing Wang, et al. Recogdrive: A reinforced cognitive frame- work for end-to-end autonomous driving.arXiv preprint arXiv:2506.08052, 2025. 1

    Pith/arXiv arXiv 2025

  34. [34]

    Jamba: A hybrid transformer-mamba language model.arXiv preprint arXiv:2403.19887, 2024

    Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al. Jamba: A hybrid transformer-mamba language model.arXiv preprint arXiv:2403.19887, 2024. 1

    Pith/arXiv arXiv 2024

  35. [35]

    Rea- sonable effectiveness of random weighting: A litmus test for multi-task learning.Transactions on Machine Learning Re- search, 2022

    Baijiong Lin, Feiyang Ye, Yu Zhang, and Ivor Tsang. Rea- sonable effectiveness of random weighting: A litmus test for multi-task learning.Transactions on Machine Learning Re- search, 2022. 4

    2022

  36. [36]

    Mmfinereason: Closing the multimodal reason- ing gap via open data-centric methods.arXiv preprint arXiv:2601.21821, 2026

    Honglin Lin, Zheng Liu, Yun Zhu, Chonghan Qin, Juekai Lin, Xiaoran Shang, Conghui He, Wentao Zhang, and Li- jun Wu. Mmfinereason: Closing the multimodal reason- ing gap via open data-centric methods.arXiv preprint arXiv:2601.21821, 2026. 2

    arXiv 2026

  37. [37]

    Smooth tchebycheff scalar- ization for multi-objective optimization.arXiv preprint arXiv:2402.19078, 2024

    Xi Lin, Xiaoyuan Zhang, Zhiyuan Yang, Fei Liu, Zhenkun Wang, and Qingfu Zhang. Smooth tchebycheff scalar- ization for multi-objective optimization.arXiv preprint arXiv:2402.19078, 2024. 5, 6

    Pith/arXiv arXiv 2024

  38. [38]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023. 1

    2023

  39. [39]

    Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vi- sion, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vi- sion, pages 216–233. Springer, 2024. 7

    2024

  40. [40]

    Learn to explain: Multimodal reasoning via thought chains for science question answering.Advances in neural information processing systems, 35:2507–2521,

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering.Advances in neural information processing systems, 35:2507–2521,

  41. [41]

    Smolvlm: Redefining small and efficient multimodal models.arXiv preprint arXiv:2504.05299, 2025

    Andr ´es Marafioti, Orr Zohar, Miquel Farr ´e, Merve Noyan, Elie Bakouch, Pedro Cuenca, Cyril Zakka, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, et al. Smolvlm: Redefining small and efficient multimodal models.arXiv preprint arXiv:2504.05299, 2025. 8 11

    Pith/arXiv arXiv 2025

  42. [42]

    Chartqa: A benchmark for question answer- ing about charts with visual and logical reasoning

    Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answer- ing about charts with visual and logical reasoning. InFind- ings of the association for computational linguistics: ACL 2022, pages 2263–2279, 2022. 7

    2022

  43. [43]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. InProceed- ings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021. 7

    2021

  44. [44]

    Infographicvqa

    Minesh Mathew, Viraj Bagal, Rub `en Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. InProceedings of the IEEE/CVF Winter Conference on Ap- plications of Computer Vision, pages 1697–1706, 2022. 7

    2022

  45. [45]

    Ocr-vqa: Visual question answering by reading text in images

    Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In2019 international conference on document analysis and recognition (ICDAR), pages 947–

  46. [46]

    Olmo 3.arXiv preprint arXiv:2512.13961, 2025

    Team Olmo, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, et al. Olmo 3.arXiv preprint arXiv:2512.13961, 2025. 7

    Pith/arXiv arXiv 2025

  47. [47]

    Per- ception test: A diagnostic benchmark for multimodal video models.Advances in Neural Information Processing Sys- tems, 36:42748–42761, 2023

    Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Re- casens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al. Per- ception test: A diagnostic benchmark for multimodal video models.Advances in Neural Information Processing Sys- tems, 36:42748–42761, 2023. 2

    2023

  48. [48]

    Rwkv: Reinventing rnns for the transformer era

    Bo Peng et al. Rwkv: Reinventing rnns for the transformer era. InEMNLP, 2023. 2

    2023

  49. [49]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, pages 8748–8763. PmLR, 2021. 1

    2021

  50. [50]

    Bond: Aligning llms with best-of-n distillation.arXiv preprint arXiv:2407.14622, 2024

    Pier Giuseppe Sessa, Robert Dadashi, L ´eonard Hussenot, Jo- han Ferret, Nino Vieillard, Alexandre Ram´e, Bobak Shariari, Sarah Perrin, Abe Friesen, Geoffrey Cideron, et al. Bond: Aligning llms with best-of-n distillation.arXiv preprint arXiv:2407.14622, 2024. 5

    Pith/arXiv arXiv 2024

  51. [51]

    Hardware co-design scaling laws via roofline modelling for on-device llms.arXiv preprint arXiv:2602.10377, 2026

    Luoyang Sun, Jiwen Jiang, Yifeng Ding, Fengfa Li, Yan Song, Haifeng Zhang, Jian Ying, Lei Ren, Kun Zhan, Wei Chen, et al. Hardware co-design scaling laws via roofline modelling for on-device llms.arXiv preprint arXiv:2602.10377, 2026. 3

    arXiv 2026

  52. [52]

    Retentive network: A successor to transformer for large language models.arXiv preprint arXiv:2307.08621, 2023

    Yutao Sun et al. Retentive network: A successor to transformer for large language models.arXiv preprint arXiv:2307.08621, 2023. 2

    Pith/arXiv arXiv 2023

  53. [53]

    Infinitevl: Synergizing linear and sparse attention for highly-efficient, unlimited-input vision-language models.arXiv preprint arXiv:2512.06450, 2025

    Hongyuan Tao, Bencheng Liao, Shaoyu Chen, Haoran Yin, Qian Zhang, Wenyu Liu, and Xinggang Wang. Infinitevl: Synergizing linear and sparse attention for highly-efficient, unlimited-input vision-language models.arXiv preprint arXiv:2512.06450, 2025. 1, 2, 3, 7

    arXiv 2025

  54. [54]

    Kimi linear: An expressive, efficient atten- tion architecture.arXiv preprint arXiv:2510.26692, 2025

    Kimi Team, Yu Zhang, Zongyu Lin, Xingcheng Yao, Jiaxi Hu, Fanqing Meng, Chengyin Liu, Xin Men, Songlin Yang, Zhiyuan Li, et al. Kimi linear: An expressive, efficient atten- tion architecture.arXiv preprint arXiv:2510.26692, 2025. 1, 2, 3, 7

    Pith/arXiv arXiv 2025

  55. [55]

    Cambrian-1: A fully open, vision-centric exploration of multimodal llms

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai C Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. Advances in Neural Information Processing Systems, 37: 87310–87356, 2024. 7

    2024

  56. [56]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017. 1, 2

    2017

  57. [57]

    An empirical study of mamba-based language models.arXiv preprint arXiv:2406.07887, 2024

    Roger Waleffe, Wonmin Byeon, Duncan Riach, Bran- don Norick, Vijay Korthikanti, Tri Dao, Albert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, et al. An empirical study of mamba-based language models.arXiv preprint arXiv:2406.07887, 2024. 1

    Pith/arXiv arXiv 2024

  58. [58]

    Hat: Hardware-aware transformers for efficient natural language processing

    Hanrui Wang et al. Hat: Hardware-aware transformers for efficient natural language processing. InACL, 2020. 3

    2020

  59. [59]

    Internvl3

    Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025. 8

    Pith/arXiv arXiv 2025

  60. [60]

    Qwen-image technical report.arXiv preprint arXiv:2508.02324, 2025

    Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng-ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, et al. Qwen-image technical report.arXiv preprint arXiv:2508.02324, 2025. 1

    Pith/arXiv arXiv 2025

  61. [61]

    Rethinking kullback-leibler di- vergence in knowledge distillation for large language mod- els

    Taiqiang Wu, Chaofan Tao, Jiahao Wang, Runming Yang, Zhe Zhao, and Ngai Wong. Rethinking kullback-leibler di- vergence in knowledge distillation for large language mod- els. InProceedings of the 31st International Conference on Computational Linguistics, pages 5737–5755, 2025. 5

    2025

  62. [62]

    Next-qa: Next phase of question-answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. InProceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 9777–9786, 2021. 2

    2021

  63. [63]

    Ava- vla: Improving vision-language-action models with active visual attention.arXiv preprint arXiv:2511.18960, 2025

    Lei Xiao, Jifeng Li, Juntao Gao, Feiyang Ye, Yan Jin, Jingjing Qian, Jing Zhang, Yong Wu, and Xiaoyuan Yu. Ava- vla: Improving vision-language-action models with active visual attention.arXiv preprint arXiv:2511.18960, 2025. 1

    Pith/arXiv arXiv 2025

  64. [64]

    Mswa: Refining local attention with multi- scalewindow attention.arXiv preprint arXiv:2501.01039,

    Yixing Xu, Shivank Nag, Dong Li, Lu Tian, and Emad Barsoum. Mswa: Refining local attention with multi- scalewindow attention.arXiv preprint arXiv:2501.01039,

    Pith/arXiv arXiv

  65. [65]

    Gated delta networks: Improving mamba2 with delta rule.arXiv preprint arXiv:2412.06464, 2024

    Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule.arXiv preprint arXiv:2412.06464, 2024. 2, 1

    Pith/arXiv arXiv 2024

  66. [66]

    Cambrian-s: Towards spatial supersens- ing in video.arXiv preprint arXiv:2511.04670, 2025

    Shusheng Yang, Jihan Yang, Pinzhi Huang, Ellis Brown, Zi- hao Yang, Yue Yu, Shengbang Tong, Zihan Zheng, Yifan Xu, Muhan Wang, et al. Cambrian-s: Towards spatial supersens- ing in video.arXiv preprint arXiv:2511.04670, 2025. 2

    Pith/arXiv arXiv 2025

  67. [67]

    Clevrer: Collision events for video representation and reasoning

    Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprint arXiv:1910.01442, 2019. 2 12

    Pith/arXiv arXiv 1910

  68. [68]

    Native sparse attention: Hardware-aligned and natively trainable sparse attention

    Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Yuxing Wei, Lean Wang, Zhiping Xiao, et al. Native sparse attention: Hardware-aligned and natively trainable sparse attention. In Proceedings of the 63rd Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 23078–23097, 2025. 2

    2025

  69. [69]

    Autodrive-r 2: Incentivizing reason- ing and self-reflection capacity for vla model in autonomous driving.arXiv preprint arXiv:2509.01944, 2025

    Zhenlong Yuan, Chengxuan Qian, Jing Tang, Rui Chen, Zi- jian Song, Lei Sun, Xiangxiang Chu, Yujun Cai, Dapeng Zhang, and Shuo Li. Autodrive-r 2: Incentivizing reason- ing and self-reflection capacity for vla model in autonomous driving.arXiv preprint arXiv:2509.01944, 2025. 1

    Pith/arXiv arXiv 2025

  70. [70]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9556–9567,

  71. [71]

    Hellaswag: Can a machine really finish your sentence? InProceedings of the 57th annual meeting of the association for computational linguistics, pages 4791–4800,

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? InProceedings of the 57th annual meeting of the association for computational linguistics, pages 4791–4800,

  72. [72]

    Vision-language models for vision tasks: A survey.IEEE transactions on pattern analysis and machine intelligence, 46(8):5625–5644, 2024

    Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey.IEEE transactions on pattern analysis and machine intelligence, 46(8):5625–5644, 2024. 1

    2024

  73. [73]

    Dreamvla: a vision-language-action model dreamed with comprehensive world knowledge.arXiv preprint arXiv:2507.04447, 2025

    Wenyao Zhang, Hongsi Liu, Zekun Qi, Yunnan Wang, Xin- qiang Yu, Jiazhao Zhang, Runpei Dong, Jiawei He, Fan Lu, He Wang, et al. Dreamvla: a vision-language-action model dreamed with comprehensive world knowledge.arXiv preprint arXiv:2507.04447, 2025. 1

    Pith/arXiv arXiv 2025

  74. [74]

    A survey on multi-task learning

    Yu Zhang and Qiang Yang. A survey on multi-task learning. IEEE transactions on knowledge and data engineering, 34 (12):5586–5609, 2021. 4

    2021

  75. [75]

    Cobra: Extending mamba to multi-modal large language model for efficient inference

    Han Zhao, Min Zhang, Wei Zhao, Pengxiang Ding, Siteng Huang, and Donglin Wang. Cobra: Extending mamba to multi-modal large language model for efficient inference. InProceedings of the AAAI Conference on Artificial Intel- ligence, pages 10421–10429, 2025. 3

    2025

  76. [76]

    Autovla: A vision- language-action model for end-to-end autonomous driving with adaptive reasoning and reinforcement fine-tuning.arXiv preprint arXiv:2506.13757, 2025

    Zewei Zhou, Tianhui Cai, Seth Z Zhao, Yun Zhang, Zhiyu Huang, Bolei Zhou, and Jiaqi Ma. Autovla: A vision- language-action model for end-to-end autonomous driving with adaptive reasoning and reinforcement fine-tuning.arXiv preprint arXiv:2506.13757, 2025. 1

    Pith/arXiv arXiv 2025

  77. [77]

    Video-star: Self-training enables video instruction tuning with any supervision.arXiv preprint arXiv:2407.06189, 2024

    Orr Zohar, Xiaohan Wang, Yonatan Bitton, Idan Szpektor, and Serena Yeung-Levy. Video-star: Self-training enables video instruction tuning with any supervision.arXiv preprint arXiv:2407.06189, 2024. 2 13 A. Details for the Search Space In this section, we provide the comprehensive architec- tural specifications for the candidate subblocks within the search...

    Pith/arXiv arXiv 2024