Pith. sign in

REVIEW 4 major objections 6 minor 70 references

A vision encoder designed from real phone latencies, not FLOPs, runs nearly 1.7× faster than prior efficient LVLM encoders while beating them on multimodal benchmarks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-30 23:58 UTC pith:CSB6DU6F

load-bearing objection Solid on-device encoder paper with real NPU-driven design and careful FastVLM re-train; the accuracy SOTA claim overstates architecture because the two-stage pretrain is never run on FastViT. the 4 major comments →

arxiv 2607.23373 v1 pith:CSB6DU6F submitted 2026-07-25 cs.CV

UltraViT: Latency-Optimized On-device Vision Encoder for Large Vision-Language Models

Ioannis Maniadis Metaxas , Adrian Bulat , Alberto Baldrati , Anestis Zaganidis , Yassine Ouali , Hyeonuk Kim , Georgios Tzimiropoulos This is my paper
classification cs.CV
keywords vision encoderon-device latencylarge vision-language modelsheterogeneous spatial mixersdense distillationgenerative pre-trainingedge deploymentpyramidal ViT
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Large vision-language models are too heavy for phones largely because their vision front-ends are still monolithic, classification-era transformers. This paper argues that the encoder itself should be redesigned for on-device latency: measure real NPU timings, then pick different spatial mixers for different pyramid stages instead of repeating one block type everywhere. It pairs that layout with a two-stage pre-training recipe—dense feature distillation from a strong teacher, then generative supervision from a frozen, capacity-mixed language model—so the encoder learns high-level semantics that transfer into LVLM alignment. Under the same fine-tuning recipe, the resulting UltraVLM beats the leading encoder-centric baseline on most benchmarks while the encoder itself is nearly 1.7× faster on a phone and natively emits far fewer tokens. A sympathetic reader cares because shrinking the vision tower, rather than only pruning tokens or the LLM, is the remaining bottleneck once language models get small enough for the edge.

Core claim

The authors claim that a pyramidal vision encoder whose stages use heterogeneous, NPU-adapted spatial mixers—chosen by real on-device latency and accuracy probes—combined with dense distillation followed by generative pre-training against a frozen capacity-mixed LLM, yields a new state of the art for efficient LVLM encoding: higher accuracy than FastVLM-style baselines under matched training data, at nearly 1.7× on-device encoder speed and with a native 16× token reduction.

What carries the argument

UltraViT’s heterogeneous macro-block pyramid: convolutional mixers early, optimized sparse and channel-partitioned single-head attention mid-network, full single-head attention late, plus a multi-scale aggregator; trained first by dense cosine distillation (with pixel-shuffle realignment) then by next-token generative loss through a frozen vision-aligned LLM that alternates 0.5B and 1.5B capacity.

Load-bearing premise

That mixer rankings measured on one phone’s INT8 NPU, plus short contrastive probes and one LVLM recipe, pick a generally optimal encoder layout rather than a device- and recipe-specific local optimum.

What would settle it

Re-run the same block-selection grid and full UltraVLM vs re-trained FastVLM comparison on a different mobile NPU family and CPU backend; if latency orderings reverse or the accuracy gap disappears under matched data, the co-design claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Edge LVLMs can drop heavyweight SigLIP-class towers without post-hoc token compressors, cutting both vision latency and LLM sequence cost.
  • Encoder design for multimodal models should treat real NPU latency tables, not FLOPs or GPU proxies, as the primary search signal.
  • Dense distillation plus frozen-LLM generative pre-training can replace or outperform pure contrastive pre-training for compact vision towers headed into generative LVLMs.
  • As LLMs shrink toward 0.5–1.5B on device, vision-encoder optimization becomes a first-order lever on end-to-end time-to-first-token.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same stage-wise mixer search could be re-run periodically as NPU operator libraries change, turning architecture choice into a living hardware-software co-design loop.
  • Capacity-mixed frozen decoders during pre-training may generalize as a cheap way to keep small vision towers from absorbing language-modeling burden in other multimodal stacks.
  • If the native 16× token cut holds under video or multi-image inputs, the design may reduce the need for separate temporal token pruners on device.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents UltraViT, a vision encoder for LVLMs designed for on-device deployment. Its contributions are (1) a four-stage pyramidal architecture in which the token mixer type is selected per stage (convolutional, conv+sparse, channel-partitioned attention, single-head attention) based on native INT8 latency measurements on a Galaxy S25 Ultra NPU plus SigLIP-style probes and LLaVA-OV fine-tunes; and (2) a two-stage pre-training scheme: dense feature distillation from a SigLIP2 teacher with a pixel-shuffle spatial realignment, followed by generative supervision from a frozen, pre-aligned Qwen2 LLM that is dynamically switched between 0.5B and 1.5B variants. Evaluated as UltraVLM under the LLaVA-OV 7.1M recipe, the system beats a re-trained FastVLM on most of 11 benchmarks (notably TextVQA +6.0, DocVQA +5.1, ChartQA +6.2) while the encoder runs ~1.7x faster on-device, and it natively emits 16x fewer tokens than LLaVA-OV.

Significance. If the results hold, this is a solid systems contribution: the first systematic on-device-latency-informed block selection for LVLM vision encoders (Sec. 3.2), a credible NPU-oriented redesign of sparse attention (S1-S4), and a two-stage pre-training pipeline that is 3x/10x more sample-efficient than the FastVLM contrastive recipe (Table 4). The empirical protocol is unusually careful for this genre: native INT8 QNN benchmarking on physical hardware with a documented methodology (7 rounds x 100 passes, reboots between models, Appendix B.4), CPU/GPU cross-checks (Appendix B.5), teacher robustness (Table 7), and end-to-end TTFT/peak-memory numbers (Table 8). These are exactly the falsifiable, reproducible measurements the field needs. However, the paper's own ablations (Table 5a) indicate the accuracy advantage is largely attributable to the pre-training recipe rather than the architecture; the durable, well-supported contribution is 'FastVLM-level accuracy at ~1.7x the on-device speed', which is still valuable but weaker than the abstract's framing.

major comments (4)
  1. [Sec. 5.2 / Sec. 6] Table 2 vs. Table 5: the headline claim 'significantly outperforming existing encoder-centric baselines' conflates architecture and pre-training recipe. The 'identical recipe' for the re-trained FastVLM baseline applies only to the LVLM fine-tuning stage; the vision encoders arrive via different pre-training (FastViT-HD's original contrastive distillation vs. UltraViT's dense distillation + generative stage). The authors' own Table 5(a) shows that under the same dense-distillation pre-training, UltraViT and FastViT are statistically indistinguishable downstream (DocVQA 63.9/62.1, InfoVQA 39.5/39.4, MMSTAR 44.6/45.9 favoring FastViT, SQA 76.9/76.3), and Table 5(b) shows the large gains come from the generative stage, which is ablated only on UltraViT. The missing control — FastViT-HD trained with the full two-stage pre-training — is load-bearing: without it, the defensible claim may be 'm
  2. [Sec. 3.2, Table 1] Table 1 does not by itself support the heterogeneity claim. The selected configuration (C, C+S, CP, A) ties with the near-homogeneous (C, C+S, A, A) on the reported average (58.5 vs. 58.5) and is slower than the all-convolutional variant (148.9 vs. 171.0 inf./s) while beating it by 2.1 avg points. Since the selection metric mixes zero-shot retrieval, classification, and LLaVA fine-tuning — the same family of probes used for the final evaluation — the reader cannot tell whether the heterogeneous layout is a genuine accuracy optimum or a latency-constrained tie-break. Please state the selection criterion explicitly (e.g., accuracy subject to a latency floor) and report variance or repeated runs for the Table 1 LLaVA fine-tunes, given that per-benchmark differences are within typical fine-tuning noise.
  3. [Sec. 3.1-3.2, Sec. 5.1, Appendix B.4] All NPU latency claims rest on a single device (Galaxy S25 Ultra, QNN, INT8). The block-ranking in Fig. 3 and the S1-S4 adaptations in Sec. 3.1 are justified entirely by this target. Appendix B.5 partially mitigates this (UltraViT is also faster on CPU/GPU), but the mixer rankings themselves could reorder on other NPUs. At minimum, the paper should (a) state explicitly that the architecture is co-optimized for the Qualcomm NPU and discuss portability risk, and (b) report INT8 accuracy (not just INT8 latency) for UltraVLM vs. the FP baseline, since the deployed artifact is quantized and no quantization accuracy drop is reported anywhere.
  4. [Sec. 5.2, Table 2] Table 2: the claim 'outperforms FastVLM across most benchmarks' should address the regressions directly: MMMU drops from 41.1 to 38.1 and RealWorldQA from 60.1 to 59.5, while POPE is a wash (87.1 vs 87.3). A short discussion of where the method loses (and why — e.g., 16x token reduction hurting fine-grained multi-discipline reasoning) would strengthen the paper's credibility and is needed before the aggregate SOTA claim is well-supported.
minor comments (6)
  1. [Abstract / Sec. 7] Abstract: 'nearly 1.7xthe speed' — missing space; also the conclusion says 'nearly 2x the speed' while the abstract says 1.7x; please make consistent.
  2. [Sec. 3.1] C_qk = 16 and C_p = C/4 are taken from SH-ViT without ablation; given the emphasis on systematic design, a brief sensitivity note on these projection widths would be welcome.
  3. [Sec. 6, Table 5(b)] Table 5(b): 'matches or exceeds the performance of the pure 1.5B model' — the dynamic variant is slightly below 1.5B on DocVQA (71.8 vs 72.0) and SQA (82.9 vs 83.1); 'matches within noise' would be more accurate. Training-cost numbers backing 'notably reducing the total training cost' are not given.
  4. [Fig. 2] Fig. 2 (left) caption is ambiguous: 'throughput when the sparse attention mixer is applied cumulatively up to the i-th stage' does not clearly convey that bars show cumulative S1-S4 adaptations; please reword.
  5. [References / Fig. 2 / Table 8] Reference [60] lacks a year; Fig. 2 caption writes 'UltraVit'; Table 8 footnote uses nonstandard notation '5122/10242' (presumably 512^2/1024^2).
  6. [Sec. 4.2] Sec. 4.2: the motivation asserts small decoders shift 'the linguistic and syntactic modeling burden onto the vision encoder'; this is plausible and supported by Table 5(b), but a citation or brief gradient-level argument would help, as this claim motivates the central design choice.

Circularity Check

0 steps flagged

No significant circularity: empirical systems paper evaluated on external benchmarks and hardware, not a self-defining derivation.

full rationale

UltraViT is an empirical architecture-and-training paper. Its load-bearing claims are (1) a heterogeneous pyramidal encoder chosen via measured INT8 NPU latencies plus probe accuracy, and (2) a two-stage pre-training recipe, both judged by external LVLM benchmarks (GQA, TextVQA, DocVQA, ChartQA, etc.) and wall-clock timings on a physical Galaxy S25 Ultra. Nothing in the derivation chain defines the reported metrics in terms of the design choices: latency figures are hardware measurements, not fitted parameters renamed as predictions; mixer selection (Sec. 3.2, Table 1) is ordinary architecture search whose final comparison (Table 2) uses a broader suite and a full LLaVA-OV recipe against a re-trained FastVLM baseline; dense distillation and frozen-LLM generative losses are trained objectives evaluated post hoc, not tautologies. Self-citations (EdgeViT, prior Samsung work) supply building blocks and baselines, not uniqueness theorems that force the result. Mild selection-on-the-same-task-family risk exists in any NAS-style paper but does not reduce the central accuracy/speed claims to inputs by construction. Score 0 is appropriate.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 3 invented entities

Load-bearing content is engineering choice plus empirical training, not a short axiom list. The claim rests on MetaFormer-style block topology, mobile-operator efficiency folklore, teacher features being good distillation targets, and the premise that phone-NPU latency tables are the right objective for LVLM encoders. Free parameters are the usual architecture and optimization knobs selected during search and training.

free parameters (5)
  • Stage channels C1–4 = [192, 512, 768, 1536] and block counts N1–4 = [4, 7, 10, 2] = [192, 512, 768, 1536], [4, 7, 10, 2]
    Final macro widths/depths chosen after discrete search; central latency/accuracy claim depends on this configuration.
  • Attention projection dims C_qk = 16 and C_p = C/4 = C_qk=16, C_p=C/4
    Fixed following SH-ViT-style choices for CP and sparse mixers; affects both speed and capacity.
  • Dense distillation loss weights 0.5 L_dense + 0.5 L_global = 0.5 / 0.5
    Hand-set mixture of patch cosine and teacher-pooled global cosine (Sec. 4.1).
  • Pretrain hyperparameters (LR, epochs, batch, data volumes) = as in Sec. 5.1
    Dense: 25 epochs, LR 1e-4, wd 0.1, bs 32k on 150M; generative: 1 epoch, LR 1e-5, bs 192 on 85M—standard fitted training schedule knobs.
  • Heterogeneous mixer assignment per stage (C, C+S, CP, A) = Conv / Conv+Sparse / CP-Attn / SH-Attn
    Selected from on-device latency grid plus accuracy ranking in Table 1; the paper’s headline architecture is this discrete choice.
axioms (5)
  • domain assumption MetaFormer residual token-mixer + MLP macro-block is an adequate universal template; only the TokenMixer needs swapping (Eqs. 1–2, citing Yu et al.).
    Architecture search never leaves this template; quality claims inherit this inductive bias.
  • domain assumption Real INT8 latency on the target mobile NPU is the correct primary efficiency objective (preferable to FLOPs/GPU proxies) for LVLM vision encoders.
    Sec. 3.2 block selection and all speed claims are grounded in this measurement philosophy.
  • domain assumption Strong contrastive teachers (SigLIP2 family etc.) provide spatial/semantic features worth dense-distilling into a compact student for later generative LVLM alignment.
    Stage-1 pretraining (Sec. 4.1) stands or falls with teacher feature usefulness.
  • ad hoc to paper A frozen, already vision-aligned LLM provides a better generative training signal for a mobile vision encoder than contrastive-only or small scratch captioners.
    Core justification for stage-2 (Sec. 4.2); supported by ablations but still a paper-specific training axiom.
  • standard math Standard deep-learning optimization and transformer/CNN operator semantics (attention, DWConv, pixel-shuffle) behave as usually assumed.
    Background math/ML machinery used throughout without re-proof.
invented entities (3)
  • UltraViT heterogeneous pyramidal encoder (incl. multi-scale aggregator) no independent evidence
    purpose: Serve as the latency-optimized vision front-end for on-device LVLMs with native token reduction.
    New composed architecture; evidence is empirical tables/timings, not an external physical entity.
  • NPU-optimized Sparse Mixer (S1–S4 adaptations) no independent evidence
    purpose: Replace EdgeViT-style sparse attention with single-head, strided QKV convs, NN upsample, and 4× value compression for mobile latency.
    Paper-specific operator redesign; validated by stage throughput in Fig. 2 left.
  • Capacity-mixed frozen-LLM generative pre-training schedule no independent evidence
    purpose: Align encoder features to autoregressive multimodal decoding cheaply by alternating 0.5B/1.5B frozen decoders.
    Training procedure introduced here; Table 5b is the internal evidence.

pith-pipeline@v1.2.0-grok45-kimik3 · 24782 in / 4327 out tokens · 82519 ms · 2026-07-30T23:58:55.860632+00:00 · methodology

0 comments
read the original abstract

Large Vision-Language Models (LVLMs) remain bottlenecked by massive computational footprints, precluding their deployment on resource-constrained edge devices. While efforts to compress LVLMs focus heavily on vision token reduction or smaller language models, the vision encoder is largely overlooked, typically deployed as a monolithic, computationally heavy feature extractor. Moreover, there is no previous effort that designs a vision encoder for LVLMs directly optimized for on-device latency. In this paper, we present UltraViT, a vision encoder for LVLMs, explicitly designed and optimized for on-device performance. Specifically, by taking into account real on-device latencies, we systematically design a pyramidal architecture that strategically integrates and adapts heterogeneous spatial mixers at the macro-block level. Furthermore, to pre-train UltraViT, we propose a novel two-stage generative pre-training strategy: cultivating rich spatial features via dense distillation, followed by direct generative supervision from a capacity-mixed frozen LLM. Compared to standard contrastive and SSL, we show that our pre-training is much more effective for achieving high-level semantic grounding for UltraViT needed for the subsequent generative multimodal alignment of LVLM training. Extensive experiments demonstrate that our on-device latency-informed design combined with our tailored training strategy establishes a new state-of-the-art for efficient LVLM encoding, significantly outperforming existing encoder-centric baselines while operating on-device at nearly 1.7xthe speed.

Figures

Figures reproduced from arXiv: 2607.23373 by Adrian Bulat, Alberto Baldrati, Anestis Zaganidis, Georgios Tzimiropoulos, Hyeonuk Kim, Ioannis Maniadis Metaxas, Yassine Ouali.

Figure 1
Figure 1. Figure 1: (a) General Structure of the Meta Architecture Block. (b): Architecture of the four different token mixers employed. (c) Overview of the full UltraViT architecture. condition on both the image and the query text. In this work, we adopt the widely-used LLaVA-OV [24] design. Unlike the bulk of prior work on efficient LVLMs, we focus on the design of the vision encoder specifically for efficient on-device dep… view at source ↗
Figure 2
Figure 2. Figure 2: Inferences per second across UltraVit stages. Left: the throughput when the sparse attention mixer is applied cumulatively up to the i-th stage. Right: stage-wise throughput with attention mixers with a varying number of attention heads. 3.2 Block Selection Process 1 2 3 4 2 10 100 1000 UltraViT Stages Images per second Sparse CP Attn Attention Convolutional [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Inferences per second for images with resolution 512x512 at each stage. We postulate that the optimal mixer operation, at different stages of the net￾work, varies as a ratio of accuracy-to￾latency trade-off. Hence, to optimize the architecture for edge deployment, we employ a systematic block selection process tailored to each stage of the net￾work. The computational dynamics of different spatial mixers: C… view at source ↗
Figure 4
Figure 4. Figure 4: Inferences per second for Ultra￾ViT vs FastViT for varying input image resolutions [PITH_FULL_IMAGE:figures/full_fig_p014_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: illustrates the overall pipeline of the proposed dense distillation strat￾egy. (Teacher) UltraViT (student) Pixel Shuffle Teacher Attention Pooling [PITH_FULL_IMAGE:figures/full_fig_p021_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

70 extracted references · 22 linked inside Pith

  1. [1]

    arXiv preprint arXiv:2509.23661 (2025)

    An, X., Xie, Y., Yang, K., Zhang, W., Zhao, X., Cheng, Z., Wang, Y., Xu, S., Chen, C., Zhu, D., et al.: Llava-onevision-1.5: Fully open framework for democratized multimodal training. arXiv preprint arXiv:2509.23661 (2025)

    Pith/arXiv arXiv 2025

  2. [2]

    In: Proceedings of the AAAI Conference on Artificial In- telligence

    Arif, K.H.I., Yoon, J., Nikolopoulos, D.S., Vandierendonck, H., John, D., Ji, B.: Hired: Attention-guided token dropping for efficient inference of high-resolution vision-language models. In: Proceedings of the AAAI Conference on Artificial In- telligence. vol. 39, pp. 1773–1781 (2025)

    2025

  3. [3]

    arXiv preprint arXiv:2511.21631 (2025)

    Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al.: Qwen3-vl technical report. arXiv preprint arXiv:2511.21631 (2025)

    Pith/arXiv arXiv 2025

  4. [4]

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., Zhong, H., Zhu, Y., Yang, M., Li, Z., Wan, J., Wang, P., Ding, W., Fu, Z., Xu, Y., Ye, J., Zhang, X., Xie, T., Cheng, Z., Zhang, H., Yang, Z., Xu, H., Lin, J.: Qwen2.5-vl technical report (2025),https://arxiv.org/abs/2502.13923

    Pith/arXiv arXiv 2025

  5. [5]

    arXiv preprint arXiv:2004.05150 (2020)

    Beltagy, I., Peters, M.E., Cohan, A.: Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 (2020)

    Pith/arXiv arXiv 2004

  6. [6]

    arXiv preprint arXiv:2504.13181 (2025)

    Bolya, D., Huang, P.Y., Sun, P., Cho, J.H., Madotto, A., Wei, C., Ma, T., Zhi, J., Rajasegaran, J., Rasheed, H., et al.: Perception encoder: The best visual em- beddings are not at the output of the network. arXiv preprint arXiv:2504.13181 (2025)

    Pith/arXiv arXiv 2025

  7. [7]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Bulat, A., Ouali, Y., Tzimiropoulos, G.: Fff: Fixing flawed foundations in con- trastive pre-training results in very strong vision-language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14172–14182 (2024)

    2024

  8. [8]

    In: The Thirty-ninth Annual Conference on Neural Information Processing Systems (2025)

    Bulat, A., Ouali, Y., Tzimiropoulos, G.: Compress & cache: Vision token compres- sion for efficient generation and retrieval. In: The Thirty-ninth Annual Conference on Neural Information Processing Systems (2025)

    2025

  9. [9]

    arXiv preprint arXiv:2405.17430 (2024)

    Cai, M., Yang, J., Gao, J., Lee, Y.J.: Matryoshka multimodal models. arXiv preprint arXiv:2405.17430 (2024)

    Pith/arXiv arXiv 2024

  10. [10]

    In: ICLR (2025)

    Cai, M., Yang, J., Gao, J., Lee, Y.J.: Matryoshka multimodal models. In: ICLR (2025)

    2025

  11. [11]

    Chen, L., Li, J., Dong, X., Zhang, P., Zang, Y., Chen, Z., Duan, H., Wang, J., Qiao, Y., Lin, D., et al.: Are we on the right way for evaluating large vision- language models? Advances in Neural Information Processing Systems37, 27056– 27087 (2024)

    2024

  12. [12]

    arXiv preprint arXiv:1904.10509 (2019)

    Child, R., Gray, S., Radford, A., Sutskever, I.: Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509 (2019)

    Pith/arXiv arXiv 1904

  13. [13]

    arXiv preprint arXiv:2402.03766 (2024)

    Chu, X., Qiao, L., Zhang, X., Xu, S., Wei, F., Yang, Y., Sun, X., Hu, Y., Lin, X., Zhang, B., et al.: Mobilevlm v2: Faster and stronger baseline for vision language model. arXiv preprint arXiv:2402.03766 (2024)

    Pith/arXiv arXiv 2024

  14. [14]

    In: Proceedings of the IEEE/CVF International Confer- ence on Computer Vision

    Cocchi,F.,Moratelli,N.,Caffagni,D.,Sarto,S.,Baraldi,L.,Cornia,M.,Cucchiara, R.: Llava-more: A comparative study of llms and visual backbones for enhanced visual instruction tuning. In: Proceedings of the IEEE/CVF International Confer- ence on Computer Vision. pp. 4278–4288 (2025)

    2025

  15. [15]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Deitke, M., Clark, C., Lee, S., Tripathi, R., Yang, Y., Park, J.S., Salehi, M., Muen- nighoff, N., Lo, K., Soldaini, L., et al.: Molmo and pixmo: Open weights and open data for state-of-the-art vision-language models. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 91–104 (2025) UltraViT: Latency-Optimized On-device Vision E...

    2025

  16. [16]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Ding, X., Zhang, X., Ma, N., Han, J., Ding, G., Sun, J.: Repvgg: Making vgg-style convnets great again. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 13733–13742 (2021)

    2021

  17. [17]

    arXiv preprint arXiv:2010.11929 (2020)

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

    Pith/arXiv arXiv 2010

  18. [18]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Han, K., Wang, Y., Tian, Q., Guo, J., Xu, C., Xu, C.: Ghostnet: More features from cheap operations. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 1580–1589 (2020)

    2020

  19. [19]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Howard, A., Sandler, M., Chu, G., Chen, L.C., Chen, B., Tan, M., Wang, W., Zhu, Y., Pang, R., Vasudevan, V., et al.: Searching for mobilenetv3. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 1314–1324 (2019)

    2019

  20. [20]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 7132–7141 (2018)

    2018

  21. [21]

    arXiv preprint arXiv:2405.19315 (2024)

    Hu, W., Dou, Z.Y., Li, L.H., Kamath, A., Peng, N., Chang, K.W.: Ma- tryoshka query transformer for large vision-language models. arXiv preprint arXiv:2405.19315 (2024)

    Pith/arXiv arXiv 2024

  22. [22]

    In: Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition

    Hudson, D.A., Manning, C.D.: Gqa: A new dataset for real-world visual reasoning and compositional question answering. In: Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition. pp. 6700–6709 (2019)

    2019

  23. [23]

    In: European conference on computer vision

    Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., Farhadi, A.: A di- agram is worth a dozen images. In: European conference on computer vision. pp. 235–251. Springer (2016)

    2016

  24. [24]

    arXiv preprint arXiv:2408.03326 (2024)

    Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., et al.: Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326 (2024)

    Pith/arXiv arXiv 2024

  25. [25]

    In: Proceedings of the 2023 conference on empirical methods in natural language processing

    Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W.X., Wen, J.R.: Evaluating object hal- lucination in large vision-language models. In: Proceedings of the 2023 conference on empirical methods in natural language processing. pp. 292–305 (2023)

    2023

  26. [26]

    Liu, Y., Zhang, Y., Ghosh, D., Schmidt, L., Yeung-Levy, S.: Data or language supervision: What makes clip better than dino? arXiv preprint arXiv:2510.11835 (2025)

    arXiv 2025

  27. [27]

    Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al.: Mmbench: Is your multi-modal model an all-around player? In: European conference on computer vision. pp. 216–233. Springer (2024)

    2024

  28. [28]

    Science China Information Sciences67(12), 220102 (2024)

    Liu, Y., Li, Z., Huang, M., Yang, B., Yu, W., Li, C., Yin, X.C., Liu, C.L., Jin, L., Bai, X.: Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences67(12), 220102 (2024)

    2024

  29. [29]

    Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer:Hierarchical visiontransformerusingshiftedwindows.In:Proceedings of the IEEE/CVF international conference on computer vision. pp. 10012–10022 (2021)

    2021

  30. [30]

    Advances in neural information processing systems35, 2507– 2521 (2022)

    Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.W., Zhu, S.C., Tafjord, O., Clark, P., Kalyan, A.: Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in neural information processing systems35, 2507– 2521 (2022)

    2022

  31. [31]

    In: Proceedings of the European conference on computer vision (ECCV)

    Ma, N., Zhang, X., Zheng, H.T., Sun, J.: Shufflenet v2: Practical guidelines for efficient cnn architecture design. In: Proceedings of the European conference on computer vision (ECCV). pp. 116–131 (2018) 18 I. Maniadis Metaxas et al

    2018

  32. [32]

    arXiv preprint arXiv:2504.05299 (2025)

    Marafioti, A., Zohar, O., Farré, M., Noyan, M., Bakouch, E., Cuenca, P., Zakka, C., Allal, L.B., Lozhkov, A., Tazi, N., et al.: Smolvlm: Redefining small and efficient multimodal models. arXiv preprint arXiv:2504.05299 (2025)

    Pith/arXiv arXiv 2025

  33. [33]

    In: Findings of the association for computational linguistics: ACL 2022

    Masry, A., Do, X.L., Tan, J.Q., Joty, S., Hoque, E.: Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In: Findings of the association for computational linguistics: ACL 2022. pp. 2263–2279 (2022)

    2022

  34. [34]

    In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision

    Mathew, M., Bagal, V., Tito, R., Karatzas, D., Valveny, E., Jawahar, C.: Info- graphicvqa. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 1697–1706 (2022)

    2022

  35. [35]

    In: Proceedings of the IEEE/CVF winter conference on applications of computer vision

    Mathew, M., Karatzas, D., Jawahar, C.: Docvqa: A dataset for vqa on document images. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision. pp. 2200–2209 (2021)

    2021

  36. [36]

    arXiv preprint arXiv:2206.02680 (2022)

    Mehta, S., Rastegari, M.: Separable self-attention for mobile vision transformers. arXiv preprint arXiv:2206.02680 (2022)

    Pith/arXiv arXiv 2022

  37. [37]

    In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision

    Methani, N., Ganguly, P., Khapra, M.M., Kumar, P.: Plotqa: Reasoning over sci- entific plots. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 1527–1536 (2020)

    2020

  38. [38]

    arXiv preprint arXiv:2304.07193 (2023)

    Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.: Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023)

    Pith/arXiv arXiv 2023

  39. [39]

    In: European conference on computer vision

    Pan, J., Bulat, A., Tan, F., Zhu, X., Dudziak, L., Li, H., Tzimiropoulos, G., Mar- tinez, B.: Edgevits: Competing light-weight cnns on mobile devices with vision transformers. In: European conference on computer vision. pp. 294–311. Springer (2022)

    2022

  40. [40]

    Advances in neural information processing sys- tems32(2019)

    Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.: Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing sys- tems32(2019)

    2019

  41. [41]

    In: International conference on machine learning

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021)

    2021

  42. [42]

    In: Proceed- ings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining

    Rasley, J., Rajbhandari, S., Ruwase, O., He, Y.: Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In: Proceed- ings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining. pp. 3505–3506 (2020)

    2020

  43. [43]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: Mobilenetv2: In- verted residuals and linear bottlenecks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4510–4520 (2018)

    2018

  44. [44]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A.P., Bishop, R., Rueckert, D., Wang, Z.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1874–1883 (2016)

    2016

  45. [45]

    In: European conference on computer vision

    Sidorov, O., Hu, R., Rohrbach, M., Singh, A.: Textcaps: a dataset for image cap- tioning with reading comprehension. In: European conference on computer vision. pp. 742–758. Springer (2020)

    2020

  46. [46]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., Rohrbach, M.: Towards vqa models that can read. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 8317–8326 (2019) UltraViT: Latency-Optimized On-device Vision Encoder 19

    2019

  47. [47]

    In: International conference on machine learning

    Tan, M., Le, Q.: Efficientnet: Rethinking model scaling for convolutional neural networks. In: International conference on machine learning. pp. 6105–6114. PMLR (2019)

    2019

  48. [48]

    arXiv preprint arXiv:2503.15485 (2025)

    Tang, Z., Lian, L., Eisape, S., Wang, X., Herzig, R., Yala, A., Suhr, A., Darrell, T., Chan, D.M.: Tulip: Towards unified language-image pretraining. arXiv preprint arXiv:2503.15485 (2025)

    Pith/arXiv arXiv 2025

  49. [49]

    Trockman, A., Kolter, J.Z.: Patches are all you need? arXiv preprint arXiv:2201.09792 (2022)

    Pith/arXiv arXiv 2022

  50. [50]

    arXiv preprint arXiv:2502.14786 (2025)

    Tschannen, M., Gritsenko, A., Wang, X., Naeem, M.F., Alabdulmohsin, I., Parthasarathy, N., Evans, T., Beyer, L., Xia, Y., Mustafa, B., et al.: Siglip 2: Multilingual vision-language encoders with improved semantic understanding, lo- calization, and dense features. arXiv preprint arXiv:2502.14786 (2025)

    Pith/arXiv arXiv 2025

  51. [51]

    Advances in Neural Information Pro- cessing Systems36, 46830–46855 (2023)

    Tschannen, M., Kumar, M., Steiner, A., Zhai, X., Houlsby, N., Beyer, L.: Image captioners are scalable vision learners too. Advances in Neural Information Pro- cessing Systems36, 46830–46855 (2023)

    2023

  52. [52]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Vasu, P.K.A., Faghri, F., Li, C.L., Koc, C., True, N., Antony, A., Santhanam, G., Gabriel, J., Grasch, P., Tuzel, O., et al.: Fastvlm: Efficient vision encoding for vision language models. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 19769–19780 (2025)

    2025

  53. [53]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Vasu, P.K.A., Gabriel, J., Zhu, J., Tuzel, O., Ranjan, A.: Fastvit: A fast hy- brid vision transformer using structural reparameterization. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 5785–5795 (2023)

    2023

  54. [54]

    Advances in neural information pro- cessing systems30(2017)

    Vaswani,A.,Shazeer,N.,Parmar,N.,Uszkoreit,J.,Jones,L.,Gomez,A.N.,Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information pro- cessing systems30(2017)

    2017

  55. [55]

    arXiv preprint arXiv:2205.14100 (2022)

    Wang, J., Yang, Z., Hu, X., Li, L., Lin, K., Gan, Z., Liu, Z., Liu, C., Wang, L.: Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205.14100 (2022)

    Pith/arXiv arXiv 2022

  56. [56]

    arXiv preprint arXiv:2409.12191 (2024)

    Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al.: Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191 (2024)

    Pith/arXiv arXiv 2024

  57. [57]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Wang, W., Xie, E., Li, X., Fan, D.P., Song, K., Liang, D., Lu, T., Luo, P., Shao, L.: Pyramid vision transformer: A versatile backbone for dense prediction with- out convolutions. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 568–578 (2021)

    2021

  58. [58]

    CVPR (2025)

    Xing, L., Huang, Q., Dong, X., Lu, J., Zhang, P., Zang, Y., Cao, Y., He, C., Wang, J., Wu, F., et al.: Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction. CVPR (2025)

    2025

  59. [59]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Yang, S., Chen, Y., Tian, Z., Wang, C., Li, J., Yu, B., Jia, J.: Visionzip: Longer is better but not necessary in vision language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19792– 19802 (2025)

    2025

  60. [60]

    In: The Thirteenth International Conference on Learning Representa- tions

    Yu, S., Tang, C., Xu, B., Cui, J., Ran, J., Yan, Y., Liu, Z., Wang, S., Han, X., Liu, Z., et al.: Visrag: Vision-based retrieval-augmented generation on multi-modality documents. In: The Thirteenth International Conference on Learning Representa- tions

  61. [61]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Yu,W.,Luo,M.,Zhou,P.,Si,C.,Zhou,Y.,Wang,X.,Feng,J.,Yan,S.:Metaformer is actually what you need for vision. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10819–10829 (2022) 20 I. Maniadis Metaxas et al

    2022

  62. [62]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al.: Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9556–9567 (2024)

    2024

  63. [63]

    In: Proceedings of the IEEE/CVF conference on computer vision and pat- tern recognition

    Yun, S., Ro, Y.: Shvit: Single-head vision transformer with memory efficient macro design. In: Proceedings of the IEEE/CVF conference on computer vision and pat- tern recognition. pp. 5756–5767 (2024)

    2024

  64. [64]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Zhai, X., Mustafa, B., Kolesnikov, A., Beyer, L.: Sigmoid loss for language im- age pre-training. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 11975–11986 (2023)

    2023

  65. [65]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Zhang, Q., Cheng, A., Lu, M., Zhang, R., Zhuo, Z., Cao, J., Guo, S., She, Q., Zhang, S.: Beyond text-visual attention: Exploiting visual cues for effective token pruning in vlms. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 20857–20867 (2025)

    2025

  66. [66]

    arXiv e-prints pp

    Zhang, Q., Cheng, A., Lu, M., Zhuo, Z., Wang, M., Cao, J., Guo, S., She, Q., Zhang, S.: [cls] attention is all you need for training-free visual token pruning: Make vlm inference faster. arXiv e-prints pp. arXiv–2412 (2024)

    2024

  67. [67]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Zhang, X., Zhou, X., Lin, M., Sun, J.: Shufflenet: An extremely efficient convolu- tional neural network for mobile devices. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 6848–6856 (2018)

    2018

  68. [68]

    arXiv preprint arXiv:2410.04417 (2024)

    Zhang, Y., Fan, C.K., Ma, J., Zheng, W., Huang, T., Cheng, K., Gudovskiy, D., Okuno, T., Nakata, Y., Keutzer, K., et al.: Sparsevlm: Visual token sparsifica- tion for efficient vision-language model inference. arXiv preprint arXiv:2410.04417 (2024)

    Pith/arXiv arXiv 2024

  69. [69]

    arXiv preprint arXiv:2306.13394 (2021)

    Zhang, Y.S.Y.Q.M., Zheng, X.L.J.Y.X., Wu, K.L.X.S.Y., Fu, R.J.C., Chen, P.: Mme: A comprehensive evaluation benchmark for multimodal large language mod- els. arXiv preprint arXiv:2306.13394 (2021)

    Pith/arXiv arXiv 2021

  70. [70]

    Zhu, J., Wang, W., Chen, Z., Liu, Z., Ye, S., Gu, L., Tian, H., Duan, Y., Su, W., Shao, J., et al.: Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479 (2025) UltraViT: Latency-Optimized On-device Vision Encoder 21 A Additional Implementation Details For the majority of our dense ...

    Pith/arXiv arXiv 2025