Pith. sign in

REVIEW 3 major objections 4 minor 45 references

Look Less, Think Faster: Joint Token-Compute Adaptation for Multimodal LLMs

T0 review · 3 major / 4 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper shows that jointly adapting visual token count and LLM compute achieves better accuracy-efficiency tradeoffs than optimizing either alone.

desk verdict A genuinely new joint token-compute scheduling framework, but the efficiency axis is an unvalidated FLOPs proxy and the 'consistently outperforms' claim is contradicted by the paper's own MMBench numbers. read the letter →

arxiv 2607.20357 v1 pith:V6JJ2QXC submitted 2026-07-22 cs.CV

Pengcheng Wang , Zhiquan Wang , Jayoung Lee , Zhuoyan Xu , Ran Xu , Saurabh Bagchi , Yin Li , Somali Chaterji This is my paper
classification cs.CV
keywords adaptiveinferencemultimodalLLMstokenpruninglayerskippingcomputebudgetdifferentiableFLOPsestimatorParetofrontierLLaVA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the two main sources of inference cost in multimodal LLMs—the number of visual tokens and the amount of LLM computation (layers and attention heads)—cannot be optimized independently. It proposes SmartVL, a framework with a vision-side token controller and an LLM-side compute controller that are trained together, sharing a single compute-budget signal. A differentiable FLOPs estimator lets the pair learn to allocate a target budget across tokens, depth, and width for each input. On seven benchmarks, SmartVL reports a better accuracy-versus-efficiency Pareto frontier than token-only, compute-only, or naively composed baselines, e.g., about 7.8% higher accuracy than AdaLLaVA at roughly 50% compute. The practical payoff: a single trained model can serve a continuous range of compute budgets without retuning thresholds.

What carries the argument

The load-bearing mechanism is a differentiable FLOPs estimator that maps continuous controller outputs (Gumbel-sigmoid token probabilities and layer/head activation probabilities) to an estimated prefill cost. Because the estimator is differentiable, both controllers receive gradient signals for budget compliance and can be trained end-to-end with a single shared budget token appended to the vision encoder and the LLM prompt. A cross-stage token survival rate, passed with a stop-gradient, makes the LLM-side decisions explicitly aware of how many tokens survived. The asymmetric budget-violation loss penalizes overshoot quadratically and undershoot linearly, with a warm-up schedule that avoids

What would settle it

Measure end-to-end wall-clock latency on a modern accelerator (e.g., an A100) for SmartVL at several budgets using a tight sparse-mask implementation, and compare the resulting latency-accuracy curves to the FLOPs-based curves; if the frontier order reverses—for instance, a baseline becomes faster at equal accuracy—the central claim collapses.

Watch

Extended reading notes

Core claim

The central discovery is that token redundancy and LLM computational requirement are coupled: how many visual tokens are worth keeping depends on how much reasoning depth remains, and vice versa. Treating these dimensions independently wastes FLOPs or causes irreversible information loss. SmartVL operationalizes this coupling by conditioning both controllers on the same budget encoding and on the actual retained token count, then training with a differentiable FLOPs estimator and an asymmetric budget-violation loss. The paper claims this yields a strictly better accuracy-efficiency tradeoff curve—at 50% FLOPs it reports 74.4% on VQAv2 versus 67.9% for AdaLLaVA, and it maintains or improves a

Load-bearing premise

The entire comparison is measured in estimated prefill FLOPs; if that differentiable estimator does not track actual wall-clock latency on real hardware, the claimed Pareto superiority may not appear in deployment.

Editorial extensions

If this is right

  • A single SmartVL model can serve any target compute budget in its trained range without per-budget retraining or manual retention-ratio thresholds.
  • At equal prefill FLOPs, joint control yields higher accuracy than optimizing token pruning or layer/head skipping alone; the paper reports an average 7.8% gain over AdaLLaVA at ~50% compute across seven benchmarks.
  • The approach transfers to larger backbones: on LLaVA-1.5-13B, SmartVL improves the Pareto frontier over AdaLLaVA (e.g., 75.45% at 7.25T versus 72.27% at 8.27T near half budget).
  • The learned allocation is content- and task-dependent—POPE favors high token retention with thin compute, while VQAv2 and TextVQA favor deeper reasoning—so no single fixed ratio is optimal.
  • Because token reduction cuts the quadratic attention term, joint control provides strictly larger savings than any single dimension alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if the FLOPs estimator remains a faithful proxy, the same joint-control design should extend to the decode stage, where memory bandwidth (KV-cache) rather than FLOPs dominates; the authors flag this as open.
  • Beyond the paper: independent Gumbel-sigmoid gating could be applied to other conditional-computation settings, such as MoE expert routing or early-exit classifiers, where the optimal number of active units is unknown a priori.
  • Beyond the paper: a direct wall-clock test on sparse-mask-capable hardware would tell whether the Pareto wins in FLOPs translate to latency wins; the authors note this integration is still needed.
  • Beyond the paper: because the budget token uses sinusoidal encodings, the controller may generalize to budgets unseen in training; a testable extension is to probe interpolation and extrapolation behavior outside the trained budget range.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes SmartVL, a framework for adaptive inference in multimodal LLMs that jointly controls visual-token retention and LLM layer/head activation under a global prefill-FLOPs budget. Two lightweight controllers, one vision-side and one LLM-side, are conditioned on a shared budget encoding and trained end-to-end with a differentiable FLOPs estimator and an asymmetric budget-violation loss. The method is evaluated on seven benchmarks against AdaLLaVA, FastV, LLaVA-PruMerge+, and AdaLLaVA-PruMerge, with Pareto frontiers over estimated prefill FLOPs. The central claim is that joint token-compute scheduling yields consistently superior accuracy-efficiency tradeoffs compared with token-only, compute-only, or naively composed baselines.

Significance. The underlying idea is timely and useful: it exposes the coupling between visual-token sparsity and LLM architectural capacity, and the proposed training scheme (shared budget encoding, Gumbel-sigmoid gates, differentiable FLOPs estimator) is a plausible way to search this joint discrete space. The paper is clearly written, covers seven benchmarks, includes a 13B scaling experiment, and provides ablations of the two controller variants. However, the evidence as presented does not establish the headline claim of consistent Pareto superiority, and the efficiency axis is an unvalidated FLOPs proxy rather than measured latency. If the method were supplemented with wall-clock validation and corrected claims, its contribution would be solid; in its current form the central claim is overstated.

major comments (3)
  1. [Abstract / §1 vs. §4.2] The claim that SmartVL "consistently outperforms" prior adaptive methods and AdaLLaVA "across all FLOPs levels" is directly contradicted by the paper's own numbers. In §4.2, MMBench at 50% FLOPs gives SmartVL 62.0 vs. AdaLLaVA-PruMerge 63.1 and AdaLLaVA 63.3; GQA at 50% gives SmartVL 59.8 vs. AdaLLaVA-PruMerge 60.1; VQAv2 at 50% gives SmartVL 74.4 vs. AdaLLaVA-PruMerge 74.5. These are cases where SmartVL is numerically worse. The abstract, §1, and the conclusion should either be narrowed to "comparable or better on most benchmarks" or the claim must be defended with error bars and a statistical test showing the residual gaps are not significant.
  2. [§3.3, Eq. (8); Figs. 1,3,5,8] All reported Pareto frontiers use the differentiable estimator \(\hat{C}\) in Eq. (8) as the x-axis, and the same estimator supplies the budget-violation loss in Eq. (10). No wall-clock latency, throughput, or energy measurement is reported anywhere. The Future Work section explicitly states that "translating FLOP reductions into wall-clock improvements will require integrating sparse execution masks with specialized hardware kernels," which concedes that the proxy may not reflect actual deployment cost. Because the central claim is practical accuracy-efficiency superiority, the paper should report end-to-end prefill latency on a concrete accelerator (e.g., A100/H100) for SmartVL and each baseline at comparable \(\hat{C}] values, or explicitly reframe the claim as FLOP-based rather than latency-based.
  3. [§4 evaluation methodology] No error bars, multiple seeds, or statistical significance tests are provided, and no code/data are released. Several headline comparisons hinge on differences of 0.1-1.3 points (e.g., VQAv2 50%: 74.4 vs. 74.5; GQA 50%: 59.8 vs. 60.1; MMBench 50%: 62.0 vs. 63.1/63.3). Without variance estimates, "consistent" superiority cannot be assessed. In addition, the §1 statement that SmartVL "outperforms AdaLLaVA by an average of 7.8% across seven benchmarks" at \(~50%\) compute lacks a supporting table with per-benchmark values; please add such a table with standard deviations.
minor comments (4)
  1. [§3.3, Eq. (9)] The asymmetric loss uses a squared penalty for overshoot and a linear penalty for undershoot. This is plausible, but the choice of the margin \(\mu\) and weights \(w_{\mathrm{over}}, w_{\mathrm{under}}\) is not ablated; a short sensitivity analysis would strengthen the paper.
  2. [§3.3, Inference] The deterministic budget projection at inference is described only qualitatively ("discards the lowest-confidence visual tokens" first, then reduces layers/heads). Please specify the exact algorithm, including the order and stopping criterion, so the procedure is reproducible.
  3. [§4.2] The phrase "consistently outperforms or matches" is more accurate than the abstract's "consistently outperforms." Please align the wording throughout the paper.
  4. [Fig. 7 caption] Typo: "comp-tute budget" should be "compute budget."

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the central claim is empirically grounded on external benchmarks; minor self-citation and FLOPs-proxy caveats do not reduce the derivation to its inputs.

full rationale

SmartVL's central claim—that joint token-compute scheduling yields superior accuracy-efficiency Pareto frontiers—is evaluated on external benchmarks (VQAv2, GQA, TextVQA, ScienceQA, POPE, VizWiz, MMBench) using standard accuracy metrics, not fitted predictions. The differentiable FLOPs estimator in Eq. (8) is an analytical, non-fitted formula for prefill cost; the same normalized ratio r enters the budget-violation loss (Eqs. 9-10) and the x-axis of the Pareto plots. This is a common and legitimate practice for constrained-efficiency methods, and it does not make the accuracy comparison circular because accuracy is independently measured. The closest self-citation is AdaLLaVA [37], whose authors overlap with the current team; however, it is used as a comparison baseline and evaluation-protocol reference, not as a load-bearing derivation of SmartVL's mechanism. The Future Work admission that 'translating FLOP reductions into wall-clock improvements will require integrating sparse execution masks with specialized hardware kernels' is a validity caveat about the FLOPs proxy, not a circular step. No parameter is fitted to evaluation data and then reported as a prediction, and no uniqueness theorem is imported from the authors' prior work. The score of 2 reflects the minor self-citation and the proxy-validation gap, not actual circularity.

Assumptions & free parameters 7 free parameters · 6 assumptions · 0 invented entities

The central claim rests on the FLOPs estimator being a faithful cost model, on Gumbel-sigmoid STE gradients being usable, and on several training choices (prefix length, loss weights, budget range, temperature) that are not specified with values. No new physical entities are introduced; the 'budget token' is a learned conditioning vector, not an entity with independent falsifiable evidence. The main unfitted choices are hyperparameters, not fitted physical constants.

free parameters (7)
  • Budget margin mu in asymmetric loss = not reported
    Eq. (9) defines a zero-penalty tolerance interval [b-mu, b]; mu is a hand-chosen hyperparameter controlling how much under-budget execution is tolerated.
  • Loss weights w_over, w_under = not reported
    Eq. (10) balances quadratic overshoot penalty and linear undershoot penalty; values are not given and affect the learned allocation.
  • Warmup duration T_warmup = not reported
    The schedule lambda(t)=min(1,t/T_warmup) delays full budget regularization; T_warmup is not specified.
  • Gumbel temperature tau = not reported
    Used in Eq. (6) for token/layer/head Gumbel-sigmoid; stated to be fixed across benchmarks but no value is given.
  • Prefix layer count P = 13B: 20; 7B: not stated
    The first P transformer blocks always execute at full capacity; P controls the minimum compute baseline and is not specified for the 7B model.
  • Head group count G = 4 for LLaMA-7B
    Attention heads are partitioned into G groups for the LH controller; G is a design choice affecting width-granularity.
  • Training budget range [b_min, b_max] = not reported
    Budgets are sampled uniformly from this range during training; the range determines generalization across the reported 20-100% operating points.
assumptions (6)
  • domain assumption Prefill FLOPs is a valid hardware-independent proxy for inference latency and compute budget.
    Invoked in §3 'Budget constraint' and used for all budget conditioning; the paper acknowledges in Future Work that FLOPs-to-wall-clock translation requires specialized sparse kernels and that decode is unaddressed.
  • domain assumption The differentiable FLOPs estimator in Eq. (8) accurately models true prefill cost of the selected token/layer/head configuration.
    Eq. (8) is used both as the training-time budget penalty and as the efficiency metric in every Pareto plot; an inaccurate estimator would make the reported tradeoffs unrepresentative of real cost.
  • standard math Gumbel-sigmoid with straight-through estimator provides usable gradients for optimizing discrete token/layer/head masks.
    Eq. (6) applies the Concrete/Gumbel-sigmoid relaxation of Maddison and Jang; this is standard practice for differentiable discrete sampling.
  • domain assumption RoPE allows physically discarding visual tokens and using the compressed contiguous sequence without positional re-encoding.
    Stated in §3.1: 'RoPE natively handles this compressed, contiguous sequence without requiring positional re-encoding.' This is a modeling assumption about token-order semantics after pruning.
  • domain assumption Training with batch-level budgets sampled uniformly from [b_min,b_max], plus sinusoidal budget encoding, generalizes to unseen budgets and individual samples.
    Described in §3.3 'Training procedure'; the paper provides no dedicated test of generalization to out-of-range budgets or per-sample budget variance.
  • domain assumption The vision encoder can be kept frozen while the budget token and token controller are trained on top of it.
    Stated in §3.3: 'the vision encoder remains frozen to ensure a stable input feature distribution for training.' This presumes frozen ViT features are sufficient for budget-conditioned token selection.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Look Less, Think Faster: Joint Token-Compute Adaptation for Multimodal LLMs." pith.science (2026). https://pith.science/paper/V6JJ2QXC

@misc{pith2026260720357,
  author       = {Pith},
  title        = {Pith review of: Look Less, Think Faster: Joint Token-Compute Adaptation for Multimodal LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V6JJ2QXC}},
  note         = {Machine review of arXiv:2607.20357}
}
read the original abstract

Multimodal Large Language Models (MLLMs) have recently demonstrated strong performance across vision-language tasks. However, their high inference cost, arising from both the large number of input visual tokens and the heavy computation of the large language model (LLM), remains a key barrier to practical deployment. Recent work attempts to reduce the cost by adaptively optimizing individual dimensions, e.g., pruning redundant visual tokens or skipping LLM layers and heads. Nonetheless, prior approaches typically treat these dimensions independently and overlook a fundamental coupling: the available compute resources must be dynamically allocated across all dimensions based on the input content. To bridge the gap, we propose SmartVL, a unified adaptive inference framework that jointly controls vision token number and model compute capability in response to varying input contents and compute budgets. SmartVL introduces a vision-side token controller that dynamically selects informative visual tokens and an LLM-side compute controller that adaptively adjusts LLM computation. Importantly, these controllers are trained to coordinate with each other so that the overall inference cost satisfies a target budget. To allow this joint scheduling, we connect the controllers using a shared budget encoding and leverage a differentiable latency estimator for end-to-end training. This design enables SmartVL to learn cross-stage allocation strategies that adapt to both input complexity and runtime compute constraints. Experiments across multiple MLLM benchmarks demonstrate that, with joint scheduling, SmartVL consistently outperforms prior adaptive methods and achieves superior accuracy-efficiency Pareto frontiers. Project page: https://www.schaterji.io/publications/2026/jointtokencompute.

Figures

Figures reproduced from arXiv: 2607.20357 by the authors.

Figure 1
Figure 1. Pareto frontier on VQAv2. SmartVL achieves a stronger accuracy– efficiency tradeoff than token-pruning methods (FastV and LLaVA-PruMerge+), compute-only control (AdaLLaVA), and the composed token+compute baseline (AdaLLaVA-PruMerge), highlighting the benefit of cross-dimensional adaptation. Driven by our insight, we propose SmartVL, a unified adaptive in￾ference framework that jointly con￾trols sequence length, mode… view at source ↗
Figure 2
Figure 2. Overview of SmartVL. A vision-side token controller and an LLM-side compute controller jointly govern three coupled compute dimensions: vision tokens (sequence length), active attention heads and FFN capacity (width), and executed transformer layers (depth). Both controllers are conditioned on the same global budget and trained end-to-end through a differentiable latency estimator, enabling them to learn content- an… view at source ↗
Figure 3
Figure 3. Accuracy–Latency Pareto comparison across benchmarks. [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Pareto frontier of different model variants (TextVQA). Our vari￾ants (L and LH) achieve a larger adapta￾tion range and better trade-off compared to AdaLLaVA / AdaLLaVA-PruMerge. 4.4 Ablation Studies L vs. LH variant. Comparing the two variants and baselines on TextVQA …
Figure 6
Figure 6. Figure 6: Optimal token-compute adaptation varies based on task. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Visualization of token-compute allocation under a 4 TFLOPs comp [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Scaling to 13B on VQAv2. SmartVL improves the accuracy-efficiency Pareto frontier over AdaLLaVA. As shown in [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

45 extracted references · 8 linked inside Pith

  1. [1]

    arXiv preprint arXiv:2308.12966 (2023)

    Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., Zhou, J.: Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966 (2023)

    arXiv 2023

  2. [2]

    arXiv preprint arXiv:2511.21631 (2025)

    Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al.: Qwen3-VL technical report. arXiv preprint arXiv:2511.21631 (2025)

    arXiv 2025

  3. [3]

    arXiv preprint arXiv:2502.13923 (2025)

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., Zhong, H., et al.: Qwen2.5-VL technical report. arXiv preprint arXiv:2502.13923 (2025)

    arXiv 2025

  4. [4]

    In: ECCV (2024)

    Chen, L., Zhao, H., Liu, T., Bai, S., Lin, J., Zhou, C., Chang, B.: An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision- language models. In: ECCV (2024)

    2024

  5. [5]

    arXiv preprint arXiv:2312.16886 (2023)

    Chu, X., Qiao, L., Lin, X., Xu, S., Yang, Y., Hu, Y., Wei, F., Zhang, X., Zhang, B., Wei, X., et al.: MobileVLM: A fast, strong and open vision language assistant for mobile devices. arXiv preprint arXiv:2312.16886 (2023)

    arXiv 2023

  6. [6]

    arXiv preprint arXiv:2402.03766 (2024)

    Chu, X., Qiao, L., Zhang, X., Xu, S., Wei, F., Yang, Y., Sun, X., Hu, Y., Lin, X., Zhang, B., et al.: MobileVLM V2: Faster and stronger baseline for vision language model. arXiv preprint arXiv:2402.03766 (2024)

    arXiv 2024

  7. [7]

    In: NeurIPS (2023)

    Dai, W., Li, J., Li, D., Tiong, A., Zhao, J., Wang, W., Li, B., Fung, P.N., Hoi, S.: Instructblip: Towards general-purpose vision-language models with instruction tuning. In: NeurIPS (2023)

    2023

  8. [8]

    In: CVPR (2025)

    Deitke, M., Clark, C., Lee, S., Tripathi, R., Yang, Y., Park, J.S., Salehi, M., Muen- nighoff, N., Lo, K., Soldaini, L., et al.: Molmo and pixmo: Open weights and open data for state-of-the-art vision-language models. In: CVPR (2025)

    2025

Show all 45 references
  1. [9]

    In: NeurIPS (2023)

    Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L.: QLORA: Efficient fine- tuning of quantized LLMs. In: NeurIPS (2023)

    2023

  2. [10]

    In: CVPR (2017)

    Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., Parikh, D.: Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In: CVPR (2017)

    2017

  3. [11]

    In: CVPR (2018)

    Gurari, D., Li, Q., Stangl, A.J., Guo, A., Lin, C., Grauman, K., Luo, J., Bigham, J.P.: Vizwiz grand challenge: Answering visual questions from blind people. In: CVPR (2018)

    2018

  4. [12]

    In: ECCV (2024)

    Huang, K., Zou, H., Xi, Y., Wang, B., Xie, Z., Yu, L.: IVTP: Instruction-guided visual token pruning for large vision-language models. In: ECCV (2024)

    2024

  5. [13]

    In: CVPR (2026)

    Huang, Y., Wang, Z., Yuan, Z., Ding, Y., Gong, R., Guo, J., Liu, X., Zhang, J.: MoDES: Accelerating mixture-of-experts multimodal large language models via dynamic expert skipping. In: CVPR (2026)

    2026

  6. [14]

    In: CVPR (2019)

    Hudson, D.A., Manning, C.D.: GQA: A new dataset for real-world visual reasoning and compositional question answering. In: CVPR (2019)

    2019

  7. [15]

    In: ICLR (2017)

    Jang, E., Gu, S., Poole, B.: Categorical reparameterization with gumbel-softmax. In: ICLR (2017)

    2017

  8. [16]

    Microsoft Research Blog (2023)

    Javaheripi, M., Bubeck, S., Abdin, M., Aneja, J., Bubeck, S., Mendes, C.C.T., Chen, W., Del Giorno, A., Eldan, R., Gopi, S., et al.: Phi-2: The surprising power of small language models. Microsoft Research Blog (2023)

    2023

  9. [17]

    In: ICML (2021) Look Less, Think Faster 17

    Jia, C., Yang, Y., Xia, Y., Chen, Y.T., Parekh, Z., Pham, H., Le, Q., Sung, Y.H., Li, Z., Duerig, T.: Scaling up visual and vision-language representation learning with noisy text supervision. In: ICML (2021) Look Less, Think Faster 17

    2021

  10. [18]

    In: ICML (2023)

    Li, J., Li, D., Savarese, S., Hoi, S.: BLIP-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In: ICML (2023)

    2023

  11. [19]

    In: EMNLP (2023)

    Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W.X., Wen, J.R.: Evaluating object hallucination in large vision-language models. In: EMNLP (2023)

    2023

  12. [20]

    IEEE Transactions on Multimedia (2026)

    Lin, B., Tang, Z., Ye, Y., Huang, J., Zhang, J., Pang, Y., Jin, P., Ning, M., Luo, J., Yuan, L.: MoE-LLaVA: Mixture of experts for large vision-language models. IEEE Transactions on Multimedia (2026)

    2026

  13. [21]

    In: AAAI (2025)

    Lin, Z., Lin, M., Lin, L., Ji, R.: Boosting multimodal large language models with visual tokens withdrawal for rapid inference. In: AAAI (2025)

    2025

  14. [22]

    In: CVPR (2024)

    Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tuning. In: CVPR (2024)

    2024

  15. [23]

    In: NeurIPS (2023), LLaVA: Large Language and Vision Assistant

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. In: NeurIPS (2023), LLaVA: Large Language and Vision Assistant

    2023

  16. [24]

    Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al.: MMBench: Is your multi-modal model an all-around player? In: ECCV (2024)

    2024

  17. [25]

    In: NeurIPS (2022)

    Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.W., Zhu, S.C., Tafjord, O., Clark, P., Kalyan, A.: Learn to explain: Multimodal reasoning via thought chains for science question answering. In: NeurIPS (2022)

    2022

  18. [26]

    In: ICLR (2017)

    Maddison, C., Mnih, A., Teh, Y.: The concrete distribution: A continuous relax- ation of discrete random variables. In: ICLR (2017)

    2017

  19. [27]

    In: ICML (2021)

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: ICML (2021)

    2021

  20. [28]

    In: ICCV (2025)

    Shang, Y., Cai, M., Xu, B., Lee, Y.J., Yan, Y.: LLaVA-PruMerge: Adaptive token reduction for efficient large multimodal models. In: ICCV (2025)

    2025

  21. [29]

    In: ICLR (2025)

    Shu, F., Liao, Y., Zhang, L., Zhuo, L., Xu, C., Zhang, G., Shi, H., Chan, L., Yu, Z., He, W., et al.: LLaVA-MoD: Making llava tiny via moe-knowledge distillation. In: ICLR (2025)

    2025

  22. [30]

    In: CVPR (2019)

    Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., Rohrbach, M.: Towards vqa models that can read. In: CVPR (2019)

    2019

  23. [31]

    Neurocomputing (2024)

    Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., Liu, Y.: Roformer: Enhanced trans- former with rotary position embedding. Neurocomputing (2024)

    2024

  24. [32]

    In: Conference on Robot Learning (2025)

    Tian, X., Gu, J., Li, B., Liu, Y., Wang, Y., Zhao, Z., Zhan, K., Jia, P., Lang, X., Zhao, H.: DriveVLM: The convergence of autonomous driving and large vision- language models. In: Conference on Robot Learning (2025)

    2025

  25. [33]

    In: CVPR (2025)

    Vasu, P.K.A., Faghri, F., Li, C.L., Koc, C., True, N., Antony, A., Santhanam, G., Gabriel, J., Grasch, P., Tuzel, O., Pouransari, H.: FastVLM: Efficient vision encoding for vision language models. In: CVPR (2025)

    2025

  26. [34]

    In: NeurIPS (2017)

    Vaswani,A.,Shazeer,N.,Parmar,N.,Uszkoreit,J.,Jones,L.,Gomez,A.N.,Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: NeurIPS (2017)

    2017

  27. [35]

    In: NeurIPS (2024)

    Wang, C., Wang, Z., Xu, X., Tang, Y., Zhou, J., Lu, J.: QuantVLM: Post-training quantization for efficient vision–language models. In: NeurIPS (2024)

    2024

  28. [36]

    arXiv preprint arXiv:2409.12191 (2024)

    Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al.: Qwen2-VL: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191 (2024)

    2024 arXiv

  29. [37]

    In: ICCV (2025)

    Xu, Z., Nguyen, K.D., Mukherjee, P., Bagchi, S., Chaterji, S., Liang, Y., Li, Y.: Learning to inference adaptively for multimodal large language models. In: ICCV (2025)

    2025

  30. [38]

    arXiv preprint arXiv:2312.16862 (2023) 18 P

    Yuan, Z., Li, Z., Huang, W., Ye, Y., Sun, L.: TinyGPT-V: Efficient multimodal large language model via small backbones. arXiv preprint arXiv:2312.16862 (2023) 18 P. Wang et al

    2023 arXiv

  31. [39]

    In: EMNLP Demo Track (2023)

    Zhang, H., Li, X., Bing, L.: Video-LLaMA: An instruction-tuned audio-visual lan- guage model for video understanding. In: EMNLP Demo Track (2023)

    2023

  32. [40]

    IEEE TPAMI (2024)

    Zhang, J., Huang, J., Jin, S., Lu, S.: Vision-language models for vision tasks: A survey. IEEE TPAMI (2024)

    2024

  33. [41]

    In: Findings of the Association for Computational Linguistics: NAACL 2025 (2025)

    Zhang, K., Li, B., Zhang, P., Pu, F., Cahyono, J.A., Hu, K., Liu, S., Zhang, Y., Yang, J., Li, C., et al.: LMMs-Eval: Reality check on the evaluation of large mul- timodal models. In: Findings of the Association for Computational Linguistics: NAACL 2025 (2025)

    2025

  34. [42]

    In: ICCV (2025)

    Zhong, Y., Liu, Z., Li, Y., Wang, L.: AIM: Adaptive inference of multi-modal llms via token merging and pruning. In: ICCV (2025)

    2025

  35. [43]

    arXiv preprint arXiv:2402.14289 (2024)

    Zhou, B., Hu, Y., Weng, X., Jia, J., Luo, J., Liu, X., Wu, J., Huang, L.: TinyLLaVA: A framework of small-scale large multimodal models. arXiv preprint arXiv:2402.14289 (2024)

    2024 arXiv

  36. [44]

    In: ICLR (2024)

    Zhu, D., Chen, J., Shen, X., Li, X., Elhoseiny, M.: MiniGPT-4: Enhancing vision- language understanding with advanced large language models. In: ICLR (2024)

    2024

  37. [45]

    In: EMCLR (2024)

    Zhu, Y., Zhu, M., Liu, N., Xu, Z., Peng, Y.: LLaVA-Phi: Efficient multi-modal assistant with small language model. In: EMCLR (2024)

    2024

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.