Pith. sign in

REVIEW 4 major objections 6 minor 69 references

A small tool-free model can skip most of an agentic MLLM’s expensive visual tool loops without losing accuracy.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 19:33 UTC pith:4YNUIARF

load-bearing objection Solid systems paper: agentic-level speculation with a separability gate delivers real speedups; the accuracy-preservation claim is a bit softer than the abstract sells because of per-benchmark best-threshold reporting. the 4 major comments →

arxiv 2603.23483 v2 pith:4YNUIARF submitted 2026-03-24 cs.CV cs.CL

SpecEyes: Accelerating Agentic Multimodal LLMs via Speculative Perception and Planning

Haoyu Huang , Jinfa Huang , Zhongwei Wan , Xiawu Zheng , Rongrong Ji , Jiebo Luo This is my paper
classification cs.CV cs.CL
keywords agentic multimodal LLMsspeculative decodingvisual tool useanswer separabilitycognitive gatingheterogeneous servinglatency and throughput
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Agentic multimodal models get stronger by repeatedly calling visual tools, but each call depends on the last, so latency grows with depth and concurrent serving collapses. SpecEyes claims that a large share of queries never need those tools: a lightweight non-agentic model can answer them from the original image alone if you can tell when to trust it. The system first screens whether tools look necessary, lets the small model guess, then accepts or rejects that guess with an answer-separability score that measures how cleanly the top logits stand out from competitors—no ground-truth labels required. Low-confidence or tool-required queries fall back to the full agentic pipeline. On V*, HR-Bench, and POPE the method reports 1.1–3.35× wall-clock speedups while matching or beating the agentic baseline’s accuracy, and the parallel front-end turns acceptance rate into higher serving throughput.

Core claim

SpecEyes shows that agentic multimodal reasoning can be accelerated at the trajectory level, not merely token by token: a tool-free small MLLM, gated by answer separability and backed by agentic fallback, can safely bypass entire tool-use chains for a large fraction of queries, cutting latency and raising concurrency while preserving or improving accuracy.

What carries the argument

Answer separability (S_sep): a scale-invariant score that standardizes the leading logit against the mean and standard deviation of the top-K logits, then aggregates (by default with the min over tokens) to decide whether to accept the small model’s answer or fall back.

Load-bearing premise

That how cleanly the small model’s top answer stands out from its nearest competitors is a reliable enough stand-in for correctness that a single threshold can accept many answers without hurting accuracy relative to the full agentic run.

What would settle it

On the same V*, HR-Bench, and POPE splits, run SpecEyes with the reported min-separability gate and show that the accepted answers fall materially below the pure agentic baseline accuracy, or that measured wall-clock speedup stays near 1× once tool time and batch effects are included.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. SpecEyes proposes agentic-level speculative acceleration for multimodal LLMs that invoke visual tools iteratively. The authors formalize the stateful sequential bottleneck of agentic depth (Eqs. 1–4), then introduce a four-phase pipeline: (I) large-model tool-necessity screening, (II) tool-free small-model speculation, (III) cognitive gating via a novel answer-separability score S_sep over top-K logits (min aggregation preferred), and (IV) agentic fallback. A heterogeneous parallel funnel converts acceptance rate into throughput gains (Eq. 16). On V*, HR-Bench, and POPE with DeepEyes and Thyme backbones plus Qwen3-VL-2B as the draft model, SpecEyes reports 1.1–3.35× wall-clock speedup while matching or exceeding baseline accuracy (e.g., DeepEyes average 81.39% → 84.26%), with ablations on threshold, batch size, and K, plus KDE evidence for min aggregation.

Significance. If the results hold under stricter evaluation of the gate, this is a useful systems contribution for agentic multimodal serving. Prior work accelerates tokens or prunes visual tokens inside a fixed agentic loop; SpecEyes is among the first to ask whether the loop is needed at all and to couple that decision to a label-free confidence gate and a heterogeneous serving funnel. The latency model (Eq. 9) and throughput approximation (Eq. 16) are clear and falsifiable; multi-backbone experiments and the promised code strengthen reproducibility. Accuracy gains on POPE and V* Relative Position, if not purely selection artifacts, would also suggest that skipping unnecessary tool trajectories can reduce hallucination—an interesting side effect beyond pure speedup.

major comments (4)
  1. [Sec. 4.1, Tab. 1] Sec. 4.1 and Tab. 1: thresholds are chosen as best operating points after collecting the full empirical confidence distribution of MS on each benchmark. The abstract and main claim of “preserving or even improving accuracy (up to +6.7%)” therefore rest on post-hoc, per-benchmark selection rather than a fixed or held-out τ. Please report (i) results with a single τ chosen on a held-out split or transferred across benchmarks, and (ii) false-accept rate of accepted answers vs. ground truth when S_min_sep ≥ τ. Without these, the accuracy half of the central claim is under-supported relative to the latency model.
  2. [Sec. 3.3, Proposition 1] Sec. 3.3, Proposition 1 and Eqs. (12)–(13): the preference for min aggregation rests on the untested premise that P(E_n) is monotonically decreasing in S_sep^(n). The paper never measures token- or answer-level error as a function of S_sep, nor reports how often MS is wrong among accepted answers. Either provide empirical calibration curves (error rate vs. S_sep bins) or soften Proposition 1 to a design heuristic and justify min aggregation solely via the KDE/Δ evidence in Fig. 3 and the Tab. 1 trade-offs.
  3. [Sec. 3.2, Tab. 1] Sec. 3.2 Phases I–III and Tab. 1: accuracy lifts (especially POPE Adv. 78.43%→85.13% and V* Pos. 82.89%→89.47%) can arise from Phase-I screening alone (routing tool-required queries away from MS) plus selective acceptance of high-confidence MS answers, independent of whether S_sep is a reliable correctness proxy. Please ablate (a) Phase I only (always accept MS when g=0), (b) Phase III only (no ML screening), and (c) full SpecEyes, reporting accuracy and βα separately so the contribution of the gate is isolated from screening and fallback.
  4. [Eq. (9), Tab. 1 HR-Bench] Eq. (9) and HR-Bench 8K (Tab. 1, Thyme): when βα is low, SpecEyes can fall below 1× (0.95×). The paper notes this but does not quantify the break-even βα or the overhead of always running Phase I+II. A short sensitivity plot of E[L_SpecEyes]/L_agent vs. βα (or measured β, α per split) would make the regime of applicability precise and strengthen the systems claim.
minor comments (6)
  1. [Fig. 1] Fig. 1 caption averages accuracy/speedup across three benchmarks; state explicitly that these are unweighted means of the seven splits (or of the three benchmark averages) so readers can reconcile with Tab. 1.
  2. [Abstract] Abstract says “up to +6.7%” accuracy; Tab. 1 DeepEyes average lift is +2.87 pp and the largest single-split lift is POPE Adv. +6.7 pp. Clarify that +6.7% is a per-split maximum, not the average.
  3. [Sec. 4.2] SpecReason is a useful contrast but consistently decelerates (0.37–0.61×). Briefly note why token/semantic speculative methods expand turns on tool-calling models so readers do not over-generalize the comparison.
  4. [Eq. (1), title page] Notation: f in Eq. (1) is overloaded with the fusion function; consider f_fuse or similar. Also “/envel⌢pe” in the author line appears to be a LaTeX artifact for corresponding-author markers.
  5. [Sec. 3.3] Sec. 3.3: state the default values of K, ε, and r once in the method section (they appear only in Sec. 4.1) so the definition of S_sep is self-contained.
  6. [Sec. 5] Future-work multi-depth speculation (D=1,2,…) is well motivated by the HR-Bench result; a one-sentence estimate of residual tool-required fraction after Phase I would help quantify the headroom.

Circularity Check

1 steps flagged

Empirical systems paper with non-circular latency/funnel math; only mild circularity is best-threshold reporting that makes accuracy-preservation partly by selection.

specific steps
  1. fitted input called prediction [Sec. 4.1 Implementation Details; Tab. 1 caption/results discussion]
    "The gating threshold is selected by running MS once per benchmark to collect the empirical confidence distribution (∼5–10min offline), from which we evenly sample multiple operating points to characterize the accuracy–latency trade-off. ... For each SpecEyes variant, we report the result at the best operating-point threshold that preserves the baseline level accuracy."

    Accuracy preservation is partly by construction of the reporting rule: among thresholds sampled from the same benchmark’s confidence distribution, they select the operating point that keeps accuracy at or above the agentic baseline, then report that accuracy is preserved (and speedup at that point). This is standard gate tuning, not a forced theoretical prediction, but the “preserving accuracy” half of the main claim is not an independent fixed-τ result.

full rationale

SpecEyes is an empirical acceleration framework, not a first-principles derivation that forces its headline numbers by definition. The stateful bottleneck (Eqs. 1–4), expected latency E[L_SpecEyes]=c_J+βc_S+(1−βα)L_agent (Eq. 9), and throughput ratio ≈1/(1−βα) (Eq. 16) follow from the heterogeneous funnel architecture and measured screening/acceptance rates; they are not tautologies of fitted targets. Answer separability S_sep (Eqs. 12–13) and Proposition 1 are design choices with an untested monotonicity premise, but that is an assumption/correctness issue, not circular reduction of a claimed prediction to its inputs. Self-citations in related work are ordinary and not load-bearing for the main claim. The only mild circularity is evaluation procedure: τ is chosen per benchmark from MS’s empirical confidence distribution, and Table 1 reports the best operating point that preserves baseline accuracy—so the “preserving accuracy” half of the claim is partly ensured by selection rather than a fixed-τ held-out test. Speedups and any accuracy gains remain measured on external benchmarks and are not forced by construction. Score 2, not higher.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 3 invented entities

The central claim is empirical systems performance under a routing policy. It rests on standard sequential-dependency modeling of tool chains, the domain premise that many queries are answerable without tools, engineered confidence scores and thresholds, and new named mechanisms (S_sep, cognitive gate, parallel funnel) rather than new physical entities. Free parameters are the usual gate/serving knobs; none are hidden global physics constants.

free parameters (5)
  • gating threshold τ
    Chosen per benchmark from the small model’s empirical confidence distribution to navigate accuracy–latency trade-off; main results use best operating points that preserve baseline accuracy.
  • top-K for separability (K)
    Default K=64; ablation shows larger K raises speedup and can hurt accuracy, so K tunes aggressiveness.
  • bottom-fraction r for S_bottom_sep
    Set to 0.2 for the bottom aggregation variant; affects alternative gates though min is default.
  • numerical epsilon ε
    ε=1e-6 in the separability denominator for stability; minor but part of the score definition.
  • max agentic tool steps
    Agentic models capped at 5 tool-use steps per query; bounds baseline latency and fallback cost.
axioms (5)
  • domain assumption Agentic tool selection forms a strict causal Markov chain so step d+1 cannot start until tool observation at d completes (Eqs. 1–2).
    Load-bearing systems premise for the sequential bottleneck and latency model in Sec. 3.1.
  • domain assumption A substantial fraction of queries can be answered correctly by a small non-agentic MLLM from the original image alone.
    Stated key insight in Sec. 3.2; without it βα is small and speedups vanish (seen on HR-Bench).
  • ad hoc to paper Token error probability is monotonically decreasing in token-level separability S_sep^(n), so min aggregation most tightly bounds answer error (Proposition 1).
    Used to justify default min aggregation; not independently proved beyond the risk-theoretic sketch.
  • domain assumption Large model binary tool-necessity judgment (Phase I) is a reliable cheap screen despite using ML rather than MS.
    Sec. 3.2 claims ML is a better judge of tool need; screening quality drives β.
  • standard math Standard softmax/logit arithmetic and batch-parallel single-turn inference for stateless models.
    Background for confidence scores and the parallel front-end cost model.
invented entities (3)
  • Answer separability score S_sep no independent evidence
    purpose: Calibration-free confidence for accepting small-model answers without oracle labels.
    Defined from standardized margin of top logit vs top-K competitors (Eq. 12) and aggregated (Eq. 13); evaluated only inside this paper’s gating experiments.
  • Cognitive gating / confidence switching via S_sep no independent evidence
    purpose: Decide accept vs agentic fallback in Phase III.
    Policy mechanism built on S_sep and threshold τ; not an external physical object.
  • Heterogeneous parallel funnel no independent evidence
    purpose: Batch-parallel small-model front-end with residual serial agentic fallback to convert acceptance rate into throughput.
    Serving architecture in Sec. 3.4; engineering construct validated by batch-size ablations.

pith-pipeline@v1.1.0-grok45 · 20706 in / 3512 out tokens · 39162 ms · 2026-07-13T19:33:37.420548+00:00 · methodology

0 comments
read the original abstract

Agentic multimodal large language models (MLLMs) (e.g., OpenAI o3 and Gemini Agentic Vision) achieve remarkable reasoning capabilities through iterative visual tool invocation. However, the cascaded perception, reasoning, and tool-calling loops introduce significant sequential overhead. This overhead, termed agentic depth, incurs prohibitive latency and seriously limits system-level concurrency. To this end, we propose SpecEyes, an agentic-level speculative acceleration framework that breaks this sequential bottleneck. Our key insight is that a lightweight, tool-free MLLM can serve as a speculative planner to predict the execution trajectory, enabling early termination of expensive tool chains without sacrificing accuracy. To regulate this speculative planning, we introduce a cognitive gating mechanism based on answer separability, which quantifies the model's confidence for self-verification without requiring oracle labels. Furthermore, we design a heterogeneous parallel funnel that exploits the stateless concurrency of the small model to mask the stateful serial execution of the large model, maximizing system throughput. Extensive experiments on V* Bench, HR-Bench, and POPE demonstrate that SpecEyes achieves 1.1-3.35x speedup over the agentic baseline while preserving or even improving accuracy (up to +6.7%), thereby boosting serving throughput under concurrent workloads.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

69 extracted references · 24 linked inside Pith

  1. [1]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736, 2022

    2022

  2. [2]

    Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    Pith/arXiv arXiv 2023

  3. [3]

    Token merging: Your vit but faster.arXiv preprint arXiv:2210.09461, 2022

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster.arXiv preprint arXiv:2210.09461, 2022

    Pith/arXiv arXiv 2022

  4. [4]

    Medusa: Simple llm inference acceleration framework with multiple decoding heads.arXiv preprint arXiv:2401.10774, 2024

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple llm inference acceleration framework with multiple decoding heads.arXiv preprint arXiv:2401.10774, 2024

    Pith/arXiv arXiv 2024

  5. [5]

    Accelerating large language model decoding with speculative sampling.arXiv preprint arXiv:2302.01318, 2023

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling.arXiv preprint arXiv:2302.01318, 2023

    Pith/arXiv arXiv 2023

  6. [6]

    Ee-llm: Large-scale training and inference of early-exit large language models with 3d parallelism.arXiv preprint arXiv:2312.04916, 2023

    Yanxi Chen, Xuchen Pan, Yaliang Li, Bolin Ding, and Jingren Zhou. Ee-llm: Large-scale training and inference of early-exit large language models with 3d parallelism.arXiv preprint arXiv:2312.04916, 2023

    Pith/arXiv arXiv 2023

  7. [7]

    Sensenova-mars: Empowering multimodal agentic reasoning and search via reinforcement learning.arXiv preprint arXiv:2512.24330, 2025

    Yong Xien Chng, Tao Hu, Wenwen Tong, Xueheng Li, Jiandong Chen, Haojia Yu, Jiefan Lu, Hewei Guo, Hanming Deng, Chengjun Xie, Gao Huang, Dahua Lin, and Lewei Lu. Sensenova-mars: Empowering multimodal agentic reasoning and search via reinforcement learning.arXiv preprint arXiv:2512.24330, 2025

    arXiv 2025

  8. [8]

    Instructblip: Towards general-purpose vision-language models with instruction tuning.Advances in neural information processing systems, 36:49250–49267, 2023

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning.Advances in neural information processing systems, 36:49250–49267, 2023

    2023

  9. [9]

    Introducing Agentic Vision in Gemini 3 Flash.https://blog.google/innovation-and-ai/tech nology/developers-tools/agentic-vision-gemini-3-flash/, January 2026

    Rohan Doshi. Introducing Agentic Vision in Gemini 3 Flash.https://blog.google/innovation-and-ai/tech nology/developers-tools/agentic-vision-gemini-3-flash/, January 2026. Accessed: 2026-02-24

    2026

  10. [10]

    Feather the throttle: Revisiting visual token pruning for vision-language model acceleration.ICCV, 2025

    Mark Endo, Xiaohan Wang, and Serena Yeung-Levy. Feather the throttle: Revisiting visual token pruning for vision-language model acceleration.ICCV, 2025

    2025

  11. [11]

    Not all layers of llms are necessary during inference.arXiv preprint arXiv:2403.02181, 2024

    Siqi Fan, Xin Jiang, Xiang Li, Xuying Meng, Peng Han, Shuo Shang, Aixin Sun, Yequan Wang, and Zhongyuan Wang. Not all layers of llms are necessary during inference.arXiv preprint arXiv:2403.02181, 2024

    Pith/arXiv arXiv 2024

  12. [12]

    Framefusion: Combining similarity and importance for video token reduction on large vision language models

    Tianyu Fu, Tengxuan Liu, Qinghao Han, Guohao Dai, Shengen Yan, Huazhong Yang, Xuefei Ning, and Yu Wang. Framefusion: Combining similarity and importance for video token reduction on large vision language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22654–22663, 2025

    2025

  13. [13]

    Deep think with confidence.arXiv preprint arXiv:2508.15260, 2025

    Yichao Fu, Xuewei Wang, Yuandong Tian, and Jiawei Zhao. Deep think with confidence.arXiv preprint arXiv:2508.15260, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    Thinking with programming vision: Towards a unified view for thinking with images.arXiv preprint arXiv:2512.03746, 2025

    Zirun Guo, Minjie Hong, Feng Zhang, Kai Jia, and Tao Jin. Thinking with programming vision: Towards a unified view for thinking with images.arXiv preprint arXiv:2512.03746, 2025

    arXiv 2025

  15. [15]

    Zipvl: Efficient large vision-language models with dynamic token sparsification, 2024

    Yefei He, Feng Chen, Jing Liu, Wenqi Shao, Hong Zhou, Kaipeng Zhang, and Bohan Zhuang. Zipvl: Efficient large vision-language models with dynamic token sparsification, 2024. URLhttps://arxiv.org/abs/2410.08584

    Pith/arXiv arXiv 2024

  16. [16]

    Deepeyesv2: Toward agentic multimodal model.arXiv preprint arXiv:2511.05271, 2025

    Jack Hong, Chenxiao Zhao, ChengLin Zhu, Weiheng Lu, Guohai Xu, and Xing Yu. Deepeyesv2: Toward agentic multimodal model.arXiv preprint arXiv:2511.05271, 2025

    Pith/arXiv arXiv 2025

  17. [17]

    Thinking with drafts: Speculative temporal reasoning for efficient long video understanding.ArXiv, abs/2512.00805, 2025

    Pengfei Hu, Meng Cao, Yingyao Wang, Yi Wang, Jiahua Dong, Jun Song, Yu Cheng, Bo Zheng, and Xiaodan Liang. Thinking with drafts: Speculative temporal reasoning for efficient long video understanding.ArXiv, abs/2512.00805, 2025. URLhttps://api.semanticscholar.org/CorpusID:283449335

    arXiv 2025

  18. [18]

    Relayllm: Efficient reasoning via collaborative decoding.ArXiv, abs/2601.05167, 2026

    Chengsong Huang, Tong Zheng, Langlin Huang, Jinyuan Li, Haolin Liu, and Jiaxin Huang. Relayllm: Efficient reasoning via collaborative decoding.ArXiv, abs/2601.05167, 2026. URLhttps://api.semanticscholar.org/ CorpusID:284544142

    arXiv 2026

  19. [19]

    Evolver: Chain-of-evolution prompting to boost large multimodal models for hateful meme detection

    Jinfa Huang, Jinsheng Pan, Zhongwei Wan, Hanjia Lyu, and Jiebo Luo. Evolver: Chain-of-evolution prompting to boost large multimodal models for hateful meme detection. InProceedings of the 31st International Conference on Computational Linguistics, pages 7321–7330, 2025. 11

    2025

  20. [20]

    Token fusion: Bridging the gap between token pruning and token merging

    Minchul Kim, Shangqian Gao, Yen-Chang Hsu, Yilin Shen, and Hongxia Jin. Token fusion: Bridging the gap between token pruning and token merging. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1383–1392, 2024

    2024

  21. [21]

    Helios: Adaptive model and early-exit selection for efficient llm inference serving.arXiv preprint arXiv:2504.10724, 2025

    Avinash Kumar, Shashank Nag, Jason Clemons, Lizy John, and Poulami Das. Helios: Adaptive model and early-exit selection for efficient llm inference serving.arXiv preprint arXiv:2504.10724, 2025

    arXiv 2025

  22. [22]

    Mini-o3: Scaling up reasoning patterns and interaction turns for visual search.ArXiv, abs/2509.07969, 2025

    Xin Lai, Junyi Li, Wei Li, Tao Liu, Tianjian Li, and Hengshuang Zhao. Mini-o3: Scaling up reasoning patterns and interaction turns for visual search.ArXiv, abs/2509.07969, 2025. URLhttps://api.semanticscholar.org/ CorpusID:281217747

    Pith/arXiv arXiv 2025

  23. [23]

    Fast inference from transformers via speculative decoding

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. InInternational Conference on Machine Learning, pages 19274–19286. PMLR, 2023

    2023

  24. [24]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational conference on machine learning, pages 19730–19742. PMLR, 2023

    2023

  25. [25]

    Hero: Rethinking visual token early dropping in high-resolution large vision-language models, 2025

    Xu Li, Yuxuan Liang, Xiaolei Chen, Yi Zheng, Haotian Chen, Bin Li, and Xiangyang Xue. Hero: Rethinking visual token early dropping in high-resolution large vision-language models, 2025. URLhttps://arxiv.org/abs/ 2509.13067

    arXiv 2025

  26. [26]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 292–305, Singapore, December

    2023

  27. [27]

    doi: 10.18653/v1/2023.emnlp-main.20

    Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.20. URL https: //aclanthology.org/2023.emnlp-main.20/

    doi:10.18653/v1/2023.emnlp-main.20 2023

  28. [28]

    Eagle: Speculative sampling requires rethinking feature uncertainty.arXiv preprint arXiv:2401.15077, 2024

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty.arXiv preprint arXiv:2401.15077, 2024

    Pith/arXiv arXiv 2024

  29. [29]

    Eagle-2: Faster inference of language models with dynamic draft trees

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-2: Faster inference of language models with dynamic draft trees. InProceedings of the 2024 conference on empirical methods in natural language processing, pages 7421–7432, 2024

    2024

  30. [30]

    Eagle-3: Scaling up inference acceleration of large language models via training-time test.arXiv preprint arXiv:2503.01840, 2025

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-3: Scaling up inference acceleration of large language models via training-time test.arXiv preprint arXiv:2503.01840, 2025

    Pith/arXiv arXiv 2025

  31. [31]

    Moe-llava: Mixture of experts for large vision-language models.IEEE Transactions on Multimedia, 2026

    Bin Lin, Zhenyu Tang, Yang Ye, Jinfa Huang, Junwu Zhang, Yatian Pang, Peng Jin, Munan Ning, Jiebo Luo, and Li Yuan. Moe-llava: Mixture of experts for large vision-language models.IEEE Transactions on Multimedia, 2026

    2026

  32. [32]

    Speculative decoding reimagined for multimodal large language models.arXiv preprint arXiv:2505.14260, 2025

    Luxi Lin, Zhihang Lin, Zhanpeng Zeng, and Rongrong Ji. Speculative decoding reimagined for multimodal large language models.arXiv preprint arXiv:2505.14260, 2025

    Pith/arXiv arXiv 2025

  33. [33]

    Accelerating multi-modal llm gaming performance via input prediction and mishit correction.arXiv preprint arXiv:2512.17250, 2025

    Ziyang Lin, Zixuan Sun, Sanhorn Chen, Xiaoyang Chen, and Roy Zhao. Accelerating multi-modal llm gaming performance via input prediction and mishit correction.arXiv preprint arXiv:2512.17250, 2025

    arXiv 2025

  34. [34]

    Efficient inference of vision instruction-following models with elastic cache

    Zuyan Liu, Benlin Liu, Jiahui Wang, Yuhao Dong, Guangyi Chen, Yongming Rao, Ranjay Krishna, and Jiwen Lu. Efficient inference of vision instruction-following models with elastic cache. InEuropean Conference on Computer Vision, pages 54–69. Springer, 2024

    2024

  35. [35]

    Video-rag: Visually-aligned retrieval-augmented long video comprehension.arXiv preprint arXiv:2411.13093, 2024

    Yongdong Luo, Xiawu Zheng, Guilin Li, Shukang Yin, Haojia Lin, Chaoyou Fu, Jinfa Huang, Jiayi Ji, Fei Chao, Jiebo Luo, et al. Video-rag: Visually-aligned retrieval-augmented long video comprehension.arXiv preprint arXiv:2411.13093, 2024

    arXiv 2024

  36. [36]

    Quota: Query-oriented token assignment via cot query decouple for long video comprehension

    Yongdong Luo, Wang Chen, Weizhong Huang, Shukang Yin, Haojia Lin, Jinfa Huang, Chaoyou Fu, Jiayi Ji, Xiawu Zheng, and Jiebo Luo. Quota: Query-oriented token assignment via cot query decouple for long video comprehension. InProceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 24160–24168, 2026

    2026

  37. [37]

    Introducing OpenAI o3 and o4-mini.https://openai.com/index/introducing-o3-and-o4-mini/ , April 2025

    OpenAI. Introducing OpenAI o3 and o4-mini.https://openai.com/index/introducing-o3-and-o4-mini/ , April 2025. Accessed: 2026-02-24

    2025

  38. [38]

    Specreason: Fast and accurate inference-time compute via speculative reasoning.arXiv preprint arXiv:2504.07891, 2025

    Rui Pan, Yinwei Dai, Zhihao Zhang, Gabriele Oliaro, Zhihao Jia, and Ravi Netravali. Specreason: Fast and accurate inference-time compute via speculative reasoning.arXiv preprint arXiv:2504.07891, 2025. 12

    Pith/arXiv arXiv 2025

  39. [39]

    Skywork r1v: Pioneering multimodal reasoning with chain-of-thought.arXiv preprint arXiv:2504.05599, 2025

    Yi Peng, Peiyu Wang, Xiaokun Wang, Yichen Wei, Jiangbo Pei, Weijie Qiu, Ai Jian, Yunzhuo Hao, Jiachun Pan, Tianyidan Xie, et al. Skywork r1v: Pioneering multimodal reasoning with chain-of-thought.arXiv preprint arXiv:2504.05599, 2025

    Pith/arXiv arXiv 2025

  40. [40]

    Toolformer: Language models can teach themselves to use tools.Advances in neural information processing systems, 36:68539–68551, 2023

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools.Advances in neural information processing systems, 36:68539–68551, 2023

    2023

  41. [41]

    Mmspec: Benchmarking speculative decoding for vision-language models.arXiv preprint arXiv:2603.14989, 2026

    Hui Shen, Xin Wang, Ping Zhang, Yunta Hsieh, Qi Han, Zhongwei Wan, Ziheng Zhang, Jingxuan Zhang, Jing Xiong, Ziyuan Liu, et al. Mmspec: Benchmarking speculative decoding for vision-language models.arXiv preprint arXiv:2603.14989, 2026

    arXiv 2026

  42. [42]

    Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face.Advances in Neural Information Processing Systems, 36: 38154–38180, 2023

    Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face.Advances in Neural Information Processing Systems, 36: 38154–38180, 2023

    2023

  43. [43]

    Codedance: A dynamic tool-integrated mllm for executable visual reasoning.ArXiv, abs/2512.17312, 2025

    Qi Song, Honglin Li, Yingchen Yu, Haoyi Zhou, Lin Yang, Song Bai, Qi She, Zilong Huang, and Yunqing Zhao. Codedance: A dynamic tool-integrated mllm for executable visual reasoning.ArXiv, abs/2512.17312, 2025. URL https://api.semanticscholar.org/CorpusID:284058227

    arXiv 2025

  44. [44]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Pith/arXiv arXiv 2023

  45. [45]

    Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, SH Cai, Yuan Cao, Y Charles, HS Che, Cheng Chen, Guanduo Chen, et al. Kimi k2. 5: Visual agentic intelligence.arXiv preprint arXiv:2602.02276, 2026

    Pith/arXiv arXiv 2026

  46. [46]

    Qwen3 technical report, 2025

    Qwen Team. Qwen3 technical report, 2025. URLhttps://arxiv.org/abs/2505.09388

    Pith/arXiv arXiv 2025

  47. [47]

    Branchynet: Fast inference via early exiting from deep neural networks

    Surat Teerapittayanon, Bradley McDanel, and Hsiang-Tsung Kung. Branchynet: Fast inference via early exiting from deep neural networks. In2016 23rd international conference on pattern recognition (ICPR), pages 2464–2469. IEEE, 2016

    2016

  48. [48]

    Look-m: Look-once optimization in kv cache for efficient multimodal long-context inference

    Zhongwei Wan, Ziang Wu, Che Liu, Jinfa Huang, Zhihong Zhu, Peng Jin, Longyue Wang, and Li Yuan. Look-m: Look-once optimization in kv cache for efficient multimodal long-context inference. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 4065–4078, 2024

    2024

  49. [49]

    Meda: Dynamic kv cache allocation for efficient multimodal long-context inference

    Zhongwei Wan, Hui Shen, Xin Wang, Che Liu, Zheda Mai, and Mi Zhang. Meda: Dynamic kv cache allocation for efficient multimodal long-context inference. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2485–2497, 2025

    2025

  50. [50]

    Fourier-vlm: Compressing vision tokens in the frequency domain for large vision-language models, 2025

    Huanyu Wang, Jushi Kai, Haoli Bai, Lu Hou, Bo Jiang, Ziwei He, and Zhouhan Lin. Fourier-vlm: Compressing vision tokens in the frequency domain for large vision-language models, 2025. URLhttps://arxiv.org/abs/25 08.06038

    2025

  51. [51]

    Divide, conquer and combine: A training-free framework for high-resolution image perception in multimodal large language models

    Wenbin Wang, Liang Ding, Minyan Zeng, Xiabin Zhou, Li Shen, Yong Luo, Wei Yu, and Dacheng Tao. Divide, conquer and combine: A training-free framework for high-resolution image perception in multimodal large language models. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 7907–7915, 2025

    2025

  52. [52]

    Efficient visual transformer by learnable token merging.IEEE transactions on pattern analysis and machine intelligence, 2025

    Yancheng Wang and Yingzhen Yang. Efficient visual transformer by learnable token merging.IEEE transactions on pattern analysis and machine intelligence, 2025

    2025

  53. [53]

    V*: Guided visual search as a core mechanism in multimodal llms.arXiv preprint arXiv:2312.14135, 2023

    Penghao Wu and Saining Xie. V*: Guided visual search as a core mechanism in multimodal llms.arXiv preprint arXiv:2312.14135, 2023

    Pith/arXiv arXiv 2023

  54. [54]

    Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation

    Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, and Zhifang Sui. Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3909–3925, Singapore, December 2023. Association for Computat...

    doi:10.18653/v1/2023.findings-emnlp.257 2023

  55. [55]

    Swift: On-the-fly self-speculative decoding for llm inference acceleration.arXiv preprint arXiv:2410.06916, 2024

    Heming Xia, Yongqi Li, Jun Zhang, Cunxiao Du, and Wenjie Li. Swift: On-the-fly self-speculative decoding for llm inference acceleration.arXiv preprint arXiv:2410.06916, 2024. 13

    Pith/arXiv arXiv 2024

  56. [56]

    Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction.arXiv preprint arXiv:2410.17247, 2024

    Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, et al. Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction.arXiv preprint arXiv:2410.17247, 2024

    Pith/arXiv arXiv 2024

  57. [57]

    Specee: Accelerating large language model inference with speculative early exiting

    Jiaming Xu, Jiayi Pan, Yongkang Zhou, Siming Chen, Jinhao Li, Yaoxiu Lian, Junyi Wu, and Guohao Dai. Specee: Accelerating large language model inference with speculative early exiting. InProceedings of the 52nd Annual International Symposium on Computer Architecture, pages 467–481, 2025

    2025

  58. [58]

    Longspec: Long-context speculative decoding with efficient drafting and verification.arXiv e-prints, pages arXiv–2502, 2025

    Penghui Yang, Cunxiao Du, Fengzhuo Zhang, Haonan Wang, Tianyu Pang, Chao Du, and Bo An. Longspec: Long-context speculative decoding with efficient drafting and verification.arXiv e-prints, pages arXiv–2502, 2025

    2025

  59. [59]

    Visionzip: Longer is better but not necessary in vision language models

    Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19792–19802, 2025

    2025

  60. [60]

    Deep but reliable: Advancing multi-turn reasoning for thinking with images, 2026

    Wenhao Yang, Yu Xia, Jinlong Huang, Shiyin Lu, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, Yuanyu Wan, and Lijun Zhang. Deep but reliable: Advancing multi-turn reasoning for thinking with images, 2026. URL https://arxiv.org/abs/2512.17306

    arXiv 2026

  61. [61]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. InThe eleventh international conference on learning representations, 2022

    2022

  62. [62]

    Recode: Unify plan and action for universal granularity control.arXiv preprint arXiv:2510.23564, 2025

    Zhaoyang Yu, Jiayi Zhang, Huixue Su, Yufan Zhao, Yifan Wu, Mingyi Deng, Jinyu Xiang, Yizhang Lin, Lingxiao Tang, Yuyu Luo, et al. Recode: Unify plan and action for universal granularity control.arXiv preprint arXiv:2510.23564, 2025

    arXiv 2025

  63. [63]

    Draft& verify: Lossless large language model acceleration via self-speculative decoding

    Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. Draft& verify: Lossless large language model acceleration via self-speculative decoding. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11263–11282, 2024

    2024

  64. [64]

    Thyme: Think beyond images.arXiv preprint arXiv:2508.11630, 2025

    Yi-Fan Zhang, Xingyu Lu, Shukang Yin, Chaoyou Fu, Wei Chen, Xiao Hu, Bin Wen, Kaiyu Jiang, Changyi Liu, Tianke Zhang, et al. Thyme: Think beyond images.arXiv preprint arXiv:2508.11630, 2025

    Pith/arXiv arXiv 2025

  65. [65]

    Skywork-r1v4: Toward agentic multimodal intelligence through interleaved thinking with images and deepresearch.arXiv preprint arXiv:2512.02395, 2025

    Yifan Zhang, Liang Hu, Haofeng Sun, Peiyu Wang, Yichen Wei, Shukang Yin, Jiangbo Pei, Wei Shen, Peng Xia, Yi Peng, et al. Skywork-r1v4: Toward agentic multimodal intelligence through interleaved thinking with images and deepresearch.arXiv preprint arXiv:2512.02395, 2025

    arXiv 2025

  66. [66]

    Pyvision-rl: Forging open agentic vision models via rl.arXiv preprint arXiv:2602.20739, 2026

    Shitian Zhao, Shaoheng Lin, Ming Li, Haoquan Zhang, Wenshuo Peng, Kaipeng Zhang, and Chen Wei. Pyvision-rl: Forging open agentic vision models via rl.arXiv preprint arXiv:2602.20739, 2026

    arXiv 2026

  67. [67]

    A stitch in time saves nine: Small vlm is a precise guidance for accelerating large vlms

    Wangbo Zhao, Yizeng Han, Jiasheng Tang, Zhikai Li, Yibing Song, Kai Wang, Zhangyang Wang, and Yang You. A stitch in time saves nine: Small vlm is a precise guidance for accelerating large vlms. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19814–19824, 2025

    2025

  68. [68]

    thinking with images

    Ziwei Zheng, Michael Yang, Jack Hong, Chenxiao Zhao, Guohai Xu, Le Yang, Chao Shen, and Xing Yu. Deepeyes: Incentivizing" thinking with images" via reinforcement learning.arXiv preprint arXiv:2505.14362, 2025

    Pith/arXiv arXiv 2025

  69. [69]

    Hierarchical skip decoding for efficient autoregressive text generation.arXiv preprint arXiv:2403.14919, 2024

    Yunqi Zhu, Xuebing Yang, Yuanyuan Wu, and Wensheng Zhang. Hierarchical skip decoding for efficient autoregressive text generation.arXiv preprint arXiv:2403.14919, 2024. 14

    Pith/arXiv arXiv 2024