Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

ToSA: Token Merging with Spatial Awareness

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ToSA merges ViT tokens with depth-derived spatial priors and beats ToMe on visual and embodied question answering.

desk verdict ToSA is a clean, plausible extension of ToMe that uses depth-derived spatial tokens to guide token merging, with consistent accuracy gains on several VQA benchmarks, but its headline runtime claim ignores the cost of estimating depth on RGB inputs and the experimental evidence is thinner than the claims. read the letter →

arxiv 2506.20066 v1 pith:VULPCMSO submitted 2025-06-24 cs.CV

Hsiang-Wei Huang , Wenhao Chai , Kuang-Ming Chen , Cheng-Yen Yang , Jenq-Neng Hwang This is my paper
classification cs.CV
keywords tokenmergingvisiontransformeraccelerationspatialawarenessdepthestimationvisualquestionansweringbipartitesoftmatchingtraining-freereductionembodied
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes ToSA, a training-free token merging method for Vision Transformers that merges tokens by combining visual feature similarity with spatial proximity derived from a depth map. It argues that in early ViT layers, where visual features mostly encode edges and textures, depth-derived spatial similarity is a more reliable guide for which tokens belong together, so a layer-wise schedule that leans on spatial similarity early and semantic similarity late better preserves scene structure. Across SpatialBench, VQAv2 counting, GQA, and OpenEQA, ToSA reports higher accuracy than the previous ToMe method at the same retained token ratios, with less than 0.6 percent throughput loss relative to ToMe.

What carries the argument

The load-bearing object is the fused score matrix $S_{\text{fused}} = \alpha S_{\text{visual}} + (1-\alpha) S_{\text{spatial}}$ fed into bipartite soft matching, the merging algorithm inherited from ToMe. The spatial term $S_{\text{spatial}}$ comes from pseudo spatial tokens: each image patch's average relative depth is quantized into 27 levels and encoded with transformer positional encoding as an $(x,y,z)$ triplet, giving the merge process an explicit spatial prior. The $\alpha$ schedule, increasing with layer depth, is what lets the method trust spatial similarity in early layers and semantic similarity later, and the reported ablation shows that this increase schedule outperforms uniform or decreasing schedules.

What would settle it

Measure ToSA's end-to-end latency including the depth-estimation forward pass (for example Depth Anything V2 on the same GPU): if the total time at 50 percent token retention exceeds ToMe's by more than the reported 0.6 percent, the runtime claim fails. Alternatively, replace the depth map with constant or random values while keeping everything else fixed; if accuracy on VQAv2 counting does not drop below ToMe's, the spatial term is not doing the causal work.

Watch

Extended reading notes

Core claim

The central claim is that token merging in ViTs should be driven by a fused similarity score $S_{\text{fused}} = \alpha S_{\text{visual}} + (1-\alpha) S_{\text{spatial}}$, where $S_{\text{visual}}$ is cosine similarity between attention-key features and $S_{\text{spatial}}$ is similarity between spatial tokens generated from patch-wise depth. With $\alpha$ increasing linearly from 0 to 1 across the 27 ViT layers, early merging is guided by spatial affinity, avoiding the pitfall of merging tokens from different objects that happen to share low-level visual features. The paper demonstrates on visual and embodied question answering benchmarks that this fused criterion outperforms semantic-only merging (ToMe) at both 50% and 10% retained tokens, and shows qualitatively that it produces more spatially coherent merging results.

Load-bearing premise

ToSA assumes a good depth map is always available and that the depth-derived spatial similarity reliably indicates which tokens should be merged; if the depth is noisy, expensive to compute, or uninformative, both the accuracy gain and the speed advantage shrink.

Editorial extensions

If this is right

  • On the existence and counting subsets of SpatialBench, ToSA improves over ToMe by 10 and 14 accuracy points respectively while retaining only 10 percent of tokens.
  • On VQAv2 counting at 50 percent token retention, ToSA reaches 64.1 percent relative counting accuracy versus ToMe's 53.1 percent, recovering a large share of the gap to the 77.1 percent full model.
  • On GQA and OpenEQA, ToSA outperforms ToMe at identical token budgets without any retraining, confirming the gain transfers beyond spatial-heavy benchmarks.
  • ToSA's inference speed is nearly identical to ToMe's (under 0.6 percent degradation), so the spatial prior does not cancel the acceleration the merging provides.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the authors leave implicit is that the same fused-score mechanism could use other auxiliary priors—optical flow for video, segmentation maps, or LiDAR range images—wherever a cheap geometric signal is available.
  • Because the depth model is excluded from the reported runtime, a natural test is end-to-end wall-clock timing that includes depth prediction; if that total exceeds ToMe's, the speed advantage shrinks despite the accuracy gain.
  • If the gains are driven mainly by early-layer spatial merging, a cheaper variant that computes spatial tokens only for the first third of layers might recapture most of the benefit at even lower overhead.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes ToSA, a training-free token merging method for Vision Transformers that augments visual feature similarity with spatial-awareness computed from depth-derived pseudo spatial tokens. A fused score matrix S_fused = αS_visual + (1−α)S_spatial is used in bipartite soft matching, with α increasing linearly across layers. Experiments on SpatialBench, VQAv2-Counting, GQA, and OpenEQA compare ToSA with ToMe, reporting accuracy improvements and a throughput table. The paper claims ToSA outperforms previous token merging methods while largely reducing ViT runtime.

Significance. If validated, the idea of using depth-based spatial tokens as an auxiliary merging criterion is a sensible extension of ToMe, particularly for tasks where spatial structure matters (counting, existence). The method is training-free and simple, and the qualitative results show more coherent merging patterns. However, the current empirical support is limited: only ToMe is compared, the efficiency claim omits the cost of depth estimation, and the α schedule is selected on the same benchmark used for reporting. These gaps make the central claims difficult to fully evaluate, but they are addressable through additional experiments and revised claims.

major comments (4)
  1. [Section IV-A, Table VI] The runtime comparison reports only the ViT encoder throughput after a depth map is already available. For SpatialBench, VQAv2-Counting, and GQA, depth is predicted by depth-anything-v2, yet that model's inference time is excluded from Table VI. Because the abstract and conclusion claim ToSA 'largely reduces the runtime of the ViT' with 'minimal additional inference cost,' the paper must either measure end-to-end wall-clock time including depth estimation or explicitly restrict the efficiency claim to settings where RGB-D depth is already provided. This is load-bearing because efficiency is a central contribution.
  2. [Section IV-D, Tables V and II] The α schedule (Eq. 3) is selected by an ablation on VQAv2-Counting that reports the highest accuracy for the 'increase' schedule, and the same benchmark's test accuracy is then reported as the method's result in Table II. This means the reported improvement over ToMe partly reflects test-set selection rather than the mechanism itself. Please use a separate validation split for schedule selection, or report results across schedules without selecting on the test set.
  3. [Section IV-C] The paper only compares ToSA with ToMe, yet the abstract claims ToSA 'outperforms previous token merging methods.' Other training-free token reduction methods (e.g., EViT, Evo-ViT, or more recent token merging variants) are not evaluated. At minimum, the claim should be scoped to 'outperforms ToMe,' or additional baselines should be added to support the broader statement.
  4. [Tables I-IV] No error bars or significance tests are reported. Several differences are small (e.g., GQA at 10% retention: 57.8 vs. 57.4; SpatialBench Size: ToSA is 3.3 points lower than ToMe). Without variance estimates, it is unclear which of the reported gains are robust. Please add repeated-run variability or significance testing, and temper statements such as 'largely outperforms' when one category (Size) is worse.
minor comments (5)
  1. [Section III-B] The sentence 'the x and y in the image ranging from the number of patches in x and y dimension' is unclear; please rephrase to specify the exact ranges of x and y and the convention used.
  2. [Table I caption] The caption uses 'Spatial Bench' while the text and other tables use 'SpatialBench'; please make this consistent.
  3. [Table VI] The unit 'im/s' should be defined as 'images per second' in the caption or in the text.
  4. [Tables II and IV] The model name 'LLaV A-OV-7B' appears with inconsistent spacing; use 'LLaVA-OneVision-7B' consistently.
  5. [Section IV-D] The text states 'We set α to 0.5 for our experiment' for the uniform schedule, but it does not explicitly state which schedule corresponds to the default setting in Eq. 3; please clarify the relationship between the default, uniform, and increase schedules.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ToSA is an empirical, training-free token-merging heuristic; its claimed gains are benchmark measurements, not outputs of a derivation that reduces to its inputs.

full rationale

ToSA does not derive a result from first principles; it proposes a merging rule (Eq. 2) and evaluates it empirically. The fused score S_fused = alpha S_visual + (1-alpha) S_spatial is defined from two input similarities, not from the benchmark outcomes, so there is no self-definitional loop. The alpha schedule (Eq. 3) is a hyperparameter; the Table V ablation selects the increase schedule on VQAv2-Counting and Table II reports VQAv2-Counting accuracy with that schedule. This is a benchmark-selection (test-set tuning) concern that can inflate that particular comparison, but it is not a circular reduction: the reported accuracy is a measured value, not a fitted parameter renamed as a prediction, and the main comparisons on SpatialBench, GQA, and OpenEQA are not selected in this way. The depth dependence is acknowledged in Sec. VI; the omission of depth-anything-v2 inference time in Table VI is an efficiency-claim gap, not circularity. Self-citations ([10], [12], [35]) appear only as related-work examples and are not load-bearing for ToSA's core mechanism. No uniqueness theorem or imported ansatz is used to force the design. The claimed spatial coherence of merged tokens follows from the algorithm's use of spatial tokens, but the accuracy gains are external empirical results, so no step in the paper's argument reduces to its own inputs by construction.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on two hand-chosen parameters (alpha schedule, depth quantization) and on three domain assumptions (predicted depth fidelity, the sufficiency of the spatial token representation, and the validity of the OpenEQA sampling). No new physical entities are introduced.

free parameters (2)
  • alpha schedule (alpha_i = i/L) = linear increase, alpha_i = i/27 for 27 layers
    Selected after an ablation on VQAv2-counting (Table V) comparing uniform 0.5, decrease, and increase schedules; the winning schedule is then used for the headline results on the same dataset.
  • depth quantization levels (z) = 27 levels
    Chosen by hand to match the x and y patch index range (0 to 26) in Section III-B; the paper does not ablate this granularity, yet it determines the resolution of the spatial score matrix.
assumptions (4)
  • domain assumption Depth-Anything-V2 provides accurate predicted depth for all evaluated RGB images
    The paper predicts depth with depth-anything-v2 for image-based benchmarks (Section IV-A) and treats this depth as ground truth for building spatial tokens; noisy or inaccurate depth would weaken the Sspatial criterion.
  • ad hoc to paper Patch-wise averaged relative depth, quantized to 27 levels, preserves spatial structure needed for token merging
    The z-index construction in Section III-B averages all pixels' relative depth per patch and divides into 27 levels; no ablation or validation is provided for this specific representation.
  • standard math The bipartite soft matching procedure from ToMe transfers unchanged to the fused score matrix
    ToSA inherits BSM from ToMe (Section III-A) and replaces the score matrix, assuming the matching and proportional attention still behave correctly.
  • domain assumption The OpenEQA evaluation with 12 uniformly sampled frames per 3D scan is a valid proxy for embodied QA
    Section IV-A introduces this sampling without comparing to the official benchmark protocol, so comparability with reported baselines rests on an unstated assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ToSA: Token Merging with Spatial Awareness." pith.science (2026). https://pith.science/paper/VULPCMSO

@misc{pith2026250620066,
  author       = {Pith},
  title        = {Pith review of: ToSA: Token Merging with Spatial Awareness},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VULPCMSO}},
  note         = {Machine review of arXiv:2506.20066}
}
read the original abstract

Token merging has emerged as an effective strategy to accelerate Vision Transformers (ViT) by reducing computational costs. However, existing methods primarily rely on the visual token's feature similarity for token merging, overlooking the potential of integrating spatial information, which can serve as a reliable criterion for token merging in the early layers of ViT, where the visual tokens only possess weak visual information. In this paper, we propose ToSA, a novel token merging method that combines both semantic and spatial awareness to guide the token merging process. ToSA leverages the depth image as input to generate pseudo spatial tokens, which serve as auxiliary spatial information for the visual token merging process. With the introduced spatial awareness, ToSA achieves a more informed merging strategy that better preserves critical scene structure. Experimental results demonstrate that ToSA outperforms previous token merging methods across multiple benchmarks on visual and embodied question answering while largely reducing the runtime of the ViT, making it an efficient solution for ViT acceleration. The code will be available at: https://github.com/hsiangwei0903/ToSA

Figures

Figures reproduced from arXiv: 2506.20066 by the authors.

Figure 1
Figure 1. A merging comparison between ToMe and ToSA. By leveraging [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The overall framework of ToSA. ToSA block is inserted between attention and MLP across each encoder layer in ViT. ToSA block takes visual [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Merging results comparison between ToMe and ToSA. We only keep 16 visual tokens (2%) from both methods for better visual comparison. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. WaveZip: Wavelet-Driven Space-Time Decoupling for Video Token Condensation

    cs.CV 2026-07 conditional novelty 6.0 of 10

    WaveZip, a training-free wavelet method for video token condensation, reports about 10x token reduction while retaining roughly 98-99% accuracy on multi-benchmark LVLM evaluation.

  2. Warehouse Spatial Question Answering with LLM Agent

    cs.CV 2025-07 conditional novelty 4.0 of 10

    An LLM agent equipped with lightweight distance and inclusion perception models achieved 95.86% accuracy on the 2025 AI City Challenge warehouse spatial QA benchmark, ranking first.

Reference graph

Works this paper leans on

40 extracted references · 23 canonical work pages · cited by 2 Pith papers

  1. [1]

    Dinov2: Learning robust visual features without supervision,

    M. Oquab, T. Darcet, T. Moutakanni, H. V . V o, M. Szafraniec, V . Khalidov, P. Fernandez, D. HAZIZA, F. Massa, A. El-Nouby et al., “Dinov2: Learning robust visual features without supervision,” Transactions on Machine Learning Research , 2024

    work page 2024

  2. [2]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning . PMLR, 2021, pp. 8748–8763

    2021

  3. [3]

    Sigmoid loss for language image pre-training,

    X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer, “Sigmoid loss for language image pre-training,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 11 975– 11 986

    work page 2023

  4. [4]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929 , 2020

    arXiv 2010

  5. [5]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” Advances in neural information processing systems , vol. 36, 2024

    2024

  6. [6]

    Llava-onevision: Easy visual task transfer,

    B. Li, Y . Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang, P. Zhang, Y . Li, Z. Liu et al. , “Llava-onevision: Easy visual task transfer,” arXiv preprint arXiv:2408.03326 , 2024

    arXiv 2024

  7. [7]

    Efficientvit: Memory efficient vision transformer with cascaded group attention,

    X. Liu, H. Peng, N. Zheng, Y . Yang, H. Hu, and Y . Yuan, “Efficientvit: Memory efficient vision transformer with cascaded group attention,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 14 420–14 430

    work page 2023

  8. [8]

    Dynamicvit: Efficient vision transformers with dynamic token sparsification,

    Y . Rao, W. Zhao, B. Liu, J. Lu, J. Zhou, and C.-J. Hsieh, “Dynamicvit: Efficient vision transformers with dynamic token sparsification,” Ad- vances in neural information processing systems , vol. 34, pp. 13 937– 13 949, 2021

    work page 2021

Show all 40 references
  1. [9]

    A-vit: Adaptive tokens for efficient vision transformer,

    H. Yin, A. Vahdat, J. M. Alvarez, A. Mallya, J. Kautz, and P. Molchanov, “A-vit: Adaptive tokens for efficient vision transformer,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022, pp. 10 809–10 818

    2022

  2. [10]

    Tempura: Temporal event masked prediction and understanding for reasoning in action,

    J.-H. Cheng, V . Wang, H. Wang, H. Zhou, Y .-H. Peng, H.-I. Liu, H.-W. Huang, K.-M. Chen, C.-Y . Yang, W. Chai et al., “Tempura: Temporal event masked prediction and understanding for reasoning in action,” arXiv preprint arXiv:2505.01583 , 2025

    2025 arXiv

  3. [11]

    Token pooling in vision transformers for image classification,

    D. Marin, J.-H. R. Chang, A. Ranjan, A. Prabhu, M. Rastegari, and O. Tuzel, “Token pooling in vision transformers for image classification,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , 2023, pp. 12–21

    2023

  4. [12]

    Zero-shot 3d question answering via voxel-based dynamic token compression,

    H.-W. Huang, F.-C. Chen, W. Chai, C.-C. Su, L. Xia, S. Jung, C.-Y . Yang, J.-N. Hwang, M. Sun, and C.-H. Kuo, “Zero-shot 3d question answering via voxel-based dynamic token compression,” in Proceed- ings of the Computer Vision and Pattern Recognition Conference (CVPR), June 20...

    2025

  5. [13]

    Token merging: Your ViT but faster,

    D. Bolya, C.-Y . Fu, X. Dai, P. Zhang, C. Feichtenhofer, and J. Hoff- man, “Token merging: Your ViT but faster,” in International Confer- ence on Learning Representations , 2023

    2023

  6. [14]

    What do vision transformers learn? a visual exploration,

    A. Ghiasi, H. Kazemi, E. Borgnia, S. Reich, M. Shu, M. Goldblum, A. G. Wilson, and T. Goldstein, “What do vision transformers learn? a visual exploration,” arXiv preprint arXiv:2212.06727 , 2022

    2022 arXiv

  7. [15]

    Spatialbot: Precise spatial understanding with vision lan- guage models,

    W. Cai, Y . Ponomarenko, J. Yuan, X. Li, W. Yang, H. Dong, and B. Zhao, “Spatialbot: Precise spatial understanding with vision lan- guage models,” arXiv preprint arXiv:2406.13642 , 2024

    2024 arXiv

  8. [16]

    Vqa: Visual question answering,

    S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh, “Vqa: Visual question answering,” in Proceedings of the IEEE international conference on computer vision , 2015, pp. 2425– 2433

    2015

  9. [17]

    Gqa: A new dataset for real- world visual reasoning and compositional question answering,

    D. A. Hudson and C. D. Manning, “Gqa: A new dataset for real- world visual reasoning and compositional question answering,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 6700–6709

    2019

  10. [18]

    Openeqa: Embodied question answering in the era of foundation models,

    A. Majumdar, A. Ajay, X. Zhang, P. Putta, S. Yenamandra, M. Henaff, S. Silwal, P. Mcvay, O. Maksymets, S. Arnaud et al. , “Openeqa: Embodied question answering in the era of foundation models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognitio...

    2024

  11. [19]

    Sp-vit: Learning 2d spatial priors for vision transformers,

    Y . Zhou, W. Xiang, C. Li, B. Wang, X. Wei, L. Zhang, M. Keuper, and X. Hua, “Sp-vit: Learning 2d spatial priors for vision transformers,” in The 33rd British Machine Vision Conference , 2022

    2022

  12. [20]

    Evo-vit: Slow-fast token evolution for dynamic vision transformer,

    Y . Xu, Z. Zhang, M. Zhang, K. Sheng, K. Li, W. Dong, L. Zhang, C. Xu, and X. Sun, “Evo-vit: Slow-fast token evolution for dynamic vision transformer,” in Proceedings of the AAAI Conference on Arti- ficial Intelligence, vol. 36, no. 3, 2022, pp. 2964–2972

    2022

  13. [21]

    Not all patches are what you need: Expediting vision transformers via token reorganizations,

    Y . Liang, C. Ge, Z. Tong, Y . Song, J. Wang, and P. Xie, “Not all patches are what you need: Expediting vision transformers via token reorganizations,” in International Conference on Learning Represen- tations, 2022

    2022

  14. [22]

    Ppt: Token prun- ing and pooling for efficient vision transformers,

    X. Wu, F. Zeng, X. Wang, and X. Chen, “Ppt: Token prun- ing and pooling for efficient vision transformers,” arXiv preprint arXiv:2310.01812, 2023

    2023 arXiv

  15. [23]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference ac- celeration for large vision-language models,

    L. Chen, H. Zhao, T. Liu, S. Bai, J. Lin, C. Zhou, and B. Chang, “An image is worth 1/2 tokens after layer 2: Plug-and-play inference ac- celeration for large vision-language models,” in European Conference on Computer Vision . Springer, 2025, pp. 19–35

    2025

  16. [24]

    Sparsevlm: Visual token sparsification for efficient vision-language model inference,

    Y . Zhang, C.-K. Fan, J. Ma, W. Zheng, T. Huang, K. Cheng, D. Gu- dovskiy, T. Okuno, Y . Nakata, K. Keutzer et al. , “Sparsevlm: Visual token sparsification for efficient vision-language model inference,” in International Conference on Machine Learning , 2025

    2025

  17. [25]

    Spatialvlm: Endowing vision-language models with spatial reasoning capabilities,

    B. Chen, Z. Xu, S. Kirmani, B. Ichter, D. Sadigh, L. Guibas, and F. Xia, “Spatialvlm: Endowing vision-language models with spatial reasoning capabilities,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 14 455– 14 465

    2024

  18. [26]

    Spatialrgpt: Grounded spatial reasoning in vision-language models,

    A.-C. Cheng, H. Yin, Y . Fu, Q. Guo, R. Yang, J. Kautz, X. Wang, and S. Liu, “Spatialrgpt: Grounded spatial reasoning in vision-language models,” in NeurIPS, 2024

    2024

  19. [27]

    Attention is all you need,

    A. Vaswani, “Attention is all you need,” Advances in Neural Informa- tion Processing Systems , 2017

    2017

  20. [28]

    Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,” in International conference on machine learning . PMLR, 2023, pp. 19 730–19 742

    2023

  21. [29]

    Instructblip: Towards general-purpose vision- language models with instruction tuning,

    W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. Hoi, “Instructblip: Towards general-purpose vision- language models with instruction tuning,” 2023

    2023

  22. [30]

    Qwen-vl: A frontier large vision-language model with versatile abilities,

    J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou, “Qwen-vl: A frontier large vision-language model with versatile abilities,” arXiv preprint arXiv:2308.12966 , 2023

    2023 arXiv

  23. [31]

    Improved baselines with visual instruction tuning,

    H. Liu, C. Li, Y . Li, and Y . J. Lee, “Improved baselines with visual instruction tuning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 26 296–26 306

    2024

  24. [32]

    Llama-vid: An image is worth 2 tokens in large language models,

    Y . Li, C. Wang, and J. Jia, “Llama-vid: An image is worth 2 tokens in large language models,” in European Conference on Computer Vision. Springer, 2024, pp. 323–340

    2024

  25. [33]

    Vila: On pre-training for visual language models,

    J. Lin, H. Yin, W. Ping, P. Molchanov, M. Shoeybi, and S. Han, “Vila: On pre-training for visual language models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 26 689–26 699

    2024

  26. [34]

    Depth anything v2,

    L. Yang, B. Kang, Z. Huang, Z. Zhao, X. Xu, J. Feng, and H. Zhao, “Depth anything v2,” Advances in Neural Information Processing Systems, vol. 37, pp. 21 875–21 911, 2024

    2024

  27. [35]

    Auroracap: Efficient, performant video detailed captioning and a new benchmark,

    W. Chai, E. Song, Y . Du, C. Meng, V . Madhavan, O. Bar-Tal, J.-N. Hwang, S. Xie, and C. D. Manning, “Auroracap: Efficient, performant video detailed captioning and a new benchmark,” arXiv preprint arXiv:2410.03051, 2024

    2024 arXiv

  28. [36]

    Longvlm: Efficient long video understanding via large language models,

    Y . Weng, M. Han, H. He, X. Chang, and B. Zhuang, “Longvlm: Efficient long video understanding via large language models,” in European Conference on Computer Vision. Springer, 2025, pp. 453– 470

    2025

  29. [37]

    Video-chatgpt: Towards detailed video understanding via large vision and language models,

    M. Maaz, H. Rasheed, S. Khan, and F. S. Khan, “Video-chatgpt: Towards detailed video understanding via large vision and language models,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024) , 2024

    2024

  30. [38]

    Video-llama: An instruction-tuned audio-visual language model for video understanding,

    H. Zhang, X. Li, and L. Bing, “Video-llama: An instruction-tuned audio-visual language model for video understanding,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , 2023, pp. 543–553

    2023

  31. [39]

    Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms,

    Z. Cheng, S. Leng, H. Zhang, Y . Xin, X. Li, G. Chen, Y . Zhu, W. Zhang, Z. Luo, D. Zhao et al., “Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms,” arXiv preprint arXiv:2406.07476, 2024

    2024 arXiv

  32. [40]

    Chat-univi: Unified visual representation empowers large language models with image and video understanding,

    P. Jin, R. Takanobu, W. Zhang, X. Cao, and L. Yuan, “Chat-univi: Unified visual representation empowers large language models with image and video understanding,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 13 700–13 710

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.