Pith. sign in

REVIEW 5 major objections 7 minor 3 cited by

What Kind of Visual Tokens Do We Need? Training-free Visual Token Pruning for Multi-modal Large Language Models from the Perspective of Graph

T0 review · 5 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Visual tokens can be pruned by graph-based information propagation, keeping representative tokens from foreground and background, so that LLaVA-NeXT runs at about 63.5% fewer FLOPs with accuracy drops of 0.95% and 2.34% on VQA2.0 and…

desk verdict G-Prune is a plausible training-free token pruning heuristic with solid benchmark results, but its 'one representative per object' explanation is mathematically shaky and the real mechanism is an unexplained early-stopped diffusion. read the letter →

arxiv 2501.02268 v1 pith:NLE4AMDZ submitted 2025-01-04 cs.CV cs.AI

Yutao Jiang , Qiong Wu , Wenhao Lin , Wei Yu , Yiyi Zhou This is my paper · ORCID
classification cs.CVcs.AI
keywords visualtokenpruningmultimodallargelanguagemodelstraining-freeaccelerationgraphinformationpropagationforegroundandbackgroundtokensLLaVA-NeXTredundancyvision-languagebenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the large stacks of visual tokens fed to multimodal large language models (MLLMs) are highly redundant, and that a training-free pruning method can remove most of them without destroying performance. Its central finding is that both foreground and background tokens matter, so pruning must keep representative tokens from every semantic region rather than only the salient objects. On this basis the paper proposes G-Prune, which builds a similarity graph over tokens, propagates importance scores through the graph, and keeps the top-scoring normalized tokens. Applied to LLaVA-NeXT, G-Prune reports about 63.5% fewer FLOPs at 70% pruning with a 0.95% accuracy drop on VQA2.0 and a 2.34% drop on TextVQA.

What carries the argument

The central object is a token-similarity graph with an information-propagation score. Given $N$ visual tokens, edges are set by cosine similarity above a threshold $s$; after softmax row-normalization, scores begin as each token's $\ell^2$-norm and evolve by repeated multiplication with the adjacency, so information flows from high-norm tokens to their semantic neighbors. Degree normalization divides each final score by the number of connected neighbors, making scores comparable across objects and regions of different sizes, and the top-$k$ tokens by normalized score are retained. This machinery does the work of turning local token statistics into a global representativeness ranking that is not biased toward the foreground.

What would settle it

Replace the $\ell^2$-norm initialization in Eq. 2 with uniform scores, keeping the same graph and iterations; if TextVQA accuracy at 90% pruning stays within about one point of G-Prune's 59.31, the norm prior is not doing the work. Similarly, a variant that clusters tokens and keeps one per cluster at controlled cluster sizes would test whether degree normalization is really compensating for component-size bias.

Watch

Extended reading notes

Core claim

The paper's central claim is that a simple graph-diffusion score can identify the visual tokens an MLLM actually needs, and that these tokens are spread across both foreground and background. G-Prune treats each visual token as a graph node, connects nodes whose cosine similarity exceeds a threshold, and iterates the update $S_t = S_0 (A')^t$, where $S_0$ is each token's $\ell^2$-norm and $A'$ is a row-normalized adjacency matrix; dividing the final score by node degree removes the bias of larger connected components. Keeping the top-$k$ normalized scores retains one or a few representative tokens per object or background region. The experiments show that random pruning collapses text-oriented VQA accuracy at 90% pruning, while G-Prune keeps TextVQA at 59.31 against ToMe's 39.02, supporting the claim that graph-selected tokens preserve fine-grained information.

Load-bearing premise

The load-bearing premise is that a token's $\ell^2$-norm measures how much information it carries, and that dividing scores by node degree makes them comparable across regions of different sizes; if either assumption fails, the top-k selection is arbitrary.

Editorial extensions

If this is right

  • At 70% pruning, LLaVA-NeXT runs at about 6.76 TFLOPs instead of 18.52 TFLOPs, a 63.57% reduction, with VQA2.0 dropping from 82.70 to 81.91 and TextVQA from 65.41 to 63.87.
  • At 90% pruning, G-Prune keeps text-oriented benchmarks usable: TextVQA at 59.31 and DocVQA at 48.94, compared with 38.36 for ToMe and 53.53 for FastV on TextVQA.
  • Because the method is training-free and does not modify the MLLM's weights or attention, it can be dropped into an existing model at inference time.
  • The graph score selects tokens from background and textured regions as well as foreground objects, which the paper argues is necessary for fine-grained tasks such as reading text.
  • The method keeps performance close to baseline across general VQA, MLLM-specific benchmarks, and text-oriented VQA at moderate pruning ratios.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The propagation step resembles a random-walk centrality measure on the similarity graph; if that reading is right, the method could be replaced by a one-step spectral or PageRank-like score with similar behavior and less iteration.
  • Since the ablation shows graph structure alone accounts for most of the gain, a testable extension is to replace the l2-norm prior with other saliency cues, such as CLS-attention or text-prompt similarity, and check whether the gains stack.
  • The degree-normalized top-k selection effectively picks one representative per clustered region; a natural extension is to make k adaptive per image or per region based on entropy or task type.
  • The method is demonstrated on a single MLLM; if the graph-score mechanism is truly model-agnostic, it should transfer to other high-resolution MLLMs, which is a direct test a reader could run.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper proposes G-Prune, a training-free visual token pruning method for multimodal large language models (MLLMs). G-Prune treats visual tokens as nodes in a graph, connects nodes by thresholded cosine similarity, propagates importance scores over the graph via a row-stochastic transition matrix, and retains the top-k tokens by a degree-normalized score. The method is evaluated on LLaVA-NeXT-8B across eight benchmarks, reporting about 63.5% FLOPs reduction with small accuracy drops on general VQA and text-oriented VQA, and favorable comparisons against Random pruning, ToMe, and FastV. Ablations show that the graph-based propagation is the main contributor, with the l2-norm initialization providing only a small additional gain. The paper releases code at a public repository.

Significance. If the empirical results hold, G-Prune is a practical, training-free acceleration tool for MLLMs: it is plug-and-play, does not require intervention in the LLM reasoning process, and retains strong accuracy even at high pruning ratios, especially on text-oriented benchmarks where existing baselines degrade sharply. The release of code and the breadth of benchmarks (general VQA, MLLM benchmarks, and text-oriented VQA) are strengths, and the reported FLOPs savings are consistent across tables. However, the conceptual contribution is currently undermined by an unverified mechanism: the paper claims that the graph propagation and degree normalization identify 'the most representative token of each object,' yet this claim is not supported by the mathematics of the proposed update, and the algorithm as stated has several inconsistencies. The paper is best viewed as an empirical heuristic paper whose central narrative needs substantial correction.

major comments (5)
  1. [Method, Eq. (5)] The claim that degree normalization in Eq. (5) reveals 'the most representative token of each object' is not supported by the mathematics of the proposed propagation. For the row-softmax matrix A' defined after Eq. (1), the stationary distribution pi of the Markov chain in Eq. (3) satisfies pi_i proportional to the row sum R_i = sum_j exp(A_ij) (by detailed balance, since A is symmetric). In an object that is a clique with near-uniform edge weight w, R_i is identical for all nodes in the object, so S'(t)_i = S(t)_i / D_i is constant within that component and top-k selection inside the object is arbitrary. In an object that is not a clique, R_i = N - 1 + (e^w - 1) D_i, so S'(t)_i is approximately (N-1)/D_i + (e^w - 1), which assigns the highest score to the lowest-degree (most peripheral) token, not to a representative or central token. The paper does not explain why stopping at t=5 avoids this behavior; Table 3 shows that t=1 already gives essentially the same accuracy (71.94 vs. 71.98 at t=5), so the result is not tied to convergence to a representative stationary distribution. The authors should either provide a correct characterization of what Eq. (5) selects or explicitly present the method as an empirically motivated heuristic without the representativeness claim.
  2. [Algorithm 1, Eqs. (4)-(5)] The degree D_i defined in Eq. (4) can be zero for any token whose cosine similarities to all other tokens are below the threshold s. In that case, Eq. (5) divides by zero and the normalized score S'(t)_i is undefined. The paper does not specify how isolated nodes are handled (e.g., setting D_i to a small constant, adding self-loops, or excluding such tokens). With the default s=0.5 and 2880 high-dimensional visual tokens, isolated nodes are a realistic possibility, yet the reported results are clean, suggesting the implementation silently applies a rule that is not described. This is a correctness gap in the algorithm as stated and should be addressed explicitly.
  3. [Method, Eq. (6) and following text] The algorithm selects the global top-k tokens by S'(t)_i via Eq. (6), but the text immediately after states that 'only the most representative token is retained to represent each object.' This description is only accurate if k equals the number of connected components. In general, if k is smaller than the number of objects, some objects receive no token; if k is larger, multiple tokens from a single object are retained. The visualizations in Fig. 6 also show multiple tokens kept in each textured region at various pruning ratios. The paper should either change the selection rule to enforce one token per connected component (which would require knowing the component decomposition) or correct the narrative to describe the actual global top-k behavior.
  4. [Experiments, Table 3 and Implementation Details] The hyperparameters s=0.5 and t=5 appear to be chosen as the best configuration from the grid in Table 3, evaluated on the same benchmarks (GQA, POPE, TextVQA) used later for the final reported results. This is a form of selection on the test benchmarks; the paper should state whether a separate validation split was used or report the full grid of results so the reader can assess the sensitivity. In addition, all benchmark numbers are single-run without error bars or a statement of determinism. For claims of small accuracy differences, such as the 0.95% drop on VQA2.0 at 70% pruning, the absence of variance information makes it difficult to judge significance.
  5. [Introduction, Fig. 1(b)] The analysis leading to the conclusion that 'both foreground and background tokens are critical' and that l2-Norm distributions of foreground and background overlap is based on segmenting the image into foreground and background areas, but the segmentation method is not specified anywhere in the paper. Without a description of how the segmentation was obtained (e.g., object detector, saliency map, manual annotation), the observation in Fig. 1(b) is not reproducible, and it is the paper's first stated contribution. The authors should specify the segmentation procedure or weaken the claim.
minor comments (7)
  1. [Datasets and Metrics] The text reads 'we evaluate OncePrue for text-oriented VQA benchmarks'; 'OncePrue' should be 'G-Prune'.
  2. [Abstract] The abstract uses 'front' instead of 'foreground' in the sentence about retaining tokens from front or background.
  3. [Experiments, comparison with ToMe] The paper refers to 'ChartVQA' but the benchmark is consistently named ChartQA elsewhere; please unify.
  4. [Experiments, Table 3 discussion] The sentence 'the difference between the highest and the lowest is only 0.32%' does not match Table 3, where the average ranges from 71.75 (t=50) to 71.98 (t=5), a difference of 0.23 points. Please correct the number.
  5. [Experiments, comparison with ToMe] The reported drop of '40.35%' for ToMe on TextVQA at 90% pruning differs from Table 1 (65.41 to 38.36, which is about 41.4% relative drop); please reconcile the percentage.
  6. [Throughout] The model name is typeset inconsistently as 'LLaVA-NeXT', 'LLaV A-NeXT', and 'LLaV A-NeXT'; a single consistent form should be used.
  7. [Algorithm 1] Eq. (6) uses lowercase n for the number of tokens while the rest of the paper uses uppercase N; please unify the notation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: G-Prune is an empirical token-pruning heuristic whose graph-score equations are not defined by the benchmark accuracies it reports.

full rationale

The paper's claimed derivation is a selection rule, not a first-principles prediction. Equations (1)-(6) define a token score: cosine-similarity graph construction, row-softmax, iterative propagation of an l2-norm-initialized score, degree normalization, and top-k selection. None of these quantities is defined in terms of the downstream VQA accuracies, so the reported accuracy drops in Table 1 are external measurements rather than consequences of the equations by construction. The 'most representative token' language is an operational description of the argmax of Eq. 5, not a theorem imported from the authors' prior work. The reference list contains no load-bearing self-citation, and no uniqueness theorem is invoked. The hyperparameters s and t are ablated in Table 3 on the same evaluation benchmarks; this is a mild model-selection concern, but it does not make the reported numbers equivalent to the method's inputs, and it does not fit a parameter to one subset and then predict a closely related quantity. The skeptic's concern that degree normalization may erase score differences after convergence is a question about the mechanism's mathematical soundness, not about circularity. Therefore the circularity score is 0.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The method introduces no new entities. Its load-bearing assumptions are the semantic validity of the similarity graph, the informativeness of l2-norm, the comparability of degree-normalized scores, and the unstated segmentation tool in Figure 1. The two tuned hyperparameters are the only free parameters.

free parameters (2)
  • similarity threshold s = 0.5
    Chosen from Table 3 sweep (0.3 to 0.9) on the same benchmarks used for final results.
  • iteration count t = 5
    Chosen from Table 3 sweep (1 to 50) on the same benchmarks used for final results.
assumptions (5)
  • domain assumption Cosine similarity with a threshold groups visual tokens into semantically coherent objects.
    Method section, Eq. 1; no independent verification that connected components correspond to objects.
  • domain assumption The l2-norm of a visual token is a valid proxy for its informativeness.
    Method section, Eq. 2; the ablation in Table 2 gives empirical support but no theoretical justification.
  • domain assumption Degree normalization makes token scores comparable across connected components of different sizes.
    Method section, Eq. 5; the paper argues this but does not test it directly.
  • domain assumption Accuracy on the eight chosen benchmarks is a valid measure of MLLM visual reasoning quality.
    Experiments section; standard practice in the field but still an assumption about evaluation validity.
  • domain assumption The foreground/background decomposition used in Figure 1 is accurate.
    Figure 1 and surrounding text; the segmentation tool is not specified, so the decomposition is not independently checkable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of What Kind of Visual Tokens Do We Need? Training-free Visual Token Pruning for Multi-modal Large Language Models from the Perspective of Graph." pith.science (2026). https://pith.science/paper/NLE4AMDZ

@misc{pith2026250102268,
  author       = {Pith},
  title        = {Pith review of: What Kind of Visual Tokens Do We Need? Training-free Visual Token Pruning for Multi-modal Large Language Models from the Perspective of Graph},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NLE4AMDZ}},
  note         = {Machine review of arXiv:2501.02268}
}
read the original abstract

Recent Multimodal Large Language Models(MLLMs) often use a large number of visual tokens to compensate their visual shortcoming, leading to excessive computation and obvious visual redundancy. In this paper, we investigate what kind of visual tokens are needed for MLLMs, and reveal that both foreground and background tokens are critical for MLLMs given the varying difficulties of examples. Based on this observation, we propose a graph-based method towards training-free visual token pruning, termed G-Prune.In particular, G-Prune regards visual tokens as nodes, and construct their connections based on their semantic similarities. Afterwards, the information flow is propagated via weighted links, and the most important tokens after iterations are kept for MLLMs, which can be front or background.To validate G-Prune, we apply it to a recent MLLM called LLaVA-NeXT, and conduct extensive experiments on a set of benchmarks.The experiment results show that G-Prune can greatly reduce computation overhead while retaining high performance on both coarse- and fine-grained tasks. For instance, G-Prune can reduce 63.57\% FLOPs of LLaVA-NeXT on VQA2.0 and TextVQA with only 0.95\% and 2.34\% accuracy drops, respectively.

Figures

Figures reproduced from arXiv: 2501.02268 by the authors.

Figure 1
Figure 1. (a) Random pruning, Foreground-preserving prun [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The overview of G-Prune. G-Prune aims to find out the important visual tokens for MLLMs, thereby reducing the [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Comparison between our G-Prune and other compression methods for the LLaVA-NeXT model tested on the TextVQA, [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Comparison between our G-Prune method and [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: The visualization of information propagation in G-Prune across different iteration num [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: The comparative visualization of ToMe, FastV, and G-Prune on LLaVA-NeXT. G-Prune effectively retains tokens [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Pruning All-Rounder: Rethinking and Improving Inference Efficiency for Large Vision Language Models

    cs.CV 2024-12 conditional novelty 6.0 of 10

    PAR trains a lightweight meta-router with self-supervised preference optimization to jointly drop redundant visual tokens and skip redundant layers in LVLMs, improving speed while keeping accuracy.

  2. Fast3D: Accelerating 3D Multi-modal Large Language Models for Efficient 3D Scene Understanding

    cs.CV 2025-07 conditional novelty 5.0 of 10

    Fast3D prunes up to 90% of object-centric visual tokens in 3D MLLMs while preserving about 96.8% of original benchmark performance, using a trained attention predictor and adaptive layer-wise pruning.

  3. Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings

    cs.CV 2024-11 conditional novelty 5.0 of 10

    DyVTE trains a lightweight hyper-network to detect when text tokens have absorbed enough image information, then removes all visual tokens at that layer, cutting MLLM FLOPs by 20-56% with small accuracy changes.

Reference graph

Works this paper leans on

45 extracted references · 9 canonical work pages · cited by 3 Pith papers

  1. [1]

    L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al

    Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    Bai, J.; Bai, S.; Chu, Y.; Cui, Z.; Dang, K.; Deng, X.; Fan, Y.; Ge, W.; Han, Y.; Huang, F.; et al. 2023 a . Qwen technical report. arXiv preprint arXiv:2309.16609

    arXiv 2023

  3. [3]

    Bai, J.; Bai, S.; Yang, S.; Wang, S.; Tan, S.; Wang, P.; Lin, J.; Zhou, C.; and Zhou, J. 2023 b . Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966

    arXiv 2023

  4. [4]

    Bolya, D.; Fu, C.-Y.; Dai, X.; Zhang, P.; Feichtenhofer, C.; and Hoffman, J. 2022. Token merging: Your vit but faster. arXiv preprint arXiv:2210.09461

    arXiv 2022

  5. [5]

    D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al

    Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877--1901

    2020

  6. [6]

    Cao, J.; Ye, P.; Li, S.; Yu, C.; Tang, Y.; Lu, J.; and Chen, T. 2024. MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 15710--15719

    work page 2024

  7. [7]

    Cao, Q.; Paranjape, B.; and Hajishirzi, H. 2023. PuMer: Pruning and merging tokens for efficient vision language models. arXiv preprint arXiv:2305.17530

    arXiv 2023

  8. [8]

    Chen, L.; Zhao, H.; Liu, T.; Bai, S.; Lin, J.; Zhou, C.; and Chang, B. 2024 a . An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. arXiv preprint arXiv:2403.06764

    arXiv 2024

Show all 45 references
  1. [9]

    Chen, Z.; Wu, J.; Wang, W.; Su, W.; Chen, G.; Xing, S.; Zhong, M.; Zhang, Q.; Zhu, X.; Lu, L.; et al. 2024 b . Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patte...

    2024

  2. [10]

    Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

    2018 arXiv

  3. [11]

    Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929

    2020 arXiv

  4. [12]

    Fu, C.; Chen, P.; Shen, Y.; Qin, Y.; Zhang, M.; Lin, X.; Yang, J.; Zheng, X.; Li, K.; Sun, X.; Wu, Y.; and Ji, R. 2023. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models. arXiv preprint arXiv:2306.13394

    2023 arXiv

  5. [13]

    Goyal, Y.; Khot, T.; Summers-Stay, D.; Batra, D.; and Parikh, D. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, 6904--6913

    2017

  6. [14]

    Guo, D.; Zhu, Q.; Yang, D.; Xie, Z.; Dong, K.; Zhang, W.; Chen, G.; Bi, X.; Wu, Y.; Li, Y.; et al. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming--The Rise of Code Intelligence. arXiv preprint arXiv:2401.14196

    2024 arXiv

  7. [15]

    A.; and Manning, C

    Hudson, D. A.; and Manning, C. D. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 6700--6709

    2019

  8. [16]

    Li, B.; Zhang, P.; Zhang, K.; Pu, F.; Du, X.; Dong, Y.; Liu, H.; Zhang, Y.; Zhang, G.; Li, C.; et al. 2024. Lmms-eval: Accelerating the development of large multimoal models

    2024

  9. [17]

    Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023 a . Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, 19730--19742. PMLR

    2023

  10. [18]

    X.; and Wen, J.-R

    Li, Y.; Du, Y.; Zhou, K.; Wang, J.; Zhao, W. X.; and Wen, J.-R. 2023 b . Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355

    2023 arXiv

  11. [19]

    Liang, Y.; Ge, C.; Tong, Z.; Song, Y.; Wang, J.; and Xie, P. 2022. Not all patches are what you need: Expediting vision transformers via token reorganizations. arXiv preprint arXiv:2202.07800

    2022 arXiv

  12. [20]

    Lin, Z.; Lin, M.; Lin, L.; and Ji, R. 2024. Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference. arXiv preprint arXiv:2405.05803

    2024 arXiv

  13. [21]

    Liu, H.; Li, C.; Li, Y.; Li, B.; Zhang, Y.; Shen, S.; and Lee, Y. J. 2024 a . Llava-next: Improved reasoning, ocr, and world knowledge

    2024

  14. [22]

    Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2024 b . Visual instruction tuning. Advances in neural information processing systems, 36

    2024

  15. [23]

    Liu, Y.; Duan, H.; Zhang, Y.; Li, B.; Zhang, S.; Zhao, W.; Yuan, Y.; Wang, J.; He, C.; Liu, Z.; et al. 2023. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281

    2023 arXiv

  16. [24]

    Long, S.; Zhao, Z.; Pi, J.; Wang, S.; and Wang, J. 2023. Beyond attentive tokens: Incorporating token importance and diversity for efficient vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10334--10343

    2023

  17. [25]

    Lu, Y.; and Zhang, J. 2022. Norm-based noisy corpora filtering and refurbishing in neural machine translation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 5414--5425

    2022

  18. [26]

    Luo, G.; Zhou, Y.; Ren, T.; Chen, S.; Sun, X.; and Ji, R. 2024. Cheap and quick: Efficient vision-language instruction tuning for large language models. Advances in Neural Information Processing Systems, 36

    2024

  19. [27]

    X.; Tan, J

    Masry, A.; Long, D. X.; Tan, J. Q.; Joty, S.; and Hoque, E. 2022. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244

    2022 arXiv

  20. [28]

    Mathew, M.; Karatzas, D.; Manmatha, R.; and Jawahar, C. V. 2020. DocVQA: A Dataset for VQA on Document Images. 2021 IEEE Winter Conference on Applications of Computer Vision (WACV)

    2020

  21. [29]

    Meta, A. 2024. Introducing meta llama 3: The most capable openly available llm to date. Meta AI

    2024

  22. [30]

    Pan, B.; Panda, R.; Jiang, Y.; Wang, Z.; Feris, R.; and Oliva, A. 2021. IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformers. Advances in Neural Information Processing Systems, 34: 24898--24911

    2021

  23. [31]

    Rao, Y.; Zhao, W.; Liu, B.; Lu, J.; Zhou, J.; and Hsieh, C.-J. 2021. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34: 13937--13949

    2021

  24. [32]

    Reid, M.; Savinov, N.; Teplyashin, D.; Lepikhin, D.; Lillicrap, T.; Alayrac, J.-b.; Soricut, R.; Lazaridou, A.; Firat, O.; Schrittwieser, J.; et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530

    2024 arXiv

  25. [33]

    Singh, A.; Natarajan, V.; Shah, M.; Jiang, Y.; Chen, X.; Batra, D.; Parikh, D.; and Rohrbach, M. 2019. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 8317--8326

    2019

  26. [34]

    M.; Hauth, A.; et al

    Team, G.; Anil, R.; Borgeaud, S.; Wu, Y.; Alayrac, J.-B.; Yu, J.; Soricut, R.; Schalkwyk, J.; Dai, A. M.; Hauth, A.; et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

    2023 arXiv

  27. [35]

    Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozi \`e re, B.; Goyal, N.; Hambro, E.; Azhar, F.; et al. 2023 a . Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  28. [36]

    Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al. 2023 b . Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  29. [37]

    Vaswani, A. 2017. Attention is All You Need. arXiv preprint arXiv:1706.03762

    2017 arXiv

  30. [38]

    Wang, H.; Dedhia, B.; and Jha, N. K. 2024. Zero-TPrune: Zero-shot token pruning through leveraging of the attention graph in pre-trained transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 16070--16079

    2024

  31. [39]

    Xu, Y.; Zhang, Z.; Zhang, M.; Sheng, K.; Li, K.; Dong, W.; Zhang, L.; Xu, C.; and Sun, X. 2022. Evo-vit: Slow-fast token evolution for dynamic vision transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, 2964--2972

    2022

  32. [40]

    Yang, J.; Li, C.; Dai, X.; and Gao, J. 2022. Focal modulation networks. Advances in Neural Information Processing Systems, 35: 4203--4217

    2022

  33. [41]

    Yao, L.; Li, L.; Ren, S.; Wang, L.; Liu, Y.; Sun, X.; and Hou, L. 2024. DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models. arXiv preprint arXiv:2405.20985

    2024 arXiv

  34. [42]

    Zhang, P.; Dong, X.; Zang, Y.; Cao, Y.; Qian, R.; Chen, L.; Guo, Q.; Duan, H.; Wang, B.; Ouyang, L.; Zhang, S.; Zhang, W.; Li, Y.; Gao, Y.; Sun, P.; Zhang, X.; Li, W.; Li, J.; Wang, W.; Yan, H.; He, C.; Zhang, X.; Chen, K.; Dai, J.; Qiao, Y.; Lin, D.; and Wang, J. 2024. Intern...

    2024 arXiv

  35. [43]

    Zhu, D.; Chen, J.; Shen, X.; Li, X.; and Elhoseiny, M. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592

    2023 arXiv

  36. [44]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  37. [45]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.