Pith. sign in

REVIEW 3 major objections 6 minor 19 references

CoreMatching: A Co-adaptive Sparse Inference Framework with Token and Neuron Pruning for Comprehensive Acceleration of Vision-Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Token pruning and neuron pruning in vision-language models are two sides of one mechanism: tokens that activate the most core neurons matter most for the answer, so pruning to them while keeping only core neurons stays near-lossless.

desk verdict Co-adaptive token+neuron pruning is a genuine new mechanism with broad empirical support, but the theory has a math error and the decoding-mask assumption is unverified. read the letter →

arxiv 2505.19235 v1 pith:OQZUVMBW submitted 2025-05-25 cs.LG cs.CV

Qinsi Wang , Hancheng Ye , Ming-Yu Chung , Yudong Liu , Yueqian Lin , Martin Kuo , Mingyuan Ma , Jianyi Zhang
show 1 more author
Yiran Chen
This is my paper
classification cs.LGcs.CV
keywords vision-languagemodelstokenpruningneuronsparsityactivationcoreneuronsco-adaptiveinferenceaccelerationplug-and-play
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Vision-language models are slow because they process hundreds of image tokens through every layer, and the two standard fixes — dropping tokens and skipping inactive neurons — have been treated as unrelated. This paper argues they are two sides of one mechanism: the neurons that fire most across the input (core neurons) also pick out the tokens that matter for the answer. It defines core tokens as the tokens activating the most core neurons and shows, on LLaVA-1.5 and other VLMs, that keeping only these tokens and only the core neurons preserves accuracy on ten tasks while cutting FLOPs roughly five-fold and wall-clock time up to ten-fold on a Titan Xp. The practical point is a training-free, plug-and-play recipe: one pass during pre-filling identifies both sparse sets, and decoding runs on the frozen core neurons alone.

What carries the argument

The load-bearing object is the intersection count $|\Gamma(x_m) \cap C_\rho^\beta(s)|$ between the set of neurons that token $x_m$ activates and the sentence-wise core-neuron set $C_\rho^\beta(s)$ (the top-$\rho$ per-token activations aggregated to the top-$\beta$ most frequent neurons). The framework equates this count with token importance via the chain: token influence on the last token is the projection $\|\mathrm{Proj}_{O_M}(\alpha_{iM}V_i)\|$ of the attention-weighted value vector onto the output direction, and this projection is proportional to $\cos(\angle(y_i,y_M))$, which is proportional to the co-activated-neuron count; replacing the final token's activation set with the core-neuron set (predictability of core neurons) yields Eq. 15. The core-token threshold is chosen adaptively by the maximum-geometric-distance 'knee' method, so no token budget is fixed. During pre-filling the framework records activations, computes core neurons at every FFN layer, selects core tokens at layer $l=2$, and the decoding stage runs on the cached core neurons.

What would settle it

Measure, during autoregressive decoding of a VLM running CoreMatching, for each generated token the fraction of its top-activated neurons that lie outside the frozen core-neuron set, and compare task accuracy against an oracle run that recomputes core neurons from the actual decoding activations at every step. If the out-of-set fraction is large for a substantial share of decoding steps, or the oracle run is meaningfully more accurate, the predictability assumption underlying Eq. 15 fails.

Watch

Extended reading notes

Core claim

The paper's central discovery is a matching relation: in a VLM's feed-forward layers, the tokens that activate the largest number of the sentence's core neurons carry the information the decoder actually uses, and those two sparse structures are generated by the same computation. Concretely, the paper defines core neurons as the top $\beta$ most frequently activated neurons per sentence (from the top $\rho$ per-token activations), and core tokens as the tokens whose activated-neuron set intersects the core-neuron set above an adaptive knee threshold. It then derives that a token's contribution to the final output is proportional to this intersection (Eq. 15), by arguing that token influence should be measured by the projection of the attention-weighted value vector onto the output direction, not by attention score alone, and that angular proximity between tokens is controlled by how many neurons they co-activate. The claimed consequence is that pruning to core tokens while running decoding on core neurons alone is nearly lossless, and the two prunings reinforce each other: core neurons select the tokens, and the surviving tokens define which neurons are needed.

Load-bearing premise

The framework freezes the core-neuron set computed from the pre-filling tokens and uses it throughout decoding, assuming that the tokens generated later activate roughly the same set of core neurons; if a generated token needs a neuron outside that set, that computation is skipped and the near-lossless claim breaks.

Editorial extensions

If this is right

  • With $\rho=0.2$ and $\beta=0.4$, keeping the top 40% of neurons costs about 3% accuracy on TextVQA (45.1% at 20% retention versus 57.8% at 100%), and the core-neuron set is nearly unchanged as the text stimulus grows.
  • Core tokens use no fixed budget: a knee-point threshold on the intersection distribution keeps roughly 10% of tokens for LLaVA-1.5-7B and 17% for 13B, with harder tasks retaining more (27.9 tokens for SciQA versus 93.9 for MM-Vet).
  • Combined sparsity produces comprehensive acceleration — a 2.1x pre-fill and 9.2x decoding speedup, a 5x FLOPs cut, and up to a 10x end-to-end speedup on a Titan Xp — with the decoding gains growing with output length.
  • The recipe transfers across architectures (LLaVA-1.5, Qwen2.5-VL, Video-LLaVA) and tasks, sometimes beating the full model on video benchmarks (48.4 versus 45.3 on ActivityNet-QA).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: core-neuron coverage of a token could be predicted from the layer-2 hidden states alone, turning the pre-fill scan into a single forward probe and removing the need to record activations at every layer.
  • If the association between projection value and output influence is as strong as Eq. 15 claims, then per-token causal deletion experiments (removing one image token and measuring the change in the final answer distribution) should reproduce the core-token ranking, which would confirm the metric outside the paper's benchmarks.
  • The near-orthogonality observation suggests the same token-neuron matching may reappear in mixture-of-experts models, where experts play the neuron role, giving a cost-geometry explanation of why expert-router-based token pruning works.
  • The frozen core-neuron mask makes a prediction about interactive use: in multi-turn dialogue the prompt semantics may shift more than within a single answer, so the pre-fill core set would need per-turn refreshment, and measuring accuracy decay across turns would bound how long a single mask stays valid.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper proposes CoreMatching, a training-free inference framework that simultaneously prunes image tokens and FFN neurons for vision-language models. In the pre-filling stage, the method computes sentence-wise 'core neurons' as the most frequently activated neurons across input tokens, and then selects 'core tokens' as the image tokens whose activated-neuron set has the largest intersection with the core-neuron set. During decoding, only the core tokens and core neurons are used. The paper reports consistent accuracy gains over token-only and neuron-only baselines on ten image understanding tasks and three hardware platforms, with a claimed 5x FLOPs reduction and 10x speedup on a Titan Xp. A theoretical section argues that core tokens correspond to a projection-guided importance criterion.

Significance. If the claims hold, CoreMatching is a useful practical contribution: it is training-free, plug-and-play, and the evaluation spans multiple architectures (LLaVA-1.5-7B/13B, Qwen2.5-VL, Video-LLaVA), tasks, and hardware, with code released. The empirical results are broad and mostly consistent. However, the theoretical analysis in Sec. 3.2 contains a concrete mathematical error (Eq. 12 incorrectly claims linear proportionality to cos where the derivation gives cos-squared), and the central assumption that core neurons computed from pre-filling tokens remain valid for decoding-stage generated tokens is asserted but not directly measured. These issues bear on the correctness of the paper's central explanatory claim, even if the pruning algorithm itself may still work empirically. The near-lossless wording is also somewhat overstated given Table 2 drops of 3-4 points on TextVQA.

major comments (3)
  1. [Sec. 3.2, Eqs. (10)-(12)] The derivation leading to Eq. (12) is mathematically incorrect. From Eq. (10) and Observation 1, substituting Wq Wk^T ≈ θI and Wv Wv^T ≈ λI yields an expression proportional to θλ <ŷ_i, ŷ_M>^2 / ||V_M||, i.e., to cos^2(∠(y_i,y_M)), not to cos(∠(y_i,y_M)). The statement immediately before Eq. (10), "Since the Softmax function is monotonic, that is, Softmax(x) ∝ x," is also false: monotonicity does not imply proportionality. The claimed consistency with Insight 1 therefore does not follow. Because Eq. (15) is the central theoretical bridge from the projection criterion to core tokens, this error must be fixed, e.g., by correcting the exponent to cos^2 and noting that this still preserves monotonicity with respect to the intersection count, or by softening the claim to a monotone relationship.
  2. [Sec. 2.1 and Eq. (15)] The load-bearing assumption that core neurons computed from pre-filling tokens remain valid for decoding-stage generated tokens is asserted but not verified for the prompt-to-answer transition. The supporting evidence in Fig. 2 only varies the length of the input prompt (50/100/150 tokens); it does not measure the overlap between the activated neurons of generated tokens and C^β_ρ(s). During decoding every generated token is computed with only the pre-filling core-neuron mask, so any generated token whose active neurons fall outside the mask is evaluated with missing components. Table 1 already shows a 4.6-point drop on TextVQA when retaining 40% of neurons alone, so neuron sparsity is not free. To support the near-lossless claim, the authors should report a direct overlap statistic (e.g., per-layer mean Jaccard or recall of Γ(x_generated) against C^β_ρ(s)) on the evaluated benchmarks, or provide an oracle comparison that recomputes core neurons during decoding. Without this, the accuracy claim in Tables 2 and 3 is fragile even if the speedup claim is not.
  3. [Sec. 3.2, Eq. (15) and Sec. 2.2] The theoretical justification of core tokens is partly circular. Core tokens are defined as the tokens with the largest |Γ(x_i) ∩ C^β_ρ(s)|, and then the derivation of Eq. (15) assumes Γ(x_M) ≈ C^β_ρ(s), i.e., the last-token activated set equals the core-neuron set. If that assumption is already the central claim, the argument does not independently validate the projection criterion. A concrete fix is to compute the projection value of Eq. (8) (or a direct proxy) for all input tokens, sort tokens by that value, and compare the resulting top set against the core tokens selected by Eq. (6). If the two sets largely coincide on held-out examples, the circularity is resolved empirically; the current manuscript only provides qualitative side-by-side examples (Fig. 5).
minor comments (6)
  1. [Sec. 3.2, Eq. (10)] The sentence "Since the Softmax function is monotonic, that is, Softmax(x) ∝ x" is a non sequitur and should be removed or rephrased; attention scores are normalized, and claims of proportionality need an explicit statement of the constant factors being absorbed.
  2. [Sec. 2.2, Eq. (5)] The assertion I(x) ∝ |Γ(x) ∩ C^β_ρ(s)| is introduced as a definitional or intuitive step, but it is not derived or tested; it should be clearly labeled as an assumption or heuristic.
  3. [Appendix B, Algorithm 1] The pseudo-code is inconsistent with the main text: the main text uses layer index l=2 as the token-pruning layer, while the algorithm loops only up to layer L and then appears to reuse the same loop in the decoding stage. The loop bounds and the stage at which token pruning happens should be clarified for both pre-filling and decoding.
  4. [Table 1 and Sec. 2.1] The text claims that retaining 40% of neurons decreases performance by "merely 3%", but Table 1 shows a drop from 57.8 to 53.2 (4.6 points, about 8% relative). Please reconcile the numbers and avoid the misleading phrasing.
  5. [Tables 4-6] Several tables contain typesetting errors: "OCRB" and "CoreMathing"/"CoreMating" appear as typos, and some row entries (e.g., PruMerge+ and FastV rows in Table 4) are concatenated without proper separators. These should be fixed in the final version.
  6. [Fig. 5] The comparison of the three metrics would be more convincing if the selected tokens were visualized on the same image with identical masks or bounding boxes, rather than in separate panels with different scales.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the central near-lossless claim is validated on external benchmarks; the theory's Eq. 15 rests on an unverified decoding-overlap assumption, not on a definitional reduction, and the CoreInfer self-citation is re-verified in Table 1.

full rationale

The paper is self-contained against external benchmarks: Tables 2/4/5/6 evaluate CoreMatching on VQAv2, GQA, SciQA, TextVQA, POPE, MME, MMB, SEED, VisWiz, MM-Vet, DocVQA, InfoVQA, ChartQA, OCRBench, AI2D, and video QA, comparing with original models and independent baselines. The central near-lossless claim is therefore anchored outside the paper's own fitted values. The theoretical route to Eq. 15 is not a reduction by construction. Eq. 8 defines the projection criterion independently of the core-token score; Eq. 15, 'ProjOM(αiM Vi) ∝ |Γ(xi)∩ Cβρ(s)|', is obtained via empirical Observation 2 (cos(Ai,AM) ∝ co-activated-neuron count, Fig. 7) plus the explicit approximation 'Γ(xM)≈ Cβρ(s)' announced in Sec. 2.1 ('they exhibit predictability'). That approximation is load-bearing but unverified: Fig. 2 varies text length (50/100/150), not the prefill-to-decoding shift, and no overlap statistic between generated-token activations and Cβρ(s) is reported. This is a missing-evidence gap and correctness risk, but it is not circular: the predictability assumption is not identical to the conclusion, and the benchmark results could in principle have falsified it. Similarly, the derivation contains mathematical slips (e.g., 'Sof tmax(x)∝x' before Eq. 10, and the cos² factor dropped between Eq. 11 and Eq. 12), and Eq. 5's identification of transmitted information with |Γ(x)∩C| is an interpretive assumption; none of these makes the output equal to the input by definition. The self-citation to CoreInfer (Wang et al., 2024) for the core-neuron concept and hyperparameters (ρ=0.2, β=0.4, Appendix D) is the only self-citation bearing on the method's ingredients, and it is not load-bearing because Table 1 independently re-verifies core-neuron importance on LLaVA-1.5-7B. No fitted parameter is renamed as a prediction: T_k is an adaptive knee threshold, not a fitted coefficient. Overall: no significant circularity; score 2 only for the minor, re-verified CoreInfer self-citation.

Assumptions & free parameters 3 free parameters · 5 assumptions · 2 invented entities

The method relies on two empirical observations about VLM weight matrices and activation geometry, plus a predictability assumption about core neurons across pre-filling and decoding. The free parameters rho, beta, and l are carried over from prior work (CoreInfer and FastV) rather than tuned on the target benchmarks, which lowers the parameter-fitting burden. The invented entities are operational constructs derived from activations, not ad hoc physical postulates, and they have independent empirical handles through accuracy experiments.

free parameters (3)
  • rho (ρ=0.2) = 0.2
    Top fraction of positive activations defining token-wise core neurons. Taken from CoreInfer, not tuned per task, but hand-chosen and load-bearing for the definition of core neurons.
  • beta (β=0.4) = 0.4
    Top fraction of most frequently activated neurons defining sentence-wise core neurons. Same source and status as rho.
  • l (layer index=2) = 2
    Layer at which core tokens are selected, chosen to match FastV. This choice determines which tokens are kept for all subsequent layers.
assumptions (5)
  • domain assumption Observation 1: Wq Wk^T ≈ θI, Wv Wv^T ≈ λI, Wd Wd^T ≈ ηI
    Empirically shown in Fig. 12-14 for LLaVA-1.5-7b, used in Eq. 10-13 to reduce attention and projection computations to inner products of normalized vectors.
  • domain assumption Observation 2: cos(∠(A_i, A_M)) is proportional to the number of co-activated neurons
    Shown in Fig. 7 and Fig. 15, used in Eq. 14 to connect activation overlap to angular similarity. The relationship is approximate and does not hold in early layers.
  • domain assumption Core neuron predictability: Γ(x_M) ≈ C^β_ρ(s)
    Stated in Sec. 2.1 and visualized in Fig. 2, used in Eq. 15 to replace the decoding token's neuron set with the precomputed core neurons. If this fails, the decoding mask is wrong.
  • ad hoc to paper Softmax(x) is proportional to x
    Used in Eq. 10 to replace softmax attention scores with unnormalized dot products. This is mathematically false; monotonicity does not imply proportionality, and it is not a harmless approximation for arbitrary logits.
  • domain assumption O_M is primarily determined by α_MM V_M
    Used in Eq. 9 to approximate the attention output by the self-token contribution. Supported by Fig. 6(c) showing large diagonal attention, but the approximation error is not quantified.
invented entities (2)
  • Core Neurons independent evidence
    purpose: Define the neuron sparsity mask for pre-filling and decoding by selecting the most frequently and strongly activated neurons for the input sentence.
    Validated by accuracy retention in Table 1 and predictability in Fig. 2. The concept is defined from the model's activations and can be tested on any VLM.
  • Core Tokens independent evidence
    purpose: Define the token sparsity mask by selecting tokens with the largest intersection of activated neurons with core neurons.
    Validated by downstream benchmark accuracy in Table 2 and by visualizations in Fig. 3-8. The definition is falsifiable: one can test whether these tokens are indeed the most important for a given input.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CoreMatching: A Co-adaptive Sparse Inference Framework with Token and Neuron Pruning for Comprehensive Acceleration of Vision-Language Models." pith.science (2026). https://pith.science/paper/OQZUVMBW

@misc{pith2026250519235,
  author       = {Pith},
  title        = {Pith review of: CoreMatching: A Co-adaptive Sparse Inference Framework with Token and Neuron Pruning for Comprehensive Acceleration of Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OQZUVMBW}},
  note         = {Machine review of arXiv:2505.19235}
}
read the original abstract

Vision-Language Models (VLMs) excel across diverse tasks but suffer from high inference costs in time and memory. Token sparsity mitigates inefficiencies in token usage, while neuron sparsity reduces high-dimensional computations, both offering promising solutions to enhance efficiency. Recently, these two sparsity paradigms have evolved largely in parallel, fostering the prevailing assumption that they function independently. However, a fundamental yet underexplored question remains: Do they truly operate in isolation, or is there a deeper underlying interplay that has yet to be uncovered? In this paper, we conduct the first comprehensive investigation into this question. By introducing and analyzing the matching mechanism between Core Neurons and Core Tokens, we found that key neurons and tokens for inference mutually influence and reinforce each other. Building on this insight, we propose CoreMatching, a co-adaptive sparse inference framework, which leverages the synergy between token and neuron sparsity to enhance inference efficiency. Through theoretical analysis and efficiency evaluations, we demonstrate that the proposed method surpasses state-of-the-art baselines on ten image understanding tasks and three hardware devices. Notably, on the NVIDIA Titan Xp, it achieved 5x FLOPs reduction and a 10x overall speedup. Code is released at https://github.com/wangqinsi1/2025-ICML-CoreMatching/tree/main.

Figures

Figures reproduced from arXiv: 2505.19235 by the authors.

Figure 1
Figure 1. Schematic diagram of CoreMatching. In the Pre-filling stage, CoreMatching calculates Core Neurons in the FFN block based on the activation. Core Neurons are the most frequently activated group of neurons. Afterwards, CoreMatching matches the neurons activated by different tokens with the core neurons, and selects a group of tokens with the largest intersection as the Core Tokens. Only the Core Tokens are passed to t… view at source ↗
Figure 2
Figure 2. Verification of the predictability of core neurons. We visualized the core neurons of the 25-th layer of Llava-1.5-7b when input text token at different lengths. ρ = 0.2, β = 0.4. We selected the first 256 neurons. It can be seen that when the input semantics are sufficient, core neurons are almost unchanged. 2. Intrinsic Relations of Two Paradigms In this section, to delve deeper into the relationship between two s… view at source ↗
Figure 3
Figure 3. (Upper) Distribution of [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Diagram of attention score and projection value. ✓ indi￾cates the token is reserved under this matric. ✗ indicates discarded. 8 7 6 5 4 3 2 1 (a) Attention score 0.01 0.02 0.03 0.04 0.05 (b) Projection Value 2800 2900 3000 3100 3200 3300 3400 3500 (c) Core Tokens [PIT…
Figure 5
Figure 5. Figure 5: Comparison of three metrics. The input is the rabbit in [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 7
Figure 7. Figure 7: The distribution of cos(Ai, AM) and [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 9
Figure 9. Figure 9: Latency comparison of token-only/neurons-only/both sparse on NVIDIA TiTAN Xp. W/T means only core tokens, W/N means only core neurons, and W/TN means CoreMatching. The number on the bar means how many seconds it took. 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 Latency Reduction (…
Figure 10
Figure 10. Figure 10: Latency comparison of token-only/neurons-only/both sparse on NVIDIA RTX 6000. decoding. When the input token length is short, CoreMatch￾ing requires only about half the memory of the original model during decoding, eliminating memory-bound limi￾tations on resource-con…
Figure 11
Figure 11. Figure 11: Number of tokens required at different layers. pronounced gains as the output token length increased. Token-only sparsity primarily accelerated the pre-filling stage, with minimal impact on the decoding stage (limited to reducing KV computation). Conversely, neuron-on…
Figure 12
Figure 12. Figure 12: Visualization of WQ@WK.T at different layers in LLaVA-1.5-7b. 0 8 16 24 32 40 48 56 0 8 16 24 32 40 48 56 Layer 0 0 8 16 24 32 40 48 56 0 8 16 24 32 40 48 56 Layer 1 0 8 16 24 32 40 48 56 0 8 16 24 32 40 48 56 Layer 2 0 8 16 24 32 40 48 56 0 8 16 24 32 40 48 56 Layer …
Figure 13
Figure 13. Figure 13: Visualization of WD@WD.T at different layers in LLaVA-1.5-7b. 0 8 16 24 32 40 48 56 0 8 16 24 32 40 48 56 Layer 0 0 8 16 24 32 40 48 56 0 8 16 24 32 40 48 56 Layer 1 0 8 16 24 32 40 48 56 0 8 16 24 32 40 48 56 Layer 2 0 8 16 24 32 40 48 56 0 8 16 24 32 40 48 56 Layer …
Figure 14
Figure 14. Figure 14: Visualization of WV @WV .T at different layers in LLaVA-1.5-7b. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_14.png]
Figure 15
Figure 15. Figure 15: Visualization of cos(∠(Ai, AM)) and co-act neurons number at different layers in LLaVA-1.5-7b. For proof of cos(∠(yi , yM)) = cos(∠(Ai , AM)), cos(∠(yi , yM)) can be written as cos(∠(yi , yM)) = ⟨AiWd, AMWd⟩/(∥yi∥∥yM∥) = Ai(WdWT d )A T M/(∥yi∥∥yM∥) = η⟨Ai , AM⟩ /(∥yi∥…
Figure 16
Figure 16. Figure 16: shows the latency comparison on NVIDIA A100. It can be seen that CoreMatching can achieve excellent acceleration effects even without memory limit. And as the batch size increases, the acceleration effect becomes more and more obvious. 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 …
Figure 17
Figure 17. Figure 17: Examples of CoreMatching sampled tokens for different inputs. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_17.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

19 extracted references · 6 canonical work pages

  1. [1]

    C., Rastegari, M., and Fara- jtabar, M

    Alizadeh, K., Mirzadeh, I., Belenko, D., Khatamifard, K., Cho, M., Del Mundo, C. C., Rastegari, M., and Fara- jtabar, M. Llm in a flash: Efficient large language model inference with limited memory. arXiv preprint arXiv:2312.11514,

    arXiv

  2. [3]

    Dynamic-llava: Efficient multimodal large language models via dynamic vision-language context sparsification

    Huang, W., Zhai, Z., Shen, Y ., Cao, S., Zhao, F., Xu, X., Ye, Z., and Lin, S. Dynamic-llava: Efficient multimodal large language models via dynamic vision-language context sparsification. arXiv preprint arXiv:2412.00876,

    arXiv

  3. [6]

    X., and Wen, J.-R

    Li, Y ., Du, Y ., Zhou, K., Wang, J., Zhao, W. X., and Wen, J.-R. Evaluating object hallucination in large vision- language models. arXiv preprint arXiv:2305.10355, 2023b. Lin, B., Ye, Y ., Zhu, B., Cui, J., Ning, M., Jin, P., and Yuan, L. Video-llava: Learning united visual represen- tation by alignment before projection. arXiv preprint arXiv:2311.10122,

    arXiv

  4. [8]

    Liu, H., Li, C., Li, Y ., and Lee, Y . J. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26296–26306, 2024a. Liu, H., Li, C., Wu, Q., and Lee, Y . J. Visual instruction tun- ing. Advances in neural information processing systems, 36, 2024b. 10 CoreMatching: A Co-...

    work page Pith review arXiv

  5. [10]

    Powerinfer: Fast large language model serving with a consumer-grade gpu

    Song, Y ., Mi, Z., Xie, H., and Chen, H. Powerinfer: Fast large language model serving with a consumer-grade gpu. arXiv preprint arXiv:2312.12456,

    arXiv

  6. [11]

    Llama: Open and efficient foundation lan- guage models

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi`ere, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302.13971, 2023a. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi`ere, B., Goyal, N., Hambro, E., Azhar, ...

    arXiv 1954

  7. [12]

    CoreInfer: Accelerating Large Language Model Inference with Semantics-Inspired Adaptive Sparse Activation

    Wang, Q., Vahidian, S., Ye, H., Gu, J., Zhang, J., and Chen, Y . Coreinfer: Accelerating large language model infer- ence with semantics-inspired adaptive sparse activation. arXiv preprint arXiv:2410.18311,

    work page Pith review arXiv

  8. [14]

    V oco-llama: Towards vision compression with large lan- guage models

    Ye, X., Gan, Y ., Huang, X., Ge, Y ., Shan, Y ., and Tang, Y . V oco-llama: Towards vision compression with large lan- guage models. arXiv preprint arXiv:2406.12275,

    arXiv

Show all 19 references
  1. [15]

    Mm-vet: Evaluating large multi- modal models for integrated capabilities

    Yu, W., Yang, Z., Li, L., Wang, J., Lin, K., Liu, Z., Wang, X., and Wang, L. Mm-vet: Evaluating large multi- modal models for integrated capabilities. arXiv preprint arXiv:2308.02490,

    arXiv

  2. [16]

    The document is organized as follows: •A- Related Work •B- Algorithm •C- Assumption Explanation •D- Experiments Settings •E- Additional Experiments •F- Visualization of Results A

    11 CoreMatching: A Co-adaptive Sparse Inference Framework for Comprehensive Acceleration of Vision-Language Models OrganizationIn this appendix, we provide in-depth descriptions of the materials that are not covered in the main paper, and report additional experimental results...

    2023

  3. [17]

    For example, in the OPT-30B, a single token activates only approximately 10% of the neurons (Alizadeh et al., 2023)

    indicates that individual tokens in LLMs exhibit significant activation sparsity. For example, in the OPT-30B, a single token activates only approximately 10% of the neurons (Alizadeh et al., 2023). Therefore, if the activated neurons can be accurately predicted in advance, a ...

    2023

  4. [18]

    CoreInfer identifies a set of core neurons that most frequently and strongly activated for each input sentence

    proposed a sentence-level adaptive activation sparsity inference method without the need of predictors. CoreInfer identifies a set of core neurons that most frequently and strongly activated for each input sentence. Experiments demonstrated that for a given input sentence, LLM...

  5. [19]

    (18) which means∥y i∥= √η∥Ai∥

    Furthermore, sinceW d is an orthogonal matrix, we have ∥yi∥2 =∥A iWd∥2 = (AiWd)(AiWd)T =A i(WdW T d )AT i =ηA iAT i =η∥A i∥2. (18) which means∥y i∥= √η∥Ai∥. Substituting this into Eq. 17 we can have cos(∠(yi, yM )) =η⟨A i, AM ⟩/(η∥A i∥∥AM ∥) = cos(∠(Ai, AM )) (19) This shows t...

    2017

  6. [2018]

    Brown, T. B. Language models are few-shot learners. arXiv preprint arXiv:2005.14165,

    2005 arXiv

  7. [2019]

    H-cot: Hijack- ing the chain-of-thought safety reasoning mechanism to jailbreak large reasoning models, including openai o1/o3, deepseek-r1, and gemini 2.0 flash thinking

    Kuo, M., Zhang, J., Ding, A., Wang, Q., DiValentin, L., Bao, Y ., Wei, W., Li, H., and Chen, Y . H-cot: Hijack- ing the chain-of-thought safety reasoning mechanism to jailbreak large reasoning models, including openai o1/o3, deepseek-r1, and gemini 2.0 flash thinking. arXiv pr...

    arXiv

  8. [2022]

    Dobi-svd: Differentiable svd for llm compression and some new perspectives

    Qinsi, W., Ke, J., Tomizuka, M., Keutzer, K., and Xu, C. Dobi-svd: Differentiable svd for llm compression and some new perspectives. In The Thirteenth International Conference on Learning Representations. Shang, Y ., Cai, M., Xu, B., Lee, Y . J., and Yan, Y . Llava- prumerge: ...

  9. [2023]

    Hippomm: Hippocampal-inspired multimodal memory for long audiovisual event understanding

    Lin, Y ., Wang, Q., Ye, H., Fu, Y ., Li, H., Chen, Y ., et al. Hippomm: Hippocampal-inspired multimodal memory for long audiovisual event understanding. arXiv preprint arXiv:2504.10739,

  10. [2024]

    Powerinfer-2: Fast large language model inference on a smartphone

    Xue, Z., Song, Y ., Mi, Z., Chen, L., Xia, Y ., and Chen, H. Powerinfer-2: Fast large language model inference on a smartphone. arXiv preprint arXiv:2406.06282,

    arXiv

  11. [2025]

    Seed-bench: Benchmarking multimodal llms with gener- ative comprehension

    Li, B., Wang, R., Wang, G., Ge, Y ., Ge, Y ., and Shan, Y . Seed-bench: Benchmarking multimodal llms with gener- ative comprehension. arXiv preprint arXiv:2307.16125, 2023a. Li, J., Li, D., Xiong, C., and Hoi, S. Blip: Boot- strapping language-image pre-training for unified vi...

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.