{"id":"73a7aaed-3b0e-4284-ad58-b10d3b6ff377","arxiv_id":"2504.17892","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Simple cluster-based averaging of visual tokens is competitive with, and sometimes better than, attention-based token selection on LLaVA and VILA benchmarks, though the advantage over prior methods is mixed.","lead":"A Stanford team tested simple ways to shrink the number of visual tokens fed to multimodal language models. They find that averaging clusters of visual tokens, and even random or spatial sampling, often beat complex attention-based selection methods.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central 'computationally simpler' claim is unsupported because no end-to-end compute is measured; token-retention percentage is not a valid proxy across methods that prune at different layers, and the authors concede overhead remains unquantified.","rationale":"The reader's weakest assumption is exactly the load-bearing concern: the paper never measures actual compute and uses token retention as a proxy for efficiency. This is not merely a missing experiment; the conclusion explicitly defers the overhead quantification, so the central 'efficient' and 'computationally simpler' claims are unsupported as stated. The accuracy finding itself is plausible and interesting, and the paper gives some independent support through consistent trends across LLaVA-7B, LLaVA-13B, and VILA-8B, but the lack of compute measurements and the absence of code or error bars prevent full verification. I also note that the 'outperforms all previous finetuning-free SoTA' claim is somewhat overstated by the paper's own tables, since VisionZip wins on several benchmarks at 7B scale; however, this could be reconciled by a normalized average and is secondary to the efficiency gap. Therefore the conditional verdict is appropriate, and this stress-test does not move it.","tokens_in":8276,"tokens_out":9450,"duration_ms":101344,"concrete_test":"Measure end-to-end prefill latency, peak activation memory, and FLOPs on LLaVA-1.5-7B using an A100 or H100 for: full 576 tokens; Cluster & Aggregate with 64 clusters, including K-means++ time; and the official finetuning-free FastV, SparseVLM, and VisionZip implementations at 64 final retained tokens. Run at least 100 images from the same benchmarks and report mean latency and accuracy together. If Cluster & Aggregate is not faster than the attention-based methods at matched benchmark accuracy, the 'computationally simpler' claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central efficiency claim rests on the assumption that reducing visual token count is equivalent to reducing compute. Section 2 presents only LLMViewer estimates at the LLM level, not measured FLOPs, latency, memory, or clustering overhead. The accuracy comparisons in Tables 3-5 and Figure 6 are reported against retained token count, but that is not a compute-equivalence metric: FastV and SparseVLM keep all tokens for early LLM layers and prune later, while Cluster & Aggregate reduces the sequence before the LLM, so the same final token count corresponds to different actual compute. Additionally, K-means++ clustering and aggregation add overhead that is never quantified. The conclusion explicitly states 'We also aim to quantify the computational overhead and net gains,' which is a clear admission that the paper's motivating efficiency claim is not yet established. If the overhead is non-negligible, or if the comparison ignores early-layer compute, the abstract's 'computationally simpler' claim could fail even though the accuracy results are internally plausible.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies compression of visual token sequences in autoregressive vision-language models. It proposes and evaluates several training-free methods: a cross-modal attention saliency score, cluster-based variants of that score, and a purely importance-agnostic \"Cluster & Aggregate\" method that clusters visual embeddings with K-means++ and averages each cluster into a single token. Experiments are run on LLaVA-1.5-7B/13B and VILA-8B across seven to eight VQA/hallucination benchmarks, with comparisons to FastV, SparseVLM, VisionZip, ToMe, and random/spatial sampling. The authors conclude that simple cluster aggregation outperforms all previous finetuning-free selection/merging methods and that attention-based saliency is volatile and largely prompt-insensitive.","tokens_in":8447,"tokens_out":5791,"duration_ms":51846,"significance":"If the empirical patterns hold, the paper makes a useful contribution by challenging the assumption that attention-based importance scoring is necessary for visual token selection: it shows that random/spatial sampling and simple aggregation are competitive, and it offers a concrete, falsifiable observation about saliency being prompt-insensitive. The comparative benchmark across three model families is a strength, and the qualitative heatmap analysis helps the reader interpret the surprising trends. However, the strongest claims are not backed by the reported data: VisionZip wins on several benchmarks in Tables 3 and 4, and no end-to-end compute or clustering overhead is measured, so the \"computationally simpler\" claim is not established. The paper provides no code, seeds, or significance tests, so reproducibility and statistical support are limited. The observations are nevertheless valuable as a stress-test of current token-reduction assumptions.","major_comments":[{"comment":"The claim that Cluster & Aggregate \"outperforms all previous finetuning-free SoTA\" is not supported by the data in Tables 3 and 4. For LLaVA-1.5-7B at 128 retained tokens, VisionZip is higher on TextVQA (56.8 vs 55.25), MME (1761.7 vs 1729.42), and MMVet (32.6 vs 30.9); at 192 tokens VisionZip is higher on TextVQA, MME, and GQA (1782.6 vs 1744.44 on MME, 59.3 vs 59.16 on GQA). Similar mixes occur for LLaVA-1.5-13B, where VisionZip wins SQA, TextVQA, MMBench, and MMVet at 64 tokens. The paper needs a more precise statement, such as reporting per-benchmark wins and an appropriate aggregate statistic, rather than \"outperforms all.\"","section":"Abstract; §3.2 (Fig. 6); Conclusion; Tables 3–4"},{"comment":"The efficiency claim (\"computationally simpler\") is not established because no end-to-end compute is measured. Fig. 2 shows only LLM-level estimates from LLMViewer, which exclude the early LLM layers where FastV and SparseVLM retain all tokens, and no measurement of K-means++/aggregation overhead is reported. The Conclusion explicitly states \"We also aim to quantify the computational overhead and net gains,\" acknowledging this gap. Without latency, FLOPs, or memory measurements, token-count retention is not a valid proxy for compute across methods that prune at different stages.","section":"§2 (Fig. 2); Conclusion"},{"comment":"The statement that \"our empirical evidence indicates no statistically significant performance differential among variants when controlling for retention percentage\" is unsupported: no error bars, number of seeds, or significance tests are reported anywhere in the paper. Since Table 2 shows differences of up to several points on TextVQA (52.97 vs 48.61 between Variant 3 with k=40 and Variant 1) and large drops on VizWiz (53.99 vs 49.58), either the statistical evidence should be provided or the sentence should be removed or explicitly weakened to a qualitative observation.","section":"§3.1 (Table 2)"},{"comment":"The VILA-8B results in Table 5 do not support the sentence that Cluster & Aggregate compression variants \"achieve superior performance compared to these baselines.\" Relative to the uncompressed VILA-8B baseline, Cluster & Aggregate loses substantially on SQA (77.94 vs 81.95), TextVQA (56.38 vs 68.54), MME (1566.2 vs 1744.41), MMBench (55.84 vs 66.83), and MMVet (32.2 vs 40.5), while improving only on VizWiz (64.14 vs 63.24) and GQA (44.02 vs 30.41). The sentence should clarify that the comparison is against the random/spatial compression baselines, not against the uncompressed model.","section":"§3.2 (Table 5)"}],"minor_comments":[{"comment":"The header row lists nine metric columns (including VizWiz) but each data row contains only eight values; the baseline row appears to be missing a VizWiz value or the column positions are shifted.","section":"Table 5"},{"comment":"The text says results are reported for \"64 retained tokens\" in Fig. 6, while the caption says 11% of visual tokens; since 11% of 576 is 63.4, please state the exact token count used and keep the two statements consistent.","section":"§3.2, Fig. 6"},{"comment":"The word \"intutions\" should be \"intuitions\"; also, \"variants\" vs \"variant's\" are used inconsistently in the same paragraph.","section":"§3.1"},{"comment":"The notation \"kvi·Qtext\" is unclear because a single key vector is multiplied by a text query matrix; please specify the shapes and the softmax axis explicitly to match the described computation.","section":"Algorithm 1"},{"comment":"The claim that \"random insertion is a simpler method and proved to yield comparable results\" is not supported by any table or figure comparing random-order and position-ordered aggregation; either add the comparison or mark this as an unreported observation.","section":"§3.2"},{"comment":"The caption says \"layer-wise visualization of saliency heatmaps\" but does not indicate which layers are shown or how heatmaps are normalized; please expand the caption so the reader can interpret the volatility claim.","section":"Fig. 5b"}],"recommendation":"major_revision","confidential_remarks":"The paper is a useful empirical characterization with interesting negative results, but the abstract and conclusion overstate what the authors' own tables show, and the efficiency claim is unmeasured. The required changes are substantial but tractable: temper the \"outperforms all\" claim, add an aggregate benchmark statistic, either measure overhead or carefully scope the efficiency statement, and provide seed-level variability. The work might also fit a workshop venue in its present scope, but as a journal submission it needs the revisions outlined in the major comments. No concerns about citation pattern or novelty disclosure."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's the one-sentence version: this paper's genuinely new result is that, across LLaVA 1.5-7B/13B and VILA-8B, simple cluster-averaging, random sampling, and spatial sampling match or beat attention-based token selection at the same retention rate. If that holds, it's a direct hit on the popular assumption that cross-modal attention scores are reliable importance signals for visual token compression. That's worth taking seriously.\n\nThe paper does a few things right. The benchmark sweep is broad—seven benchmarks, three model scales, two families. The authors include visualizations of saliency instability across layers and prompt invariance, which support their qualitative argument. They are also candid: the conclusion admits the computational overhead still needs to be quantified.\n\nThe soft spots are real but not fatal. First, the headline claim—'outperforms all previous finetuning-free SoTA methods'—is an overreach. Their own Tables 3 and 4 show VisionZip ahead on several benchmarks, especially SQA and MMVet at 128 tokens, and on the 13B model VisionZip wins SQA at every retention. 'Competitive with, and on average better than' would be defensible. Second, there are no error bars, seeds, or statistical tests, yet the text says there is 'no statistically significant performance differential among variants.' You can't assert that without numbers. Third, the efficiency argument isn't backed by measured compute. The stress-test note is correct: token count is not a compute-equivalent metric because FastV and SparseVLM prune at later LLM layers while Cluster & Aggregate prunes before the LLM, and the K-means++ overhead is unquantified. The abstract's 'computationally simpler' is unsupported on the evidence here. Fourth, Table 2 has inconsistencies—retention percentages drifting to 6-15% and some suspicious repeated numbers—that need cleanup, along with code and data release.\n\nNone of this kills the core observation. The accuracy comparisons are plausible, the method is simple enough to be reproducible, and the negative result about saliency is likely robust. But the paper as written needs revision: soften the SoTA claim, provide at least seed-level variability, measure actual latency/FLOPs including clustering cost, and release the code.\n\nRecommendation: send it to review, don't desk reject. The negative result is important enough to engage with, and the gaps are fixable. If those fixes land, this becomes a useful contribution to the efficient-VLM literature.","headline":"A small empirical paper with a real negative result—saliency-agnostic token compression matches attention-based selection—but the SoTA overclaim and unmeasured compute overhead need fixing.","tokens_in":8989,"tokens_out":3088,"would_cite":true,"duration_ms":30261,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that averaging visual-token clusters, not attention-based selection, is the most effective training-free way to compress visual tokens in vision-language models.","keywords":["visual token compression","cluster aggregation","attention saliency","training-free inference","vision-language models","K-means++","token selection","multimodal efficiency"],"falsifier":"Run LLaVA-1.5-7B end to end on the same GPU with the full 576-token baseline, Cluster & Aggregate at 64 tokens, and an attention-based method at 64 tokens, measuring wall-clock latency and energy including the K-means++ step; if Cluster & Aggregate is not faster than the baseline, or is slower than the attention-based method at matched accuracy, the paper's efficiency claim fails. A complementary test: on a fine-grained localization benchmark, if attention-selected tokens clearly beat cluster-averaged tokens at equal retention, the paper's conclusion that saliency is not predictive would fail.","tokens_in":8043,"feed_emoji":"🗜️","tokens_out":9164,"duration_ms":86985,"temperature":0.7,"pith_summary":"The paper is trying to establish that the most effective way to compress images for vision-language models is not to decide which visual tokens matter, but to group similar tokens and average each group into one token. Its proposed method, Cluster & Aggregate, clusters visual token embeddings with K-means++ after the vision encoder and replaces each cluster with its mean embedding, requiring no fine-tuning and no importance scores. On LLaVA-1.5-7B/13B and VILA-8B, the paper reports that this method matches or outperforms earlier token-selection and merging algorithms, including attention-based ones, at the same 11% token retention. It also reports that attention-based saliency rankings are volatile across layers and barely change with the text prompt, which challenges the common assumption that attention reveals which visual tokens are semantically important. If these results hold, efficient multimodal inference would not need learned or attention-based importance scoring at all.","feed_headline":"Cluster averaging beats attention-based token selection","feed_subtitle":"A training-free method that averages visual-token clusters matches or beats finetuning-free pruning on LLaVA and VILA.","key_machinery":"The central object is the Cluster & Aggregate pipeline: after the vision encoder and the projector, visual token embeddings are grouped with K-means++ on embedding similarity, and all tokens in a cluster are replaced by one averaged embedding; the resulting aggregate embeddings are concatenated, in random order, and fed to the LLM alongside text tokens. The method works in the shared pre-LLM embedding space, requires no fine-tuning, and has no saliency score or per-layer pruning decisions. The paper contrasts this with its Algorithm 1 saliency score, a cross-modal attention measure that ranks visual tokens by the attention they receive from text tokens. That contrast carries the argument: if a mean of a cluster preserves enough information to match or beat attention-selected tokens, then attention-based importance is not doing useful work for compression.","core_discovery":"The paper's central claim, stated in its own terms, is that simple cluster-level token aggregation outperforms prior finetuning-free state-of-the-art methods for visual token selection and merging while being computationally simpler. In the paper's experiments, Cluster & Aggregate retains 64 of 576 visual tokens from LLaVA-1.5-7B and achieves higher average accuracy on eight visual question-answering benchmarks than the paper's saliency-based variants and than finetuning-free versions of VisionZip; the pattern persists at 128 and 192 tokens and on LLaVA-1.5-13B and VILA-8B. Alongside this, the paper documents two puzzling trends in attention-based saliency: the highlighted regions often do not coincide with intuitively informative image regions, and the saliency ranking of visual tokens changes very little when the text prompt changes. The authors take these observations as evidence that the visual encoding pipeline is highly redundant and that attention-based importance metrics are not reliable guides for selecting which visual tokens to keep.","pith_inferences":["Not in the paper, but a direct corollary of its retention-rate finding: replacing K-means++ with a cheaper clustering routine should preserve most of the accuracy, since the paper's own results show performance tracks the number of retained tokens more than the clustering method.","If the reported prompt-invariance of saliency generalizes, prompt-conditioned compression could be simplified to content-based compression computed once per image, with little accuracy loss.","A direct wall-clock and energy comparison that includes clustering overhead would settle whether the accuracy-versus-retention tradeoff becomes a real speedup; the paper explicitly leaves that measurement for future work."],"forward_implications":["The paper's experiments imply that attention-based importance scoring is not necessary: saliency-agnostic aggregation matches or exceeds it at equal token retention.","Performance depends more on how many tokens remain than on how they are chosen; the clustering variants show no statistically significant differences once retention is controlled.","Because Cluster & Aggregate is training-free and runs after the projector, it can be applied to any existing LLaVA-style autoregressive VLM without fine-tuning.","In some benchmarks, such as VizWiz, compressed token sequences yield higher accuracy than the full 576-token baseline, indicating redundancy and noise in the uncompressed encoding."],"supporting_citations":[{"why":"Supplies the LLaVA-1.5-7B/13B models and the 576-token visual encoding used as the primary testbeds and baseline.","marker":"[12]"},{"why":"VisionZip is the main finetuning-free state-of-the-art comparison that Cluster & Aggregate claims to outperform on average.","marker":"[20]"},{"why":"SparseVLM provides an attention-based token sparsification baseline compared across retained-token counts.","marker":"[25]"},{"why":"FastV supplies a layer-wise attention-based pruning baseline used in the same comparisons.","marker":"[3]"},{"why":"ToMe establishes the token-merging baseline inside the vision encoder against which post-projector clustering is contrasted.","marker":"[2]"},{"why":"PruMerge supplies the intuition, tested and set aside, that clustering over keys has semantic advantages.","marker":"[17]"},{"why":"LLMViewer provides the FLOPs, prefill-time, memory-access, and activation-memory estimates used to motivate token reduction.","marker":"[23]"},{"why":"VILA supplies the VILA-8B model used to show Cluster & Aggregate transfers beyond the LLaVA family.","marker":"[11]"}],"fun_headline_variants":["Cluster averaging tops attention for visual token selection","Average visual token clusters, beat attention-based pruning","Simple cluster averaging wins for visual token compression","Attention saliency unreliable; cluster merging excels","Cluster-level token averaging beats finetuning-free pruning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's efficiency claim assumes that cutting the visual token count is a valid proxy for real computation savings; since it reports only LLM-level estimates and never measures the added cost of K-means++ clustering and aggregation, high clustering overhead would break the link between the accuracy-versus-retention tradeoff and actual speedup.","fun_headline_variants_meta":{"raw":{"variants":["Cluster averaging tops attention for visual token selection","Average visual token clusters, beat attention-based pruning","Simple cluster averaging wins for visual token compression","Attention saliency unreliable; cluster merging excels","Cluster-level token averaging beats finetuning-free pruning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000542,"raw_usage":{"total_tokens":2575,"prompt_tokens":902,"completion_tokens":1673,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":518,"completion_tokens_details":{"reasoning_tokens":1604}},"tokens_in":518,"tokens_out":1673,"duration_ms":11989,"temperature":1.0,"reasoning_tokens":1604,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:29:24.579349+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run LLaVA-1.5-7B end to end on the same GPU with the full 576-token baseline, Cluster & Aggregate at 64 tokens, and an attention-based method at 64 tokens, measuring wall-clock latency and energy including the K-means++ step; if Cluster & Aggregate is not faster than the baseline, or is slower than the attention-based method at matched accuracy, the paper's efficiency claim fails. A complementary test: on a fine-grained localization benchmark, if attention-selected tokens clearly beat cluster-averaged tokens at equal retention, the paper's conclusion that saliency is not predictive would fail.","supporting_citations":[{"cited_title":"Visionzip: Longer is better but not necessary in vision language models","cited_arxiv_id":null,"evidence_quote":"VisionZip is the main finetuning-free state-of-the-art comparison that Cluster & Aggregate claims to outperform on average."},{"cited_title":"An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models","cited_arxiv_id":null,"evidence_quote":"FastV supplies a layer-wise attention-based pruning baseline used in the same comparisons."},{"cited_title":"Token merging: Your vit but faster","cited_arxiv_id":null,"evidence_quote":"ToMe establishes the token-merging baseline inside the vision encoder against which post-projector clustering is contrasted."},{"cited_title":"Llava-prumerge: Adaptive token reduction for efficient large multimodal models","cited_arxiv_id":null,"evidence_quote":"PruMerge supplies the intuition, tested and set aside, that clustering over keys has semantic advantages."},{"cited_title":"Vila: On pre-training for visual language models","cited_arxiv_id":null,"evidence_quote":"VILA supplies the VILA-8B model used to show Cluster & Aggregate transfers beyond the LLaVA family."}],"review_version":1}