{"id":"d3163f0e-ade8-447a-b014-936494736bf2","arxiv_id":"2411.17773","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A new grouping mechanism, VisToG, uses semantic tokens and isolated attention inside a pretrained CLIP vision encoder to compress visual tokens for MLLMs, retaining roughly 98% performance while cutting inference time by over 27%.","lead":"VisToG is a method that cuts the number of image tokens fed to a multimodal language model from 576 to 128 or 64 by grouping similar image patches inside a pretrained vision encoder. It keeps about 98% of the original accuracy while reducing inference time by over 27%, which could make vision-language models cheaper to run.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No evidence that VisToG's learned token groups are semantically structured; a same-token-count learned-pooling control is needed before the 'grouping' claim can be separated from generic token reduction.","rationale":"The strongest claim is an efficiency/accuracy trade-off, but the paper's evidence conflates token reduction with semantic grouping. The reader's identified discrepancy between abstract 98.1% and Table 2's 97.5% is real but not fatal because Table 2 explicitly covers only four datasets; using all six Table 1 numbers gives about 98.1% under Eq. (6). The more under-supported step is the causal story: VisToG retains performance because semantic tokens recover meaningful segments. No measurement of the assignment matrix, no segmentation-quality evaluation, and no direct same-token-count control against non-semantic pooling is reported; LLaVA-AvgPool appears only in figures and qualitative comparisons. Since the training signal is a single LLM loss and the Gumbel assignments are unconstrained, degenerate uniform pooling is a plausible alternative explanation. This does not invalidate the paper's empirical result, but it makes the contribution's novelty and the interpretation of the PRT conditional on an unverified mechanism. A matched AvgPool control is the minimal experiment that would settle it.","tokens_in":11375,"tokens_out":12901,"duration_ms":126208,"concrete_test":"Retrain the LLaVA-AvgPool baseline at 128 output tokens under the exact Stage-2 recipe used for VisToG (same 624K data, same LLM, same training budget), and compare per-benchmark PRT and average PRT against VisToG in Table 1. If the average PRT gap is smaller than run-to-run noise (or less than about 1 point), the central 'grouping' claim is not supported; if VisToG beats AvgPool by a clear margin, the grouping mechanism is validated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The manuscript's stated contribution (Sec. 3.2) is that learnable semantic tokens group image patches into semantic segments without mask supervision, and Sec. 1 motivates this by arguing that random token sampling only works when every semantic segment is covered. But VisToG is never shown to produce semantic groups. The aggregate PRT figures (Tables 1-2) are consistent with any learned token-reduction mechanism: LLaVA-AvgPool, a non-semantic adaptive average pooling baseline, is mentioned in Sec. 4.4 and Fig. 5 as lying between VisToG and random, but no side-by-side table at the same 128-token count and training budget is given. Because Eqs. (2)-(4) are supervised only by the LLM's next-token loss via the straight-through Gumbel assignment, the rows of the assignment matrix could collapse to near-uniform or input-agnostic groupings; the claimed 'eliminate redundant visual tokens utilizing the prior knowledge of the pre-trained vision encoder' would then not be the operative mechanism. This is load-bearing for the central claim because 'maintains 98.1% via visual token grouping' is not established if a generic pooling baseline at equal token count matches the PRT.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes VisToG, a method to reduce the number of visual tokens fed to a multimodal large language model (MLLM). VisToG inserts learnable semantic tokens into a frozen CLIP ViT encoder, uses an isolated attention mask to keep image patch tokens unaffected, and applies a Gumbel-softmax grouping layer that assigns patch tokens to semantic groups and merges them into a smaller set of tokens before the visual projector. Training is done in two stages: first the visual connector is aligned on image-text pairs without grouping, then the grouping layer, connector, and LLM are fine-tuned on visual instruction data. The paper evaluates VisToG on GQA, ScienceQA, TextVQA, POPE, MME, and MMB, reporting that it maintains 98.1% of the LLaVA-1.5 performance while reducing inference time by over 27%.","tokens_in":11579,"tokens_out":8656,"duration_ms":71023,"significance":"If the reported results hold, VisToG offers a practical way to cut MLLM inference cost with a modest performance drop, and the use of a reproduced LLaVA-1.5 baseline is a strength that makes the main comparisons fair. However, the key mechanistic claim—that the benefit comes from semantic grouping rather than from generic learned token reduction—is not yet supported, because no matched learnable-pooling control is reported. The claimed PRT and inference-time numbers also need clarification. The contribution is incremental but potentially useful; the paper contains falsifiable predictions that the grouping structure matters, which additional experiments can test.","major_comments":[{"comment":"The abstract and conclusion claim that VisToG maintains 98.1% of the original performance, but Table 2, which defines PRT over GQA, TextVQA, POPE, and MME, reports 97.5% for the 128-token model. The 98.1% figure matches an average over all six datasets in Table 1 (including SQA and MMB), so the manuscript is not consistent about which datasets the headline PRT uses; please state the exact set and report the corresponding value.","section":"Abstract, Table 2, Sec. 4.3"},{"comment":"Equation (4) defines the merged token as VISi = Semi + Wo * (Σ_j \\hat A_{i,j} Wv Img_j) / (Σ_j \\hat A_{i,j} Wv). The denominator is a sum of projected vectors, not a scalar, so the normalization is ill-defined; with the one-hot assignment from Eq. (3) the expression degenerates. Please correct the denominator to a scalar normalization (e.g., Σ_j \\hat A_{i,j}) or provide the intended operation, because this equation is the core of the grouping mechanism.","section":"Sec. 3.2, Eq. (4)"},{"comment":"The paper's central claim is that VisToG works by grouping image patches into semantic segments, but no controlled baseline with a learnable pooling mechanism (e.g., a learned attention-pooling or cross-attention layer with the same number of output tokens and the same training budget) is provided. LLaVA-AvgPool is a non-learned adaptive average pooling and is compared only on GQA and POPE in Fig. 5, not on the full benchmark suite at 128 tokens. Without such a control, the results are consistent with any learned token-reduction mechanism, and the specific contribution of semantic grouping is not established.","section":"Sec. 4.4, Fig. 5"},{"comment":"The claimed 'over 27% inference time reduction' is not supported with sufficient detail: the paper does not report the actual inference time of VisToG, nor the measurement protocol (batch size, number of generated tokens, warmup, input resolution, prefill vs. decode time). Please provide the raw times in a table and describe the protocol so the efficiency claim can be reproduced.","section":"Sec. 4.4, Eq. (7)"},{"comment":"No variance or number of seeds is reported for any result, including the stochastic components (Gumbel noise in grouping and random token sampling for LLaVA-rand). The paper itself notes in Sec. 4.3 that LLaVA-rand has 'high variance,' so reporting mean and standard deviation across multiple runs is necessary to assess the significance of the differences, especially the small gaps between VisToG and the baseline.","section":"Sec. 4.3, Table 1"},{"comment":"The LLaVA-1.5-Q-Former row lists #Tokens=576, while the text in Sec. 4.2 states that the number of queries is set to 64. If Q-Former is used to compress tokens, the token count should be 64; if it is 576, the comparison is not a compression setting. Please correct this inconsistency and ensure the comparison is at the claimed token count.","section":"Table 1"}],"minor_comments":[{"comment":"'Large Lanugae Model' is a typo for 'Large Language Model'.","section":"Sec. 3.1"},{"comment":"Fig. 3 visualizes LLaVA-rand, not VisToG; adding a visualization of VisToG's learned group assignments would directly support the semantic grouping claim.","section":"Fig. 3"},{"comment":"Table 2 reports PRT only for the 128-token VisToG; the corresponding 64-token PRT can be inferred from Table 1 but should be reported explicitly for completeness.","section":"Table 2"},{"comment":"Eq. (2) is called a Gumbel-Softmax operation, but Eq. (3) uses a one-hot straight-through estimator; please clarify the relationship between the soft assignment used for gradients and the hard assignment used for the merged output.","section":"Sec. 3.2, Eq. (2)-(3)"},{"comment":"The notation t_ij/|D_i| in Eq. (7) is unclear; specifying the averaging over samples and datasets would improve reproducibility.","section":"Sec. 4.4, Eq. (7)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript addresses a timely problem and the reproduced LLaVA baseline is a strength. However, the current evidence does not yet separate the semantic-grouping mechanism from generic learned pooling, and the headline PRT is not consistent with Table 2. I recommend a major revision that adds the controlled baseline, corrects Eq. (4), and clarifies the efficiency measurement. The paper is within the journal's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful paper on reducing visual token counts in MLLMs without wrecking accuracy. The method is a clean adaptation of GroupViT's token grouping to the MLLM setting, with two genuinely nice touches: isolated attention that leaves the pretrained image tokens untouched, and a two-stage training scheme that makes grouping instruction-aware. The experiments are honestly run against a reproduced LLaVA-1.5 baseline, and the 27% inference speedup at 128 tokens with roughly 97-98% retained performance on most benchmarks is a real plus.\n\nWhere I'd push back: the abstract says 98.1% PRT, but Table 2 lists 97.5% for 128 tokens. That turns out to be because Table 2 averages only GQA/TextVQA/POPE/MME while the 98.1% uses all six benchmarks; the paper never states this, which is sloppy. More substantive: no error bars or seeds are reported anywhere, despite the paper itself noting that random token sampling has high variance. That matters for a method whose whole point is robust reduction. And the TextVQA drop (57.3 to 54.5) is the largest and is exactly the place where token grouping should be most fragile, so the 'maintains performance' claim needs qualification.\n\nOn the stress-test concern: I think it is partially right. The paper never shows that the learned groups are semantically coherent. Figure 5 does include LLaVA-AvgPool as a same-token-count control, and VisToG beats it on GQA and POPE, which suggests grouping does more than generic pooling. But a figure without numbers, across only two benchmarks, is weaker than a proper table, and the paper provides no visualization of the learned assignment matrix. A referee should ask for that. Still, the central empirical result does not depend on proving the groups are semantic; it depends on whether the method retains performance, and the evidence for that is reasonably solid.\n\nBottom line: worth a serious referee. If I were editing, I'd send it out with requests for error bars, a table-form comparison to AvgPool at matching token counts and training budget, and a clearer statement of which benchmarks the abstract's 98.1% is computed over. Then it would be a solid acceptance.","headline":"A solid, incremental token-reduction method for MLLMs; the headline PRT is slightly overstated and the 'semantic grouping' claim outruns the evidence, but the core result and ablations are worth a serious look.","tokens_in":12160,"tokens_out":3981,"would_cite":true,"duration_ms":34577,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that grouping similar image patches into semantic tokens preserves 98.1% of a vision-language model's accuracy while cutting inference time by over 27%.","keywords":["multi-modal large language models","visual token compression","visual token grouping","inference acceleration","vision transformer","semantic tokens","Gumbel-Softmax assignment","instruction-aware grouping"],"falsifier":"Run VisToG and the full 576-token baseline on a benchmark of dense small text or fine-grained detail, such as a page with many tiny labels, and compare per-image accuracy: if grouping drops the answer region into a large averaged group and the model fails on questions the baseline answers, the central claim that grouping is near-lossless for instruction-relevant detail is falsified.","tokens_in":11149,"feed_emoji":"⚡","tokens_out":6031,"duration_ms":49598,"temperature":0.7,"pith_summary":"VisToG is a visual-token compression method for multimodal large language models that tries to cut inference cost without retraining the vision encoder and without segmentation masks. The paper's claim is that adding a small set of learnable semantic tokens inside the frozen CLIP vision encoder lets the model group similar image patches, merge each group into a single token, and still answer questions nearly as well as the full 576-token model. On six standard benchmarks, the 128-token version retains 98.1% of the baseline's average performance while reducing inference time by more than 27%. The reason this matters is that the language model's attention cost grows with the number of input tokens, so cheaper, smaller visual token sets make high-resolution and video inputs more practical.","feed_headline":"Visual token grouping cuts MLLM inference time 27% at 98.1% accuracy","feed_subtitle":"A grouping layer inside the vision encoder merges similar image patches, using 128 tokens where 576 were needed.","key_machinery":"The load-bearing mechanism is the grouping layer placed inside the vision encoder, together with an isolated-attention mask. The grouping layer introduces N learnable semantic tokens, computes a similarity matrix between semantic tokens and image patch tokens, applies a Gumbel-Softmax with a straight-through estimator to make the one-hot assignment differentiable, and forms each output token as a weighted sum of the patches assigned to its group, added to the semantic token. The mask M is set to False whenever an image token attends to a semantic token, so the frozen image representations are unchanged while semantic tokens can aggregate information from all patches. This design transfers the clustering burden to the pretrained encoder's prior knowledge and makes the grouping trainable under instruction supervision in the second training stage.","core_discovery":"On the paper's own terms, the central discovery is that the redundancy of image tokens can be exploited inside the pretrained vision encoder rather than in the downstream connector. VisToG concatenates N learnable semantic tokens with the image patch tokens before the transformer layers of a frozen CLIP encoder, uses a Gumbel-Softmax assignment to softly allocate each patch to one semantic group, and then replaces each group by a weighted average of its patches, producing 128 (or 64) visual tokens instead of 576. An isolated-attention mask forbids the original image patches from attending to the semantic tokens, so the pretrained representations stay intact while the semantic tokens learn to collect patches into instruction-relevant groups during visual instruction tuning. With this mechanism, the paper reports the 128-token model keeping average performance at 98.1% of the reproduced LLaVA-1.5 baseline and cutting inference time by over 27%, and the 64-token model staying close behind. The largest observed drop is on TextVQA, where fine-grained text reading suffers, which the paper treats as a sign of redundancy limits rather than a refutation.","pith_inferences":["Editorial inference: the same grouping layer could be applied to video by treating frames as additional patches or by inserting a temporal grouping step, since the paper explicitly leaves video experiments for future work.","Editorial inference: if the TextVQA drop reflects a real limitation, a hybrid token allocation that keeps fine-grained tokens for high-detail regions and groups only low-detail background would likely recover much of the loss while keeping most of the speed gain.","Editorial inference: the method's reliance on CLIP's patch embeddings suggests it will work best on images whose semantic structure matches CLIP's image-text alignment; domain-shifted inputs such as medical or satellite imagery may need re-trained semantic tokens."],"forward_implications":["If the claim holds, visual token budgets for MLLMs can be cut by roughly a factor of 4 (576 to 128) with only a small benchmark-quality loss, making longer inputs such as high-resolution images more affordable.","Because the grouping layer is trained during instruction tuning, the token groups become task-dependent rather than fixed, so the same image can be compressed differently for different questions.","The isolated-attention result implies that adding auxiliary tokens to a frozen vision transformer does not have to disturb its original representations, a general recipe for injecting learnable structure into pretrained encoders.","The 64-token variant's near-parity on most benchmarks suggests the effective information content of many images is far below 576 tokens, and that the bottleneck is coverage of semantic regions rather than sheer token count."],"supporting_citations":[{"why":"supplies the semantic-token grouping design (learnable tokens, assignment) that VisToG inserts into the vision encoder.","marker":"[28]"},{"why":"provides the LLaVA architecture, two-stage training recipe, and reproduced 576-token baseline against which all retention numbers are measured.","marker":"[17]"},{"why":"supplies the frozen CLIP vision encoder whose pretrained language-aligned representations are the prior knowledge the grouping exploits.","marker":"[20]"},{"why":"provides the straight-through estimator that makes the one-hot Gumbel-Softmax assignment differentiable.","marker":"[25]"},{"why":"is the DeCo baseline using adaptive average pooling that VisToG compares against on retain rate and inference time.","marker":"[30]"},{"why":"is the VoCo-LLaMA baseline at 128 tokens that VisToG compares against in the main results table.","marker":"[31]"}],"fun_headline_variants":["Grouping vision tokens inside encoder slashes MLLM cost 27%","VisToG: 4x fewer visual tokens, 27% faster, 98.1% accurate","Group similar patches, cut MLLM inference time 27%","MLLM speeds up 27% by merging similar visual tokens"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that a frozen, pre-trained vision encoder can split an image into meaningful semantic groups using only a few learned semantic tokens, with no segmentation labels, and that replacing every patch in a group by the group average leaves intact whatever detail the question needs.","fun_headline_variants_meta":{"raw":{"variants":["Grouping vision tokens inside encoder slashes MLLM cost 27%","VisToG: 4x fewer visual tokens, 27% faster, 98.1% accurate","Group similar patches, cut MLLM inference time 27%","MLLM speeds up 27% by merging similar visual tokens"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000195,"raw_usage":{"total_tokens":1380,"prompt_tokens":988,"completion_tokens":392,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":604,"completion_tokens_details":{"reasoning_tokens":306}},"tokens_in":604,"tokens_out":392,"duration_ms":4054,"temperature":1.0,"reasoning_tokens":306,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:19:51.176236+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run VisToG and the full 576-token baseline on a benchmark of dense small text or fine-grained detail, such as a page with many tiny labels, and compare per-image accuracy: if grouping drops the answer region into a large averaged group and the model fails on questions the baseline answers, the central claim that grouping is near-lossless for instruction-relevant detail is falsified.","supporting_citations":[{"cited_title":"Groupvit: Semantic segmentation emerges from text supervision","cited_arxiv_id":null,"evidence_quote":"supplies the semantic-token grouping design (learnable tokens, assignment) that VisToG inserts into the vision encoder."},{"cited_title":"Visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"provides the LLaVA architecture, two-stage training recipe, and reproduced 576-token baseline against which all retention numbers are measured."}],"review_version":1}