{"id":"48396500-14dd-4876-98c2-8a5779e53dec","arxiv_id":"2504.19627","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A self-supervised training method that selects and merges instruction-relevant vision tokens, cutting LVLM compute while claiming stable VQA accuracy and better dense perception.","lead":"VCM trains large vision-language models to keep only the image pieces relevant to a question, cutting the number of visual tokens fed to the language model to a quarter of the original. This could make image assistants much cheaper to run while preserving answer quality, if the reported gains hold up.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Inference-time length selection is undefined: the only L formula uses the response text, yet all evaluations fix token counts; the paper's central dynamic-selection claim is unverified.","rationale":"The reader's weakest assumption (deterministic keyword-count-to-length mapping and unspecified r-to-L rule) is precisely the gap I find most load-bearing. The paper's novelty claim is a dynamic vision concept model: the number and location of concepts should depend on instructions. Yet every quantitative evaluation fixes the number of vision tokens (144, 160, 136, 128, 64), so dynamic behavior is never measured. The training-time length formula needs the response keywords, which do not exist at inference, and the 'mask ratio r' statement in Section 3.4 provides no explicit function. This is an internal recipe gap, not an outside-consensus disagreement. Secondary issues (85% vs 63-73% FLOPs from Table 10; missing 'fine-tune CLIP without VCM' baselines for Tables 2-3) are real but addressable. Because the missing inference rule and dynamic evaluation are straightforwardly fixable and the fixed-length results remain informative, CONDITIONAL is the right verdict.","tokens_in":23850,"tokens_out":6530,"duration_ms":72541,"concrete_test":"Run the released inference recipe with only the instruction available: set the response keyword count to 0 (or specify the r-to-L rule) and compute L per sample on GQA and POPE; report the distribution of L and accuracy versus the fixed-144/128 rows of Table 10. If token counts are constant or the instruction keyword count does not shift L as Figure 2 predicts, the dynamic-selection claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing weakness is that VCM's adaptive behavior, the core novelty, has no specified inference-time length rule. Section 3.3 defines the target length as L = floor(M*S*(1 - (Nkey - Nmin)/(Nmax - Nmin))) with Nkey the keyword-count difference between instruction and response, but at inference the response is absent. Section 3.4 says length is controlled 'based on the mask ratio r' without giving a map from r, or from the instruction alone, to L. In every reported experiment (Tables 1, 7, 8, 10), the model is evaluated at a fixed token count (144, 160, 136, 128, or 64), so there is no evidence that per-instance lengths were ever chosen dynamically. As written, the central claim that VCM 'dynamically determines the required vision concepts' (Section 1) cannot be tested. Appendix I acknowledges the coarse min-max estimate as a limitation, but that passage does not resolve the missing recipe.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VCM (Vision Concept Modeling), a self-supervised framework for adaptive vision token compression in LVLMs. VCM learns to select a small, instruction-dependent set of vision concepts via implicit contrastive learning: a keyword selector highlights instruction-relevant text keywords, a forward-backward dynamic programming algorithm aligns variable-length retained token segments to a target concept length, and segment merging converts selected tokens into concept-level features. The authors report strong VQA performance at reduced token counts (e.g., 144 tokens for LLaVA-1.5-7B), improved dense-perception results on region-level VQA, zero-shot classification, open-vocabulary detection, and open-vocabulary segmentation, and claim an 85% FLOP reduction. The paper also includes extensive ablations, high-resolution and video experiments, and a derivation of the gradient of the VCM loss.","tokens_in":24024,"tokens_out":4632,"duration_ms":45290,"significance":"If the reported results hold, VCM would be a valuable addition to vision token compression: it introduces a theoretically motivated forward-backward objective for variable-length concept extraction, a parallelizable segment-merging operation, and evidence of improved dense visual features. The main VQA comparison at 144 tokens is internally consistent and broadly competitive with, and in several benchmarks superior to, existing token-reduction methods. However, the paper's central claim of dynamic, instruction-dependent length selection is not actually tested, because the inference-time mapping from instruction to token count is never specified and all experiments use fixed token budgets. The headline 85% FLOP reduction is contradicted by the paper's own Table 10 (63% at 128 tokens, 73% at 64 tokens). The dense-perception improvements lack a control that isolates VCM from additional instruction fine-tuning and possible encoder unfreezing. The information-domain scalar S is tuned on the same benchmarks used for evaluation. These issues are load-bearing for the paper's main contributions and require substantial revision before the results can be considered established.","major_comments":[{"comment":"The inference-time length selection is not specified. The target length in Section 3.3 is L = floor(M*S*(1 - (Nkey - Nmin)/(Nmax - Nmin))), where Nkey is the keyword-count difference between instruction and response; at inference the response is absent. Section 3.4 states that the model can 'precisely output vision concepts of different lengths based on the mask ratio r', but no formula or algorithm maps r (or the instruction alone) to a concrete L. All reported experiments (Tables 1, 7, 8, 10) evaluate fixed token counts (144, 160, 136, 128, or 64), so the central claim that VCM 'dynamically determines the required vision concepts' (Section 1) is not tested. Appendix I acknowledges the coarse min-max estimation as a limitation, but this does not resolve the missing inference recipe. The authors should specify the inference-time length rule and evaluate instances with per-instruction variable lengths.","section":"§3.3, §3.4"},{"comment":"The claim of '85% fewer FLOPs for LLaVA-1.5-7B' is contradicted by the paper's own measurements. Table 10 reports FLOPs of 4.62T for LLaVA-v1.5, 1.71T at 128 vision tokens, and 1.24T at 64 vision tokens, which correspond to reductions of approximately 63% and 73%, respectively. The asymptotic derivation in Appendix D gives a ratio near 3/25 (an 88% reduction) by assuming the sequence length is scaled by 1/8, but the experimental token counts (e.g., 576 to 128, or 2880 to 160 in Table 7) do not match that assumption. The theoretical and empirical efficiency numbers must be reconciled, and all claims should be stated to match the measured values.","section":"Abstract; Table 10; Appendix D"},{"comment":"The dense-perception improvements are confounded with additional fine-tuning and possible image-encoder unfreezing. Table 2 compares LLaVA-v1.5 without VCM to LLaVA-v1.5 with VCM, but the VCM variant undergoes additional instruction fine-tuning, and Table 1 shows that unfreezing the image encoder ('*') is itself a design choice. Without a control that applies the same training recipe (including any unfreezing and the same instruction data) minus the VCM loss, the gains on RefCOCO region-level VQA, COCO panoptic classification, and open-vocabulary detection/segmentation cannot be attributed to VCM. Please add such a no-VCM fine-tuned baseline.","section":"Tables 2 and 3; §4.3"},{"comment":"The information-domain scalar S is selected by ablation on the same four benchmarks (SciQA, VizWiz, POPE, MME) that appear in the main evaluation in Table 1. This constitutes model selection on the evaluation set, so the reported gains on those benchmarks partially reflect tuning rather than an intrinsic property of VCM. The paper should either fix S on a separate validation split or derive S from a criterion that does not involve the evaluation benchmarks.","section":"Table 4; §4.4"},{"comment":"The keyword selector in Eq. (2) uses both instruction tokens HI and response tokens HR: K = Softmax(MHSA([HI; HR])(GV)^T). Since the response is unavailable at inference time, the manuscript does not describe how keyword selection is performed at inference, nor how the mask ratio r is derived in practice. This is an integral part of the claimed dynamic behavior, not a minor implementation detail, and it must be specified together with the length-selection rule.","section":"§3.2, Eq. (2)"}],"minor_comments":[{"comment":"The caption uses 'KMeans' while the text uses 'K-Means'; please unify the spelling.","section":"Figure 4 caption"},{"comment":"The approximation that discards d*σ_n^2/32 and d^3/512 in the denominator is not justified in general; the variance σ_n^2 can be substantial, so the reported ratio should state the regime in which the approximation holds.","section":"Appendix D, Eq. (13)"},{"comment":"There are duplicated references (e.g., GUI-R1 appears as [5] and [61]; MiniGPT-4 as [7] and [70]; MQT-LLaVA as [12] and [59]); these should be consolidated.","section":"References"},{"comment":"Algorithm 1 uses the term 'Norm(Ntext)' without defining it in the pseudocode; please define it consistently with Eq. (4) in Section 3.3.","section":"Algorithm 1"},{"comment":"The equality p(ZV|YV) = Σ_l α(t,l)β(t,l) for all t is a standard CTC property, but it is stated without a derivation; citing or briefly deriving this property would improve accessibility.","section":"Eq. (4)"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid core idea and the forward-backward derivation is plausible, but the central dynamic-selection claim is as yet unverified because inference-time length selection is unspecified and all experiments use fixed token counts. The FLOP claim in the abstract is overstated relative to Table 10, and the dense-perception tables lack the appropriate no-VCM fine-tuned control. I would ask the authors to supply the missing inference rule, reconcile the efficiency numbers, and add the control baseline. The choice of S on the evaluation benchmarks is a serious methodological concern that should also be addressed. These are fixable within the scope of a revision, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nVCM is a genuinely fresh mechanism for instruction-guided vision token compression. The forward-backward alignment over variable-length concept sequences, with segment merging, is not in the cited pruning/merging baselines (FastV, SparseVLM, VisionZip, MQT-LLaVA), and the gradient derivation in Appendix C checks out. The VQA results at 144 tokens are competitive with LLaVA-1.5's full 576, and the parallelized segment-merging pseudocode is a concrete engineering contribution. So the paper deserves a serious referee.\n\nBut the main claim of the paper is not actually tested. The target length L is defined from the difference in keyword counts between the response and the instruction. At inference the response does not exist. Section 3.4 says the mask ratio r gives a handle to control length, but no formula or lookup maps r to L. Every experiment in the paper runs at a fixed token budget (144, 160, 136, 128, or 64). So the headline property, that VCM dynamically determines the required number of vision concepts per instruction, has no supporting evaluation. That is a load-bearing gap, not a presentation nit. The authors can fix it by specifying an inference-only estimator (say, instruction keywords alone) and evaluating with per-instance variable lengths.\n\nSecond, the abstract's '85% fewer FLOPs' is contradicted by Table 10, which shows 63% to 73% reduction on LLaVA-1.5-7B. The 85% figure comes from a theoretical Appendix D calculation that assumes the total sequence length drops by 1/8, which ignores the text tokens that dominate after compression. Use the measured numbers.\n\nThird, the dense-perception gains in Tables 2 and 3 lack the right control: compare against fine-tuning the same CLIP encoder on the same instruction data without the VCM loss. Without that, the improvement could be ordinary fine-tuning, not the proposed mechanism. This is fixable and should be minor in revision.\n\nFinally, the information-domain scalar S is chosen by ablation on four VQA benchmarks that also appear in the main table; that is selection on the evaluation set. Not disqualifying, but it should be disclosed or tuned on a held-out split.\n\nAs it stands, the paper is a promising mechanism wrapped in a central unverified claim. I would send it to review, with a clear request to fix the inference recipe and rerun the main experiments under variable lengths. A reader working on token reduction will learn something from the formulation even if the current numbers are provisional.\n\nBest.","headline":"VCM offers a genuinely new CTC-style token-selection mechanism, but the paper never specifies inference-time length choice, leaving its core adaptivity claim untested.","tokens_in":24625,"tokens_out":3434,"would_cite":true,"duration_ms":35216,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"VCM lets a vision-language model answer questions from a small set of instruction-selected vision concept tokens, cutting FLOPs by up to 85%.","keywords":["vision concept model","vision token compression","large vision-language models","instruction fine-tuning","implicit contrastive learning","forward-backward algorithm","dynamic programming","open-vocabulary perception"],"falsifier":"Run VCM's length estimator on a held-out set of VQA pairs by computing the keyword-count difference from each instruction, predicting the concept length, then sweeping a token-pruning tool over many lengths with an evaluator to find the true minimum; if the formula's prediction is no better than a constant or random baseline, the load-bearing mapping is false. A cheaper check is to replace the formula with the oracle response-derived length and show the gains vanish.","tokens_in":1862,"feed_emoji":"🖼️","tokens_out":2344,"duration_ms":86067,"temperature":0.7,"pith_summary":"The paper tries to establish a new operating principle for large vision-language models: instead of feeding an LLM all of an image's vision tokens, a model can be taught to model visual concepts and keep only the tokens that a given instruction actually needs, along with their spatial locations. VCM is a self-supervised framework that learns this without concept-level labels, by exploiting the correlation between text keywords and the minimum number of vision tokens required to answer correctly. If it works as claimed, LVLMs become cheaper to train and run, and their vision encoders become better at dense perception tasks such as open-vocabulary detection and segmentation. The sympathetic reading is that VCM replaces a fixed token budget with an instruction-dependent one, and makes the pruning itself semantically meaningful.","feed_headline":"VCM cuts LLaVA-1.5 vision FLOPs by 85% while holding accuracy","feed_subtitle":"The method keeps only the image concepts a question needs, and the fine-tuning also sharpens the encoder's dense perception.","key_machinery":"The load-bearing object is a trainable vision-concept selector attached between the vision encoder and the LLM. It computes a per-token 'retain vs blank' probability, estimates the number of concepts to keep from the text prior, and optimizes the chosen alignment with a forward-backward algorithm: the target concept sequence is padded with blanks, and the loss is the negative log probability of all feasible alignments, $\\mathcal{L}_{VCM} = -\\log p(Z^V \\mid Y^V)$, with gradient $\\partial \\mathcal{L}_{VCM}/\\partial y^V_t(z^V_l) = p(z^V_l \\mid y^V_t) - \\gamma(t,l)$. The segment-merging (SM) operation then compresses each retained run of tokens into one weighted-average concept feature, preserving position information. The keyword selector, trained with a semantic-alignment loss during pretraining, provides the text prior that drives both the length estimate and the per-token scores.","core_discovery":"VCM claims that the right unit of vision processing for LVLMs is not the individual patch token but the 'vision concept': a contiguous, spatially grounded segment of tokens whose count and position should be selected by the instruction. The framework learns this selection in two stages: a pretraining stage aligns global text, vision and language-model features through a keyword selector and a semantic alignment loss, and an instruction fine-tuning stage trains a binary classifier to mark each vision token as retained or blank. The target length for retained concepts is estimated from the keyword-count difference between response and instruction by the formula $L = \\lfloor M S (1 - (N_{\\text{key}} - N_{\\min})/(N_{\\max} - N_{\\min})) \\rfloor$ with $N_{\\max}=10$, $N_{\\min}=-35$, $S=1/4$. A forward-backward dynamic program over an extended target sequence of retained and blank symbols supplies a differentiable loss whose gradient is $p(z \\mid y) - \\gamma$, and a parallel segment-merging operation turns the chosen path into weighted-average concept features for the LLM. The paper's stated result is that this reduces computational cost (85% fewer FLOPs for LLaVA-1.5-7B) while matching or improving VQA scores, and that the VCM-trained CLIP encoder transfers to zero-shot classification, open-vocabulary detection, and semantic segmentation.","pith_inferences":["A testable extension the paper does not pursue is to replace the response-dependent length formula with a predictor that sees only the instruction at inference; the paper substitutes the mask ratio r for the absent response but never specifies the mapping, so an instruction-only length network is the obvious next step.","Because the VCM-trained CLIP encoder improves F-VLM and Cat-Seg, the same fine-tuning step could plausibly be used as a drop-in upgrade for any frozen-ViT system, including retrieval or few-shot classifiers that never generate text.","The 85 percent FLOP reduction is computed from sequence-length asymptotics, so real wall-clock gains will vary by hardware and implementation; the paper's parallel segment-merging speedup is necessary for the benefit to materialize in practice."],"forward_implications":["Instruction-conditioned token budgets work: a model can drop from 576 to 128-144 vision tokens for LLaVA-1.5 while keeping a higher average across 11 VQA benchmarks than the full-token baseline.","The same trained vision encoder transfers to dense tasks: VCM improves CLIP ViT's K-Means grouping, zero-shot classification on COCO panoptic masks, and open-vocabulary detection and segmentation, e.g. $AP_{50}^{\\text{novel}}$ on OV-COCO rises from 9.2 to 12.6 with ViT-L/14.","The gain carries to high-resolution and video settings: with LLaVA-NeXT, VCM keeps only 160 of 2880 tokens and stays close to the full model; with Video-LLaVA, 136 tokens beat token-reduction baselines at similar token counts.","The framework scales with data and model size: increasing training steps or moving to a 13B LLM improves VQA averages under the same 144-token budget.","The method is architecture-general: Qwen2-VL keeps most of its performance with 576 of 1326 vision tokens, roughly a 57% reduction."],"supporting_citations":[{"why":"VisionZip supplies the token-length sweep used to define the minimum required vision length, and provides the strongest token-reduction baseline the method must beat.","marker":"[14]"},{"why":"LLaVA-1.5 is the base architecture, and its instruction-tuning dataset provides the 5K samples and training data; the main claims are measured against it.","marker":"[18]"},{"why":"FastV is the main inference-acceleration baseline and the source of the FLOP-analysis framing that motivates reducing vision tokens.","marker":"[17]"},{"why":"SparseVLM is a token-sparsification baseline used for performance and FLOP comparisons across benchmarks.","marker":"[13]"},{"why":"MQT-LLaVA is a variable-length query-compression baseline that the paper argues is not a vision concept model because it lacks spatial grounding.","marker":"[12]"},{"why":"LLaVA-NeXT supports the high-resolution generalization experiment where VCM keeps only 160 of 2880 tokens.","marker":"[1]"},{"why":"F-VLM provides the frozen-CLIP object-detection setup onto which the VCM-trained encoder is transplanted for OV-COCO evaluation.","marker":"[50]"},{"why":"Cat-Seg provides the cost-aggregation segmentation setup for evaluating the VCM-trained encoder on ADE-150 and ADE-847.","marker":"[52]"}],"fun_headline_variants":["VCM trims vision FLOPs 85% by processing only needed concepts","Vision concept model cuts LLaVA FLOPs 85%, boosts encoder","Teach LVLMs to see concepts, not tokens: 85% fewer FLOPs","VCM: select vision concepts per question, slash compute 85%","Implicit concept learning makes LVLMs 85% cheaper, smarter"],"cache_read_input_tokens":26752,"weakest_assumption_plain":"The whole method rests on one empirical mapping: the minimum number of vision tokens an answer needs is a fixed function of the keyword-count difference between instruction and response, with constants chosen from a single 5,000-example study, and this same mapping is used at inference even though the response is absent.","fun_headline_variants_meta":{"raw":{"variants":["VCM trims vision FLOPs 85% by processing only needed concepts","Vision concept model cuts LLaVA FLOPs 85%, boosts encoder","Teach LVLMs to see concepts, not tokens: 85% fewer FLOPs","VCM: select vision concepts per question, slash compute 85%","Implicit concept learning makes LVLMs 85% cheaper, smarter"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000223,"raw_usage":{"total_tokens":1506,"prompt_tokens":1041,"completion_tokens":465,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":657,"completion_tokens_details":{"reasoning_tokens":362}},"tokens_in":657,"tokens_out":465,"duration_ms":4709,"temperature":1.0,"reasoning_tokens":362,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:47:58.500035+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run VCM's length estimator on a held-out set of VQA pairs by computing the keyword-count difference from each instruction, predicting the concept length, then sweeping a token-pruning tool over many lengths with an evaluator to find the true minimum; if the formula's prediction is no better than a constant or random baseline, the load-bearing mapping is false. A cheaper check is to replace the formula with the oracle response-derived length and show the gains vanish.","supporting_citations":[{"cited_title":"An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models","cited_arxiv_id":null,"evidence_quote":"FastV is the main inference-acceleration baseline and the source of the FLOP-analysis framing that motivates reducing vision tokens."},{"cited_title":"Cat-seg: Cost aggregation for open-vocabulary semantic segmentation","cited_arxiv_id":null,"evidence_quote":"Cat-Seg provides the cost-aggregation segmentation setup for evaluating the VCM-trained encoder on ADE-150 and ADE-847."}],"review_version":1}