{"id":"c7b48d00-56c9-46fc-ba80-447cb482d901","arxiv_id":"2607.16326","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A training-free, text-driven pre-LLM pruning method that keeps ~99% of LVLM performance while retaining only ~11-22% of visual tokens.","lead":"CRISP speeds up vision-language models by discarding most image tokens before the language model runs, using the text question to decide which image regions to keep. It reports preserving roughly 97-100% of baseline performance while cutting compute by 2-3x, making large multimodal models cheaper to deploy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cosine-relevance proxy in Eq. (2) is unvalidated; if it fails, CRISP's 'text-driven' claim is unsupported.","rationale":"This concern is load-bearing because the central claim of the paper is that CRISP performs 'text-driven' pre-LLM pruning, and this claim rests entirely on the assumption in Eq. (2) that cosine similarity in the shared embedding space reveals query-relevant visual tokens. If that assumption is false, Stage 1 amounts to selecting tokens by an arbitrary projection-dependent similarity, and the method's reported edge over text-agnostic baselines (e.g., FasterVLM, DivPrune) cannot be attributed to textual guidance. The ablation in Table III shows that removing Stage 1 degrades performance, but it does not isolate the role of the specific cosine ranking; a random or scrambled word embedding might produce a similar drop. The paper provides no calibration of the cosine scores against human judgments, ground-truth attention, or object locations, and it gives no statistics on how often the noun-less fallback is triggered. Other issues—alpha tuned on the same benchmarks, AVG% excluding MM-Vet, and the efficiency claim of '>2x' being only true for decoding speed—are real but secondary; they affect the strength of the reported numbers, not the conceptual core. A simple scrambling test would settle whether the text-driven mechanism is genuine. If it fails, the paper's contribution is significantly weakened, so the reader's conditional verdict should remain until this is tested.","tokens_in":11073,"tokens_out":7515,"duration_ms":69504,"concrete_test":"Run CRISP on LLaVA-1.5-7B at a 128-token budget with the extracted noun embeddings e_j replaced by embeddings of random words (or by a random permutation of the original e_j across words), keeping all other components fixed. If the average performance retention (AVG%, excluding MM-Vet per the paper's convention) stays within ~1 point of the reported 99.5% on GQA, MME, POPE, TextVQA, and MMB, then the cosine selection is not actually using text semantics, and the 'text-driven' claim is unsupported. If retention drops significantly, the mechanism is validated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that cosine similarity in Eq. (2) between projected visual tokens and word-level embeddings is a valid measure of visual-textual relevance. The paper asserts (§III-B.2) that both lie in the same semantic space, but a shared input embedding space does not guarantee that cosine ranking aligns with answer-relevant image regions. Visual tokens are patch-level features; word embeddings are trained for language modeling. Their geometries may be incompatible, and high cosine could be driven by generic visual statistics or low-level correlations rather than semantic grounding. If this proxy is weak, Stage 1 selects tokens by an unvalidated heuristic, and the claimed advantage of 'text-driven' pre-LLM pruning collapses into a text-agnostic or spurious selection. The ablation (Table III) shows that removing Stage 1 hurts, but that only proves that some word-dependent selection helps, not that the cosine ranking is semantically meaningful. The paper also provides no statistics on how often the noun-extraction fallback (CLS attention) is triggered, and it does not calibrate the cosine scores against ground-truth relevance or full-model attention.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"CRISP is a training-free, pre-LLM visual token pruning method. It first extracts nouns from the textual query, averages their LLM word embeddings, and scores projected visual tokens by cosine similarity to these word embeddings (Eq. 2). Per-word quotas are allocated using CLS attention from the vision encoder, and the top tokens per word form a 'critical' set. A second diversity stage iteratively adds tokens with minimal maximum cosine similarity to the current set until the token budget K is reached. Experiments on LLaVA-1.5-7B and LLaVA-NeXT-7B at budgets of 192/128/64 and 320 tokens compare CRISP with 10+ baselines, reporting average retention rates up to 99.8% and more than 2x decoding speedup.","tokens_in":11344,"tokens_out":9464,"duration_ms":89942,"significance":"Text-driven pruning before the LLM is a sensible and useful design point: it avoids the text-agnostic nature of prior pre-LLM heuristics and the efficiency cost and noisy attention of intra-LLM pruning. The paper provides broad comparisons on two LVLMs, ablations of both stages, and efficiency measurements. If the reported retention rates are robust, CRISP would be a practical plug-and-play module. The main weaknesses are a selectively averaged headline metric, a hyperparameter selected on the evaluation benchmarks, and an unvalidated cosine-relevance assumption; these are addressable but currently make the central empirical claim stronger than fully supported.","major_comments":[{"comment":"The headline 'AVG.%' is undefined and silently excludes MM-Vet. From the numbers in Table I, CRISP at 128 tokens has per-benchmark retention rates whose mean over the eight non-MM-Vet benchmarks is 99.5%, matching the table; including MM-Vet raises the mean to about 100.1%. Since MM-Vet is listed as a benchmark in §IV-A.2 and appears in the same table, the exclusion is unexplained and changes the headline. Please define AVG, report it with and without MM-Vet, or justify the exclusion.","section":"§IV-B, Tables I–II"},{"comment":"The only free hyperparameter alpha is selected on the same benchmarks used for the main results: Table III sweeps alpha over MME, GQA, SQA, POPE, MMB and chooses 0.3 as 'best overall'; these benchmarks (plus others) appear in Tables I–II. This is test-set model selection and inflates the comparison. Please select alpha on a held-out validation set or show that the relative ranking over baselines is stable across alpha in [0.1, 0.9]. Also justify the separate rule that alpha is set to 0.5 when K < 100.","section":"§IV-C, Table III"},{"comment":"The method's text-driven claim rests on cosine similarity between projected visual tokens and word-level embeddings in the LLM input space. A shared embedding space does not by itself guarantee that cosine ranking identifies answer-relevant image regions; patch tokens are trained through a projector and language-model loss, not as a visual-semantic index for arbitrary query nouns. The ablation 'w/o stage-1 selection' shows that some word-conditioned selection helps, but it does not validate the specific cosine ranking, and the CLS-attention fallback for noun-less queries is called 'rare' without a reported frequency. Please validate Eq. (2) against full-model cross-modal attention or human-annotated regions on a sample, report how often the fallback triggers, and compare against an alternative text-conditioned relevance score.","section":"§III-B.2, Eq. (2)"},{"comment":"The abstract says CRISP 'reduc[es] inference cost and latency by more than 2 times.' Table IV shows a >2x decoding-speedup for both models, but end-to-end total time improves by only 1.48x (LLaVA-1.5, 128 tokens) and 1.87x (LLaVA-NeXT, 640 tokens). The 2x total-time speedup holds only for LLaVA-NeXT at 320 tokens. Please state whether 'latency' refers to decoding speed or end-to-end time and qualify the claim accordingly.","section":"§IV-D, Table IV; Abstract"}],"minor_comments":[{"comment":"Please state whether all evaluations use greedy decoding and are deterministic. If they are deterministic, a brief note would clarify why repeated-run variance is not reported; if not, multiple seeds or standard errors are needed.","section":"§IV-A.1"},{"comment":"The paper would benefit from a short pseudo-code or algorithm block for the two-stage selection, since the description is spread over Eqs. (1)–(5) and the text.","section":"General"},{"comment":"The AVG% rows should be annotated with a formula. As written, it is not clear whether MME's raw score or its percentage retention is averaged, and whether MM-Vet is included.","section":"Tables I–II"},{"comment":"The fallback to CLS attention for noun-less queries is described as 'rare'; please quantify it on the evaluation benchmarks, since it affects the interpretation of the text-driven component.","section":"§IV-A.1"}],"recommendation":"major_revision","confidential_remarks":"The central idea is plausible and the comparison is broad, but the headline metric and hyperparameter selection need tightening before the claims can be accepted. I do not see a fundamental design error; the issues are fixable with additional analysis and clearer reporting. I would also encourage the authors to release code to support reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"CRISP is a clean, training-free way to prune visual tokens before the LLM while still conditioning selection on the text query. The idea is simple: project visual tokens, extract nouns from the question, compute cosine similarity between each visual token and each noun's embedding, allocate a quota to each noun based on CLS attention, and then fill the remaining budget with diverse tokens. It does not touch the LLM, so it keeps the speed advantage of pre-LLM pruning. On the reported benchmarks it consistently beats the baselines at three token budgets on LLaVA-1.5 and one on LLaVA-NeXT, and the ablations make sense: both stages contribute, and the quota allocation helps. The POPE results, where the method beats the full model, are the kind of result that makes you believe the text-driven signal is doing something real.\n\nSoft spots are mostly empirical hygiene. No code, no error bars, and the alpha=0.3 default is chosen on the same benchmarks used for the headline numbers. The AVG% metric silently drops MM-Vet because two baselines don't report it, which is defensible but should be said out loud. The bigger conceptual question, which the stress-test note raises, is whether cosine similarity between projected patch embeddings and word embeddings is a trustworthy relevance signal. The paper asserts it because they live in the same semantic space, but shared input space is not the same as aligned geometry. That said, the POPE performance and the consistent gains over text-agnostic baselines make me think the proxy is capturing something real, even if it is not perfectly calibrated. This is a caveat, not a fatal flaw; the method would be stronger with a small analysis of what the selected tokens look like and how often the noun-less fallback triggers.\n\nI'd send this to a serious referee. It's a useful engineering contribution with a clear and honest presentation, and the weaknesses are standard for the subfield and fixable in revision. The cosine-relevance assumption deserves probing, but the empirical case is strong enough to warrant a proper review rather than a desk reject.","headline":"A genuinely text-driven pre-LLM pruning method that looks like it works; the main doubts are empirical hygiene and an unvalidated cosine-relevance assumption, not the method's logic.","tokens_in":11828,"tokens_out":3125,"would_cite":true,"duration_ms":31028,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CRISP claims that visual token pruning for vision-language models can be done before the language model while still following the text query, preserving 97–100% accuracy even when discarding 78–89% of visual tokens.","keywords":["visual token pruning","LVLM inference acceleration","training-free efficient inference","text-driven pruning","cross-modal similarity","token diversity","LLaVA","visual-language models"],"falsifier":"Take an image where the object named in the question (say 'dog') is small and the background contains a large green area whose projected tokens are more cosine-similar to the word embedding of 'dog' (e.g., due to color or texture correlations). If CRISP retains those grass tokens and drops the small dog region, its accuracy on a simple existence question should drop below that of a text-agnostic CLS-attention pruner or random retention at the same budget. Running such a controlled image set is a direct test of the similarity-as-relevance assumption.","tokens_in":11008,"feed_emoji":"⚡","tokens_out":3728,"duration_ms":32445,"temperature":0.7,"pith_summary":"CRISP is a training-free visual token pruning method that acts entirely before the language model in a vision-language model, yet is driven by the text query. It first picks the visual tokens most similar to the nouns in the question, using the shared embedding space of the projector output and the LLM's word embeddings, then adds semantically diverse tokens to preserve scene context. The paper reports that on LLaVA-1.5-7B retaining only 22.2% of visual tokens preserves 99.5% of full-model accuracy, and on LLaVA-NeXT-7B keeping 11.1% of tokens retains 96.9% accuracy, while inference becomes more than twice as fast. The central claim is that text-awareness does not require entering the LLM: a similarity computation before the LLM can capture query-relevant visual evidence, and a diversity stage supplies the context that pure text-matching misses.","feed_headline":"Prune 89% of visual tokens, retain 97% accuracy","feed_subtitle":"A pre-LLM pruning method uses the question text to choose which image tokens to keep, cutting inference cost by more than half.","key_machinery":"The load-bearing object is the cross-modal relevance matrix S, where S_{i,j} is the cosine similarity between projected visual token v_i and the averaged word embedding e_j of an extracted query noun. It drives Stage 1's per-word top-k selection and quota allocation; Stage 2 uses the same cosine-similarity geometry to select the remaining token with the largest distance to the current set (minimax dissimilarity). Together they turn a text-agnostic pre-LLM pruner into a text-driven one at no extra LLM cost.","core_discovery":"The discovery is that the projected visual tokens and the LLM's word embeddings already lie in a common semantic space, and cosine similarity in that space is a usable relevance signal for pruning. CRISP extracts the content-bearing nouns from the query, averages their subword embeddings to get word-level embeddings, and keeps for each noun the visual tokens with the highest cosine similarity, with per-word budgets allocated by how much the vision encoder's CLS attention favors those tokens. A second stage then iteratively adds, from the remaining tokens, the one least similar to anything already selected, ensuring contextual coverage. The paper argues that this combination—critical-first se","pith_inferences":["If the shared-space similarity assumption holds broadly, one could extend the same selection idea to prune or reweight tokens for other modalities (e.g., audio or video tokens), or to pick which image regions to feed at higher resolution.","The noun-extraction step is the main brittleness: for queries with no concrete nouns (e.g., 'What is wrong with this image?') CRISP falls back to CLS attention, essentially becoming text-agnostic; a testable extension is to use verb or attribute embeddings as additional grounding cues.","The quota-allocation via CLS attention introduces a second, text-agnostic signal; the paper does not analyze cases where CLS attention and text similarity disagree, so a targeted study of such conflicts could indicate when the two stages fight rather than complement."],"forward_implications":["Deploying CRISP before the LLM reduces FLOPs and KV-cache roughly in proportion to the pruning ratio, yielding a >2× decoding speedup on tested models.","Because it never touches the LLM's attention, CRISP is compatible with FlashAttention and other optimized inference stacks that intra-LLM pruning disrupts.","The two-stage design is architecture-agnostic: it transfers from LLaVA-1.5 to LLaVA-NeXT with no retraining, suggesting it can be plugged into other LVLMs.","At high pruning ratios (retaining 11–22% of tokens) CRISP keeps 97–100% of averaged benchmark performance, including gains on hallucination-focused POPE where it can exceed the full-token baseline."],"fun_headline_variants":["Text-driven token pruning speeds LVLM inference 2x","Prune 89% of visual tokens, keep 97% accuracy","Pre-LLM pruning uses query text to drop tokens","Semantic alignment enables efficient visual token pruning","Cut visual tokens by 89% without losing accuracy"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"Equation (2) treats cosine similarity between projected visual tokens and word-level LLM embeddings as a measure of how relevant a patch is to the query; if the projector and embedding spaces are not aligned well enough for that ranking to track the actual answer-relevant regions, Stage 1 will retain the wrong tokens and retention will fall.","fun_headline_variants_meta":{"raw":{"variants":["Text-driven token pruning speeds LVLM inference 2x","Prune 89% of visual tokens, keep 97% accuracy","Pre-LLM pruning uses query text to drop tokens","Semantic alignment enables efficient visual token pruning","Cut visual tokens by 89% without losing accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000841,"raw_usage":{"total_tokens":3477,"prompt_tokens":697,"completion_tokens":2780,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":441,"completion_tokens_details":{"reasoning_tokens":2700}},"tokens_in":441,"tokens_out":2780,"duration_ms":31349,"temperature":1.0,"reasoning_tokens":2700,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T03:37:19.164916+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take an image where the object named in the question (say 'dog') is small and the background contains a large green area whose projected tokens are more cosine-similar to the word embedding of 'dog' (e.g., due to color or texture correlations). If CRISP retains those grass tokens and drops the small dog region, its accuracy on a simple existence question should drop below that of a text-agnostic CLS-attention pruner or random retention at the same budget. Running such a controlled image set is a direct test of the similarity-as-relevance assumption.","supporting_citations":[],"review_version":1}