{"id":"3256fec1-f8e6-4cd9-b62c-b3a6c0bc0573","arxiv_id":"2608.04496","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"By repeatedly re-scoring image tokens after each selection and discounting already-covered evidence, DIVE keeps vision-language model accuracy nearly intact while removing up to 89% of the visual tokens.","lead":"DIVE is a new method for speeding up vision-language models by choosing which parts of an image to keep for the model to look at, choosing them one at a time so each new piece adds information the previous choices missed. It reports keeping 98% of a model's original accuracy while using only about 1 in 9 of the image tokens.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'No Update' control is degenerate: row-normalized initialization makes the frozen one-shot score pure prompt alignment, so the controlled evidence for iterative residual feedback does not test what it claims.","rationale":"Read in good faith, DIVE's contribution is the select-update-re-evaluate mechanism, and the strongest controlled evidence for that mechanism is Table 6 and Figure 4(a). Both compare against a frozen-score variant. That variant is constructed so that the initial scores strip out the visual residual-energy term: Eq. (5) normalizes rows, so all candidate norms are equal at t=0, and Eq. (8) makes the score a monotone function of prompt alignment alone. The ablation therefore conflates 'one-shot scoring' with 'prompt-only scoring.' A static scorer that also uses original visual norms as an importance signal is the appropriate baseline for the claim that set-dependent scoring beats static scoring, and that baseline is absent. This is a design flaw in the ablation, not a question of integrity. The headline comparison to existing static pruning methods is credible and supports the practical claim that DIVE preserves accuracy at equal token budgets, but it does not isolate the mechanism. The reader's CONDITIONAL verdict remains appropriate; the concern strengthens the conditions by adding a specific missing control, but it does not overturn the verdict.","tokens_in":26102,"tokens_out":7379,"duration_ms":67663,"concrete_test":"Re-run the Table 6 ablation with No Update modified to freeze scores that retain the visual residual-energy term, e.g. g_0(i) = exp(lambda * phi_0(i)) * ||x_i||_2^2, where x_i are the raw layer-2 visual hidden states from Eq. (4), using the same lambda=1.5, eta=0.8, h=3 and the same six benchmarks. If this static top-64 baseline reaches roughly 95-97% relative average instead of 61.97%, the claimed necessity of iterative updates is not established; if it remains far below DIVE's 97.20%, the mechanism claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 3.1, Eq. (5) initializes both visual and textual residuals by row-wise L2 normalization, so at t=0 every visual candidate satisfies ||x_i^(0)||_2 = 1. Equation (8) then gives g_0(i) = exp(lambda * phi_0(i)), and since exp is monotone, the one-shot ranking used by the 'No Update' ablation in Table 6 is exactly the top-64 prompt-alignment ranking, with no visual-residual term. The 35-point gap between No Update (61.97%) and DIVE (97.20%) therefore does not establish that set-dependent residual feedback beats a reasonable one-shot scorer; it only shows that selecting by prompt alignment alone is much worse than the iterative procedure. The paper itself calls No Update 'rather than serving as a competitive static pruning baseline,' which concedes that this control is not a meaningful static top-k baseline. The external comparisons in Table 1 with real static methods (best 95.17% at 64 tokens) support the headline practical claim, but the paper's causal claim that residual re-evaluation is necessary within its formulation, and the 'Frozen Score' comparisons in Figure 4(a), rest on this degenerate baseline. This is the weakest load-bearing element of the central mechanism claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"DIVE proposes a training-free framework for visual-token pruning in vision-language models (VLMs). Instead of scoring all visual tokens once and retaining the top-k, DIVE iteratively selects the highest-scoring remaining token, updates both visual and textual residual states by subtracting the one-sided projection onto the selected token's direction, and re-evaluates the remaining candidates. The selection layer is fixed after the first two LLM blocks, and the retained tokens are restored to their original order before being passed to the frozen VLM. Experiments across LLaVA-1.5, LLaVA-NeXT, Qwen2-VL, and LLaVA-OV on eight image and two video benchmarks report that DIVE preserves a higher fraction of the full-token model's average performance than existing static or greedy methods at matched token budgets; the headline claim is 98.2% retained average performance with 64 of 576 visual tokens. The paper includes component ablations, parameter sensitivity, a residual-energy proposition, complexity analysis, and a qualitative analysis of selected tokens.","tokens_in":26235,"tokens_out":4696,"duration_ms":46699,"significance":"If the reported results hold, DIVE provides a practical, training-free improvement for efficient VLM inference, with consistent gains across model families and token budgets. The paper is generally careful: it releases code, gives a precise algorithm including initialization and update formulas, proves the monotone residual-energy decrease (Appendix A.3), reports an explicit complexity expression (Appendix A.4), and compares against both static top-k methods and greedy subset-selection methods (Table 10). The main scientific claim, that set-dependent residual feedback is superior to one-shot scoring at equal budgets, is plausible and practically supported by the external comparisons against FastV, VisionZip, PruneSID, MMTok, CDPruner, and SCOPE. However, the paper's own controlled evidence for the mechanism is weaker than presented: the 'No Update' ablation is degenerate, and the hyperparameter configuration was fixed based on the same LLaVA-1.5-7B 64-token setting used in the headline result. These issues do not invalidate the empirical comparisons, but they do need to be addressed before the causal claim about iterative re-evaluation can be accepted.","major_comments":[{"comment":"The 'No Update' ablation does not isolate the effect of iterative residual feedback. Because the algorithm initializes every visual residual by row-wise L2 normalization, ||x_i^(0)||_2 = 1 for all candidates, so at t=0 the score in Eq. (8) reduces to g_0(i) = exp(lambda * phi_0(i)). The frozen one-shot ranking used by 'No Update' is therefore purely a prompt-alignment ranking and contains no visual residual-energy term. The 61.97% versus 97.20% gap in Table 6 conflates the removal of iterative updates with the removal of the visual residual signal. The manuscript's own statement that No Update is 'rather than serving as a competitive static pruning baseline' concedes this point, but Section 4.4 still concludes that 'set-dependent residual feedback is necessary within the DIVE formulation.' That conclusion is not supported by the current ablation. Please add a static control that uses both prompt alignment and a meaningful visual-saliency term (for example, the original pre-normalization norm or an attention-based score) so that the comparison isolates the benefit of re-evaluating after each selection.","section":"§3.1, Eq. (5), Eq. (8), Table 6"},{"comment":"The global configuration lambda = 1.5, eta = 0.8, h = 3 was selected after inspecting the LLaVA-1.5-7B 64-token setting, and the sensitivity table (Table 7) is computed on the same benchmarks that appear in the main results. The text states only that the configuration was 'fixed' and then 'examined'; it does not state whether a separate validation split was used or whether the authors observed the test-set results before choosing lambda, eta, and h. As written, the headline 98.2% figure for the 64-token LLaVA-1.5-7B setting is a result of searching over the parameter grid on that same benchmark set. This does not necessarily invalidate the cross-backbone and cross-budget findings, especially since the sensitivity range in Table 7 is modest (96.06% to 97.20%), but the selection protocol must be disclosed so the reader can interpret how much of the reported advantage is a tuned operating point.","section":"Appendix C.2 and §3.3"}],"minor_comments":[{"comment":"The text refers to 'No Update' in Table 6 and 'Frozen Score' in Figure 4 for the same zero-update schedule; please unify the terminology to avoid confusion.","section":"Table 6 and Figure 4"},{"comment":"Several numeric entries are visually corrupted by missing spaces, for example '13.5962.8577.0' and '78.8585.7' in the SparseVLM and DIVE rows. Please regenerate the table with clean alignment.","section":"Table 5"},{"comment":"The assumption that visual and textual hidden states from the same multimodal sequence can be compared directly by inner products is stated but not validated. The ablation in Table 6 shows that removing language guidance costs only about one relative percentage point (96.13% vs. 97.20% for DIVE), so the prompt-alignment term is a minor contributor to the overall result; the paper should either provide a probe validating the alignment or temper the language about the importance of prompt-conditioned scoring.","section":"§3.2"},{"comment":"The average metric in Eq. (16) is computed from unrounded benchmark scores, which is good, but Table 1 and Table 6 use different subsets of benchmarks for the same 64-token LLaVA-1.5-7B setting (eight vs. six). Please state this explicitly in the table captions or the text so the different Avg. values (98.21% vs. 97.20%) are not perceived as inconsistent.","section":"Appendix B.1"},{"comment":"Panel (c) reports 'decode-normalized E2E' but the axis label and legend do not make clear that decoding time is replaced by the No-Update mean; Appendix B.4 describes this, but the figure itself should carry a short note.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The paper's external benchmark results are quite strong and the method is cleanly described, but the central mechanistic claim is currently supported by a degenerate ablation. The authors should be asked to provide a proper static baseline that retains the visual residual term, and to state the hyperparameter selection protocol. If those are fixed, the paper could become a solid contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, DIVE is a clean, well-specified iterative selection method for visual-token pruning in VLMs, and its benchmark claims hold up on the reported numbers. Second, the paper's internal ablation for the mechanism is weaker than it looks: the 'No Update' control selects by prompt alignment alone, so the 35-point gap to full DIVE does not show that residual re-evaluation beats a reasonable static scorer.\n\nThe new thing is the select-update-re-evaluate loop (Eqs. 1–3): after picking a token, DIVE removes the positively aligned component from both visual and query residuals, then re-scores. That coupling of visual and textual residual feedback is not in the cited static or greedy baselines. The method is training-free, the complexity analysis in Appendix A.4 is correct, and the experiments are broad: LLaVA-1.5/NeXT, Qwen2-VL, image and video, several token budgets, plus an efficiency table and latency–performance sweeps. The comparison against greedy coverage and diversity methods in Table 10 is the right control, and DIVE leads there too.\n\nSoft spots, in proportion. The stress-test note is right: with row-normalized initialization, the frozen score at t=0 is exp(λ·prompt-alignment), so 'No Update' is not a meaningful static top-k baseline. The paper itself concedes this, saying it is 'not a competitive static pruning baseline'; that makes the causal claim in Section 4.4 weaker than the prose suggests. But this does not sink the paper. The headline claim rests on Table 1 and Table 10 against real static and greedy methods, where DIVE wins at equal token budgets. So the flaw is in the mechanism-validity narrative, not in the practical result.\n\nTwo smaller issues. There are no error bars or repeated runs anywhere, so a few of the small margins at high budgets are unquantified. And λ, η, and h were selected after looking at Table 7 on the same LLaVA-1.5-7B 64-token setting; the sensitivity table shows performance is stable across values, so this is mild, but it should be stated as such. The shared-representation assumption for comparing visual and text inner products is plausible and partially validated by the w/o L-G ablation, which only loses about a point.\n\nWho this is for: anyone working on efficient VLM inference or token pruning. The paper is honest, reproducible in principle (code link included), and the method is simple enough to be drop-in. It does not change the field, but it is a genuinely useful benchmark point.\n\nRecommendation: send to peer review. A serious referee should ask for a real static one-shot baseline (e.g., one-shot scoring with residual norms, no updates), error bars, and a cleaner statement of the hyperparameter selection. With those, the mechanism claim could be made solid. As it stands, the practical claim is already solid.","headline":"Solid, useful empirical systems paper for VLM token pruning; the mechanism ablation is overstated because the 'No Update' control is degenerate, but the headline result survives on external baselines.","tokens_in":26890,"tokens_out":3092,"would_cite":true,"duration_ms":25802,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DIVE claims that one-shot visual-token scoring is the wrong basis for pruning in vision-language models: iterative select-update-re-evaluate preserves 98.2% of full-model accuracy while cutting 88.9% of visual tokens.","keywords":["vision-language models","visual token pruning","iterative selection","residual feedback","prompt conditioning","inference efficiency","training-free","subset selection"],"falsifier":"Two direct checks would settle the claim: measure the top singular values of the cross-covariance between the row-normalized visual and text hidden states at the pruning layer, since near-zero values would mean the language-guidance term carries no signal, or find any benchmark and token budget where DIVE's frozen-score ranking matches the iterative procedure's accuracy, contradicting the reported 97.2% versus 62.0% gap at 64 tokens.","tokens_in":25785,"feed_emoji":"⚡","tokens_out":10206,"duration_ms":78506,"temperature":0.7,"pith_summary":"The paper argues that one-shot visual-token scoring is the wrong basis for pruning in vision-language models because a token's usefulness depends on which tokens are already retained. It introduces DIVE, a training-free procedure that constructs the retained set one token at a time: pick the highest-scoring remaining token, discount the visual and text evidence that token explains, and re-score the rest. On LLaVA-1.5-7B this preserves 98.2% of the full model's average performance while dropping 88.9% of the visual tokens, and the advantage holds across the LLaVA family, Qwen2-VL, and video-capable models. The source of the gain is set-dependent residual feedback, not a better static importance score.","feed_headline":"Iterative selection keeps 98% of VLM accuracy on 89% fewer tokens","feed_subtitle":"Re-scoring tokens after each pick favors complementary evidence, not just salient patches.","key_machinery":"The load-bearing object is the one-sided feedback operator $F_u(z) = [\\langle z, u \\rangle]_+ u$, applied to the visual and query residuals after every selection. It removes only the component of each remaining residual that is positively aligned with the newly selected token's direction, so the update is monotone in residual energy and leaves orthogonal or negatively aligned components untouched. Because the same direction $u_t$ is applied to both the visual state $X^{(t)}$ and the prompt state $Q^{(t)}$, the discount covers prompt directions that the selected token has already addressed, coupling the two modalities inside the selection score.","core_discovery":"The central discovery is that a visual token's marginal value is state-dependent: a high-scoring patch loses most of its worth once the retained set already covers the same region, while a previously secondary patch becomes valuable. DIVE operationalizes this by keeping evolving visual residuals $x_i^{(t)}$ and prompt residuals $q_m^{(t)}$, initialized by row-wise $\\ell^2$ normalization of the visual and text rows of the same multimodal hidden-state sequence, taken after two Transformer blocks. At each step the score $g_t(i) = \\exp(\\lambda \\phi_i^{(t)}) \\|x_i^{(t)}\\|_2^2$ multiplies residual energy by a prompt-alignment factor; the selected token supplies the feedback direction $u_t$, and both residuals are discounted by subtracting $\\eta$ times their positive projection onto $u_t$. Repeating this select–update–re-evaluate loop for $K$ steps yields a compact set of complementary, prompt-relevant evidence, and the original token representations are restored to input order before the frozen VLM processes them.","pith_inferences":["The shared-representation assumption could be stress-tested at deeper layers: if DIVE's language-guidance contribution shrinks when selection happens after more than two blocks, the early-layer alignment is doing the work rather than the residual feedback per se.","The one-sided projection update is a greedy orthogonalization of the visual and query residuals, which suggests a formal connection to coverage maximization or submodular objectives that the paper does not claim.","Because DIVE only replaces the token-selection step and leaves the VLM frozen, it should compose with other inference accelerators such as KV-cache eviction, quantization, or speculative decoding.","For models with no hidden-state access, a black-box proxy of residual evidence—for instance attention-weighted saliency—could carry the same select-update-re-evaluate idea to API-only VLMs."],"forward_implications":["At matched token budgets, DIVE outperforms static top-k baselines (FastV, SparseVLM, VisionZip, PruneSID) and greedy set-selection methods (MMTok, CDPruner, SCOPE) across the eight image benchmarks and on MVBench and VideoMME.","DIVE's No Update variant, which freezes the same scores into a one-shot ranking, falls from 97.2% to 62.0% average relative performance at 64 tokens, isolating iterative residual feedback as the source of the gain.","Residual feedback reduces within-set redundancy, measured by mean maximum cosine similarity among retained tokens, and the reduction is largest at the tightest 64-token budget.","The selection overhead is mostly amortized: the full update schedule adds 30.7% to prefill latency but only 1.8% to end-to-end latency with 64 retained tokens and 32 generated tokens.","Fixed hyperparameters ($\\lambda=1.5$, $\\eta=0.8$, top-3 prompt pooling, pruning after block 2) transfer across backbones, resolutions, image and video tasks, and token budgets without retuning."],"supporting_citations":[{"why":"Provides FastV, the attention-based static pruning baseline DIVE is designed to beat.","marker":"Chen et al. 2024"},{"why":"SparseVLM, the prompt-relevance one-shot baseline that motivates DIVE's language-guidance term.","marker":"Zhang et al. 2025d"},{"why":"VisionZip, the strongest redundancy-based static competitor at tight budgets.","marker":"Yang et al. 2025"},{"why":"PruneSID, a greedy importance-diversity baseline used in the main comparisons.","marker":"Fang et al. 2026"},{"why":"SCOPE, a greedy saliency-coverage selector that also updates coverage after each pick and is compared in Table 10.","marker":"Deng et al. 2025"},{"why":"Token merging, cited as evidence that a token's usefulness depends on the already-retained set.","marker":"Bolya et al. 2022"},{"why":"LLaVA-1.5, the primary 576-token backbone and evaluation setting.","marker":"Liu et al. 2023a"},{"why":"LMMS-Eval, the evaluation harness that supplies the benchmark protocol and metrics.","marker":"Zhang et al. 2025b"}],"fun_headline_variants":["Iterative residual scoring keeps 98% accuracy on 89% fewer tokens","DIVE: re-score token value after each pick, retain 98% accuracy with 89% fewer tokens","Complementary evidence: iterative selection retains 98% accuracy on 89% fewer tokens","State-dependent token value: iterative pruning keeps 98% accuracy on 89% fewer tokens","DIVE: update residual evidence each step, keep 98% accuracy with 89% fewer tokens"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The prompt-alignment weight assumes that row-normalized visual and text hidden states from the same multimodal sequence share a coordinate space, so inner products measure prompt relevance; if those representations are not actually aligned, the language-guidance term could steer selection toward irrelevant patches.","fun_headline_variants_meta":{"raw":{"variants":["Iterative residual scoring keeps 98% accuracy on 89% fewer tokens","DIVE: re-score token value after each pick, retain 98% accuracy with 89% fewer tokens","Complementary evidence: iterative selection retains 98% accuracy on 89% fewer tokens","State-dependent token value: iterative pruning keeps 98% accuracy on 89% fewer tokens","DIVE: update residual evidence each step, keep 98% accuracy with 89% fewer tokens"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001806,"raw_usage":{"total_tokens":7119,"prompt_tokens":964,"completion_tokens":6155,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":6037}},"tokens_in":580,"tokens_out":6155,"duration_ms":31417,"temperature":1.0,"reasoning_tokens":6037,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:38:44.774992+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Two direct checks would settle the claim: measure the top singular values of the cross-covariance between the row-normalized visual and text hidden states at the pruning layer, since near-zero values would mean the language-guidance term carries no signal, or find any benchmark and token budget where DIVE's frozen-score ranking matches the iterative procedure's accuracy, contradicting the reported 97.2% versus 62.0% gap at 64 tokens.","supporting_citations":[],"review_version":2}