{"id":"a250b1bf-a3f1-41df-8ea0-262e483b62b0","arxiv_id":"2607.17052","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Evolutionary search improves fixed-budget vision-block skipping, but capability-specific routes are unstable and fail to transfer across models and sources.","lead":"This paper uses evolutionary search to find fixed-size sets of vision blocks that can be skipped in vision-language models without retraining, and asks whether different question types need different routes. The search builds stronger routes than naive block ranking, but capability-specific routes do not transfer reliably across models or datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Search-vs-baseline comparison is evaluated on the same 876-example set used to freeze the searched route; the central claim has no sealed test.","rationale":"The reader's weakest assumption—selection-induced optimism from using the 876 set for both freezing routes and comparing methods—is precisely the point I find most load-bearing. The paper's own wording supports this: §4.1 calls the 876 partition 'method-selection evidence,' and §5 reiterates that it is 'not a pristine held-out benchmark.' Yet the abstract's central claim is stated categorically, and §7 says the search question is answered 'consistently yes.' The only sealed post-freeze audit (IIIT5K) evaluates a different comparison (OCR-specific vs. shared route), so the main search-vs-baseline result has no external validation. The concern is concrete: the frozen route is the winner among a small set of finalists on the evaluation set, while the baselines are not. The magnitude of possible inflation is in the same range as the reported gains. I do not recommend changing the verdict because the paper is transparent about the limitation, and the effect might survive a held-out test (especially at more aggressive budgets where the gap to contiguous/random is large). CONDITIONAL is the right verdict: the authors need to either report the finalist-mean comparison or run a fresh multi-capability sealed evaluation before the 'consistently yes' claim can be accepted. My read does not change the reader's verdict, so I mark UNCHANGED.","tokens_in":12730,"tokens_out":5976,"duration_ms":62824,"concrete_test":"Using the existing search logs, compute the mean (and 95% bootstrap CI) of the 876-set accuracy across all deduplicated finalists (typically 3–6 per family) at each budget, not just the single best finalist. Compare this finalist-mean to the independent-ranking, contiguous, and random baselines on the same 876 set. If the finalist-mean advantage over independent ranking shrinks to near zero or crosses zero, then the reported advantage is an artifact of selecting the best on the evaluation set rather than an intrinsic property of the search procedure. If the finalist-mean remains above the upper bound of the bootstrap CI, the concern is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's primary positive claim—that source-balanced evolutionary search reliably beats independent ranking, contiguous removal, and random selection at matched budgets—is established in §4.2 by comparing the frozen searched route against those controls on the same 876-example selection partition. However, the searched route is not an arbitrary output of the search; §3.5 states that after three-seed search and a 300-example development filter, 'at most three routes with the lowest matching scalar loss advance to the 876-example image-disjoint selection set, and the route with the lowest selection loss is frozen.' Thus the frozen route is the best of several candidates evaluated on the exact set used for the headline comparison, whereas the independent/contiguous/random baselines are not selected on that set. With n=876, the standard error of an accuracy difference is roughly 1.3–1.5 percentage points; a best-of-3 selection can inflate measured accuracy by a comparable amount, which covers a large part of the reported 1.03–3.54 pp Qwen gains and 4.91 pp SmolVLM2 gain. The authors explicitly acknowledge this in §4.1 and §5: the 876 set is 'method-selection evidence rather than... a sealed test set,' and only the 250-example IIIT5K audit is genuinely fresh. But IIIT5K tests OCR-specific vs. shared routes, not the search-vs-baseline comparison. Therefore the central claim 'combinatorial search reliably improves route construction' currently rests on an unsealed, selection-biased comparison. This is the load-bearing weakness of the paper. A secondary issue is that the scalar loss used for selection (combining mean damage, worst-source, and variability) is not the same as the reported overall accuracy, but this does not remove the selection bias because the scalar loss is correlated with accuracy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies training-free, fixed-budget pruning of vision-encoder blocks in VLMs by replacing selected blocks with identity operations. It introduces a source-balanced evolutionary search for shared and capability-specific block routes, and compares the searched routes against independent-ranking, contiguous-removal, and random baselines at matched skip budgets on Qwen2.5-VL-3B and SmolVLM2-2.2B. The central positive claim is that combinatorial search reliably improves route construction; the central negative claim is that capability-specific routes are not stable or transferable. The paper also reports a sealed IIIT5K audit showing that the SmolVLM2 OCR-specific route underperforms the shared route by 13.6 pp.","tokens_in":13105,"tokens_out":6699,"duration_ms":66338,"significance":"If the positive claim were validated on properly sealed data, the paper would make a useful contribution: it demonstrates that block interactions matter in VLM vision encoders, that simple independent rankings do not compose, and that a source-balanced evolutionary search can find better fixed-cardinality routes. The negative result on capability-specific routing is also valuable and is supported by a genuinely fresh IIIT5K evaluation. The paper is commendably reproducible: frozen routes are listed in Table 2, search configurations are hashed, code is released, matched budgets are enforced, and the authors explicitly disclose that the 876-example set is method-selection evidence rather than a sealed test set. However, the headline search-vs-baseline comparison is evaluated on the same 876-example set used to freeze the searched route, so the positive claim currently rests on an in-sample comparison. The IIIT5K audit does not test the search-vs-baseline claim.","major_comments":[{"comment":"The central comparison is partially in-sample. The frozen route is chosen as the lowest-loss finalist on the 876-example selection set (§3.5), and §4.2 then compares that route against independent, contiguous, and random baselines on the same 876 examples. The baselines are not selected on this set, so the searched route's advantage includes selection-induced optimism. At n=876, the standard error of an accuracy difference is roughly 1.3–1.5 pp; selecting the best of up to three finalists can inflate the apparent advantage by a comparable amount, which covers much of the reported Qwen gains (1.03, 1.03, 3.54 pp) and a nontrivial part of the SmolVLM2 gain (4.91 pp). The authors explicitly acknowledge in §4.1 and §5 that the 876 set is 'method-selection evidence rather than... a sealed test set,' but the IIIT5K audit in §4.5 is fresh only for the shared-vs-OCR comparison, not for search-vs","section":"§3.5, §4.2"},{"comment":"The Qwen search-vs-baseline results are reported as point differences without the paired bootstrap intervals promised in §3.5. For example, the searched shared route beats independent ranking by 1.03 pp at K=4 and 1.03 pp at K=6, and by 3.54 pp at K=8, but no intervals are given. With n=876, 1.03 pp is within plausible sampling noise, so the reader cannot judge whether these differences are meaningful. The SmolVLM2 replication reports intervals, but the Qwen numbers—which are the primary model's main evidence—do not. Report paired bootstrap intervals for all headline comparisons. If those intervals are computed on the 876 set after finalist selection, state explicitly that they do not correct for model selection.","section":"§4.2, §3.5"},{"comment":"The cross-architecture generalization rests on a single SmolVLM2 budget (K=4). The SmolVLM2 replication runs only the four-block study, while the Qwen study covers K=4, 6, and 8. The abstract and conclusion say 'search transfers across architectures' and 'combinatorial search reliably improves route construction,' but the evidence for transfer is one architecture at one budget. This is a scope issue rather than an internal error, but it should be reflected in the claims: the manuscript should either add SmolVLM2 results at additional budgets or replace 'reliably' with a statement scoped to the tested budgets.","section":"§4.2, §4.4, Abstract"}],"minor_comments":[{"comment":"The progressive ablation reports a 4.89% vision-encoder latency improvement for a route that is not listed in Table 2. Clarify which route this is and why this latency number is not used in §4.6.","section":"§2.2"},{"comment":"The text has a typo: 'T able 1' instead of 'Table 1'.","section":"Table 1"},{"comment":"The latency measurement is an unlocked same-VM comparison; the caption discloses this, but the main text should state prominently that the speedup is not a fixed-clock or edge-device measurement. It already does in the figure caption, but consider moving that caveat into the body text.","section":"§4.6, Figure 8"},{"comment":"The 'mean of three random routes' is reported without showing the spread of the three individual random routes. Reporting the individual values or a range would be more informative, since a single good random route could drive the mean.","section":"§4.2"},{"comment":"The figure compares Qwen at K=6 with SmolVLM2 at K=4. The caption is transparent, but the visual pairing may be misread as a matched comparison. Consider labeling each panel with its budget more prominently.","section":"Figure 5"},{"comment":"The scalar loss weights (0.50/0.30/0.20 and 0.45/0.30/0.15/0.10) are introduced without sensitivity analysis. A brief note that the conclusions are robust or not to these weights would strengthen the paper.","section":"§3.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's honesty about the selection set is admirable, but the abstract overstates the evidence: the main positive claim has no sealed test. The negative capability-transfer result is well supported by the IIIT5K audit, but the search-vs-baseline claim needs either a genuinely fresh evaluation set or a substantial re-scoping to 'method-selection evidence.' Because the code, frozen routes, and hashes are released, a revision that adds a sealed comparison or softens the central claim is feasible. The work is within the journal's scope if the claims are made proportionate to the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hey — quick take. This is a serious, unusually honest empirical paper, but the central positive claim isn't sealed yet. What's new: a source-balanced evolutionary search for fixed-budget vision-block routes in VLMs, with matched-budget comparisons against independent, contiguous, and random routes across two architectures and five capabilities. The paper also documents low cross-seed route overlap and shows that an OCR-specific route trained on one source transfers badly to IIIT5K. That negative transfer result is on genuinely fresh data and looks solid.\n\nThe paper does a lot right. It ships frozen routes, pinned model revisions, seeded search, paired bootstrap intervals, and explicit limitation statements. The progressive ablation showing that independently safe blocks become damaging in combination is persuasive. On the capability-specialization question, the paper is properly cautious: it labels the Qwen K=6 result borderline because the interval touches zero, and the SmolVLM2 sign reversal for OCR is a good sanity check.\n\nThe soft spot is the load-bearing comparison. Section 4.2's search-vs-baseline results are all measured on the 876-example set, and that same set was used to freeze the searched route (pick the lowest-scalar-loss finalist from at most three candidates). The paper itself calls this method-selection evidence, not a sealed test. But the headline conclusion 'combinatorial search reliably improves route construction' depends on differences of 1.03–3.54 pp on Qwen and 4.91 on SmolVLM2. With n=876, best-of-3 selection can plausibly inflate those numbers by a comparable amount. So the central claim currently rests on an unsealed comparison. The IIIT5K audit doesn't fix this, because it compares OCR-specific vs. shared routes, not search vs. baselines.\n\nI also would have liked per-seed accuracy for the three search runs, since the Jaccard overlaps show the finalists are quite different. And the scalar loss used for finalist selection isn't the reported accuracy, which makes the selection bias a bit harder to quantify.\n\nThat said, this is not a slipshod paper and the main weakness is fixable. Referee it, and require a genuinely held-out evaluation of the search-vs-baseline comparison plus per-seed variance. If the search advantage survives a sealed test, it's a useful result for VLM efficiency. Even if it doesn't, the capability-transfer evidence and the methodological candor make the paper worth publishing after revision.","headline":"A transparent and well-executed empirical study, but the central claim that evolutionary search beats naive pruning is evaluated on the same set used to freeze the searched routes, so the headline result is not yet sealed.","tokens_in":13619,"tokens_out":3270,"would_cite":false,"duration_ms":33239,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T45","68T20"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that choosing which vision blocks to skip is a combinatorial search problem, and a source-balanced evolutionary search finds stronger fixed-budget routes than independent ranking or random selection.","keywords":["vision-language models","vision block pruning","evolutionary search","structured pruning","capability-specific routing","combinatorial optimization","model efficiency","OCR transfer"],"falsifier":"Take the same two models and the same search and control methods, but freeze all routes on the 876-example set and then evaluate on a genuinely sealed benchmark from sources not used in any screening. If the searched routes' advantage over independent ranking shrinks or reverses there, the selection-set reuse rather than the search itself would be the likely explanation.","tokens_in":12589,"feed_emoji":"🧬","tokens_out":5567,"duration_ms":48793,"temperature":0.7,"pith_summary":"The paper asks whether a vision-language model must run every vision block for every question, and whether the best blocks to skip can be found by looking at each block's importance alone. It shows that they cannot: blocks that look safe individually can become harmful when skipped together, so choosing a fixed set of K blocks to skip is a combinatorial search problem. The paper introduces a source-balanced evolutionary search that consistently constructs stronger same-budget routes than independent ranking, contiguous removal, or random selection on two architectures. It also shows that capability-specific routes — routes tuned for OCR, counting, spatial reasoning, attributes, or object existence — are not universally better or transferable. A shared route often beats a capability-specific route, especially on a sealed transfer test, so the paper concludes that combinatorial search reliably improves route construction but capability labels do not define stable vision pathways.","feed_headline":"Evolutionary search beats ranking for vision-block pruning","feed_subtitle":"Skipping vision blocks is a combinatorial problem; searched routes gain up to 4.91 points over naive ranking.","key_machinery":"A route is a set of exactly K vision-transformer blocks replaced by identity operations. The key mechanism is the source-balanced evolutionary search: a genetic algorithm with deterministic parent pairing, fixed-K crossover that keeps the intersection of two parental routes, and one-swap mutation, guided by Pareto survival on a multi-objective vector that balances mean damage, worst-source damage, and source variability (plus collateral damage for capability-specific routes). This machinery lets the search evaluate complete block combinations rather than composing individual importance scores, which is necessary because residual blocks interact.","core_discovery":"On the paper's own terms, the central discovery is that route quality is non-additive: the damage from skipping a set of blocks is not predicted by the sum of individual single-block damages. A source-balanced evolutionary search treats a route as one complete set of exactly K blocks and optimizes a multi-objective loss over capability-source cells, including worst-source and variability terms. This search beats independent ranking by 1.03, 1.03, and 3.54 percentage points on Qwen at four, six, and eight skipped blocks, and by 4.91 points on SmolVLM2 at four skipped blocks. Capability specialization gives a borderline 2.17-point gain on Qwen at six blocks, driven by a 7.10-point OCR gain, bu","pith_inferences":["A learned router that conditions on image statistics or question text rather than a known capability label is a natural next step; the paper's oracle-label assumption is a deployment bottleneck.","The interaction measure I(i,j) could be used to predict when independent ranking will fail, potentially giving a cheaper screening test before running full search.","The paper's search advantage is measured on the same selection set used to freeze routes; a strictly sealed multi-source evaluation would test whether the advantage persists without selection optimism.","Since the paper only varies skip budgets K=4,6,8 on one model and K=4 on another, extending to larger budgets and more architectures would show whether the search advantage grows monotonically."],"forward_implications":["Independent layer-importance scores should not be used to compose multi-block reductions in vision encoders; they miss block interactions.","Any fixed-budget depth pruning of a vision-language model is better done by combinatorial search than by ranking, especially at larger skip budgets.","Capability-specific routes should be validated on held-out sources before deployment; a route named for OCR may not transfer across OCR benchmarks.","The same skip budget has different costs across architectures, so pruning budgets need to be set per model, not globally.","Route search can reduce executed vision depth and give modest end-to-end speedups, but the checkpoint itself is not smaller until parameters are physically removed."],"fun_headline_variants":["Evolutionary search beats ranking for vision-block routes","Searched routes beat ranking, but capability labels unstable","Whole-route search beats block-sum ranking in VLMs","Vision pruning: search gains up to 4.91 points over ranking","Non-additive damage: why search beats ranking for vision blocks"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The headline comparisons use the 876-example selection set both to pick the frozen route and to measure the search advantage, so the results assume that selection over these 876 examples does not materially inflate the apparent gain of the searched routes.","fun_headline_variants_meta":{"raw":{"variants":["Evolutionary search beats ranking for vision-block routes","Searched routes beat ranking, but capability labels unstable","Whole-route search beats block-sum ranking in VLMs","Vision pruning: search gains up to 4.91 points over ranking","Non-additive damage: why search beats ranking for vision blocks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000274,"raw_usage":{"total_tokens":1498,"prompt_tokens":789,"completion_tokens":709,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":626}},"tokens_in":533,"tokens_out":709,"duration_ms":6810,"temperature":1.0,"reasoning_tokens":626,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T19:09:13.290216+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same two models and the same search and control methods, but freeze all routes on the 876-example set and then evaluate on a genuinely sealed benchmark from sources not used in any screening. If the searched routes' advantage over independent ranking shrinks or reverses there, the selection-set reuse rather than the search itself would be the likely explanation.","supporting_citations":[],"review_version":1}