REVIEW 3 major objections 6 minor 17 references
The paper argues that choosing which vision blocks to skip is a combinatorial search problem, and a source-balanced evolutionary search finds stronger fixed-budget routes than independent ranking or random selection.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 19:09 UTC pith:ULFM3HKJ
load-bearing objection A transparent and well-executed empirical study, but the central claim that evolutionary search beats naive pruning is evaluated on the same set used to freeze the searched routes, so the headline result is not yet sealed. the 3 major comments →
Searching for Task-Specific Vision Paths: Evolutionary Block Pruning Across Vision-Language Models
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
On the paper's own terms, the central discovery is that route quality is non-additive: the damage from skipping a set of blocks is not predicted by the sum of individual single-block damages. A source-balanced evolutionary search treats a route as one complete set of exactly K blocks and optimizes a multi-objective loss over capability-source cells, including worst-source and variability terms. This search beats independent ranking by 1.03, 1.03, and 3.54 percentage points on Qwen at four, six, and eight skipped blocks, and by 4.91 points on SmolVLM2 at four skipped blocks. Capability specialization gives a borderline 2.17-point gain on Qwen at six blocks, driven by a 7.10-point OCR gain, bu
What carries the argument
A route is a set of exactly K vision-transformer blocks replaced by identity operations. The key mechanism is the source-balanced evolutionary search: a genetic algorithm with deterministic parent pairing, fixed-K crossover that keeps the intersection of two parental routes, and one-swap mutation, guided by Pareto survival on a multi-objective vector that balances mean damage, worst-source damage, and source variability (plus collateral damage for capability-specific routes). This machinery lets the search evaluate complete block combinations rather than composing individual importance scores, which is necessary because residual blocks interact.
Load-bearing premise
The headline comparisons use the 876-example selection set both to pick the frozen route and to measure the search advantage, so the results assume that selection over these 876 examples does not materially inflate the apparent gain of the searched routes.
What would settle it
Take the same two models and the same search and control methods, but freeze all routes on the 876-example set and then evaluate on a genuinely sealed benchmark from sources not used in any screening. If the searched routes' advantage over independent ranking shrinks or reverses there, the selection-set reuse rather than the search itself would be the likely explanation.
If this is right
- Independent layer-importance scores should not be used to compose multi-block reductions in vision encoders; they miss block interactions.
- Any fixed-budget depth pruning of a vision-language model is better done by combinatorial search than by ranking, especially at larger skip budgets.
- Capability-specific routes should be validated on held-out sources before deployment; a route named for OCR may not transfer across OCR benchmarks.
- The same skip budget has different costs across architectures, so pruning budgets need to be set per model, not globally.
- Route search can reduce executed vision depth and give modest end-to-end speedups, but the checkpoint itself is not smaller until parameters are physically removed.
Where Pith is reading between the lines
- A learned router that conditions on image statistics or question text rather than a known capability label is a natural next step; the paper's oracle-label assumption is a deployment bottleneck.
- The interaction measure I(i,j) could be used to predict when independent ranking will fail, potentially giving a cheaper screening test before running full search.
- The paper's search advantage is measured on the same selection set used to freeze routes; a strictly sealed multi-source evaluation would test whether the advantage persists without selection optimism.
- Since the paper only varies skip budgets K=4,6,8 on one model and K=4 on another, extending to larger budgets and more architectures would show whether the search advantage grows monotonically.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies training-free, fixed-budget pruning of vision-encoder blocks in VLMs by replacing selected blocks with identity operations. It introduces a source-balanced evolutionary search for shared and capability-specific block routes, and compares the searched routes against independent-ranking, contiguous-removal, and random baselines at matched skip budgets on Qwen2.5-VL-3B and SmolVLM2-2.2B. The central positive claim is that combinatorial search reliably improves route construction; the central negative claim is that capability-specific routes are not stable or transferable. The paper also reports a sealed IIIT5K audit showing that the SmolVLM2 OCR-specific route underperforms the shared route by 13.6 pp.
Significance. If the positive claim were validated on properly sealed data, the paper would make a useful contribution: it demonstrates that block interactions matter in VLM vision encoders, that simple independent rankings do not compose, and that a source-balanced evolutionary search can find better fixed-cardinality routes. The negative result on capability-specific routing is also valuable and is supported by a genuinely fresh IIIT5K evaluation. The paper is commendably reproducible: frozen routes are listed in Table 2, search configurations are hashed, code is released, matched budgets are enforced, and the authors explicitly disclose that the 876-example set is method-selection evidence rather than a sealed test set. However, the headline search-vs-baseline comparison is evaluated on the same 876-example set used to freeze the searched route, so the positive claim currently rests on an in-sample comparison. The IIIT5K audit does not test the search-vs-baseline claim.
major comments (3)
- [§3.5, §4.2] The central comparison is partially in-sample. The frozen route is chosen as the lowest-loss finalist on the 876-example selection set (§3.5), and §4.2 then compares that route against independent, contiguous, and random baselines on the same 876 examples. The baselines are not selected on this set, so the searched route's advantage includes selection-induced optimism. At n=876, the standard error of an accuracy difference is roughly 1.3–1.5 pp; selecting the best of up to three finalists can inflate the apparent advantage by a comparable amount, which covers much of the reported Qwen gains (1.03, 1.03, 3.54 pp) and a nontrivial part of the SmolVLM2 gain (4.91 pp). The authors explicitly acknowledge in §4.1 and §5 that the 876 set is 'method-selection evidence rather than... a sealed test set,' but the IIIT5K audit in §4.5 is fresh only for the shared-vs-OCR comparison, not for search-vs
- [§4.2, §3.5] The Qwen search-vs-baseline results are reported as point differences without the paired bootstrap intervals promised in §3.5. For example, the searched shared route beats independent ranking by 1.03 pp at K=4 and 1.03 pp at K=6, and by 3.54 pp at K=8, but no intervals are given. With n=876, 1.03 pp is within plausible sampling noise, so the reader cannot judge whether these differences are meaningful. The SmolVLM2 replication reports intervals, but the Qwen numbers—which are the primary model's main evidence—do not. Report paired bootstrap intervals for all headline comparisons. If those intervals are computed on the 876 set after finalist selection, state explicitly that they do not correct for model selection.
- [§4.2, §4.4, Abstract] The cross-architecture generalization rests on a single SmolVLM2 budget (K=4). The SmolVLM2 replication runs only the four-block study, while the Qwen study covers K=4, 6, and 8. The abstract and conclusion say 'search transfers across architectures' and 'combinatorial search reliably improves route construction,' but the evidence for transfer is one architecture at one budget. This is a scope issue rather than an internal error, but it should be reflected in the claims: the manuscript should either add SmolVLM2 results at additional budgets or replace 'reliably' with a statement scoped to the tested budgets.
minor comments (6)
- [§2.2] The progressive ablation reports a 4.89% vision-encoder latency improvement for a route that is not listed in Table 2. Clarify which route this is and why this latency number is not used in §4.6.
- [Table 1] The text has a typo: 'T able 1' instead of 'Table 1'.
- [§4.6, Figure 8] The latency measurement is an unlocked same-VM comparison; the caption discloses this, but the main text should state prominently that the speedup is not a fixed-clock or edge-device measurement. It already does in the figure caption, but consider moving that caveat into the body text.
- [§4.2] The 'mean of three random routes' is reported without showing the spread of the three individual random routes. Reporting the individual values or a range would be more informative, since a single good random route could drive the mean.
- [Figure 5] The figure compares Qwen at K=6 with SmolVLM2 at K=4. The caption is transparent, but the visual pairing may be misread as a matched comparison. Consider labeling each panel with its budget more prominently.
- [§3.3] The scalar loss weights (0.50/0.30/0.20 and 0.45/0.30/0.15/0.10) are introduced without sensitivity analysis. A brief note that the conclusions are robust or not to these weights would strengthen the paper.
Circularity Check
Headline search-vs-baseline comparison is evaluated on the same 876-example set used to freeze the searched route; the central claim therefore rests on an in-sample selection advantage rather than a sealed test.
specific steps
-
fitted input called prediction
[§3.5, §4.2, §5]
"At most three routes with the lowest matching scalar loss advance to the 876-example image-disjoint selection set, and the route with the lowest selection loss is frozen. ... The 876-example set is used for method selection and is not described as sealed evidence because earlier discovery work touched its underlying benchmark sources. ... Only the 250-example IIIT5K audit is a genuinely fresh post-freeze source-transfer test."
The frozen route is chosen as the lowest-loss candidate on the 876-example set, and §4.2 then reports that same route's accuracy on that same 876-example set against independent, contiguous, and random controls. The search route is therefore selected on the evaluation set, while the controls are not, so the reported gains (1.03–3.54 pp on Qwen; 4.91 pp on SmolVLM2) partly reflect best-of-at-most-three selection optimism on the exact examples used for the comparison. The paired bootstrap resamples example IDs but does not resample the selection procedure, so the intervals understate this in-sample advantage. The paper candidly labels the 876 set as method-selection evidence rather than a sealed test, but the central claim that combinatorial search reliably improves route construction is not
full rationale
The paper's mathematical machinery is internally consistent: the source-balanced objectives, Pareto survival, crossover, and mutation are not defined in terms of the headline results, and no self-citation chain is load-bearing. The circularity arises at the evaluation step. §3.5 freezes the route with the lowest loss on the 876-example selection set, and §4.2 uses that same set to compare the frozen route against non-selected baselines. This is a fitted-input-called-prediction pattern: the searched route is effectively a parameter selected on the evaluation data, and its reported superiority over independent/contiguous/random routes is at least partly an artifact of that selection. The paper explicitly acknowledges the limitation in §4.1 and §5, which is honest, but the acknowledgment does not remove the circularity from the central empirical claim. The IIIT5K audit is genuinely post-freeze and supports the secondary claim about non-transferable OCR pathways, but it does not validate the primary search-vs-baseline claim. The bootstrap intervals quantify sampling noise over examples, not selection-induced optimism, so the reported confidence intervals are too narrow for the claim they support. Overall, the derivation is partially circular: the central positive result is not backed by a sealed comparison, though the paper's own caveats prevent this from being a fully concealed circularity.
Axiom & Free-Parameter Ledger
free parameters (5)
- Scalar loss weights (shared route) =
0.50, 0.30, 0.20
- Scalar loss weights (capability route) =
0.45, 0.30, 0.15, 0.10
- Evolutionary budgets =
Qwen: pop 16, 3 generations, K=4,6,8; SmolVLM2: pop 12, 2 generations, K=4
- Search seeds =
20260715, 20260716, 20260717
- Finalist selection cap =
at most 3 finalists per route family
axioms (6)
- domain assumption Identity operation is a valid substitute for a skipped block without fine-tuning or compensation
- domain assumption Capability labels are known (oracle) and each question maps to exactly one capability
- ad hoc to paper Equal-weight source-balanced cell aggregation is the correct optimization objective
- domain assumption The 876-example selection partition is representative of the capability distributions
- domain assumption Greedy decoding and short-answer exact-match accuracy are valid evaluation metrics
- domain assumption The evolutionary search with small populations and few generations finds routes near the true optimum
read the original abstract
Vision-language models normally execute the same complete vision encoder for every question, even when OCR, counting, object, attribute, and spatial queries may not require identical computation. We study whether fixed-budget combinations of vision blocks can be skipped without fine-tuning. A shared K-block route skips one searched set of exactly K blocks for every question, while a capability-specific K-block policy selects one same-size route using a known capability label. We introduce a source-balanced evolutionary search and compare it with independent ranking, contiguous removal, and random routes at matched budgets. Experiments use Qwen2.5-VL-3B-Instruct, SmolVLM2-2.2B-Instruct, and an 876-example image-disjoint selection split. Search transfers across architectures: on SmolVLM2, the searched shared four-block route beats independent construction by 4.91 percentage points. Capability specialization is less stable. On Qwen, the six-block capability policy beats the shared route by 2.17 points, driven by a 7.10-point OCR gain. On sealed IIIT5K, however, the SmolVLM2 OCR-specific route trails its shared route by 13.6 points. Combinatorial search reliably improves route construction, but capability labels do not define universally transferable vision pathways.
Figures
Reference graph
Works this paper leans on
-
[6]
URL https://arxiv.org/abs/2306.13394. Searching for Task-Specific Vision Paths 14 Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. InIEEE Conference on Computer Vision and Pattern Recognition,
-
[12]
Fangyu Liu, Guy Emerson, and Nigel Collier
URL https://arxiv.or g/abs/2305.10355. Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. InTransactions of the Association for Computational Linguistics, 2023a. URL https://arxiv.org/abs/2205.00363. Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xucheng Yin, Cheng-lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench:...
-
[13]
URL https://arxiv.org/abs/2507.23362
doi: 10.1145/3746027.3755122. URL https://arxiv.org/abs/2507.23362. Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect.arXiv preprint arXiv:2403.03853,
-
[14]
Anand Mishra, Karteek Alahari, and C
URL https://arxiv.org/abs/2403.03853. Anand Mishra, Karteek Alahari, and C. V. Jawahar. Scene text recognition using higher order language priors. InBritish Machine Vision Conference,
-
[17]
Ce Zhang, Kaixin Ma, Tianqing Fang, Wenhao Yu, Hongming Zhang, Zhisong Zhang, Haitao Mi, and Dong Yu
URL https://proceedings.neurips.cc/paper_files/paper/2025/hash/8 8441fee0c0dacceb95e4ad77ece5d0a-Abstract-Conference.html. Ce Zhang, Kaixin Ma, Tianqing Fang, Wenhao Yu, Hongming Zhang, Zhisong Zhang, Haitao Mi, and Dong Yu. Vscan: Rethinking visual token reduction for efficient large vision-language models.Transactions on Machine Learning Research,
2025
-
[18]
URL https://arxiv.org/abs/2505.22654. Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, et al. Sparsevlm: Visual token sparsification for efficient vision-language model inference. InInternational Conference on Machine Learning,
-
[19]
URL https: //arxiv.org/abs/2410.04417
-
[2002]
doi: 10.1109/4235.996017. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394,
-
[2012]
Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh
doi: 10.5244/C.26.127. Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. InAdvances in Neural Information Processing Systems,
-
[2017]
URL https://arxiv.org/abs/1612.00837. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models.arXiv preprint arXiv:2203.15556,
-
[2019]
URL https://arxiv.org/abs/1810.12440. Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. InInternational Conference on Learning Representations,
-
[2020]
Saeed Khaki, Nima Safaei, and Kamal Ginotra
URL https://arxiv.org/abs/2001.08361. Saeed Khaki, Nima Safaei, and Kamal Ginotra. Understanding pruning regimes in vision-language models through domain-aware layer selection.arXiv preprint arXiv:2603.20275,
Pith/arXiv arXiv 2001
-
[2021]
Sreetama Sarkar, Saket Gurukar, Yuexi Zhang, Monami Banerjee, and Ashwin Chandra
URL https://arxiv.org/abs/2106.02034. Sreetama Sarkar, Saket Gurukar, Yuexi Zhang, Monami Banerjee, and Ashwin Chandra. Flashvlm: Exploiting layer redundancy via visual attention for efficient vision-language inference.OpenReview preprint,
-
[2022]
URL https://arxiv.org/abs/2203.15556. Hugging Face. Smolvlm2-2.2b-instruct model card. Hugging Face model repository,
-
[2023]
Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and T
URL https://arxiv.org/abs/2210.09461. Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and T. Meyarivan. A fast and elitist multiobjective genetic algorithm: Nsga-ii.IEEE Transactions on Evolutionary Computation, 6(2):182–197,
-
[2024]
URL https://arxiv.org/abs/2401.15024. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923,
-
[2025]
Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman
URL https://arxiv.org/abs/2502.13923. Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. InInternational Conference on Learning Representations,
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.