{"id":"c36a44b7-d2e8-46d3-b9bd-caa62d15c5ea","arxiv_id":"2507.20613","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A TPE-guided search over layer-wise pruning ratios and KV cache bit-widths compresses LLaVA-1.5 7B/13B with small accuracy loss, outperforming Wanda and SparseGPT on most tested benchmarks.","lead":"This paper proposes an adaptive search method that assigns different sparsity and KV cache quantization levels to different layers of vision-language models, then reports that the tuned models beat uniform-compression baselines on several benchmarks. It matters because it is an engineering recipe for running large multimodal models on smaller devices with less memory.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline claim of uniform 50%-sparsity advantage is not established because the TPE search objective is Wikitext-2 perplexity, yet the benchmarks in Table II are multimodal; the paper never measures the correlation between this proxy and VQA accuracy.","rationale":"The reader identified the same load-bearing weakness: the search objective (Wikitext-2 PPL) is never shown to correlate with downstream multimodal accuracy. My stress-test agrees that this is the most insecure link in the argument. The paper's own tables provide partial evidence: in Table II, the 13B POPE result is worse than baselines, so 'outperforms all benchmarks' is false as stated. No error bars or repeated trials are reported, and the 'Ours' method differs from baselines in two ways (metric and allocation) without an ablation separating them. The claimed insight about U-shaped sparsity allocation and early-layer precision is derived from single runs, so it may be noise. The proposed correction is concrete and relatively cheap: isolate the two variables, measure the proxy correlation, and report variance. This would either support the central claim or reduce the claim to a metric improvement, which is still useful but much weaker. The verdict remains CONDITIONAL because the engineering contribution is plausible and the code is promised, but the stronger claims require the missing ablations and correlation analysis.","tokens_in":471,"tokens_out":724,"duration_ms":61846,"concrete_test":"Run a controlled ablation on LLaVA-1.5 Vicuna-7B at 50% overall sparsity: (1) uniform sparsity with the proposed metric; (2) TPE-searched allocation with the vanilla Wanda metric; (3) TPE-searched allocation with the proposed metric; (4) uniform sparsity with Wanda and SparseGPT as controls. Evaluate all four on VQAv2, SQA, TextVQA, POPE, and Wikitext-2, with three seeds or evaluation runs and standard deviations. If (2) does not consistently beat (1) and (4), the adaptive allocation contributes nothing beyond the metric, and the central claim fails.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central empirical claim is that the proposed layer-wise sparsity allocation plus new pruning metric beats uniform SparseGPT and Wanda at 50% overall sparsity on VQAv2, SQA, TextVQA, and POPE. The allocation is selected by Algorithm 1, whose objective is Wikitext-2 perplexity (Section III.C: 'PPL is chosen as it reflects model performance'). The paper never validates that minimizing text-only PPL improves multimodal benchmarks. If the proxy correlation is weak, the profile chosen by TPE is effectively a random or overfit configuration, and the observed Table II gains could reflect the new pruning metric alone rather than the adaptive allocation. The paper also never isolates the metric from the allocation, and the 13B POPE row in Table II (83.94 vs. SparseGPT 86.90 and Wanda 85.01) directly contradicts the Section IV.B statement that 'our method outperforms others on all benchmarks.' Thus the load-bearing assertion—that the full system beats uniform baselines due to adaptive allocation—is not supported by the presented evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an adaptive compression framework for large multimodal models (LMMs) that jointly searches layer-wise unstructured sparsity ratios and KV cache quantization bit-widths using the Tree-structured Parzen Estimator (TPE), together with a new pruning metric defined in Eq. (4). The method is evaluated on LLaVA-1.5 7B and 13B on VQAv2, SQA, TextVQA, POPE, and Wikitext-2 perplexity, and the authors claim that it outperforms uniform SparseGPT and Wanda at 50% overall sparsity, and that the adaptive KV cache bandwidth allocation outperforms the reverse allocation.","tokens_in":10086,"tokens_out":5590,"duration_ms":59972,"significance":"If the results hold, the framework would offer a practical recipe for deploying compressed LMMs on resource-constrained devices, and the combination of pruning and KV cache quantization with a TPE-based search is a timely and relevant direction. The paper also reports a new pruning metric that is computationally cheap and potentially useful. However, the current evidence is insufficient to support the central claims: the search objective (PPL on Wikitext-2) is not shown to transfer to multimodal benchmarks, the pruning metric is not ablated separately from the allocation, and the claims of universal superiority are directly contradicted by the paper's own Table II.","major_comments":[{"comment":"The text states that 'our method outperforms others on all benchmarks,' but in the Vicuna-13B part of Table II, the POPE column shows Ours = 83.94 while SparseGPT = 86.90 and Wanda = 85.01, i.e., our method is worse than both baselines on this benchmark. This is a direct factual contradiction of the claim. The authors must either correct the claim to acknowledge exceptions or explain why this particular benchmark is less indicative of the method's value.","section":"Section IV.B, Table II"},{"comment":"The search objective in Algorithm 1 is Wikitext-2 perplexity, yet the headline results are on multimodal VQA benchmarks. The paper asserts that 'PPL is chosen as it reflects model performance' but never provides any evidence of a correlation between text-only PPL and multimodal task accuracy. In fact, Table II provides a counterexample: for Vicuna-13B, Ours has the best PPL (6.66) but the worst POPE among the three compression methods (83.94 vs. SparseGPT 86.90 and Wanda 85.01). Without a correlation analysis or a multimodal validation split in the search, the selected sparsity and bandwidth profiles may be overfit to PPL, and the observed VQA gains cannot be attributed to the adaptive allocation rather than to the new pruning metric or other confounds. Please add such an analysis or adapt the search objective to a multimodal metric.","section":"Section III.C, Algorithm 1, and Table II"},{"comment":"The pruning metric in Eq. (4) is never ablated separately from the sparsity allocation search. As a result, it is impossible to determine whether the improvements over SparseGPT/Wanda come from the new metric, from the adaptive layer-wise allocation, or from both. The central claim that the framework's allocation is beneficial requires an ablation with at least three settings: (i) uniform sparsity + Wanda metric, (ii) uniform sparsity + proposed metric, (iii) adaptive allocation + proposed metric. Only then can the reader assess the contribution of each component.","section":"Section III.E and Table II"},{"comment":"The pseudocode in Algorithm 1 rejects profiles with 'mean(Pt) > σoverall' (continue to next trial), while the text in Section III.C states that 'we ensure the average sparsity meets or exceeds the specified overall sparsity before proceeding.' These two statements are inconsistent. If the code is interpreted literally, the accepted profiles satisfy mean(Pt) ≤ σoverall, i.e., the actual sparsity is below the stated 50% target, which would give the method an unfair advantage over baselines that are pruned to exactly 50%. If the text is correct, then the pseudocode condition should be 'mean(Pt) < σoverall' instead. Please clarify and verify that the effective overall sparsity is exactly (or at least) the same for all methods in the comparison.","section":"Algorithm 1, line 4"},{"comment":"The search space for each layer's sparsity is limited to ±5% around the overall sparsity in 2.5% increments (e.g., {45%, 47.5%, 50%, 52.5%, 55%} for 50% overall). This is a narrow range, and the paper does not compare the TPE search against a random search over the same space or against a uniform allocation with the same pruning metric. Without such a comparison, the claimed advantage of 'adaptive' allocation over uniform allocation is not isolated; the observed differences could be within the noise of the restricted search space.","section":"Section III.C"}],"minor_comments":[{"comment":"The abstract contains grammatical errors, e.g., 'our framework automatic allocation' should be 'our framework's automatic allocation,' and the phrases 'sets a new standard in LMM optimization' and 'superiority over state-of-the-art techniques' are promotional and not supported by the data, especially given the POPE exception in Table II.","section":"Abstract and Section IV.B"},{"comment":"The statement 'PPL is chosen as it reflects model performance' conflates text-only perplexity with overall model performance, including multimodal capabilities. It would be more precise to say that PPL is a convenient proxy and to provide evidence for its correlation with the downstream benchmarks used in the paper.","section":"Section III.C"},{"comment":"The 'Opposite' row is described as 'the reverse allocation,' but it is not specified whether this means the exact reversed profile of 'Ours' or simply swapping the layer groups that receive 6-bit and 8-bit. Please define this clearly for reproducibility.","section":"Table IV"},{"comment":"The activation term in the proposed metric is sqrt(||X_j||_2), while Wanda uses ||X_j||_2 without the square root. No justification is given for this square root, and it is not discussed in the ablation. A brief explanation or a sensitivity analysis would strengthen the metric's description.","section":"Eq. (4)"},{"comment":"Table I compares compressed LLaVA models to uncompressed models such as BLIP-2 and InstructBLIP. Since the compressed models start from a strong pretrained baseline, the comparison is not apples-to-apples. The text should clarify that this table is only a reference to show the compressed models remain competitive with existing uncompressed LMMs, not a head-to-head comparison of compression methods.","section":"Table I"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a relevant problem and the TPE-based search plus new pruning metric is an interesting direction. However, the central claim of universal superiority is contradicted by the authors' own Table II (Vicuna-13B POPE), and the missing ablation isolating the pruning metric from the allocation leaves the source of gains unclear. The proxy-correlation issue (PPL vs. multimodal benchmarks) is also load-bearing and needs a direct empirical check. These are fixable with additional experiments and corrected claims, so I recommend major revision rather than rejection. The authors should be asked to provide the ablations and correlation analysis, and to temper the overclaims in the abstract and Section IV.B."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Honest incremental compression paper. The genuinely new bit is the combination of TPE-based layer-wise sparsity allocation with KV cache bandwidth search on LMMs, plus a Wanda-style metric that adds row/column L2 norms and a log. They ship code, test on LLaVA-1.5 7B/13B and LLaMA 1/2, and report gains over uniform SparseGPT/Wanda on VQAv2, SQA, TextVQA at 50% sparsity. The allocation insights (U-shaped sparsity; early layers need higher KV precision) are plausible and useful.\n\nThe soft spots are real, though. The search objective is Wikitext-2 perplexity; the evaluation is multimodal. The paper never checks whether PPL tracks VQA accuracy, so we don't know if the TPE-selected profiles are anything more than random good configurations. That's the load-bearing gap. Second, the pruning metric is never ablated from the allocation search: the observed gains could come from the metric alone, not the adaptive allocation. Third, the \"outperforms all baselines\" claim in Section IV.B is directly contradicted by the 13B POPE row (Ours 83.94 vs SparseGPT 86.90). Fourth, the paper cites refs 13-15 (layer-wise sparsity allocation methods) but never compares numerically against them, which is a notable omission. Fifth, the Wikitext PPL numbers in Table II are the search objective itself, so those are selected rather than predicted; a held-out split or repeated-search variance would help.\n\nNone of this kills the paper. The central result—modest gains on most benchmarks—is probably true, and the code is there to check. The overclaims should be trimmed, and the proxy correlation plus an ablation of the metric would turn a conditional into a solid accept. I'd send this to peer review with a request for major revision. I wouldn't cite it yet, but I'd happily read a revised version.","headline":"A credible but overclaimed compression paper: TPE-based layer-wise sparsity and KV bandwidth allocation works on most benchmarks, but the search proxy is unvalidated and one table row contradicts the 'outperforms all' claim.","tokens_in":10642,"tokens_out":2909,"would_cite":false,"duration_ms":31276,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adaptive per-layer allocation of sparsity and KV-cache bit widths, found by a Tree-structured Parzen Estimator using text perplexity as the objective, lets LLaVA-1.5 models shed half their weights and compress the KV cache while keeping…","keywords":["large multimodal models","model compression","pruning","KV cache quantization","layer-wise sparsity allocation","tree-structured parzen estimator","post-training pruning","LLaVA"],"falsifier":"On LLaVA-1.5 7B, take a set of random sparsity profiles at 50% overall sparsity, measure each profile's Wikitext-2 perplexity and VQAv2 accuracy, and compute the rank correlation between the two. If the correlation is not strongly negative, the search objective is not selecting for the downstream task. More directly, run the TPE search twice, once optimizing Wikitext-2 perplexity and once optimizing a held-out VQAv2 split, and compare the two profiles' test-set VQAv2 scores; if the perplexity-optimized profile does not match the VQA-optimized one, the paper's claim that the searched allocation is the best one fails.","tokens_in":9630,"feed_emoji":"🤖","tokens_out":9764,"duration_ms":92605,"temperature":0.7,"pith_summary":"The paper is trying to establish that uniform compression is the wrong default for large multimodal models. It shows that a Bayesian search over layer-wise sparsity ratios and KV-cache quantization bit widths, guided only by text perplexity, produces compressed LLaVA-1.5 models (7B and 13B) that outperform uniform SparseGPT and Wanda at 50% overall sparsity on VQAv2, ScienceQA, TextVQA, and POPE. If true, this means substantial memory savings for edge deployment without any fine-tuning or weight adjustment, and it reveals where compression hurts least: keep early and late layers dense, and keep early KV caches in higher precision. The caveat to keep in mind is that the search objective is Wikitext-2 perplexity, and the paper assumes this tracks multimodal task accuracy.","feed_headline":"Per-layer search keeps multimodal LLMs accurate at 50% sparsity","feed_subtitle":"A Bayesian optimizer allocates pruning and KV-cache bits per layer, beating uniform SparseGPT and Wanda on four VQA benchmarks.","key_machinery":"The load-bearing mechanism is the trial loop in Algorithm 1: a Tree-structured Parzen Estimator proposes a full layer-wise sparsity profile (and, for the KV cache, a bit-width profile), the model is pruned or quantized with that profile, and Wikitext-2 perplexity is measured; the estimator then models which profiles produced low perplexity and proposes the next trial. This loop is cheap because the pruning metric in Equation 4 is post-training and needs no fine-tuning: each weight is scored by a log-scaled combination of its row and column L2-normalized magnitudes multiplied by the input activation norm. The search runs about 30 to 50 trials, with each trial taking one to two minutes on an A100 for LLaVA-1.5 7B.","core_discovery":"The central claim is that adaptively allocating compression resources across layers beats applying the same sparsity ratio or the same quantization bit width to every layer. With an overall 50% unstructured sparsity budget, the TPE-searched layer-wise sparsity profile combined with a new pruning metric beats Magnitude, SparseGPT, and Wanda on all four benchmarks for LLaVA-1.5 Vicuna-7B, and on all but POPE for Vicuna-13B. Under a constraint that half the layers use 8-bit and half use 6-bit KV cache quantization, the searched placement of those bit widths beats the reversed ('Opposite') placement on all four benchmarks, often by several points on ScienceQA and TextVQA. The paper also reports the allocation patterns it finds: sparsity follows a U-shape across depth (denser at early and late layers), while KV cache precision favors earlier layers.","pith_inferences":["The search objective is text-only Wikitext-2 perplexity; if perplexity does not rank-order profiles by their visual-question-answering accuracy, the searched profiles are arbitrary. This could be tested by re-running the search with a small VQAv2 validation split as the objective and comparing the resulting profiles and accuracies.","The same TPE machinery could tune other per-layer choices, such as weight quantization precision or attention-head counts, since nothing in Algorithm 1 is specific to sparsity or KV bits; the conclusion already gestures at mixed precision.","The 'Opposite' allocation row in Table IV shows the performance gradient across layer placements is steep, so an even finer-grained search over mixes beyond half-8-bit/half-6-bit might recover some of the remaining loss."],"forward_implications":["At 50% overall sparsity, the 7B model loses about 2.4 points on VQAv2 (78.5 to 76.1) and 1.2 points on ScienceQA (66.8 to 65.6), far less than the 13-plus-point drops from magnitude pruning.","Adding searched 8/6-bit KV cache quantization on top of the pruned 7B model costs less than one point on most benchmarks, whereas a naive 6-bit-everywhere allocation collapses ScienceQA.","The discovered profiles give a reusable heuristic for other transformer-based multimodal models: spend sparsity budget in the middle layers and KV precision in the early layers.","The same search transfers to single-modality LLaMA-1 and LLaMA-2 models, so the benefit is not specific to vision-language training."],"supporting_citations":[{"why":"Supplies the SparseGPT one-shot pruning baseline that the adaptive allocation must outperform on the four benchmarks.","marker":"[19]"},{"why":"Supplies the Wanda baseline and the activation-aware magnitude metric that the paper's pruning metric extends.","marker":"[20]"},{"why":"Supplies the Tree-structured Parzen Estimator algorithm used to propose layer-wise compression profiles.","marker":"[25]"},{"why":"Supplies the LLaVA-1.5 model family (Vicuna 7B/13B) used as the base models for all compression and evaluation.","marker":"[31]"},{"why":"Supplies the uniform asymmetric quantization scheme applied to the KV cache in the bandwidth-allocation experiments.","marker":"[24]"},{"why":"Provides the VQAv2 benchmark, one of the four downstream tasks on which compressed models are scored.","marker":"[32]"},{"why":"Provides the ScienceQA benchmark, the multimodal reasoning task where naive 6-bit KV quantization collapses and allocation matters most.","marker":"[33]"},{"why":"Provides the TextVQA benchmark for measuring the effect of compression on reading text inside images.","marker":"[34]"},{"why":"Provides the POPE hallucination benchmark, the one task where the 13B pruned model does not beat the baselines.","marker":"[35]"}],"fun_headline_variants":["Adaptive per-layer sparsity and KV cache beats uniform pruning","Bayesian search tunes per-layer compression for multimodal LLMs","Per-layer sparsity and KV cache search keeps multimodal LLMs accurate","Adaptive compression allocation outperforms SparseGPT and Wanda","Per-layer pruning and cache bits beat uniform methods in LMMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The search chooses profiles by minimizing Wikitext-2 perplexity, and the paper assumes that ordering carries over to visual question answering and hallucination benchmarks; if that correlation is weak, the selected profiles are arbitrary and the claimed edge over uniform allocation is not established.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive per-layer sparsity and KV cache beats uniform pruning","Bayesian search tunes per-layer compression for multimodal LLMs","Per-layer sparsity and KV cache search keeps multimodal LLMs accurate","Adaptive compression allocation outperforms SparseGPT and Wanda","Per-layer pruning and cache bits beat uniform methods in LMMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000351,"raw_usage":{"total_tokens":1903,"prompt_tokens":919,"completion_tokens":984,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":897}},"tokens_in":535,"tokens_out":984,"duration_ms":9940,"temperature":1.0,"reasoning_tokens":897,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T13:25:45.900951+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On LLaVA-1.5 7B, take a set of random sparsity profiles at 50% overall sparsity, measure each profile's Wikitext-2 perplexity and VQAv2 accuracy, and compute the rank correlation between the two. If the correlation is not strongly negative, the search objective is not selecting for the downstream task. More directly, run the TPE search twice, once optimizing Wikitext-2 perplexity and once optimizing a held-out VQAv2 split, and compare the two profiles' test-set VQAv2 scores; if the perplexity-optimized profile does not match the VQA-optimized one, the paper's claim that the searched allocation is the best one fails.","supporting_citations":[{"cited_title":"A simple and effective pruning approach for large language models,","cited_arxiv_id":null,"evidence_quote":"Supplies the Wanda baseline and the activation-aware magnitude metric that the paper's pruning metric extends."},{"cited_title":"Improved baselines with visual instruction tuning,","cited_arxiv_id":null,"evidence_quote":"Supplies the LLaVA-1.5 model family (Vicuna 7B/13B) used as the base models for all compression and evaluation."},{"cited_title":"Quantization and training of neural networks for efficient integer- arithmetic-only inference,","cited_arxiv_id":null,"evidence_quote":"Supplies the uniform asymmetric quantization scheme applied to the KV cache in the bandwidth-allocation experiments."},{"cited_title":"Making the v in vqa matter: Elevating the role of image understanding in visual question answering,","cited_arxiv_id":null,"evidence_quote":"Provides the VQAv2 benchmark, one of the four downstream tasks on which compressed models are scored."},{"cited_title":"Learn to explain: Multimodal reasoning via thought chains for science question answering,","cited_arxiv_id":null,"evidence_quote":"Provides the ScienceQA benchmark, the multimodal reasoning task where naive 6-bit KV quantization collapses and allocation matters most."},{"cited_title":"Towards vqa models that can read,","cited_arxiv_id":null,"evidence_quote":"Provides the TextVQA benchmark for measuring the effect of compression on reading text inside images."}],"review_version":1}