{"id":"4cd265d8-5527-4e15-800f-0a38f733edbc","arxiv_id":"2608.06901","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A variance-based, retraining-free pruning framework for vision-language models that allocates per-layer sparsity and outperforms Wanda and SparseGPT at high sparsity.","lead":"PORTA is a pruning method for vision-language models that removes weights based on how much their activations vary across inputs, without retraining. It aims to compress one model once and use it across many image and text tasks, and it outperforms several existing pruning baselines at high sparsity.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'prune once' claim rests on an unreported hyperparameter alpha: the main CIFAR-10 numbers match per-sparsity best-alpha values tuned on CIFAR-10 in Appendix D, so downstream labels may have leaked into pruning decisions.","rationale":"The reader's weakest_assumption is exactly the load-bearing concern: the layer-wise allocation depends on alpha, and the paper does not show that alpha transfers across tasks without downstream tuning. I find concrete corroborating evidence in the numbers themselves: Table 3's CIFAR-10 values at 60% and 65% coincide with the per-sparsity best alpha in Appendix D Table A4, implying alpha was selected on a downstream task. This directly threatens the central task-agnostic claim, because pruning ratios are part of the pruning decision and CIFAR-10 labels would have been used to choose them. The other reader concerns are real but less central: the MSCOCO calibration/evaluation overlap is mitigated by the calibration-robustness results in Fig. 4 and Appendix E, and missing error bars are a reporting issue rather than a threat to the central mechanism. The alpha issue, by contrast, determines whether the headline experiment is a valid task-agnostic evaluation at all. The method may still work; the fix is to disclose alpha values and a label-free selection rule. I therefore keep the verdict conditional, matching the reader, and recommend no change to the verdict.","tokens_in":17059,"tokens_out":5924,"duration_ms":59600,"concrete_test":"Re-run Table 3 at 55%, 60%, and 65% sparsity with a single fixed alpha (e.g., alpha=10) and with a label-free sparsity schedule, e.g., alpha = c*log(S) fitted only on generic calibration data. Report the exact alpha used in every row. If the Table 3 numbers require the per-sparsity best alpha from CIFAR-10 in Table A4, the task-agnostic claim fails; if a fixed or label-free schedule reproduces them, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Algorithm 1 (Sec. 3.3) requires a strength hyperparameter alpha, but the main tables do not state which alpha was used. Appendix D, Table A4 sweeps alpha on CIFAR-10 at each sparsity level. The CIFAR-10 entries in Table 3 match the per-sparsity best values in Table A4 for 60% (95.06 with alpha=10) and 65% (90.49 with alpha=30). This is strong evidence that alpha was selected using a downstream classification task. If so, the central 'task-agnostic, prune-once' claim fails for the reported experiments: the layer-wise sparsity allocation is informed by CIFAR-10 labels, and the CIFAR-10 column is a selected result rather than an independent zero-shot evaluation. The paper's Appendix D text says larger alpha is preferred at higher sparsity, but it gives no label-free rule for setting alpha, and no closed-form schedule is proposed. Because alpha is not reported in Table 3, the reader cannot determine whether the retrieval and VQA results inherit the same tuning policy. The importance score itself is plausible and the robustness ablations in Sec. 4.4 are useful, but the audit trail for the headline 'prune once' workflow is incomplete.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PORTA, a retraining-free, task-agnostic pruning framework for vision-language models. The importance score combines per-feature activation variance with weight magnitude (Eq. 2), and layer-wise sparsity is allocated through a score based on normalized output covariance (Eqs. 10-12) and a strength parameter alpha in Algorithm 1. Experiments cover CLIP, BLIP, and Qwen2-VL on classification, retrieval, and VQA, with comparisons to Wanda, SparseGPT, ECoFLaP, and Multiflow. The paper reports strong high-sparsity results and includes ablations on calibration source, sample count, the zero-mean assumption, and the diagonal covariance approximation.","tokens_in":17267,"tokens_out":5913,"duration_ms":64927,"significance":"If the central claim holds, PORTA is a practically useful step: a single pruned VLM that can be deployed across heterogeneous tasks without re-pruning or retraining, using only generic calibration data. The variance-based motivation is well supported by the axis-removal experiment in Table 2, the cross-combination study in Table 6, and the calibration-robustness experiments in Fig. 4 and Appendix E. The paper also provides a clean derivation of the layer score and explicitly ablates its two modeling assumptions. Code release and the multi-architecture evaluation are additional strengths. The main weakness is that the hyperparameter alpha, which controls the layer-wise ratio allocation, is tuned per sparsity level on CIFAR-10 in Appendix D but is not reported for the main results, leaving the task-agnostic 'prune once' claim incompletely audited.","major_comments":[{"comment":"The main tables do not report the value of alpha used for each sparsity level, and Appendix D shows that the CIFAR-10 numbers in Table 3 track per-sparsity best-alpha values: at 60% sparsity the reported CIFAR-10 accuracy of 95.06 is the best entry for alpha=10 in Table A4, and at 65% sparsity the reported 90.49 is the best entry for alpha=30. This is strong evidence that the layer-wise allocation strength was selected on CIFAR-10 labels, which conflicts with the paper's stated protocol of pruning without downstream task-specific samples. Please report alpha for every main-table result, provide a label-free rule or closed-form schedule for setting alpha, and clarify whether the retrieval and VQA results in Tables 3-5 were obtained with the same alpha as the classification column or with per-task selection.","section":"Sec. 4.3 / Appendix D / Table A4"},{"comment":"Even if alpha is reported, the 'prune once' claim requires that a single alpha transfers across tasks and modalities. The current evidence is limited to CIFAR-10 classification on CLIP (Table A4) and one visualization at 60% sparsity (Fig. 3). Since Table A4 itself shows that the preferred alpha shifts with target sparsity (from roughly 5-10 at 50-55% to 30 at 65%), the paper should either give a principled, label-free schedule or demonstrate with a fixed alpha, held out from downstream validation, that the results in Tables 3-5 remain competitive across all task families. Without this, the central task-agnostic claim is not auditable.","section":"Algorithm 1 / Sec. 3.3"},{"comment":"The paper describes all evaluations as 'fully zero-shot,' but hyperparameter selection on CIFAR-10 in Appendix D means the CIFAR-10 column at least is a selected result rather than an independent zero-shot evaluation. The authors should state explicitly that alpha tuning was performed on a downstream dataset, or remove this apparent leak by reporting alpha values and a label-free selection mechanism.","section":"Sec. 4.1 / 'zero-shot' protocol"}],"minor_comments":[{"comment":"The notation for input activations is inconsistent: Sec. 3.2 uses X in R^{B x T x D_in} while Sec. 3.3 uses X in R^{T x D_in}; please unify the notation and clarify whether the variance in Eq. (1) is computed over both batch and token dimensions or over tokens only.","section":"Sec. 3.2 / Sec. 3.3"},{"comment":"The returned ratios r_i are not explicitly constrained to [0,1]; if C u_i exceeds 1 for some layer, the allocation would be invalid. Please specify clipping or normalization and confirm that the final global sparsity exactly matches the target S.","section":"Algorithm 1"},{"comment":"The caption states alpha=10 for the 60% sparsity visualization, but Table 3 does not report which alpha was used for the corresponding 60% results; please align the visualization with the evaluated configuration.","section":"Fig. 3 caption"},{"comment":"The abstract and contribution statements describe the importance criterion as 'activation variation,' but Eq. (2) is Var * |W|; the weight-magnitude factor should be acknowledged in the summary of the method.","section":"Abstract / Sec. 1"},{"comment":"Table 1 reports max-min dispersion for layers labeled L1, L20, L30 but does not state which model and which layers are used; please add the experimental details so the reader can interpret the modality-sensitivity claim.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"I am not questioning the variance-based importance score itself; the cross-combination ablation and the calibration robustness experiments are credible and useful. The blocking issue is the missing audit trail for alpha. If the authors supply the alpha values used in all main tables and either a label-free selection rule or a fixed-alpha sensitivity analysis over the full task suite, I would support acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. This is a real contribution, not a repacking. The variance-times-magnitude importance score is a natural but genuinely useful variant of Wanda, and the layer-wise ratio mechanism is new in this form. The authors do the work: Table 2 (variance ablation removal), Table 6 (cross-combination of score and ratio), and Table 7 (zero-mean and diagonal covariance assumptions) are exactly the ablations you'd want. Evaluation breadth is impressive—CLIP, BLIP, Qwen2-VL, plus SDXL and OpenFlamingo in the appendix. Calibration robustness across Flickr30K, MSCOCO, Visual Genome, NoCaps, and PASCAL-50S is a strong point. Code release helps too. The importance score is not circular: variance is computed from calibration activations, not fit to downstream labels.\n\nThe soft spots are real but narrower than the stress-test suggests. The alpha strength hyperparameter in Algorithm 1 is not reported in the main tables, and Appendix D shows the CIFAR-10 numbers at 60% and 65% sparsity match the per-sparsity best alpha values. That is a genuine audit failure for the 'no task-specific samples at pruning time' claim. I checked the 55% row: the reported CIFAR-10 (95.58) corresponds to alpha=30, but the best in Table A4 at 55% is 96.40 with alpha=10. So the pattern is not a clean one-to-one match at every sparsity level. Still, two of three matching the tuned best is enough to require an explanation. The paper either needs to report alpha for every main result or provide a label-free rule for setting alpha based only on target sparsity. Without that, the 'prune once' workflow is not fully auditable.\n\nTwo more minor points. Retrieval is calibrated and evaluated on MSCOCO, which is in-distribution for that task, though the calibration robustness results in Fig. 4 and Table A5 mitigate the concern. Main tables lack error bars; the appendix shows variance across calibration samples is small, so this is minor.\n\nThe stress-test claim that the central claim 'fails' overstates it. The importance score itself is not fit to CIFAR-10; alpha is a scalar that modulates layer-wise allocation. Tuning one scalar on a validation set is standard practice, but it undercuts the strict zero-shot story and should be disclosed. The paper is otherwise carefully done and the experiments are believable.\n\nWho is this for? Anyone working on post-training compression of VLMs or importance-score design. It deserves a serious referee. I'd send it with a request for alpha reporting, out-of-domain retrieval evaluation, and standard errors in main tables. Not a desk reject.","headline":"A solid, well-ablated VLM pruning paper whose 'prune once' claim is undercut by an unreported per-sparsity alpha hyperparameter that appears to track CIFAR-10 tuned values.","tokens_in":17858,"tokens_out":4540,"would_cite":true,"duration_ms":39955,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PORTA prunes a vision-language model once using activation-variance importance and adaptive layer-wise sparsity, so the same compressed model transfers to classification, retrieval, and VQA tasks without retraining.","keywords":["vision-language models","post-training pruning","task-agnostic pruning","retraining-free compression","activation variance","layer-wise sparsity allocation","zero-shot evaluation","model compression"],"falsifier":"Run PORTA on a held-out VLM with the $\\alpha$ values used in the paper and a fixed generic calibration set, then measure zero-shot performance across several unseen tasks; if the same $\\alpha$ does not reproduce the reported rankings, or if an alternative calibration set shifts performance by far more than 1%, the prune-once claim would be falsified. A sharper test is to find one VLM layer whose high-variance features are redundant and whose low-variance features are task-critical; pruning by variance should fail dramatically there.","tokens_in":1904,"feed_emoji":"✂️","tokens_out":1875,"duration_ms":88655,"temperature":0.7,"pith_summary":"PORTA is a pruning framework for vision-language models that aims to make the pruned model usable across many downstream tasks immediately after a single pruning pass, with no retraining or re-pruning. The paper argues that the usual magnitude-based importance measures, borrowed from large language models, are biased by the different activation scales of vision and text branches, and that the variance of activations along the token/patch dimension is a stable, modality-agnostic signal of a feature's representational utility. PORTA multiplies this per-feature variance by weight magnitude to score weights, and additionally assigns each layer a pruning ratio from the variability of its output features. Across CLIP, BLIP, Qwen2-VL, OpenFlamingo, and SDXL text encoders, the paper reports competitive zero-shot results on classification, retrieval, VQA, and captioning, with the clearest gains over Wanda and SparseGPT appearing at high sparsity.","feed_headline":"Prune once, deploy everywhere: variance scores keep VLMs accurate","feed_subtitle":"One pruned model serves classification, retrieval, and VQA—no re-pruning, no fine-tuning.","key_machinery":"The object carrying the argument is the per-feature activation variance vector $v_j = \\mathrm{Var}_{b,t}(X_{b,t,j})$ for input features of each linear layer, combined with the weight matrix as $S_{i,j} = v_j |W_{i,j}|$ for weight importance. The layer-wise allocation uses the normalized input covariance $\\hat\\Sigma_X = \\Sigma_X / \\mathrm{Tr}(\\Sigma_X)$ and the score $s_{\\mathrm{layer}} = \\frac{1}{D_{\\mathrm{out}}} \\sum_o w_o \\hat\\Sigma_X w_o^\\top$, the average output variance per channel, which is also written as a variance-weighted average of column norms $\\|W_{:,j}\\|_2^2$. Algorithm 1 converts these scores into pruning ratios $r_i = C (1 - p_i)^\\alpha$ with a single strength parameter $\\alpha$, after weighting by each layer's parameter count. The core work is to replace magnitude-based outlier reasoning with variance-based reasoning at both the weight level and the layer level.","core_discovery":"The central claim is that activation variance, not activation magnitude, is the right importance signal for pruning vision-language models. In CLIP-style contrastive models the vision and text encoders have very different activation scales and outlier patterns, so a magnitude-based score such as Wanda's prunes the two branches unevenly; variance, measured per input feature across tokens or patches, is shown to have similar dispersion in both branches and to track representation utility. Removing high-variance features from CLIP destroys retrieval accuracy, while removing low-variance features leaves it nearly intact. On top of that, PORTA computes a layer score from the normalized input covariance and the squared norms of output weight vectors, and uses it to allocate smaller pruning ratios to layers whose outputs use more of the feature space. The paper claims that together these two mechanisms let a VLM be pruned once with generic image-text calibration data and then applied zero-shot to heterogeneous tasks, outperforming Wanda, SparseGPT, ECoFLaP, and Multiflow at high sparsity.","pith_inferences":["If the variance-based signal generalizes beyond the tested families, the same scoring could be applied to other multimodal architectures such as audio-text, video-text, or interleaved generation models without redesigning the calibration step.","The reuse of variance statistics for both weight scoring and ratio allocation suggests the method could fold into quantization-aware compression pipelines, since both need activation statistics and the paper already shows a 2:4 structured-pruning extension.","A testable extension is to check whether the same $\\alpha$ value transfers across architectures; the paper's sensitivity study shows the preferred $\\alpha$ drifts with target sparsity, so an automatic rule for $\\alpha$ from target sparsity and model depth would make the prune-once workflow fully parameter-free.","The axis-removal result implies variance could serve as a diagnostic for other structure-discovery tasks, such as identifying which feature dimensions a modality encoder actually uses."],"forward_implications":["One pruned VLM can be stored once and served to classification, retrieval, VQA, and captioning workloads without re-pruning or fine-tuning, removing the per-task compression pipeline.","High-sparsity deployment becomes practical: at 65% sparsity on CLIP, PORTA reports a 12.6% mean improvement over SparseGPT and 21.5% over Wanda across the eight reported metrics.","Calibration becomes cheap and robust: PORTA reports roughly 1% variation across Flickr30K, MSCOCO, and Visual Genome calibration at 60% sparsity, and near-flat performance from 64 to 2048 calibration samples.","The same variance-based importance transfers to generative VLMs: Qwen2-VL on ScienceQA reaches 54.30 average accuracy at 50% sparsity, the best among pruned baselines.","Pruning overhead drops: PORTA prunes CLIP in about 195 seconds, 3.96x faster than SparseGPT and 2.06x faster than ECoFLaP."],"supporting_citations":[{"why":"Defines the magnitude-based Wanda baseline and the LLM activation-outlier assumptions that PORTA argues do not transfer to vision branches.","marker":"[35]"},{"why":"SparseGPT, the Hessian-based one-shot pruning baseline PORTA compares against on retrieval and classification.","marker":"[12]"},{"why":"Multiflow formalizes task-agnostic VLM pruning and is the strongest task-agnostic baseline; PORTA compares with and without Multiflow's fine-tuning.","marker":"[11]"},{"why":"ECoFLaP supplies a VLM-specific layer-wise ratio baseline and an importance score (Wanda) that PORTA's ratio mechanism is compared with.","marker":"[36]"},{"why":"Evidence that calibration-data choice shifts Wanda and SparseGPT results, motivating PORTA's variance-based robustness to calibration.","marker":"[45]"},{"why":"CLIP is the primary testbed architecture for the zero-shot classification and retrieval experiments.","marker":"[33]"},{"why":"BLIP provides the second contrastive VLM architecture used to test transfer of the pruning criterion.","marker":"[24]"},{"why":"Qwen2-VL is the generative VLM used for the ScienceQA VQA experiments.","marker":"[42]"},{"why":"Shows that final CLIP layers carry most semantic representation, supporting the layer-wise sparsity allocation pattern PORTA produces.","marker":"[13]"},{"why":"Supports the observation that higher target sparsity benefits from a wider layer-wise allocation range, used to explain the $\\alpha$ schedule.","marker":"[8]"}],"fun_headline_variants":["Variance, not magnitude, for one-shot VLM pruning","Retraining-free VLM pruning via activation variance","Prune once, no fine-tune: VLM variance scores","Activation variance guides retraining-free VLM pruning","Task-agnostic VLM pruning using variance scores"],"cache_read_input_tokens":19968,"weakest_assumption_plain":"The load-bearing premise is that one strength parameter $\\alpha$, combined with the layer output-variance scores, transfers across tasks, modalities, and sparsity levels without per-task recalibration; the paper's own sensitivity table shows the best $\\alpha$ drifts as target sparsity changes, so if $\\alpha$ must be retuned per deployment the 'prune once' advantage narrows.","fun_headline_variants_meta":{"raw":{"variants":["Variance, not magnitude, for one-shot VLM pruning","Retraining-free VLM pruning via activation variance","Prune once, no fine-tune: VLM variance scores","Activation variance guides retraining-free VLM pruning","Task-agnostic VLM pruning using variance scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000795,"raw_usage":{"total_tokens":3508,"prompt_tokens":959,"completion_tokens":2549,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":2469}},"tokens_in":575,"tokens_out":2549,"duration_ms":21752,"temperature":1.0,"reasoning_tokens":2469,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:41:06.855919+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run PORTA on a held-out VLM with the $\\alpha$ values used in the paper and a fixed generic calibration set, then measure zero-shot performance across several unseen tasks; if the same $\\alpha$ does not reproduce the reported rankings, or if an alternative calibration set shifts performance by far more than 1%, the prune-once claim would be falsified. A sharper test is to find one VLM layer whose high-variance features are redundant and whose low-variance features are task-critical; pruning by variance should fail dramatically there.","supporting_citations":[{"cited_title":"ICLR (2024)","cited_arxiv_id":null,"evidence_quote":"Defines the magnitude-based Wanda baseline and the LLM activation-outlier assumptions that PORTA argues do not transfer to vision branches."},{"cited_title":"In: ICML (2023)","cited_arxiv_id":null,"evidence_quote":"SparseGPT, the Hessian-based one-shot pruning baseline PORTA compares against on retrieval and classification."},{"cited_title":"In: International Conference on Learning Repre- sentations (ICLR) (2024)","cited_arxiv_id":null,"evidence_quote":"ECoFLaP supplies a VLM-specific layer-wise ratio baseline and an importance score (Wanda) that PORTA's ratio mechanism is compared with."},{"cited_title":"In: ICML (2021) 18 Kang et al","cited_arxiv_id":null,"evidence_quote":"CLIP is the primary testbed architecture for the zero-shot classification and retrieval experiments."},{"cited_title":"In: International Conference on Machine Learning (ICML) (2025)","cited_arxiv_id":null,"evidence_quote":"Supports the observation that higher target sparsity benefits from a wider layer-wise allocation range, used to explain the $\\alpha$ schedule."}],"review_version":1}