{"id":"6f7e730e-e806-4678-a720-055a8daa9dfb","arxiv_id":"2411.14345","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A consensus of multiple similarity metrics selects which layers to prune, preserving accuracy, reducing computation, and improving robustness in the tested models.","lead":"This paper introduces Consensus, a layer pruning criterion that ranks how similar a network remains after removing each layer, using several representation-similarity metrics at once. The authors report that pruned models keep accuracy, cut FLOPs substantially, and show better robustness to adversarial and out-of-distribution inputs than single-metric pruning.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The triple-win claim rests on uncontrolled baseline numbers and single-run comparisons; the aggregation benefit is not yet statistically established.","rationale":"The reader's weakest-assumption analysis focuses on whether the final-representation distance is a faithful proxy for layer importance. That is a legitimate heuristic concern, but the method's validity can be tested empirically, and the paper's own CKA comparison is the relevant evidence. The more immediate threat is that the empirical evidence does not control for training variability or baseline protocols. Literature numbers are explicitly taken from original papers, no confidence intervals are reported, and the only aggregation-versus-individual comparison has a maximum gap of 0.60 pp without variance. Without multiple seeds, this is insufficient to distinguish a real effect from noise. If the proposed controlled re-run shows Consensus consistently ahead, the conclusion stands; if not, the triple-win claim reduces to run-to-run variation. The reader's CONDITIONAL verdict remains appropriate, with acceptance contingent on a controlled multi-seed evaluation.","tokens_in":16137,"tokens_out":4582,"duration_ms":49901,"concrete_test":"Run the full iterative pruning protocol for Consensus, CKA, and each individual metric on ResNet56/CIFAR-10 with identical fine-tuning hyperparameters and at least 5 random seeds. At matched FLOP-reduction levels (e.g., 37.52%, 48.78%, 60.04%, 75.05%, 78.80%), report mean plus/minus standard deviation of clean accuracy, CIFAR-10.2, CIFAR-C, and FGSM deltas, with paired confidence intervals. If Consensus does not beat CKA by more than the standard error at a majority of compression levels, or does not beat the best individual metric by a consistent margin, the triple-win claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is the evaluation protocol used to support the central claim, not the final-representation proxy itself. Section 4 states that for comparison with prior methods the authors 'report the results of each method according to the original paper.' Thus Tables 1-4 compare Consensus numbers produced by this pipeline against CKA, DECORE, HALP, and other baselines taken from external papers, which may use different fine-tuning schedules, epochs, learning rates, and data augmentation. The only direct comparison in an apparently shared protocol is Figure 2, and it reports single trajectories without seeds or confidence intervals. The claimed margins there, up to 4.14-4.29 pp on FGSM, are within normal run-to-run variability of fine-tuned CIFAR ResNets, so they could reflect training noise rather than the Consensus criterion. The single-iteration individual-metric experiment in Section 4 ('Effectiveness of the Proposed Consensus Criterion') also reports only a maximum margin of 0.60 pp with no variance, so aggregation over multiple metrics is not shown to be reliably better than its components. Consequently, the central claim that combining multiple similarity metrics results in more robust and trustworthy pruned models is not yet supported by controlled evidence; the paper needs same-protocol, multi-seed comparisons against CKA and against each constituent metric.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Consensus, a layer-pruning criterion that aggregates multiple representation-similarity metrics by summing per-metric layer rankings, and claims a 'triple win': low accuracy drop, high FLOP/latency reduction, and improved robustness to adversarial attacks and OOD shifts. The method is evaluated on CIFAR-10 and ImageNet with ResNet architectures, on robustness benchmarks (FGSM, CIFAR-10.2, CIFAR-C, ImageNet-C), on shallow ResNets, on Transformers for wearable-sensor HAR, and with GreenAI cost estimates. The authors report state-of-the-art or on-par accuracy at high FLOP reductions and robustness gains over the CKA baseline, and they attribute the gains to avoiding shortcut learning.","tokens_in":16372,"tokens_out":3285,"duration_ms":33232,"significance":"If the central claim holds, the paper makes a useful practical contribution: a simple, code-available pruning criterion that improves robustness and compression simultaneously, without adversarial training. The evaluation breadth is commendable, and the method's rank-aggregation formulation is easy to reproduce. However, the evidence for the aggregation benefit and for the robustness advantage over single-metric baselines is not yet statistically established, because the controlled comparisons lack multiple seeds and the SOTA comparisons use numbers from heterogeneous original papers.","major_comments":[{"comment":"The paper explicitly states 'For a fair comparison, we report the results of each method according to the original paper.' This means the baseline numbers in Tables 1-4 come from different training and fine-tuning pipelines (different epochs, learning rates, augmentation, and hardware), so the comparisons are not controlled. The only apparently shared-protocol comparison is Figure 2, which shows single trajectories without seeds or confidence intervals. The headline robustness margins of 4.14 and 4.29 pp on FGSM at 41.28% and 56.29% FLOP reduction are within the typical run-to-run variance of fine-tuned CIFAR ResNets, so they do not yet demonstrate that Consensus is more robust than CKA. The authors should run a same-protocol, multi-seed comparison against CKA (and at least one other SOTA baseline) and report means and standard deviations or confidence intervals.","section":"§4, 'Comparison with the State of the Art' (Tables 1-4)"},{"comment":"The single-pruning-iteration experiment on ResNet32 reports mean accuracy across benchmarks and a maximum margin of 0.60 pp over the best individual metric, with no variance or per-benchmark breakdown. This is too weak to support the central claim that combining multiple similarity metrics is reliably better than its components. The authors should provide multi-iteration pruning trajectories for each constituent metric and for Consensus, with multiple seeds, and report the distribution of accuracy deltas on each robustness benchmark.","section":"§4, 'Effectiveness of the Proposed Consensus Criterion'"},{"comment":"The paper repeatedly attributes Consensus's robustness gains to avoiding 'shortcut learning' (e.g., Section 4: 'our results suggest that our criterion yields pruned models that avoid shortcut opportunities'; Section 5: 'it avoids shortcut learning'), but no direct evidence for shortcut avoidance is provided. The manuscript does not test for reliance on spurious correlations, background or texture shortcuts, or any shortcut-detection metric. Since this is a load-bearing part of the 'trustworthy pruned models' claim, the authors should either add a concrete shortcut-detection experiment (e.g., evaluating on a biased variant of the dataset or using feature attribution) or soften the claim to a hypothesis.","section":"§4-§5, shortcut learning claims"},{"comment":"The Consensus score is a simple sum of rankings over the chosen metric set S, and S is a free design choice (the paper does not study how varying the composition of S affects pruning outcomes). The claim that consensus is beneficial depends on which metrics are included and on their interpolation parameters. The paper should include an ablation that varies the metric set (e.g., each individual metric, pairs, and the full set) and shows the effect on the pruning trajectory, to establish that the aggregation itself, rather than any single component metric, drives the reported gains.","section":"§3, Algorithm 1, and §4, ablation of metric set"}],"minor_comments":[{"comment":"The left panel's label 'FSGM' should be 'FGSM'.","section":"Figure 1 caption"},{"comment":"'botton-right' should be 'bottom-right'.","section":"Figure 2 caption"},{"comment":"Typos: 'traning' should be 'training'; 'mantaining' should be 'maintaining'.","section":"Section 1 and Section 3"},{"comment":"The row labeled 'Consensus20 (Ours)' should probably be 'Consensus (Ours)' to match the other rows; please check this row and the corresponding run.","section":"Table 1"},{"comment":"The carbon emission and cost reduction figures (68.75% and 66.99%) are reported with a footnote referring to an impact calculator, but the hardware, runtime, power draw, and training time assumptions are not given. Please state these assumptions for reproducibility.","section":"§4, 'GreenAI and Computational Costs'"},{"comment":"The Transformer experiments show accuracy improvements across datasets, but there is no comparison to any prior layer-pruning method on these HAR datasets. The claim that the results 'confirm its effectiveness' would be stronger with a CKA baseline or a standard filter-pruning baseline.","section":"§4, 'Effectiveness in Transformer Architectures'"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is clearly written and the code is available, which is a strength. However, the evaluation protocol needs substantial strengthening before the central claim can be accepted: the SOTA tables rely on numbers from other papers, and the only same-protocol comparison lacks seeds and confidence intervals. The shortcut-learning explanation is asserted, not demonstrated. I would advise the editor that a major revision requiring controlled multi-seed experiments and a shortcut-avoidance test is appropriate; the core idea is promising but the evidence is not yet conclusive."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the Consensus idea is plausible and worth a referee's time, but the triple-win claim is not yet supported by controlled evidence. The paper's own Section 4 explains that state-of-the-art comparisons use numbers from the original papers, so the tables are not same-protocol comparisons. The one place a shared protocol appears (Figure 2) has no error bars or seeds, so the headline FGSM margins of ~4pp could easily be fine-tuning noise.\n\nWhat is actually new: combining several representation-similarity metrics by rank aggregation for layer pruning is not in the cited prior work. The code is public, which is a real plus. The experiments are broad — CIFAR-10 and ImageNet on ResNets, shallow variants, a couple of Transformer/tabular cases, and robustness/OOD benchmarks. The GreenAI estimates are rough but a reasonable extra.\n\nSoft spots, in proportion: the evaluation protocol is the load-bearing one. Tables 1-4 mix numbers from different papers, so even the +2.04pp ImageNet margin over CKA cannot be attributed to the criterion. Figure 2 reports single trajectories; FGSM differences of 4.14-4.29pp are within normal run-to-run variation for fine-tuned CIFAR ResNets. The single-iteration experiment shows a 0.60pp max advantage of Consensus over its own constituents, again without variance, so aggregation is not shown to be reliably better. The shortcut-learning explanation is asserted, not tested. The representation proxy (using the pre-classification layer for the whole network) is reasonable but never interrogated with ablations. None of these are fatal; they are all fixable.\n\nWho it's for: pruning researchers and practitioners who want a cheap, simple layer-removal criterion. A serious referee should engage with the work; the main ask should be a controlled, multi-seed comparison against CKA and each metric individually, with the exact fine-tuning protocol.\n\nMy recommendation: accept for peer review, with the expectation that the evaluation protocol is tightened before publication.","headline":"A plausible new criterion for layer pruning, but the triple-win claim rests on uncontrolled baselines and single runs.","tokens_in":16890,"tokens_out":2769,"would_cite":false,"duration_ms":26535,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Combining several similarity metrics into one pruning score identifies layers whose removal least hurts the model, keeping accuracy high and improving adversarial robustness.","keywords":["layer pruning","similarity metrics","Consensus criterion","adversarial robustness","FLOPs reduction","representation similarity","out-of-distribution generalization","structured pruning"],"falsifier":"Compare the consensus ranking against exhaustive leave-one-layer-out accuracy after fine-tuning across several random seeds: if the layer with the smallest summed rank is not among the layers whose removal costs the least accuracy, then the representation-distance proxy is not tracking true layer importance.","tokens_in":15961,"feed_emoji":"⚡","tokens_out":7130,"duration_ms":61159,"temperature":0.7,"pith_summary":"Layer pruning removes whole layers from a trained network to cut compute, but existing criteria rank layers with a single similarity metric and can pick layers that hurt accuracy or generalization. This paper argues that combining several similarity metrics into one ranking, called the Consensus criterion, finds layers whose removal least disturbs the network's final feature representation. The claim is a triple win: pruned models stay accurate, run with up to 78.8% fewer FLOPs, and hold up better under adversarial attacks and out-of-distribution shifts than models pruned with a single metric such as CKA (centered kernel alignment). If true, this makes pruning a more trustworthy compression tool and a cheap robustness defense, since no adversarial training is required.","feed_headline":"Consensus layer pruning trims 78.8% FLOPs, resists attacks","feed_subtitle":"Merging several similarity metrics into one ranking identifies the safest layers to remove while preserving accuracy.","key_machinery":"The Consensus criterion is the central object: for each layer, compute several similarity scores between the pruned and unpruned networks' final representations, convert each metric's scores into a rank ordering of layers, sum the ranks across metrics, and remove the layer ranked most similar overall. The similarity metrics come from a metric space for stochastic neural network representations, including Procrustes and Bures distances and interpolated variants, so the comparison is invariant to rotations and accounts for noise structure in neural responses. The load-bearing operation is the rank-sum: it turns incomparable score magnitudes from different metrics into a single ordering, and it operationalizes the paper's thesis that no single metric should decide layer importance.","core_discovery":"The paper's central discovery is that a pruning criterion formed by summing rank positions across multiple representation-similarity metrics identifies low-importance layers more reliably than any single metric. Concretely, for each candidate layer the method removes it, extracts the feature map at the layer before classification, measures how similar that representation is to the unpruned network's under several metrics, ranks layers per metric, and prunes the layer with the smallest summed rank. Across ResNet models on CIFAR-10 and ImageNet, Consensus matches or beats state-of-the-art pruning methods in accuracy drop at equal FLOP reduction, reaches 78.80% FLOP reduction with minimal accuracy loss, and improves accuracy under FGSM, CIFAR-C, CIFAR-10.2, and ImageNet-C by up to roughly 4 percentage points over the CKA criterion. The authors interpret the gains as avoiding shortcut learning: a single metric can overfit to clean-image statistics, while a consensus of metrics selects layers that preserve features useful under distribution shift.","pith_inferences":["If the rank-sum interpretation is the active ingredient, the same consensus procedure could be applied to filter pruning or to LLM layer dropping, where single-criterion rankings are known to be unstable; the paper notes the LLM extension but does not test it.","A testable extension is to use the dispersion of rank positions across metrics as a confidence measure: layers on which metrics agree may be safer to prune than layers on which they disagree, something the paper does not examine.","The robustness gains suggest that representation-preservation pruning could serve as a zero-cost complement to adversarial training; combining Consensus with adversarial training is a natural next experiment the paper does not run.","The method's dependence on a single reference representation from the unpruned network means its ranking may drift as training data changes; re-running selection on different data subsets would test how stable the pruned layer choice is."],"forward_implications":["Pruned models can reach 78.80% FLOP reduction on ResNet56/CIFAR-10 while keeping accuracy nearly unchanged, with gains that are often larger than single-criterion baselines at the same compression level.","Adversarial robustness improves without adversarial training: consensus-pruned models beat CKA-pruned ones by up to roughly 4 percentage points on FGSM and 1.2 percentage points on CIFAR-10.2 at matched compression.","The same criterion transfers to shallow ResNets (ResNet32/44) and to a Transformer on tabular human-activity data, cutting FLOPs up to about 80% with small accuracy change.","Reported energy and carbon savings follow mechanically from the FLOP reduction: the paper estimates 66.99% lower energy and 68.75% lower carbon emissions on ResNet56.","Because the method needs no architectural search or specialized sparse hardware, it offers a drop-in layer-pruning criterion for standard residual networks."],"supporting_citations":[{"why":"Provides the CKA layer-pruning criterion that Consensus extends and the main baseline it must beat.","marker":"[47]"},{"why":"Supplies the family of similarity metrics for stochastic neural network representations that form the Consensus criterion.","marker":"[10]"},{"why":"Motivates the claim that single pruning criteria give biased importance rankings, which Consensus addresses.","marker":"[25]"},{"why":"Defines the adversarial-robustness evaluation protocol and baseline results that Consensus claims to improve.","marker":"[29]"},{"why":"Supplies the shortcut-learning concept used to explain why single-metric pruned models fail under distribution shift.","marker":"[17]"},{"why":"Grounds the shortcut-learning explanation with a formal account of when models rely on spurious features.","marker":"[23]"},{"why":"Provides a layer-pruning baseline and the iterative pruning/fine-tuning protocol the paper follows.","marker":"[27]"}],"fun_headline_variants":["Consensus layer pruning cuts 78.8% FLOPs, boosts attack robustness","Prune with consensus: up to 78.8% FLOPs off, more robust","Triple-win pruning: consensus criterion cuts costs, boosts robustness","Consensus of metrics finds safe layers to prune: 78.8% FLOPs saved","Layer pruning backed by consensus: low drop, high speed, more robust"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking is only meaningful if how much removing a layer changes the network's final feature representation faithfully measures how important that layer is, and if summing rank positions across metrics preserves that signal.","fun_headline_variants_meta":{"raw":{"variants":["Consensus layer pruning cuts 78.8% FLOPs, boosts attack robustness","Prune with consensus: up to 78.8% FLOPs off, more robust","Triple-win pruning: consensus criterion cuts costs, boosts robustness","Consensus of metrics finds safe layers to prune: 78.8% FLOPs saved","Layer pruning backed by consensus: low drop, high speed, more robust"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000781,"raw_usage":{"total_tokens":3438,"prompt_tokens":920,"completion_tokens":2518,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":2411}},"tokens_in":536,"tokens_out":2518,"duration_ms":16751,"temperature":1.0,"reasoning_tokens":2411,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:16:07.409238+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the consensus ranking against exhaustive leave-one-layer-out accuracy after fine-tuning across several random seeds: if the layer with the smallest summed rank is not among the layers whose removal costs the least accuracy, then the representation-distance proxy is not tracking true layer importance.","supporting_citations":[{"cited_title":"Effective Layer Pruning Through Similarity Metric Perspective","cited_arxiv_id":"2405.17081","evidence_quote":"Provides the CKA layer-pruning criterion that Consensus extends and the main baseline it must beat."},{"cited_title":"Duong and et al","cited_arxiv_id":null,"evidence_quote":"Supplies the family of similarity metrics for stochastic neural network representations that form the Consensus criterion."},{"cited_title":"Rethinking the prun- ing criteria for convolutional neural network","cited_arxiv_id":null,"evidence_quote":"Motivates the claim that single pruning criteria give biased importance rankings, which Consensus addresses."},{"cited_title":"On the effect of pruning on adversarial robustness","cited_arxiv_id":null,"evidence_quote":"Defines the adversarial-robustness evaluation protocol and baseline results that Consensus claims to improve."},{"cited_title":"Shortcut learning in deep neural networks","cited_arxiv_id":null,"evidence_quote":"Supplies the shortcut-learning concept used to explain why single-metric pruned models fail under distribution shift."},{"cited_title":"Hermann and et al","cited_arxiv_id":null,"evidence_quote":"Grounds the shortcut-learning explanation with a formal account of when models rely on spurious features."},{"cited_title":"Discriminative layer prun- ing for convolutional neural networks","cited_arxiv_id":null,"evidence_quote":"Provides a layer-pruning baseline and the iterative pruning/fine-tuning protocol the paper follows."}],"review_version":1}