{"id":"5ed61ab3-291b-42c2-b240-e53d7187c59c","arxiv_id":"2506.04513","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A pruning method that iteratively generates both layer-pruned and filter-pruned candidates, keeps the one with the highest CKA similarity to the parent, and achieves high FLOPs reduction on small ResNets.","lead":"Deep learning models can be made smaller and faster by removing either whole layers or individual filters, but most pruning methods remove only one kind of structure. This paper proposes a method that, at each step, generates both a layer-pruned and a filter-pruned version of the model and keeps whichever one remains most similar to the original, measured with a representation similarity score.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never specifies how CKA is computed when parent and candidate differ in depth or width; without an explicit layer/channel alignment, Eq. 3's selection rule is undefined and the central claim is not testable.","rationale":"The reader's weakest assumption identifies the unspecified CKA alignment as the key gap, and I agree that this is the most load-bearing concern. The paper's central claim is that CKA similarity to the parent is an effective selection criterion; but without knowing how representations are aligned across architecturally different networks, the comparison is not well-defined. This is not a matter of external consensus but of internal precision: Algorithm 1 and Eq. 3 provide no mapping between feature maps when depth or width changes. The paper does report a Random Walk control, which strengthens the claim that some selection is better than random, but it does not address whether the specific CKA alignment is principled; a poorly chosen alignment could make the method equivalent to an arbitrary heuristic. Since the code is available, the concern is testable, and the achievable fix is a clear description or an ablation. Therefore the verdict remains CONDITIONAL: the core idea is plausible and empirically promising, but the missing details currently prevent full verification. I do not see a reason to reject or accept outright; the condition is that the authors must specify and justify the CKA alignment, and ideally show robustness to alternative alignments. The overclaiming (e.g., calling a -2.91 pp drop on ResNet110 'negligible') is a secondary issue that does not change the technical verdict.","tokens_in":14790,"tokens_out":3565,"duration_ms":43630,"concrete_test":"Inspect the released code (https://github.com/NascimentoG/PruningEverything) to extract the exact CKA computation: identify which tensors are passed for a layer-pruned candidate and a filter-pruned candidate, and how layer correspondences are established after removing a block or channels. Then, run an ablation that replaces the implemented alignment with an alternative (e.g., comparing only global average pooled final features, or matching layers by block index ignoring removed blocks) over the ResNet56/CIFAR-10 setting; if the decision in Eq. 3 flips for a substantial fraction of iterations or the reported 72.67% FLOPs result changes by more than the reported accuracy delta, the central claim is sensitive to underspecified alignment and the method as presented is not reproducible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The entire method rests on comparing CKA(F, F'_l) and CKA(F, F'_f) in Eq. 3, yet neither Section III nor Algorithm 1 states which feature maps are compared. Layer pruning removes a layer, so the remaining layers have different indices than the parent; filter pruning changes channel counts, so per-layer activation tensors have different dimensions. CKA between such pairs is not well-defined without a precise alignment rule: e.g., which parent layer is paired with which pruned layer after a removal, and whether the full activation maps are used or only the final output. If the alignment rule is chosen arbitrarily (e.g., comparing only the final logits, or matching layers by position), the selection in Eq. 3 can favor one candidate for reasons unrelated to representation preservation. This is load-bearing because the paper's core claim—that CKA-guided selection preserves predictive ability better than random choice—depends on CKA being a faithful, reproducible measure of parent-to-candidate similarity. The Random Walk control validates that the choice matters, but not that the specific CKA comparison is correct; a different alignment could invert the decision and change the pruning paths reported in Section IV. The missing detail also blocks verification: reviewers cannot reimplement the algorithm without guessing this alignment.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an iterative structured-pruning method that, at each step, generates two candidate subnetworks from a parent network—one by pruning layers and one by pruning filters—and selects the candidate whose internal representation is most similar to the parent's, as measured by Centered Kernel Alignment (CKA). The selected candidate becomes the parent for the next iteration, after a short fine-tuning step. The authors claim that this greedy, CKA-guided alternation between layer and filter pruning yields highly sparse models that preserve predictive accuracy, and they report state-of-the-art FLOP reductions on CIFAR-10 (e.g., 86.37% on ResNet56 and 95.82% on ResNet110), competitive results on ImageNet with ResNet50, and additional experiments on Transformers, robustness to adversarial/out-of-distribution samples, and carbon-emission reduction.","tokens_in":15053,"tokens_out":3083,"duration_ms":34086,"significance":"If the central claim holds, the paper addresses a genuine gap in structured pruning: most existing methods remove only layers or only filters, whereas this method systematically alternates between both. The reported FLOP reductions, especially 95.82% on ResNet110 with a limited accuracy drop, would be notable. The Random Walk comparison in Section IV provides a meaningful external control, and the public release of code is a strength. However, the missing specification of how CKA is computed across architectures of different depth and width, together with the absence of error bars and the loose protocol for choosing operating points, currently prevents full verification of the method and its claims.","major_comments":[{"comment":"The selection rule in Eq. (3) compares CKA(F, F'_l) and CKA(F, F'_f), but the paper never states which feature maps are used when parent and candidate differ in depth (layer pruning removes a layer, changing layer indices) and width (filter pruning changes channel counts). Without an explicit alignment rule—for example, matching residual blocks by position, using only the final logits, or some interpolation—Eq. (3) is undefined, and the central claim that CKA-guided selection preserves predictive ability is not testable or reproducible. The authors must specify the alignment precisely and justify that it is not arbitrarily favoring one candidate type.","section":"Section III, Eq. (2) and Algorithm 1"},{"comment":"All main accuracy and FLOP results are reported as point estimates with no error bars, seeds, or variance, while the Random Walk baseline is averaged over only three runs. Given that the iterative process involves fine-tuning and stochasticity, the claimed improvements of +0.19 pp, +0.09 pp, and +0.66 pp over Random Walk in Table III could easily fall within run-to-run noise. The authors should report means and standard deviations over at least three to five independent seeds for both the proposed method and the Random Walk baseline.","section":"Section IV, Tables I and III"},{"comment":"The Random Walk control shows that the choice of pruning path matters, but it does not establish that the CKA similarity signal is the operative cause of the improvement. A stronger and more direct baseline is to select the candidate with higher validation accuracy after the same fine-tuning procedure, or to compare against a criterion that directly estimates accuracy. Without such a baseline, the paper's claim that representation similarity is an effective proxy for predictive preservation remains indirect, since the accuracy gain might stem from other properties of the greedy path selection.","section":"Section IV, 'Is CKA Better than Random Choice?' and Table III"},{"comment":"There are internal numerical inconsistencies that must be corrected: the text in Section IV says 'achieving a reduction of 86.57%' while the abstract and Table I report 86.37% for ResNet56; moreover, the 86.37% row has a delta of -1.36 pp, which is described in the abstract as a 'minimal drop' and elsewhere as 'negligible'. The authors should reconcile these numbers and characterize the accuracy drops more carefully, especially since -1.36 pp and -2.91 pp (ResNet110 at 95.82%) are not obviously negligible.","section":"Section IV and Abstract"}],"minor_comments":[{"comment":"The 10-epoch pre-fine-tuning mentioned in Section IV is not shown in Algorithm 1; please include it or explicitly annotate the algorithm with a note that this step occurs before line 4.","section":"Algorithm 1 and Section IV"},{"comment":"The definition of HSIC uses m examples, but the feature maps from the parent and pruned networks may have different spatial dimensions; please clarify whether CKA is computed on aligned, per-example vectors of fixed size and how that alignment is obtained.","section":"Section III, Eq. (1)"},{"comment":"There are typos and reference formatting issues, including 'Centred' vs 'Centered', 'Manson el al.' instead of 'Manson et al.', and several incomplete author lists such as 'A. D. et al.' and 'S. G. et al.'; these should be fixed in a final version.","section":"Throughout"},{"comment":"The caption says the ideal point is 'intentionally misused' and placed 'on the top right'; please clarify whether this is a mistake or a deliberate illustrative choice, and correct the notation so the axes are unambiguous.","section":"Figure 1 caption"},{"comment":"The Transformer experiments reportedly use 10 layers with '128 heads' each; for a tabular dataset this seems implausibly large and likely a typo. Please verify and correct the architecture description.","section":"Appendix VI-C"},{"comment":"The sentence 'we discount the amount of 10-epoch from the remaining fine-tuning epochs' is unclear; please specify how the discounting is applied to the total fine-tuning budget.","section":"Section IV, 'The Role of Fine-tuning'"}],"recommendation":"major_revision","confidential_remarks":"The paper has a compelling high-level idea and the Random Walk baseline is a positive step, but the missing CKA alignment specification is a load-bearing gap that must be fixed before the results can be trusted. In addition, the absence of any variance estimates and the apparent post-hoc selection of operating points make it hard to assess whether the improvements over random selection are real. If the authors can provide a precise algorithmic specification and rigorous experimental reporting, the paper could become a solid contribution; otherwise the central claim remains unverifiable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the core idea is genuinely new and the Random Walk experiment is a good control, but the paper never specifies how CKA is computed when parent and candidate differ in depth or width. That is load-bearing, because Eq. 3 is the whole method. As written, no one can reimplement it.\n\nWhat I like: The proposed alternation between layer and filter pruning, guided by representation similarity, is a real departure from Pons et al., who exhaust layers first, and Muralidharan et al., whose alternation is hand-tuned. The KL-divergence criterion applied to both structures is sensible. The Random Walk baseline is the right control: it isolates the selection rule from everything else, and the result—your CKA pick prunes further before accuracy drops—supports the basic claim. Code availability also helps.\n\nWhere it gets soft: The CKA alignment problem is not a minor omission. A layer-pruned network has fewer layers than its parent; a filter-pruned network has different channel counts. The paper defines CKA on 'internal representations' but never says which layer of the parent is compared with which layer of the candidate, or whether the comparison uses full activation tensors, pooled maps, or the final logits. Until that is specified, Eq. 3 is not an algorithm. The stress-test note is right, and the appendix doesn't fix it either. Second, there are no seeds or error bars anywhere. The Random Walk was run three times, but no variance is reported, and the 'Ours' operating points in Table I seem selected after the fact; there is no pre-registered FLOP target. Third, the strongest baseline is missing: after fine-tuning both candidates, you could simply pick the one with higher validation accuracy. That would tell you whether CKA is adding any signal beyond direct accuracy. Random Walk is a floor, not a ceiling. Finally, the tone overshoots—'milestone', 'new chapter'—and the carbon-emission numbers are just FLOPs translated with a calculator.\n\nVerdict: this deserves a serious referee, but as submitted it is not reproducible. The central idea is plausible and the results are consistent with it, so the fixes are achievable: specify the alignment, add error bars, and compare against accuracy-based selection. I'd engage with it, but only after those changes.","headline":"A plausible pruning alternation idea whose decision rule is under-specified: the CKA comparison between parent and candidate is never defined across different depths and widths.","tokens_in":15590,"tokens_out":2751,"would_cite":false,"duration_ms":31212,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A greedy similarity rule — keep whichever candidate, layer-pruned or filter-pruned, is most CKA-similar to its parent — lets pruning remove both structures at once, reaching 95.82% FLOPs reduction on ResNet110 with a 2.91-point accuracy…","keywords":["structured pruning","layer pruning","filter pruning","Centered Kernel Alignment","representation similarity","FLOPs reduction","ResNet","network compression"],"falsifier":"Fix a small architecture and data set, e.g., ResNet32 on CIFAR-10, and enumerate every layer-or-filter pruning path up to a target sparsity, fine-tuning each candidate. At each decision node, record which of the two candidates has higher post-fine-tuning accuracy and compare that with which candidate has higher CKA to its parent under an explicit layer- and channel-alignment rule. If the higher-CKA candidate is not the higher-accuracy candidate in a substantial fraction of nodes, the selection rule's predictive claim is false; because the paper does not specify the alignment, the test must state which alignment it used.","tokens_in":14604,"feed_emoji":"✂️","tokens_out":11826,"duration_ms":102328,"temperature":0.7,"pith_summary":"The paper tries to settle a decision that most pruning methods dodge: when compressing a network, should the next cut remove a layer or a filter? Its answer is to generate one candidate of each type, compare each candidate's internal representations with those of the unpruned parent using Centered Kernel Alignment (CKA), and keep the more similar one. Repeating this greedy choice, with brief fine-tuning between rounds, produces networks that are simultaneously shallower and narrower. If the claim holds, the payoff is compression beyond what layer-only or filter-only pruning reaches: 72.67% FLOPs reduction on ResNet56 with accuracy up 0.19 points, 86.37% with a 1.36-point drop, and 95.82% on ResNet110 with a 2.91-point drop, along with improved robustness to adversarial and out-of-distribution inputs. The authors position the rule as a structure-agnostic route to pruning that extends to ResNet50 on ImageNet and to Transformers on tabular data.","feed_headline":"Similarity score steers pruning past 95% FLOP reduction","feed_subtitle":"Comparing pruned candidates with their parent via CKA lets pruning remove layers and filters while accuracy holds.","key_machinery":"Centered Kernel Alignment (CKA), the normalized Hilbert-Schmidt Independence Criterion between feature-map kernels of two networks, is the object that carries the decision: $$CKA(R, R_{F'}) = \\frac{HSIC(R,R_{F'})}{\\sqrt{HSIC(R,R) \\cdot HSIC(R_{F'},R_{F'})}}.$$ It is used as the rule in Eq. (3): keep the layer-pruned candidate if $CKA(F,F'_l) \\geq CKA(F,F'_f)$, otherwise keep the filter-pruned candidate. Both candidates are produced by the same KL-divergence pruning criterion, a short fine-tuning is applied before each comparison, and the selected candidate becomes the parent of the next iteration, giving an $O(\\log n)$ search over the binary tree of possible pruning paths.","core_discovery":"The paper's central claim is that representation similarity to the parent is a sufficient guide for deciding which structure to prune next. At each iteration, two candidates are built from the current parent $F$ using the same pruning criterion: one by removing a layer, $F'_l$, and one by removing a filter, $F'_f$. After brief fine-tuning, the method computes $CKA(F,F'_l)$ and $CKA(F,F'_f)$ and keeps the layer candidate when the scores are equal, because layer removal also improves inference efficiency; otherwise it keeps the more similar candidate. The winner becomes the parent for the next round, so the process walks a binary decision tree of structural choices in $O(\\log n)$ time. On the paper's evidence this rule yields accuracy-preserving or accuracy-improving pruned models at FLOPs reductions that single-structure baselines cannot reach, including 86.37% on ResNet56 and 95.82% on ResNet110.","pith_inferences":["Editorial inference: because CKA needs only feature maps, the same selection rule could serve as a label-free reward signal for deciding where to prune or when to stop, a use the paper does not pursue.","Editorial inference: the paper reports FLOPs and estimated carbon, not measured latency or energy on specific hardware; a direct extension would test whether the CKA-chosen paths also minimize real runtime, which FLOPs do not guarantee.","Editorial inference: the unspecified layer/channel alignment between parent and candidate is the main extension point; once an alignment rule is fixed, the binary layer-versus-filter tree naturally widens into a multi-branch tree over blocks, heads, or residual branches, which the paper names only as future work."],"forward_implications":["If the claim holds, structured pruning no longer has to commit to one structure: the CKA rule removes layers and filters in alternation, pushing compression past the layer-only ceiling of roughly 75.5% FLOPs on CIFAR-10/ResNet56.","If the claim holds, the rule transfers across datasets and architectures: it raises accuracy on ResNet32/44 and on Transformers over tabular data, matches strong baselines on ImageNet/ResNet50, and reaches 86.37% and 95.82% FLOPs reduction on ResNet56 and ResNet110 with small accuracy cost.","If the claim holds, high-sparsity pruned models also improve robustness rather than merely preserving accuracy: pruned ResNets improve FGSM adversarial robustness and CIFAR-C/CIFAR-10.2 out-of-distribution performance at around 90% FLOPs reduction.","If the claim holds, the decision rule is the contribution rather than any single score: the paper reports positive results with CKA, Linear, Gaussian Stochastic, and Wasserstein similarities and with both KL-divergence and ℓ1 pruning criteria.","If the claim holds, the FLOPs savings turn into estimated environmental and financial savings: up to 83.31% lower CO2 emissions and up to 83.67% lower estimated financial costs on the pruned ResNet models."],"supporting_citations":[{"why":"Defines Centered Kernel Alignment, the similarity metric that drives the choice in Eq. (3).","marker":"[9]"},{"why":"Defines HSIC, the dependence estimator from which CKA is normalized.","marker":"[21]"},{"why":"Supplies the KL-divergence criterion used to score and remove either layers or filters in each iteration.","marker":"[23]"},{"why":"Defines the layer-pruning method whose ceiling motivates mixing layer and filter removal, and serves as a key comparison baseline.","marker":"[6]"},{"why":"Supports generating the two candidates with similar capacity so that the CKA comparison is fair.","marker":"[22]"},{"why":"Supplies the argument that the choice of which parameters to prune matters more than the number pruned, and the recovery-based fine-tuning rationale used before CKA comparison.","marker":"[8]"},{"why":"Documents the prior approach that alternates width and depth pruning by trial and error, which the proposed rule replaces.","marker":"[10]"},{"why":"Supplies the ResNet architectures that serve as the main testbed across CIFAR-10 and ImageNet.","marker":"[24]"}],"fun_headline_variants":["CKA-guided pruning cuts 95.8% FLOPs on ResNet110","Similarity to parent steers pruning past 95% FLOP reduction","Prune layers and filters together with CKA, cut 95.8% FLOPs","Iterative CKA picks best subnetwork, pruning everything at once"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the similarity score between a parent network and a pruned candidate — computed from internal activity maps that no longer line up one-to-one after layers or channels are removed — reliably measures how much accuracy the candidate keeps, even though the paper does not specify exactly which layers are compared or how activations are matched.","fun_headline_variants_meta":{"raw":{"variants":["CKA-guided pruning cuts 95.8% FLOPs on ResNet110","Similarity to parent steers pruning past 95% FLOP reduction","Prune layers and filters together with CKA, cut 95.8% FLOPs","Iterative CKA picks best subnetwork, pruning everything at once"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000642,"raw_usage":{"total_tokens":3007,"prompt_tokens":1053,"completion_tokens":1954,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":669,"completion_tokens_details":{"reasoning_tokens":1868}},"tokens_in":669,"tokens_out":1954,"duration_ms":14894,"temperature":1.0,"reasoning_tokens":1868,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:40:53.740339+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fix a small architecture and data set, e.g., ResNet32 on CIFAR-10, and enumerate every layer-or-filter pruning path up to a target sparsity, fine-tuning each candidate. At each decision node, record which of the two candidates has higher post-fine-tuning accuracy and compare that with which candidate has higher CKA to its parent under an explicit layer- and channel-alignment rule. If the higher-CKA candidate is not the higher-accuracy candidate in a substantial fraction of nodes, the selection rule's predictive claim is false; because the paper does not specify the alignment, the test must state which alignment it used.","supporting_citations":[{"cited_title":"When layers play the lottery, all tickets win at initialization,","cited_arxiv_id":null,"evidence_quote":"Supports generating the two candidates with similar capacity so that the CKA comparison is fair."},{"cited_title":"Similarity of neural network representations revisited,","cited_arxiv_id":null,"evidence_quote":"Defines Centered Kernel Alignment, the similarity metric that drives the choice in Eq. (3)."},{"cited_title":"Measuring statistical dependence with hilbert-schmidt norms,","cited_arxiv_id":null,"evidence_quote":"Defines HSIC, the dependence estimator from which CKA is normalized."},{"cited_title":"Neural network pruning with residual-connections and limited-data,","cited_arxiv_id":null,"evidence_quote":"Supplies the KL-divergence criterion used to score and remove either layers or filters in each iteration."},{"cited_title":"Effective layer pruning through similarity metric perspective,","cited_arxiv_id":null,"evidence_quote":"Defines the layer-pruning method whose ceiling motivates mixing layer and filter removal, and serves as a key comparison baseline."},{"cited_title":"What makes a good prune? maximal unstructured pruning for maximal cosine similarity,","cited_arxiv_id":null,"evidence_quote":"Supplies the argument that the choice of which parameters to prune matters more than the number pruned, and the recovery-based fine-tuning rationale used before CKA comparison."},{"cited_title":"Compact language models via pruning and knowledge distillation,","cited_arxiv_id":null,"evidence_quote":"Documents the prior approach that alternates width and depth pruning by trial and error, which the proposed rule replaces."}],"review_version":1}