{"id":"4404cc9c-1c03-4e91-9f16-570b8d69a45c","arxiv_id":"2505.07411","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"An automated pruning pipeline that skips fine-tuning when accuracy drop is small and freezes less sensitive layers, cutting pruning time up to 9.61x with similar final accuracy.","lead":"ICE-Pruning speeds up neural network pruning by skipping fine-tuning after pruning steps where accuracy barely drops and by freezing insensitive layers during the fine-tuning that does occur. The paper reports up to 9.61x faster pruning than existing pipelines while keeping accuracy roughly the same on the tested image models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported speedups likely exclude Stage 1 auto-tuning cost, so the 9.61x end-to-end claim is not yet supported.","rationale":"The reader's weakest assumption concerns Stage 1 hyperparameter transfer from a random subset to the full dataset; my concern is about Stage 1 time not being counted in the reported speedups. Both are Stage-1 issues, but they are distinct: one is about accuracy transfer, the other about time accounting. The time-accounting gap is the more load-bearing for the central quantitative claim, because the headline 'up to 9.61x' is an end-to-end claim. The paper provides useful evidence in other respects: code is available, ablations isolate the contributions of each component, and accuracy is generally comparable or better than baselines. However, the missing Stage 1 cost and the ambiguous freezing warm-up step make the speedup number unsupported as an end-to-end measure. This reinforces the reader's CONDITIONAL verdict: the claim is plausible but requires revised time reporting and a clear statement of what is included in the measured time. No verdict shift is needed; the conditional acceptance should now explicitly require total-pipeline time, including auto-tuning and freezing warm-up, to be reported.","tokens_in":11772,"tokens_out":9183,"duration_ms":81855,"concrete_test":"Re-run the DenseNet-121/TinyImageNet 80% pruning experiment using the complete Algorithm 1, recording the total wall-clock time including Stage 1 hyperparameter search, the number of candidates, and the subsample fraction. Recompute the speedup versus AAP and baseline using this total time. Also re-run one setting while instrumenting the freezing warm-up step separately. If the end-to-end speedup falls substantially below 9.61x (e.g., below 5x), the headline should be revised to specify the search-excluded speedup or the search cost should be amortized and reported explicitly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is a wall-clock speedup of up to 9.61x for the complete pruning pipeline. Algorithm 1 shows ICE-Pruning has two stages: Stage 1 searches hyperparameters on a randomly sampled subset, and Stage 2 applies the chosen hyperparameters to the full dataset. The times reported in Table III (e.g., 432s for DenseNet-121/TinyImageNet at 80% pruning) appear to be PFT times from Algorithm 2 (Stage 2 only). The paper never reports the wall-clock duration of Stage 1, the number of hyperparameter candidates evaluated, or the subsampling ratio. Baselines and AAP do not have an equivalent auto-tuning stage, so an end-to-end comparison would include Stage 1 in ICE-Pruning's total time. With, for example, 20 candidates and 10% data, Stage 1 could add roughly two full-pipeline equivalents, materially shrinking the headline speedup. A related gap is that the freezing strategy (Section III-B) requires an initial pruning-plus-fine-tuning step to measure layer weight changes and decide which layers to freeze, but Algorithm 2 simply calls Freeze(M, eta) without showing this warm-up cost; if that step is not in the measured time, the speedup is further overstated. The reported 9.61x therefore measures only a portion of the pipeline, not the end-to-end pruning time promised in the abstract.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes ICE-Pruning, an iterative structured-pruning pipeline that wraps existing pruning criteria rather than introducing a new one. The pipeline has two stages: Stage 1 auto-tunes six hyperparameters (accuracy-drop threshold, frozen-layer percentage, base learning rate, maximum LR change, the pruning-level parameter p, and shape parameter beta) by running the pruning-fine-tuning loop on a randomly subsampled dataset; Stage 2 reruns the loop on the full dataset with the selected hyperparameters. Three mechanisms reduce fine-tuning cost: skipping fine-tuning when the post-pruning accuracy drop is below a threshold, freezing layers whose normalized weight changes are small, and a pruning-aware LR scheduler that lowers the maximum LR as the average model width shrinks. The evaluation compares ICE-Pruning with a naive iterative baseline and with AAP on ResNet-152/CIFAR-10, DenseNet-121/TinyImageNet, and WRN-101-2/ImageNet, and reports speedups up to 9.61x at 80% pruning with similar or better accuracy in most cells.","tokens_in":1612,"tokens_out":1690,"duration_ms":94044,"significance":"If the headline speedup survives a complete accounting of all pipeline stages, this is a useful practical contribution: it is a model-agnostic wrapper, it supports multiple pruning criteria, it includes an ablation study of each component, and the code is released. The paper is explicit that it introduces no new pruning criterion, and the experimental claims are falsifiable through the reported times and accuracies. The main open risk is whether the 9.61x figure is an end-to-end wall-clock speedup or only a Stage-2 fine-tuning speedup; this is the difference between a strong systems result and a more modest one. The accuracy comparisons would also be more persuasive with repeated runs and variance estimates.","major_comments":[{"comment":"The headline 9.61x speedup is not yet supported as an end-to-end wall-clock claim. In Algorithm 1, Stage 1 runs the complete PFT loop for every candidate hyperparameter on sampled data, and Stage 2 reruns PFT on the full dataset; the Time values in Table III appear to report only the Stage-2 PFT time. The paper never reports the number of hyperparameter candidates evaluated, the data subsampling ratio, or the duration of Stage 1. With, for example, 20 candidates on 10% of the data, Stage 1 alone would add roughly two full-pipeline equivalents of compute, which could materially shrink the reported speedup. In addition, the freezing strategy of Section III-B requires a warm-up pruning-plus-fine-tuning step to measure layer weight changes before Algorithm 2 calls Freeze(M, eta); the cost of that warm-up is not visible in the time figures. Please report a complete time budget that separates Stage 1 search cost, the freezing warm-up, and Stage 2, and recompute the speedups on the total.","section":"Algorithm 1, Table III"},{"comment":"The experimental protocol is inconsistent with the algorithm description. Section IV-A says that for TinyImageNet and ImageNet, fine-tuning is performed on subsampled datasets, whereas Algorithm 1's Stage 2 calls PFT with the full dataset D. The sampling fractions are not given, and no distinction is made between the subset used for Stage 1 and any subset used in Stage 2. If the Table III numbers for DenseNet-121 and WRN-101-2 were obtained with subsampled fine-tuning data, then the comparison is not against full-data baseline and AAP runs on equal terms, and the 9.61x speedup applies to a different protocol than the one in the algorithm. Please clarify precisely which data each stage uses and report the subsampling ratios.","section":"Section IV-A, Algorithm 1"},{"comment":"The 'maintaining similar accuracy' claim is supported only loosely. The experiments appear to be single runs with no error bars, so differences in the table cannot be distinguished from noise. In several cells ICE-Pruning is a few points below a baseline: DenseNet-121 at 70% pruning (54.50 vs 57.00 for the baseline), WRN-101-2 at 70% (54.97 vs 55.65), and ResNet-152 at 80% (91.10 vs 91.20 for AAP). Please add repeated-seed experiments with means and standard deviations for at least the central comparisons, or explicitly qualify the accuracy claim as comparable in single-run evaluation.","section":"Section IV-E, Table III"},{"comment":"The transferability of the Stage-1 hyperparameter search to the full dataset is asserted rather than demonstrated. The paper states that the subsampled data has a similar distribution to the whole dataset, but it reports no sampling fraction, no sensitivity analysis over that fraction, and no comparison of the objective in Eq. (2) on sampled versus full data. Since the six searched hyperparameters determine the speed-accuracy trade-off, an unrepresentative subset would directly invalidate the reported results. Please report the size of the search space and at least one experiment varying the subsampling ratio (for example 5%, 10%, and 25%) to validate the transfer.","section":"Section III-D"}],"minor_comments":[{"comment":"The title in the PDF reads 'An I terative Cost-Efficient...' with a stray space in 'Iterative'; please fix the typo.","section":"Title"},{"comment":"Equation (1) contains unmatched parentheses and is ambiguous as typeset; please rewrite it with explicit parentheses and define the domain of each variable in the same display.","section":"Equation (1)"},{"comment":"In the paragraph after Eq. (2), 'since accuracyPTis always larger than Delta-A' should read 'since PT is always larger than Delta-A'; the current wording is ungrammatical.","section":"Section III-D"},{"comment":"Please state explicitly that the hyperparameter set lambda in Eq. (2) is the tuple (theta, eta, LR_base, Delta, p, beta) from Algorithm 2; the link is currently implicit.","section":"Section III-D, Algorithm 2"},{"comment":"The four fine-tuning cases described in the text are not labeled in the figure; add labels or a legend so the reader can map cases 1-4 to the plotted curves.","section":"Figure 1(b)"}],"recommendation":"major_revision","confidential_remarks":"The paper is a plausible engineering contribution whose main risk is that the headline speedup may not be end-to-end once Stage 1 and the freezing warm-up are included. The single-run accuracy comparisons are a further concern but could be addressed by adding seeds. I would ask the authors to resolve the time-budget question before publication; if the total speedup is still substantial, the paper would be publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ICE-Pruning is a sensible engineering contribution: a pipeline that skips fine-tuning when the accuracy drop is below a threshold, freezes layers ranked by one-step weight-change sensitivity, and uses a pruning-aware LR schedule. The ablation is honest and shows each piece helps. That part is real and worth building on.\n\nThe problem is the headline. The 9.61x speedup is measured on Stage 2 only. Stage 1, which grid-searches six hyperparameters on a subsampled dataset, is never timed. With dozens of candidates and a subsample that still requires a full pruning pass per candidate, that cost will eat a large fraction of the claimed gain. The paper also never states the candidate count or the subsampling ratio. So the end-to-end speedup over the baseline and AAP, as promised in the abstract, is not established. This is not a nitpick; it is the central quantitative claim.\n\nThere are smaller issues. The freezing strategy needs a warm-up pruning-plus-fine-tuning step to measure layer changes, and it isn't clear whether that cost is inside the reported times. Results are single-run with no error bars, and in a few cells ICE-Pruning is a couple of points below the baseline, so 'similar accuracy' is only loosely supported. The LR formula has an unmatched parenthesis, and the search space is not described. The paper also cites their own ICE-Pick but never compares to it, which is odd for an 'efficient pruning pipeline' paper.\n\nNone of this kills the idea. The rank-stability observation and the gating mechanism are plausible and the ablation supports them. With code, a full end-to-end timing breakdown, and a comparison to ICE-Pick, this would be a useful systems paper. I would send it to review, but I'd tell the authors the headline claim needs to be rescoped and the missing measurements added.","headline":"A useful pruning pipeline whose headline speedup likely ignores auto-tuning cost.","tokens_in":12580,"tokens_out":2803,"would_cite":false,"duration_ms":26016,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ICE-Pruning speeds deep-network pruning up to 9.61x by skipping unnecessary fine-tuning.","keywords":["network pruning","iterative pruning","fine-tuning","layer freezing","learning rate scheduling","hyperparameter auto-tuning","model compression","structured pruning"],"falsifier":"Run the Stage 1 auto-tuning on several disjoint random subsets of the same dataset for the same model, then apply each resulting hyperparameter set to the full dataset. If the chosen threshold and freezing percentage vary so much across subsets that the reported speedups or accuracies do not reproduce, the transfer assumption that underpins the method is false.","tokens_in":11572,"feed_emoji":"✂️","tokens_out":10664,"duration_ms":87355,"temperature":0.7,"pith_summary":"The paper argues that fine-tuning after every pruning step is wasteful, because over-parameterized networks often lose little accuracy at intermediate steps. ICE-Pruning therefore fine-tunes only when the measured accuracy drop crosses a threshold, freezes the layers that change least during fine-tuning, and lowers the maximum learning rate as the network gets narrower. These three mechanisms cut the number and cost of fine-tuning rounds. The paper reports that this speeds pruning by up to 9.61x while keeping final accuracy close to, and sometimes above, existing pipelines.","feed_headline":"Skipping needless fine-tuning speeds pruning up to 9.61x","feed_subtitle":"The pipeline only trains when accuracy actually drops, so final accuracy stays close while time falls.","key_machinery":"The mechanism that carries the argument is a threshold-gated fine-tuning loop. Each pruning step is followed by an accuracy test, and fine-tuning is triggered only when the drop is at least $\\theta$. The second component is layer freezing: after one fine-tuning, per-layer weight changes are normalized by the layer weights, and the $\\eta$ fraction of layers with the smallest changes stay frozen thereafter. The third is a pruning-aware learning-rate schedule, $LR_{\\max} = LR_{\\mathrm{base}} - \\Delta / (1 + (\\alpha / (2(1-p) - \\alpha))^\\beta)$, where $\\alpha$ is the fraction of unpruned parameters, so narrower models receive lower maximum learning rates. An auto-tuning objective $(\\mathit{PT} + \\Delta A)/\\max(\\mathit{PT}, \\Delta A)$ balances pruning time $\\mathit{PT}$ and accuracy loss $\\Delta A$. The same pruning criteria as in existing pipelines can be dropped in, making the efficiency gains independent of the choice of criterion.","core_discovery":"On the paper's own terms, the central claim is that an iterative pruning pipeline can become much cheaper without introducing a new pruning criterion. After each pruning step the model is tested; fine-tuning runs only if the accuracy drop reaches a threshold. Layers whose normalized weight changes are small after the first fine-tuning are frozen in later steps, and a pruning-aware learning-rate scheduler caps the maximum learning rate based on how much of the network remains. An auto-tuning stage picks the few pipeline hyperparameters on a sampled subset and then applies them to the full dataset. The evaluation reports up to 9.61x faster pruning than a baseline iterative pipeline and a state-of-the-art automatic pipeline, with comparable or better final accuracy in most cases.","pith_inferences":["The same test-then-spend-compute gating could be applied to other expensive training loop operations, such as validation or checkpointing, whenever a cheap metric indicates that extra compute would not change the outcome.","The reported results are on convolutional vision models; the natural stress test is a model family with less stable layer-sensitivity rankings, where the one-step freezing rule could misfire.","Varying the sample size in Stage 1 across several random subsets would show how sensitive the chosen settings are, turning the representative-sample assumption into a measurable quantity.","An ablation that runs the full pipeline with always-fine-tune and never-fine-tune extremes would isolate how much of the 9.61x comes from skipping fine-tuning versus from freezing and learning-rate scheduling."],"forward_implications":["For over-parameterized models, many intermediate pruning steps can skip fine-tuning entirely, so a target compression ratio is reached with substantially fewer training epochs.","The pipeline works with existing structured-pruning criteria, so teams can keep their preferred criterion and still get the time savings.","The speedup grows with pruning ratio, reaching 9.61x at 80% pruning, because that is where the largest number of fine-tuning rounds are avoided.","Final accuracy stays competitive with the baseline and automatic pipelines, and in some reported settings it is higher, so the time savings do not come from accepting a lower-quality model.","The auto-tuning stage removes the need for manual tuning of the introduced hyperparameters, making the pipeline plug-and-play."],"supporting_citations":[{"why":"Provides the L1-norm filter-pruning criterion that ICE-Pruning wraps and uses in most experiments.","marker":"[6]"},{"why":"Supplies the iterative prune-then-fine-tune baseline and the knowledge-distillation practice used in the CIFAR-10 experiments.","marker":"[3]"},{"why":"Defines the automatic-pipeline baseline that ICE-Pruning compares against for accuracy and time.","marker":"[7]"},{"why":"Supplies one of the alternative pruning criteria, entropy-based filter pruning, used in the criterion comparison.","marker":"[4]"},{"why":"Motivates the layer-freezing strategy by showing that freezing layers accelerates fine-tuning in transfer learning.","marker":"[19]"},{"why":"Provides the lottery-ticket hypothesis that underpins the rewind-and-resume strategy of the comparison baseline, which ICE-Pruning avoids.","marker":"[23]"},{"why":"Shows that narrower networks have narrower loss landscapes, which justifies lowering the maximum learning rate as pruning increases.","marker":"[27]"},{"why":"Supplies the learning-rate-schedule idea that ICE-Pruning adapts into its pruning-aware maximum learning-rate formula.","marker":"[28]"}],"fun_headline_variants":["Prune up to 9.61x faster by fine-tuning only when accuracy drops","Fine-tune only when needed, prune up to 9.61x faster","Skip needless fine-tuning, prune up to 9.61x faster","Selective fine-tuning prunes up to 9.61x faster","Pruning gets 9.61x faster with on-demand fine-tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that tuning on a small random sample of the data gives settings (the accuracy-drop threshold, freezing percentage, and learning-rate schedule) that still work when the pipeline runs on the full dataset.","fun_headline_variants_meta":{"raw":{"variants":["Prune up to 9.61x faster by fine-tuning only when accuracy drops","Fine-tune only when needed, prune up to 9.61x faster","Skip needless fine-tuning, prune up to 9.61x faster","Selective fine-tuning prunes up to 9.61x faster","Pruning gets 9.61x faster with on-demand fine-tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001268,"raw_usage":{"total_tokens":5196,"prompt_tokens":960,"completion_tokens":4236,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":4134}},"tokens_in":576,"tokens_out":4236,"duration_ms":27422,"temperature":1.0,"reasoning_tokens":4134,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:17:06.710946+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the Stage 1 auto-tuning on several disjoint random subsets of the same dataset for the same model, then apply each resulting hyperparameter set to the full dataset. If the chosen threshold and freezing percentage vary so much across subsets that the reported speedups or accuracies do not reproduce, the transfer assumption that underpins the method is false.","supporting_citations":[{"cited_title":"Pruning filters for efficient convnets,","cited_arxiv_id":null,"evidence_quote":"Provides the L1-norm filter-pruning criterion that ICE-Pruning wraps and uses in most experiments."},{"cited_title":"Linearly Replaceable Filters for Deep Network Channel Pruning,","cited_arxiv_id":null,"evidence_quote":"Supplies the iterative prune-then-fine-tune baseline and the knowledge-distillation practice used in the CIFAR-10 experiments."},{"cited_title":"Automatic attention pruning: Improv- ing and automating model pruning using attentions,","cited_arxiv_id":null,"evidence_quote":"Defines the automatic-pipeline baseline that ICE-Pruning compares against for accuracy and time."},{"cited_title":"FreezeOut: Accelerate Training by Progressively Freezing Layers,","cited_arxiv_id":null,"evidence_quote":"Motivates the layer-freezing strategy by showing that freezing layers accelerates fine-tuning in transfer learning."},{"cited_title":"The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks,","cited_arxiv_id":null,"evidence_quote":"Provides the lottery-ticket hypothesis that underpins the rewind-and-resume strategy of the comparison baseline, which ICE-Pruning avoids."},{"cited_title":"S-Cyc: A Learning Rate Schedule for Iterative Pruning of ReLU-based Networks","cited_arxiv_id":"2110.08764","evidence_quote":"Supplies the learning-rate-schedule idea that ICE-Pruning adapts into its pruning-aware maximum learning-rate formula."}],"review_version":1}