{"id":"833337f4-7d61-4691-8a7d-17db2a30e0a7","arxiv_id":"2501.01118","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"PruneFuse combines pruning at initialization with weight fusion and knowledge distillation to make active learning data selection cheaper and to initialize the final model.","lead":"PruneFuse uses a randomly pruned, smaller network to pick which data samples to label for active learning, then copies the trained small network's weights into the full network to speed up training. It reports accuracy gains over standard active learning on CIFAR and Tiny-ImageNet while using fewer FLOPs for the selection step.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The pruning-surrogate selection advantage is never isolated from the fusion initialization: every PruneFuse final model is initialized with pruned-model weights, so the reported gains may reflect fusion rather than better data selection.","rationale":"The reader's weakest assumption is on point: the load-bearing premise is that weight-magnitude pruning at initialization yields a surrogate whose selections transfer to the dense model. This is asserted qualitatively in Section 3 ('strong correlation') and never isolated experimentally. The ablations isolate fusion (Fig. 4/6) and KD (Tables 3/7), but selection source is only varied while keeping fusion on. The data actually contain a hint against selection importance: on CIFAR-100 at b=10%, PruneFuse with Random selection and no KD (40.07%) beats PruneFuse with Least Confidence and no KD (39.27%), and this is well above the full-model AL baseline's Random (37.06%) and Greedy k (38.28%). Since fusion is constant within PruneFuse rows, the selection metric contributes little or nothing at low budget. This does not disprove the method's practical value — fusion plus a pruned selector may still be a cheap and effective pipeline — but it does undercut the specific claim that the pruned surrogate's selections are superior. A clean factorial test would settle it. No internal inconsistency or fabrication is apparent; the issue is missing control conditions and error bars, which is why the conditional verdict is retained.","tokens_in":16118,"tokens_out":4268,"duration_ms":43739,"concrete_test":"Re-run the CIFAR-10/ResNet-56 experiment at b=10% and b=30% as a factorial: final training from random init vs. fused init, crossed with three selection sources — pruned-model LC, random, and full-model LC — with KD disabled. If a randomly initialized full ResNet-56 trained on PruneFuse's selected subset is not more accurate than the same protocol trained on full-model LC data, the surrogate-selection claim is unsupported. Also, if fusion plus random selection matches fusion plus LC selection, the gain is from fusion.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the pruned surrogate θp selects better data is never tested independently of the fusion initialization. Section 3 asserts a 'strong correlation between θ and θp' and that selection using θp 'effectively minimizes the loss when θ is trained on sp,' but every reported PruneFuse final model is initialized by copying the trained pruned weights into θ (Section 3.4) and then refined with KD (Section 3.5). The baseline AL final model is trained from random initialization on its selected subset. Thus the comparisons in Tables 1, 2, 4, and 5 confound two variables: data selection source and model initialization/training protocol. The fusion effect is visible in Fig. 4 and in Table 3, where PruneFuse with Random selection and no KD reaches 40.07% on CIFAR-100 at b=10%, exceeding PruneFuse with Least Confidence (39.27%) and nearly matching the with-KD LC result (40.26%). That pattern suggests the reported advantage over the AL baseline is largely a fusion/initialization effect, not evidence for the structural-correlation premise in Eq. 1.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PruneFuse, a training pipeline that (i) prunes a randomly initialized dense network to obtain a small surrogate model, (ii) uses that surrogate for iterative active-learning data selection, and (iii) after the labeling budget is exhausted, fuses the trained pruned weights into the original dense network and refines it with knowledge distillation. The authors claim that PruneFuse lowers the computational cost of data selection, achieves higher accuracy than standard active-learning baselines, and accelerates training. The empirical section reports experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet-200 with ResNet variants, comparing against a standard AL baseline and the SVP proxy-model method, plus ablations of the fusion and KD components.","tokens_in":16346,"tokens_out":3091,"duration_ms":31219,"significance":"If the central claim were established, the paper would make a useful practical contribution: using a pruned-at-initialization surrogate for active learning is a natural efficiency idea, and the fusion step is an original twist that could improve initialization in low-label regimes. The paper contains a substantial amount of experimental work: multiple datasets, architectures, pruning ratios, label budgets, selection metrics, and an ablation study on fusion and KD. However, the experiments as presented do not isolate the data-selection effect of the pruned surrogate from the effect of the fusion initialization, and the theoretical motivation in Section 3 is asserted rather than tested. The reported gains, especially at small label budgets, may be fully attributable to the fusion-KD pipeline applied to random data. The significance of the specific 'pruned surrogate selects better data' claim is therefore presently unverified, although the fusion component itself appears to have empirical support.","major_comments":[{"comment":"The paper's central claim is that the pruned surrogate θp selects more informative data than the original model (Eq. 1 and the 'strong correlation' assertion in Section 3). This claim is never tested in isolation. Every PruneFuse final model is initialized via fusion with the trained pruned weights (Section 3.4) and refined with KD (Section 3.5), while the baseline AL final model is trained from random initialization on the selected subset. Table 3 (CIFAR-100, ResNet-56, p=0.5, without KD) shows PruneFuse with Random selection reaching 40.07% at b=10%, which exceeds PruneFuse with Least Confidence (39.27%) and nearly matches the with-KD LC result (40.26%). This pattern indicates that the gain over the AL baseline (35.99%) is driven largely by the fusion initialization, not by surrogate selection. I request an ablation that trains the final dense model from random initialization on PruneFuse-selected data (no fusion, no KD) and compares it with the baseline AL final model on the same data, and also a PruneFuse-with-Random condition without fusion. Without these conditions, the superiority of the surrogate-based selection is not established.","section":"Section 3 and Tables 1, 3, 4, 5"},{"comment":"Eq. (1) is not used as an optimization objective or to derive any algorithmic step. The expression E_{(x,y)∈s_p}[l(x,y; θ, θp)] is never defined precisely (the loss depends on both θ and θp in an unspecified way), and the arg-min over subsets is not computed. The subsequent text asserts that structural coherence between θp and θ implies that θp-based selection 'effectively minimizes the loss,' but no measurement of correlation between θp and θ selections is provided. Either remove the formal framing and present the method as a heuristic, or provide evidence (e.g., agreement of selected sets, or target-model performance on selections made by θp vs. θ under identical training protocols) that supports the claimed relationship.","section":"Section 3, Eq. (1)"},{"comment":"The paper reports only averaged accuracies over three runs, with no error bars, standard deviations, or significance tests. Several headline comparisons are within a fraction of a percentage point: e.g., in Table 1 on CIFAR-10 at b=10%, Baseline AL is 80.53% and PruneFuse p=0.8 is 80.11% (a decrease), while PruneFuse p=0.5 is 80.92%. Across the tables, many differences are smaller than 1 point, and the direction is inconsistent (e.g., CIFAR-100 p=0.8 at 10% is 36.49 vs. baseline 35.99, a gain of only 0.5). Given that three runs rarely give tight confidence intervals for such differences, I ask that the authors report variance or error bars, and identify which differences are statistically meaningful. This is necessary to support the claim of 'consistently outperforms' across budgets and settings.","section":"Section 4, 'Experimental Setup'"},{"comment":"The computational-efficiency claim in Fig. 3 is presented as 'total number of FLOPs utilized by PruneFuse, compared to the baseline Active Learning method,' but the methodology for computing FLOPs is not described. It is unclear whether the figure includes: (a) the cost of training the pruned model from scratch in every AL round (as described in Section 3.3), (b) the fusion and KD fine-tuning cost, and (c) the fact that the baseline AL also retrains the dense model each round. The FLOPs comparison should be defined precisely for both methods, including all training and selection stages, so that the claimed computational savings can be verified and not merely inferred from the smaller parameter count.","section":"Section 4.1 and Fig. 3"}],"minor_comments":[{"comment":"Section 3.1 states that after pruning, 'we retain the weights of the pruned network before training,' but Section 3.3 says that at the start of each training cycle, 'a fresh θp is generated' and trained from scratch. Please clarify whether the pruned subnetwork is re-generated (with a new random initialization) each AL round or re-used with the same initial pruned weights.","section":"Section 3.1 vs Section 3.3"},{"comment":"Algorithm 1 contains typographical issues: 'Prune(θ, p) // Structure pruning' should be 'structured pruning,' and line 7 has awkward syntax 'Dk = topk[Dj ∈ U ]j∈[k]' that should be written clearly as selecting the top-k scored samples.","section":"Appendix A, Algorithm 1"},{"comment":"In the CIFAR-10 without-KD row for Random selection, the entries '90.2091.10' and '91.8292.89' appear to be two numbers concatenated without a separator; please fix the formatting.","section":"Appendix A, Table 7"},{"comment":"The related-work section discusses SVP and SubSelNet but does not compare with several common modern active-learning baselines (e.g., BADGE, TypiClust, or recent coreset variants) that are also designed for efficient selection. Adding such comparisons or at least a discussion of where PruneFuse stands would strengthen the positioning, though it is not essential for the central claims.","section":"Section 2, Related Work"},{"comment":"The caption of Figure 2 states 'Pruning θ to θp tailors the loss landscape from 2a to 2b,' but the subfigures are not explicitly labeled in the caption (they are just '(a)' and '(b)' with no titles in the text). Please make the correspondence between subfigures and the described trajectories explicit.","section":"Figure 2 caption"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an arXiv preprint (labeled NeurIPS 2024 submission) rather than a journal submission; I am evaluating the technical content regardless. The main concern is the confounding of the data-selection and fusion effects. If the authors can provide the requested ablations (target trained from scratch on PruneFuse-selected data; fusion with random selection; variance estimates), the paper could be publishable as an empirical method paper, even if the theoretical framing in Eq. (1) is dropped. I do not see evidence of circularity or parameter-fitting-to-results, and the method is clearly presented. The scope fits a machine-learning venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper: the fusion idea is doing the work, and the data-selection claim is untested. The paper bundles pruning-based selection with weight fusion into the target model, and every PruneFuse run uses the fused initialization. The baseline active-learning model trains from scratch. So every comparison in Tables 1, 2, 4, and 5 confounds the two. The authors' own Table 3 shows it: at b=10% on CIFAR-100, PruneFuse+Random without KD hits 40.07%, PruneFuse+LC without KD hits 39.27%, and with KD LC is 40.26%. The selection criterion barely moves the needle; the fusion/KD protocol is what jumps from the 35.99% baseline. The structural-correlation argument in Section 3 is asserted, not measured. There is no ablation where the same selected subset is trained from random init versus fused init, which is exactly what you'd need to isolate selection quality.\n\nWhat the paper does well: it is clearly written, the SVP comparison is the right benchmark, the FLOPs accounting is a useful practical contribution, and the fusion/KD ablations are honest attempts to unpack the method. The idea of reusing the trained pruned proxy as an initialization is a reasonable and cheap trick, and Figure 4 suggests it genuinely accelerates convergence. That is a real, albeit modest, empirical finding.\n\nSoft spots beyond the confound: no error bars despite reporting three-run averages; no code; the Eq. 1 objective is decorative—it doesn't constrain the method or guide the evaluation. None of these are fatal, but they compound the interpretation problem.\n\nNet: this is not a paper that supports the claim 'pruned proxy selects better data.' It may be a paper about a useful initialization trick. That is fixable with one or two ablations, plus standard deviation reporting. I'd send it to a serious referee with a request for exactly that isolation—it deserves referee time, not because the conclusion is established, but because the failure mode is instructive and the fix is cheap.","headline":"PruneFuse's fusion trick seems to drive the reported gains; the data-selection advantage it claims is never isolated, but the work is fixable and worth a referee's time.","tokens_in":16861,"tokens_out":2905,"would_cite":false,"duration_ms":30012,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that PruneFuse can pick the most informative training samples with a pruned copy of the network and then fuse what that copy learned back into the full network, cutting data-selection compute while matching or beating…","keywords":["active learning","data selection","network pruning","pruning at initialization","network fusion","knowledge distillation","efficient deep learning","surrogate model"],"falsifier":"Train the full pipeline but replace the pruned surrogate's chosen examples with examples chosen by a same-size network that has no structural relation to the target, for instance a differently initialized architecture with the same parameter count. If the fused model's final accuracy stays the same, the 'strong correlation' premise is not doing the work. A second check: fuse the trained surrogate's weights into the dense model and compare against fusing the same number of random weights in the same positions; if accuracy does not drop, the fusion step is not transferring learned knowledge.","tokens_in":15922,"feed_emoji":"⚡","tokens_out":10275,"duration_ms":86267,"temperature":0.7,"pith_summary":"PruneFuse is a training pipeline that tries to make active-learning data selection cheap enough for real use. Instead of training the full network to decide which unlabeled samples are worth labeling, it first prunes the untrained target network down to a small channel-pruned surrogate, trains that surrogate on the small labeled set, and uses the surrogate's uncertainty or diversity scores to pick the next batch. When the labeling budget is reached, the trained surrogate is not discarded: its weights are copied back into the matching positions of the untrained dense network, and that fused model is trained with knowledge distillation from the surrogate. The paper argues that because the surrogate is structurally a subnetwork of the target, the samples it selects are nearly as good as those the full model would select, while the fusion gives the dense model a better starting point than random initialization. Across CIFAR-10, CIFAR-100, and Tiny-ImageNet-200, the reported result is that PruneFuse beats standard active learning and the selection-via-proxy baseline at most label budgets while using a fraction of the selection compute.","feed_headline":"Cut data-selection cost with a pruned copy, then fuse it","feed_subtitle":"A pruned copy picks samples; its learned weights fuse back into the full model for faster, better training.","key_machinery":"The load-bearing object is the channel-pruned surrogate $\\theta_p$, created by one-shot structured pruning at initialization: each channel is scored by its L2 magnitude and the lowest-scoring channels are removed to a target sparsity $p$, leaving a subnetwork whose surviving filters line up one-to-one with filters of the dense network. That structural correspondence is what lets the method do two things: use $\\theta_p$ as a cheap scorer for least-confidence, entropy, or greedy k-centers acquisition, and then transfer knowledge through filter-wise fusion $\\theta_F = \\mathrm{Fuse}(\\theta, \\theta_p^*)$, where trained weights fill their matching slots and the untouched weights remain random. A knowledge-distillation loss with $\\theta_p^*$ as teacher then refines $\\theta_F$, combining cross-entropy on the labeled set with softened logit matching. The mechanism as a whole converts the usual proxy-model weakness, structural mismatch between selector and target, into the method's main asset.","core_discovery":"The central claim is that a single untrained network can serve both roles in efficient active learning: a pruned version selects the data, and the dense version becomes the final model. Formally, the paper defines the goal as choosing a subset $s_p$, scored by the pruned model $\\theta_p$, whose expected loss on the original model $\\theta$ is as close as possible to training $\\theta$ on the full dataset, and it justifies the surrogate by asserting that $\\theta_p$ 'maintains a structure that is essentially identical to $\\theta$' with a 'strong correlation' between them. Once the budget is reached, PruneFuse fuses the trained pruned weights $\\theta_p^*$ into the untrained $\\theta$ by copying each surviving filter into its corresponding location and leaving the rest untouched, producing $\\theta_F = \\mathrm{Fuse}(\\theta, \\theta_p^*)$, then fine-tunes $\\theta_F$ on the selected labels with a cross-entropy plus distillation loss. The paper reports that this combined procedure yields higher final accuracy than training the dense model on the same selected subset from scratch, faster convergence, and lower FLOPs during selection, with the best accuracy at high pruning ratios such as $p=0.5$ and the lowest compute at $p=0.8$.","pith_inferences":["Editorial inference: the design implies that structural coherence between a channel-pruned and dense network, not just smaller size, is doing the work; a same-size but architecturally unrelated selector would test that directly.","Editorial inference: because the pruned model is retrained from scratch each selection round, warm-starting it across rounds could yield additional compute savings that the paper does not quantify.","Editorial inference: the fusion step should transfer to other structured architectures, such as attention heads in transformers, but the paper only demonstrates CNNs on image classification.","Editorial inference: the reported gains at low label budgets suggest the fused initialization may be most valuable in annotation-scarce regimes; a budget sweep below 10% would show where the advantage saturates."],"forward_implications":["With PruneFuse, data selection runs on a pruned copy of the target network, so the FLOPs spent on querying drop with sparsity; the reported ResNet-56 selector at $p=0.8$ has about 0.03 million parameters versus 0.85 million for the full model.","The subset picked by the pruned surrogate transfers to the dense model, matching or beating standard active learning accuracy at every reported label budget on CIFAR-10, CIFAR-100, and Tiny-ImageNet-200.","Fusing the trained pruned weights into the untrained dense network yields faster early convergence and higher final accuracy than training from scratch on the same selected subset, as shown in the fusion ablations.","The pipeline works with multiple acquisition functions (Least Confidence, Entropy, Greedy k-centers) and across pruning ratios, giving users a compute-versus-accuracy knob.","Compared with selection-via-proxy, PruneFuse reaches higher target-model accuracy with a smaller or comparable selector, so the structural coherence argument beats simply using a small external proxy."],"supporting_citations":[{"why":"Provides the selection-via-proxy baseline that PruneFuse is directly compared against and whose discarded-proxy limitation motivates the fusion step.","marker":"[15]"},{"why":"Supports pruning at initialization as a way to obtain a small trainable surrogate network before any training.","marker":"[39]"},{"why":"Supplies the greedy k-centers acquisition criterion used as one of the three data-selection metrics.","marker":"[6]"},{"why":"Provides the knowledge-distillation formulation used to refine the fused model with the trained pruned network as teacher.","marker":"[22]"},{"why":"Supplies the structural pruning routine that generates the pruned surrogate at the sparsities used in the experiments.","marker":"[45]"}],"fun_headline_variants":["Prune to select data, fuse to train faster","A pruned twin selects samples, then fuses for better training","Cut data-selection FLOPs with a pruned model, then fuse it","PruneFuse: prune for selection, fuse for initialization boost","Cheap data selection via pruning, then fuse the weights back"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central bet is that a smaller, magnitude-pruned copy of an untrained network will pick just as useful training examples as the full network would, because the two are structurally similar enough that what is good for one is good for the other; the paper's experiments never isolate this surrogate-selection effect from the separate boost given by fusion, so that bet is asserted rather than demonstrated.","fun_headline_variants_meta":{"raw":{"variants":["Prune to select data, fuse to train faster","A pruned twin selects samples, then fuses for better training","Cut data-selection FLOPs with a pruned model, then fuse it","PruneFuse: prune for selection, fuse for initialization boost","Cheap data selection via pruning, then fuse the weights back"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000749,"raw_usage":{"total_tokens":3334,"prompt_tokens":945,"completion_tokens":2389,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":2299}},"tokens_in":561,"tokens_out":2389,"duration_ms":14540,"temperature":1.0,"reasoning_tokens":2299,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:34:16.325844+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the full pipeline but replace the pruned surrogate's chosen examples with examples chosen by a same-size network that has no structural relation to the target, for instance a differently initialized architecture with the same parameter count. If the fused model's final accuracy stays the same, the 'strong correlation' premise is not doing the work. A second check: fuse the trained surrogate's weights into the dense model and compare against fusing the same number of random weights in the same positions; if accuracy does not drop, the fusion step is not transferring learned knowledge.","supporting_citations":[{"cited_title":"Pruning from scratch","cited_arxiv_id":null,"evidence_quote":"Supports pruning at initialization as a way to obtain a small trainable surrogate network before any training."},{"cited_title":"Depgraph: Towards any structural pruning","cited_arxiv_id":null,"evidence_quote":"Supplies the structural pruning routine that generates the pruned surrogate at the sparsities used in the experiments."}],"review_version":1}