{"id":"f9311617-f356-4ea3-9f2f-95cc697c95f9","arxiv_id":"2411.17796","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":15,"one_line_summary":"An iterative block-coordinate-descent version of the Combinatorial Brain Surgeon outperforms one-shot Wanda pruning on DeiT and Mistral-7b, but without code release or error bars.","lead":"The paper introduces iCBS, a pruning method that iteratively solves small block-constrained quadratic optimization problems over subsets of a neural network's weights, starting from a one-shot pruned model. On DeiT and Mistral-7b it reports higher validation accuracy than the Wanda baseline at the same density levels, at the price of much higher compute, about 7.9 days for the 7B model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"iCBS's reported gains over Wanda rest on single-run, validation-tuned evaluations; without code and seeds the central empirical claim is not yet established.","rationale":"The paper gives a clear derivation of the per-block QCBO and a modular algorithm, and the reported improvements are large and consistent across three models. That is real evidence in the paper's favor. However, the central claim is empirical and the comparison is underpowered: iCBS curves are single runs, hyperparameters were selected on the validation set used for reporting (Section V.A), and no code is released. The reader's concern about the 16-sample Hessian in Eq. (9) is one path to non-reproducibility, but the more load-bearing gap is the lack of any statistical control over the evaluation itself. A controlled multi-seed comparison with a separated test set and a batch-size sweep would settle whether the improvement is genuine. My recommendation therefore matches the reader's CONDITIONAL: accept only if the reproducibility evidence is supplied.","tokens_in":19475,"tokens_out":10773,"duration_ms":94102,"concrete_test":"Release code and run a controlled comparison on Mistral-7b at 30% density: hold out a test split (e.g., the LM Evaluation Harness tasks), use a separate validation split for any hyperparameter selection, run iCBS with 5 random seeds and with batch_size_pruning in {16, 64, 128}, and compare mean accuracy against Wanda. The central claim is supported only if iCBS's mean improvement exceeds the pooled standard error and persists when the Hessian is estimated from larger batches.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that iCBS beats Wanda at equal densities—is not yet supported because the evaluation is not controlled enough to rule out selection effects. Section V.A states that for Garment Classifier and DeiT, up to 100 hyperparameter configurations were searched with Ray Tune on the validation accuracy that is later reported, and for Mistral hyperparameters were hand-picked because tuning was infeasible. The iCBS curves in Figs. 4 and 5 are single runs with no error bars, and no code is released. Since iCBS also starts from the best-performing Wanda variant and has many more free parameters than the one-shot baselines, the reported +7.7% to +21.6% gains could be an artifact of validation-set overfitting rather than a property of the block-coordinate-descent construction. The internal mechanism is also fragile: Eq. (9) estimates the block Hessian from per-sample gradients on a batch of 16 for Mistral-7b, giving a rank-at-most-16 approximation to a 4096 x 4096 Hessian; without multi-seed or batch-size ablations, there is no evidence that the solved local QCBO improves the true validation objective.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes iCBS, an iterative block-coordinate-descent pruning method that extends the Combinatorial Brain Surgeon to large models. Starting from an initial one-shot pruning (Wanda or magnitude), it fixes extreme-score weights, selects blocks by weight scores, and at each step solves a quadratic constrained binary optimization problem derived from a second-order Taylor expansion of the loss (Eq. 8) using a batch-estimated gradient and Hessian (Eq. 9). The authors report validation-accuracy gains over Wanda/magnitude baselines on Fashion-MNIST (Garment Classifier), ImageNet (DeiT-base), and seven-task LM evaluation (Mistral-7b), with only a small fraction of weights optimized (Table III), at the cost of substantially more compute (Table IV).","tokens_in":19723,"tokens_out":8318,"duration_ms":71509,"significance":"If the reported gains hold, iCBS is a meaningful advance in optimization-based pruning: it shows that a Hessian-based combinatorial objective can be applied to 7B-parameter models by block decomposition, and it explicitly trades compute for one-shot pruning quality. The derivation of the block objective from Eq. (4) via Eq. (7) is transparent and algebraically consistent, and the modular design (weight scoring, fixing, tabu, plug-in QCBO solver) is a useful contribution. However, the empirical support is currently not fully controlled: the main comparisons are single runs, hyperparameters were selected on the same validation metric that is reported, and the Hessian is approximated from very small batches; these issues should be resolved before the central claim can be considered established.","major_comments":[{"comment":"For the Garment Classifier and DeiT, up to 100 hyperparameter configurations were tuned with Ray Tune on the validation accuracy that is later reported in Figs. 3-4, and the final parameters were obtained by averaging over the ~20 best configurations. Since iCBS has many more free parameters than Wanda (num_epochs, num_steps, block_size, num_restarts, batch sizes, alpha, lambda, tabu_frac, fix_frac_prune, fix_frac_keep), the reported improvements of +21.6%, +14.2%, and +7.7% (Fig. 6) may be inflated by validation-set overfitting. Please provide a tuner-free or nested-evaluation protocol: tune on a separate validation split, report test-set curves, or at least compare against Wanda under the same tuning budget.","section":"Section V.A / Table V"},{"comment":"The iCBS curves appear to be single runs with no error bars, and the caption of Fig. 3 states that error bars are included only for the baselines. Because iCBS involves random batch draws, random layer selection, and stochastic simulated-annealing restarts, run-to-run variance is expected. Without confidence intervals or multiple seeds, the claimed gains over Wanda cannot be distinguished from noise. Please report means and standard deviations over at least 3-5 independent runs (or a justified deterministic protocol), for all methods and all plotted densities.","section":"Section V.B, Figs. 4 and 5"},{"comment":"The block Hessian is estimated as H_B ≈ (1/n) A^T A using n = batch_size_pruning samples; for Mistral-7b this is n=16, giving a rank-at-most-16 approximation to the 4096×4096 Hessian of each block. The paper gives no evidence that this batch-local quadratic model is sufficiently accurate to improve the true validation objective, nor any ablation on batch_size_pruning, block size, or number of solver restarts. Please add such ablations, or a direct check that solutions of Eq. (8) consistently reduce the validation loss, to support the claimed mechanism behind the gains.","section":"Section IV.A, Eq. (9), and Algorithm 1"},{"comment":"The abstract highlights a quality-time tradeoff, but the paper only reports total runtime at density 50% (Table IV) and convergence curves (Figs. 4b and 5b) without a systematic accuracy-vs-cost comparison. Please quantify the tradeoff by varying num_steps, num_epochs, and block_size and plotting accuracy against pruning time, since this tradeoff is presented as a principal advantage over one-shot pruning.","section":"Abstract and Section V.B, Table IV"}],"minor_comments":[{"comment":"The symbol n is used for the block size in Eq. (8) and for the batch size in Eq. (9); please use distinct notation to avoid ambiguity.","section":"Section IV.A, Eq. (9)"},{"comment":"The number of weights listed as \"Optimized\" is an upper bound, since blocks can revisit weights after tabu-list expiration; please state this explicitly in the caption or text.","section":"Table III"},{"comment":"The caption says error bars are included for all baselines, but iCBS is not a baseline; please clarify that the iCBS curve is a single run with no error bars.","section":"Fig. 3 caption"},{"comment":"The pseudocode uses num_steps_per_epoch while the text and Table V use num_steps; please align the terminology.","section":"Algorithm 1"},{"comment":"The constrained simulated annealing solver is described only by a reference; please provide the temperature schedule, number of sweeps, and stopping criteria to make the experiments reproducible.","section":"Appendix A"},{"comment":"The manuscript does not state whether source code and exact experimental configurations will be released; please add an availability statement, which is particularly important given the many implementation details in Appendix B.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the paper's central empirical claim is plausible and the algorithmic derivation is sound, but the evaluation currently rests on single runs and validation-set-tuned hyperparameters. I would encourage requiring an artifact/reproducibility statement with code and seeds before acceptance, and asking for at least a small multi-seed study on the two smaller models plus a batch-size ablation for the Hessian estimate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before you look at this. First, the engineering is real: the paper defines a per-block quadratic binary formulation (Eq. 8) that generalizes one-shot CBS, solves it iteratively with weight-fixing, extremal selection, and a tabu list, and reports consistent gains over Wanda at low densities on three models — +21.6 points on a small Fashion-MNIST MLP, +14.2 on DeiT at 20% density, +7.7 on Mistral-7b at 30%. Second, the evaluation is not tight enough yet to fully back those numbers: no code released, the iCBS curves appear to be single runs without error bars, and for Garment and DeiT the hyperparameters were tuned with up to 100 Ray Tune configs on the same validation accuracy that is later reported.\n\nThe genuinely new parts are the iterative block-wise scheme and the block-selection machinery: fixing roughly three-quarters of the weights, choosing the rest by gradient-based extremal scores, and tabu lists to stop the optimizer cycling. The most striking number in the paper is Table III — for Mistral, optimizing over 0.2% of the weights buys most of the gain. The derivation is self-contained and honest about assumptions, runtimes are reported plainly (7.9 days for Mistral at density 50%), and the gradient term in Eq. (8) is a real improvement over the zero-gradient CBS assumption.\n\nThe soft spots are real but not disqualifying. The Hessian for Mistral is a rank-at-most-16 approximation to a 4096×4096 block (batch of 16 samples); the ridge term keeps it well-posed, and the consistent gains across models and densities suggest the batch-local quadratic is doing useful work, not just chasing noise. The sharper concern is selection on the validation metric: single runs plus tuning on the reported accuracy could inflate the gains. That said, the low-density gains are large enough that they probably are not an artifact, and Mistral's hyperparameters were hand-picked rather than tuned. What is missing is evidence: a couple of seeds per density, a batch-size ablation, and code. Citing their own BCD solver paper and the annealing solver is legitimate — those are the actual ingredients.\n\nBottom line: this deserves a serious referee. The right path is conditional acceptance pending code, multi-seed runs, and ideally a held-out split for reporting. If the gains replicate, it is a solid addition to the pruning toolbox — and worth a reading-group slot.","headline":"A real engineering contribution that generalizes CBS to iterative block-wise pruning with weight-fixing and tabu search, reporting large gains over Wanda — plausible and worth refereeing, but the single-run, validation-tuned evaluation needs code and seeds before the numbers can be trusted.","tokens_in":20353,"tokens_out":4885,"would_cite":true,"duration_ms":42883,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Iterative block-wise pruning matches or beats one-shot baselines on LLMs by solving small quadratic binary problems.","keywords":["neural network pruning","block coordinate descent","Combinatorial Brain Surgeon","large language models","Hessian approximation","quadratic constrained binary optimization","model compression","weight scoring"],"falsifier":"Re-run the Mistral-7b experiment at density 30 percent with fresh random batches from C4 used for gradient and Hessian estimation; if the reported accuracy gain over per-output Wanda does not reproduce across independent batches, or if a randomly scrambled Hessian of the same scale produces the same gain, the load-bearing role of the local quadratic model is falsified.","tokens_in":19239,"feed_emoji":"✂️","tokens_out":6497,"duration_ms":50839,"temperature":0.7,"pith_summary":"This paper introduces iterative Combinatorial Brain Surgeon (iCBS), a pruning method that starts from a one-shot pruning such as Wanda or magnitude pruning and repeatedly improves it by solving a small quadratic binary optimization problem over a block of weights using block coordinate descent. The central claim is that this iterative, block-wise procedure scales to large models such as Mistral-7b and DeiT and achieves higher validation accuracy at the same density than one-shot baselines such as Wanda. The authors report that only a small fraction of weights needs to be optimized (0.07 for DeiT and 0.002 for Mistral-7b) to obtain these gains. If correct, this shows that Hessian-based pruning is feasible for large language models, at the cost of substantially more compute, and that the per-block optimization could be offloaded to hardware accelerators or quantum solvers.","feed_headline":"Pruning beats Wanda by solving small blocks iteratively","feed_subtitle":"iCBS improves Mistral-7b and DeiT accuracy at the same density while touching only a tiny fraction of weights.","key_machinery":"The carrying object is the per-block quadratic constrained binary optimization problem of Eq. (8): binary variables $x_i$ indicate whether weight $i$ is pruned, and the objective combines a linear gradient term, a quadratic Hessian term, and a ridge term, with a cardinality constraint that exactly $k$ of the $n$ selected weights are pruned. The block Hessian is approximated from per-sample gradients as $H_B \\approx \\frac{1}{n} A^T A$, so only the Hessian entries for the selected block are needed. Weight-scoring methods (Wanda, gradient, magnitude) choose which weights are fixed as always pruned or kept and which weights enter each block, while a tabu list prevents re-selecting the same weights; the block solver is a constrained simulated annealing routine that searches only feasible solutions.","core_discovery":"Starting from a cheap one-shot pruning, iCBS repeatedly selects a block of $n$ candidate weights using weight-scoring methods and a tabu list, estimates the block gradient and Hessian from a small data batch via $H_B \\approx \\frac{1}{n} A^T A$, solves a quadratic constrained binary optimization problem that decides which weights in the block to prune or un-prune, and applies the solution. The central discovery is that this local second-order correction, repeated over blocks, improves the pruned model's accuracy on Fashion-MNIST, ImageNet-1K, and seven language-modeling tasks compared with Wanda, magnitude, and gradient baselines at the same densities, despite optimizing over only a small fraction of the weights. The improvement is largest at low densities (up to +21.6 percent accuracy for the Garment Classifier at density 10 percent, +14.2 percent for DeiT at density 20 percent, and +7.7 percent for Mistral-7b at density 30 percent) and shrinks as density approaches the unpruned model.","pith_inferences":["Editorial extension: the fixing and selection steps imply most weights are easy to classify as keep or prune, so expensive optimization only needs to touch a small 'disputed' boundary; this supports a two-stage view of pruning where cheap scoring handles the bulk and optimization handles the margin.","Editorial extension: applying iCBS on top of SparseGPT instead of Wanda could combine SparseGPT's weight-reconstruction benefits with iCBS's iterative rebalancing.","Editorial extension: using richer selection scores, such as activation variance or range rather than mean activation, is a natural and testable way to improve which weights enter each block.","Editorial extension: the reported Mistral-7b runtime of 7.9 days suggests that, without algorithmic or hardware speedups, iCBS is currently a post-processing tool for one-shot pruning rather than a routine training-time method."],"forward_implications":["Hessian-based second-order pruning is not limited to small networks; it can improve large language models and vision transformers at the same density as one-shot baselines.","One-shot pruning results serve as a warm start, and iterative block solves refine them without ever forming the full $N \\times N$ Hessian.","The method exposes a quality-time tradeoff: more epochs, more steps, or larger blocks cost more compute but buy accuracy, which one-shot methods do not offer.","Because each block problem is a small quadratic constrained binary program, faster specialized solvers, including quantum or quantum-inspired hardware, could directly enable larger blocks and better pruning.","The gains come from optimizing only a small fraction of weights, suggesting that the hard decisions are concentrated among weights that cheap scoring cannot confidently classify."],"supporting_citations":[{"why":"Supplies the Wanda one-shot baseline and the per-output aggregation that iCBS improves upon.","marker":"[7]"},{"why":"Provides the Combinatorial Brain Surgeon QCBO formulation that the per-block problem Eq. (8) is derived from.","marker":"[14]"},{"why":"Gives the per-sample-gradient approximation of the Hessian used in Eq. (9).","marker":"[26]"},{"why":"Origin of Taylor-expansion-based pruning with a diagonal Hessian assumption, background for the second-order approach.","marker":"[12]"},{"why":"Introduces full-Hessian single-weight pruning, the precursor to the combinatorial formulation.","marker":"[13]"},{"why":"Supplies the gradient and ridge terms and discussion of their use in the quadratic pruning objective.","marker":"[18]"},{"why":"SparseGPT is the other one-shot optimization-based pruning method that the paper positions itself against.","marker":"[8]"},{"why":"Shows second-order pruning applied to large language models, motivation for scaling Hessian-based methods.","marker":"[5]"}],"fun_headline_variants":["iCBS prunes large models with small block steps","Block-wise pruning boosts accuracy over Wanda","Small-block pruning iteratively beats Wanda","iCBS: iterative block pruning for LLMs and vision"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the quadratic loss approximation built from a small batch of pruning data (batch size 16 for Mistral-7b) is accurate enough that solving the local binary problem improves the model's accuracy on the validation tasks.","fun_headline_variants_meta":{"raw":{"variants":["iCBS prunes large models with small block steps","Block-wise pruning boosts accuracy over Wanda","Small-block pruning iteratively beats Wanda","iCBS: iterative block pruning for LLMs and vision"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000644,"raw_usage":{"total_tokens":3004,"prompt_tokens":1035,"completion_tokens":1969,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":651,"completion_tokens_details":{"reasoning_tokens":1907}},"tokens_in":651,"tokens_out":1969,"duration_ms":13821,"temperature":1.0,"reasoning_tokens":1907,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:54:36.337202+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the Mistral-7b experiment at density 30 percent with fresh random batches from C4 used for gradient and Hessian estimation; if the reported accuracy gain over per-output Wanda does not reproduce across independent batches, or if a randomly scrambled Hessian of the same scale produces the same gain, the load-bearing role of the local quadratic model is falsified.","supporting_citations":[{"cited_title":"Optimal brain surgeon and general network pruning","cited_arxiv_id":null,"evidence_quote":"Provides the Combinatorial Brain Surgeon QCBO formulation that the per-block problem Eq. (8) is derived from."},{"cited_title":"Q-CHOP: Quantum constrained hamiltonian optimization","cited_arxiv_id":null,"evidence_quote":"Gives the per-sample-gradient approximation of the Hessian used in Eq. (9)."},{"cited_title":"Woodfisher: Efficient second-order approximation for neural network compression","cited_arxiv_id":null,"evidence_quote":"Supplies the gradient and ridge terms and discussion of their use in the quadratic pruning objective."}],"review_version":1}