{"id":"265f80f5-e5b8-45a9-b7a4-08a72e34c355","arxiv_id":"2412.11629","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A pruning-plus-quantization framework that uses Bayesian optimization to choose per-layer bit widths reports roughly 30 percent memory savings on 7B-13B LLMs at approximately unchanged zero-shot accuracy.","lead":"QPruner is a pipeline that prunes large language models, assigns 4-bit or 8-bit precision to each layer, and uses Bayesian optimization to pick the bit-width mix before fine-tuning. It reports about 30 percent lower peak memory on 7B and 13B models with accuracy close to a pruning-only baseline, which matters for deploying LLMs on limited hardware.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"QPruner3's accuracy gains may be test-set-selected maxima: the Bayesian optimization loop scores P(b) on the same benchmarks reported in Table 1, with no validation split described.","rationale":"The reader's weakest assumption correctly identifies the most load-bearing flaw: the absence of any validation split in the Bayesian optimization loop, combined with Appendix D's explicit description of constructing a Pareto front from 50 evaluations on the same task benchmarks, makes test-set selection the natural reading of the experimental protocol. I agree with the reader's verdict because the paper's central accuracy claim depends on QPruner3 outperforming LLM-Pruner after per-task selection on the test data; this is not a fair comparison and the reported numbers cannot be taken as independent evidence. The paper does have some merits: the memory numbers are internally consistent, the motivating experiment in Section 2.2 is plausibly described, and the combination of pruning, mixed-precision quantization, and LoftQ/LoRA is a reasonable engineering direction. However, those merits do not rescue the headline claim. The concern is not about disagreement with current consensus or about the novelty of the components; it is a correctness risk in the evaluation protocol itself. A revision that separates configuration selection from final evaluation, reports variance across seeds, and releases code would be needed before the accuracy gain can be assessed. I do not see a different load-bearing concern that should replace this one, so the reader's verdict stands unchanged.","tokens_in":13527,"tokens_out":4421,"duration_ms":39603,"concrete_test":"Rerun the QPruner3 optimization for LLaMA-7B at 50% pruning using a held-out validation split (or an unrelated calibration set such as Alpaca) to score P(b), fix the selected bit-width configuration, and then evaluate it once on the BoolQ and WinoGrande test sets. Also apply the same per-task test-set selection to LLM-Pruner's hyperparameters to quantify selection bias. If the selected configuration's test accuracy falls to the QPruner2 or LLM-Pruner level, the claimed 6% gain is an artifact of test-set selection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing concern is selection-on-test leakage in the Bayesian optimization loop. Algorithm 1 (Section 3.2) defines the objective in terms of P(bt+1) measured after fine-tuning, but neither Section 4 nor Appendices C or D describes a validation split. Appendix D says that for 50%-pruned LLaMA-7B the authors 'fine-tuned 10 sets of configurations as the initialization' and ran 40 iterations to obtain '50 data points for constructing the Pareto front' and 'ensure the best configuration was found'; Figure 3 plots Pareto-front scatter points for BoolQ and WinoGrande, the same benchmarks used in Table 1. If P(b) is accuracy on the benchmark test sets, each QPruner3 entry in Table 1 is a per-task maximum selected after evaluating many bit-width configurations on that task's test set, not an unbiased estimate of the chosen configuration's accuracy. The headline comparison (BoolQ 48.37 vs. 43.76) then contrasts a test-set-selected configuration against LLM-Pruner, which received no per-task selection, so the claimed gain is expected even if QPruner's actual transfer performance is no better than the baseline. The memory-savings component is largely unaffected, but the central 'maintains or improves accuracy' claim is not supported by the reported protocol.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes QPruner, a framework that combines structured pruning (using LLM-Pruner as the pruning backbone), layer-wise mixed-precision quantization initialized by mutual information between layer outputs and model predictions, and Bayesian optimization to refine the per-layer bit-width configuration, followed by LoRA/LoftQ fine-tuning. Experiments on LLaMA-7B, LLaMA-13B, and Vicuna-7B at 20%, 30%, and 50% pruning rates report peak memory savings of roughly 30% or more and, for the Bayesian-optimized variant QPruner3, accuracy improvements over LLM-Pruner on several zero-shot commonsense benchmarks. The paper claims that QPruner can surpass baseline methods in accuracy by up to 6% while saving at least 30% of memory.","tokens_in":13782,"tokens_out":7137,"duration_ms":53595,"significance":"If the accuracy gains were genuine, QPruner would be a practically valuable recipe for low-memory fine-tuning of pruned LLMs. The memory-savings component (Table 1) is consistent and is a real contribution: quantizing to 4/8 bits with LoftQ initialization reduces peak memory substantially. The accuracy claims, however, are not credible as reported because the Bayesian optimizer selects configurations on the same test sets used for final evaluation (Appendix D, Figures 3-4). Thus QPruner3's numbers are selected maxima, and the comparison to LLM-Pruner is unfair. The mutual-information-only and uniform variants (QPruner1/2) are less affected and show modest gains, but the headline 6% improvement is unsupported.","major_comments":[{"comment":"The Bayesian optimization objective P(b) is the model's accuracy after fine-tuning on a given configuration, and Appendix D reveals that the Pareto-front data points are generated per benchmark (BoolQ, WinoGrande, ARC-c, ARC-e, HellaSwag, OBQA, PIQA). These are exactly the seven tasks in Table 1, and no validation split or held-out test set is mentioned anywhere in Section 4 or the appendices. Consequently, each QPruner3 entry in Table 1 is the maximum over the 50 evaluated configurations on that task's test set. Comparing those selected maxima to LLM-Pruner, which is not subjected to per-task selection, does not measure generalization; the 48.37 vs. 43.76 BoolQ difference at 50% pruning is exactly the kind of improvement one would expect from test-set selection. This is load-bearing for the central accuracy claim and invalidates the QPruner3 rows as unbiased evidence.","section":"Section 3.2, Algorithm 1; Appendix D; Figures 3-4; Table 1"},{"comment":"QPruner3 is described as 'the mixed-precision quantization after further optimization using Bayesian methods based on QPruner2,' and Algorithm 1 initializes b0 from the mutual-information computation. Appendix D, however, states that for the reported LLaMA-7B 50%-pruning experiments, the GP was initialized with '10 sets of configurations' whose precisions were 'randomly selected between 4-bit and 8-bit,' with no indication that the QPruner2 configuration was included. The reported QPruner3 results therefore do not actually demonstrate the Bayesian refinement of the mutual-information allocation that the method section promises; they demonstrate optimization from a random starting population. This inconsistency should be resolved, and the incremental benefit of the mutual-information initialization should be measured.","section":"Section 4.1 and Appendix D"}],"minor_comments":[{"comment":"The paper contains two separate 'Limitation' sections, one directly before the references and one as Appendix G; they should be merged or cross-referenced.","section":"Overall structure"},{"comment":"Table 3 reports memory values as parenthetical numbers (e.g., '61.93(41.32)') without a header specifying that the parenthetical is peak memory in GB; this makes the table difficult to read, and the memory value 41.32 GB for LLM-Pruner is larger than the unquantized model's Table 1 memory, which should be explained.","section":"Table 3"},{"comment":"Equation (7) computes mutual information from layer outputs, but the required discretization of continuous activations (number of bins or a density estimator) is not specified; this hampers reproducibility.","section":"Section 3.2, Eq. (7)"},{"comment":"Algorithm 1 refers to an acquisition function α(b) but does not state which acquisition function is used (e.g., expected improvement, UCB); the Optuna settings in Appendix B are also not detailed enough to reproduce the GP model.","section":"Algorithm 1"},{"comment":"Section 4 states 'QPruner performes' (typo for 'performs') and the abstract's claim of 'up to 6%' accuracy improvement is never precisely anchored to a specific baseline-task-pruning-rate triple in the main text.","section":"Section 4.1 and Abstract"},{"comment":"Appendix F gives usage instructions but no public code repository link; providing the implementation would be necessary for the claimed results to be auditable.","section":"Appendix F"}],"recommendation":"major_revision","confidential_remarks":"The selection-on-test leakage described in major comment 1 is the kind of issue that the editor should weight heavily. The paper's empirical core is the Table 1 comparison, and without a validation-split protocol the headline accuracy numbers cannot be trusted. If the authors cannot provide a corrected experimental setup with a genuine held-out test evaluation (or at least demonstrate that the BO objective was evaluated on training data only), the revised manuscript should not be accepted. I also note that the memory-savings claim is independent of the leakage and appears solid."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a reasonable engineering integration of LLM-Pruner, mutual-information-based mixed-precision quantization, and LoftQ, with consistent ~30% memory savings; but the headline accuracy gains for QPruner3 are likely fitted to the test sets themselves, so the central performance claim is not supported as reported.\n\nWhat's genuinely useful: the framework is straightforward—prune with LLM-Pruner, assign bit-widths by layer importance, optionally run Bayesian optimization over bit-width configurations, then LoftQ fine-tune. The memory numbers across Table 1 are internally consistent: QPruner1/2/3 all cut peak memory by roughly 30-50% relative to LLM-Pruner, and the trend holds at 20/30/50% pruning on both LLaMA-7B and Vicuna-7B. The ablation at 20% pruning covers data type, LoRA init, LoftQ iterations, and importance estimation; that's more than many compression papers do. I also appreciate the candid limitation note about Bayesian optimization being time-consuming.\n\nThe soft spot is load-bearing and the stress-test note is right. Algorithm 1 says the objective measures P(b) after fine-tuning, but the paper never says P(b) is evaluated on a validation split. Appendix D then describes running 50 configurations on the 50%-pruned LLaMA-7B and plotting the Pareto front on BoolQ and WinoGrande—exactly the same benchmarks reported in Table 1. So QPruner3's accuracy is a per-task selected maximum over 50 evaluated bit-width configs, not an unbiased estimate for a fixed configuration. The BoolQ comparison (48.37 vs 43.76) then pits a test-set-selected configuration against LLM-Pruner, which got no such selection. The expected gap from selection alone could account for most of the claimed 6%, so the claim that QPruner3 'outperforms baseline by up to 6%' is unsupported.\n\nIt's also worth noting smaller issues: no seed variance, no code, only one pruning baseline, and the LLaMA-13B table has odd formatting with inconsistent memory values for what should be the same configuration. None of these are fatal by themselves, but they compound the evaluation problem.\n\nWho this is for: someone working on LLM compression might find the combination of mutual-information bit allocation and LoftQ after structured pruning worth trying, but the evaluation needs a held-out split, error bars, and a uniform-precision baseline with the same search budget before the accuracy claim is credible.\n\nMy recommendation: a serious referee should see this because the integration is reasonable and the memory results are solid, but in current form the accuracy claim collapses on selection-on-test leakage. It deserves a major-revision path, not acceptance.","headline":"Sensible engineering integration with a fatal evaluation flaw: QPruner3's accuracy gains are likely test-set-selected maxima, so the central claim is unsupported.","tokens_in":14347,"tokens_out":2650,"would_cite":false,"duration_ms":22716,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"QPruner weds structured pruning to per-layer 4/8-bit quantization, and at 50% pruning on LLaMA-7B it reports higher accuracy than the half-precision baseline on BoolQ while using about 30% less memory.","keywords":["structured pruning","mixed-precision quantization","Bayesian optimization","large language models","parameter-efficient fine-tuning","low-rank adaptation","mutual information","memory efficiency"],"falsifier":"Run the full QPruner pipeline on LLaMA-7B at 50% pruning with the seven downstream tasks held out during Bayesian optimization, select the bit-width configuration on a separate validation set, and report the held-out test accuracy; if the configuration chosen without test information no longer beats LLM-Pruner on BoolQ or saves less than 30% memory, the central claim as stated fails.","tokens_in":13285,"feed_emoji":"🧠","tokens_out":7681,"duration_ms":62346,"temperature":0.7,"pith_summary":"Structured pruning removes whole groups of neurons from an LLM, but the accuracy loss it causes normally has to be repaired by fine-tuning, which costs more memory than pruning saved. QPruner treats pruning, quantization, and fine-tuning as one pipeline: after pruning, each layer is assigned a 4- or 8-bit precision according to how strongly its output predicts the task answer, and a Bayesian optimizer then searches the layer-wise precision assignments under a memory ceiling. The paper's central result is that this combined scheme can beat the standard pruned half-precision baseline on several zero-shot commonsense benchmarks while cutting peak memory by roughly 30% at aggressive pruning rates. Concretely, on LLaMA-7B pruned to 50%, the optimized QPruner3 configuration reports 48.37% on BoolQ versus 43.76% for the baseline, using 16.65 GB instead of 23.89 GB. If the result holds, resource-constrained users can fine-tune and deploy smaller, quantized LLMs without the accuracy penalty usually associated with structured pruning.","feed_headline":"QPruner cuts LLM memory by 30% and lifts accuracy on pruned tasks","feed_subtitle":"Per-layer 4/8-bit assignment plus Bayesian search beats the half-precision baseline on BoolQ at 50% pruning.","key_machinery":"The load-bearing object is the bit-width vector $b=[B_1,\\ldots,B_L]$ with each $B_i$ restricted to 4 or 8 bits and constrained by a total memory budget $M(b)\\le M_{\\max}$. The allocation runs in two stages: mutual information $I(X_i;Y)$ ranks every layer's contribution to the task and fixes an initial precision assignment, and a Gaussian-process model of the map from bit-width vectors to (accuracy, memory) then proposes new configurations through an acquisition function $\\alpha(b)$, fine-tunes each candidate, and iterates toward a Pareto front. Performance recovery uses low-rank adapters initialized by alternating quantization and low-rank approximation, so only small adapter matrices are trained.","core_discovery":"The paper's central claim is that after structured pruning, the model's layers are no longer equally important, so a single uniform bit width either wastes memory or damages prediction. QPruner discovers that per-layer precision should be driven by the mutual information $I(X;Y)$ between each layer's output and the model's final prediction, then refined by Bayesian optimization that trades accuracy against memory. With this allocation, the quantized pruned model can be recovered with low-rank adapter fine-tuning and in several configurations ends up more accurate than the unquantized pruned baseline at lower peak memory. The authors state the advantage grows with pruning rate: at 50% pruning on LLaMA-7B the best configuration improves BoolQ accuracy from 43.76% to 48.37% and lowers memory from 23.89 GB to 16.65 GB, about a 30% reduction.","pith_inferences":["The same per-layer importance signal could be reused to choose pruning ratios per layer, not just quantization widths, making the two compression decisions jointly optimal.","A decisive check is to hold out all seven benchmarks during Bayesian optimization: if the 'best' configuration is selected on a validation set instead, the reported gains may shrink or vanish.","Because only 4-bit and 8-bit options are considered, the Pareto front may miss useful 3-bit or mixed 4/8/16 configurations for extremely memory-starved deployments.","The mutual-information importance estimator is cheap and could replace Hessian-based sensitivity in other structured-pruning pipelines that currently fine-tune at full precision."],"forward_implications":["At 50% pruning on LLaMA-7B, the optimized QPruner3 configuration reports 48.37% on BoolQ versus 43.76% for the LLM-Pruner baseline, with peak memory dropping from 23.89 GB to 16.65 GB, about a 30% reduction.","On Vicuna-7B at 50% pruning, QPruner3 also beats the baseline on BoolQ (61.56% vs 59.51%) and WinoGrande (53.39% vs 52.01%) while using less than two-thirds of the baseline's memory.","The ablation results show the main gain comes from the mixed-precision allocation itself, since replacing NF4 with FP4 or changing the low-rank initialization changes scores only slightly.","Bayesian refinement (QPruner3 over QPruner2) usually improves or preserves accuracy, with the largest gaps appearing at 30% and 50% pruning where layer imbalances are strongest."],"supporting_citations":[{"why":"Supplies the structured-pruning method, the dependency-group importance estimates, and the baseline that QPruner is compared against.","marker":"(Ma et al., 2023)"},{"why":"LoftQ initialization alternates quantization with low-rank approximation, which QPruner uses to recover performance after quantization.","marker":"(Li et al., 2023)"},{"why":"Provides the NF4 4-bit quantization scheme and QLoRA-style fine-tuning used for 4-bit layers and low-rank adapters.","marker":"(Dettmers et al., 2024)"},{"why":"Supplies the Bayesian optimization machinery, including the Gaussian-process model and acquisition function, used to refine bit-width assignments.","marker":"(Frazier, 2018)"},{"why":"LoRA defines the low-rank adapter parameterization that QPruner fine-tunes after pruning and quantization.","marker":"(Hu et al., 2021)"},{"why":"The Alpaca instruction-following data set provides the 50k public samples used for pruning calibration and fine-tuning.","marker":"(Taori et al., 2023)"}],"fun_headline_variants":["QPruner slashes LLM memory by 30% and boosts BoolQ accuracy","Per-layer bits via Bayesian search: QPruner cuts memory 30%","QPruner trims memory 30% with smarter bit allocation","Mutual information picks bit widths; QPruner saves memory","QPruner reduces memory 30% and improves pruned task accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The Bayesian optimizer's performance scores appear to come from the same benchmark tasks used to report final accuracy, with no described validation split, so the reported QPruner3 numbers may be selected maxima rather than unbiased estimates.","fun_headline_variants_meta":{"raw":{"variants":["QPruner slashes LLM memory by 30% and boosts BoolQ accuracy","Per-layer bits via Bayesian search: QPruner cuts memory 30%","QPruner trims memory 30% with smarter bit allocation","Mutual information picks bit widths; QPruner saves memory","QPruner reduces memory 30% and improves pruned task accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000656,"raw_usage":{"total_tokens":2995,"prompt_tokens":928,"completion_tokens":2067,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":1970}},"tokens_in":544,"tokens_out":2067,"duration_ms":14391,"temperature":1.0,"reasoning_tokens":1970,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:45:04.260119+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full QPruner pipeline on LLaMA-7B at 50% pruning with the seven downstream tasks held out during Bayesian optimization, select the bit-width configuration on a separate validation set, and report the held-out test accuracy; if the configuration chosen without test information no longer beats LLM-Pruner on BoolQ or saves less than 30% memory, the central claim as stated fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Bayesian optimization machinery, including the Gaussian-process model and acquisition function, used to refine bit-width assignments."}],"review_version":1}