{"id":"3589d673-faab-4c19-9606-ac0da01c61f1","arxiv_id":"2505.07289","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A 25% pruning plus 4-bit quantization configuration retains roughly 20% more benchmark performance than 3-bit quantization alone at matching theoretical compression rates, across two LLMs.","lead":"The authors tested combining two model-shrinking methods, pruning and quantization, on two 7-8B language models. They report that a balanced mixture keeps more of the model's abilities than using quantization alone at the same theoretical size.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (1) as printed contradicts Table I: 25% pruning + 4-bit quantization has TCr 56.25% by the formula, not the 81.25% the paper needs for the headline matched-TCr comparison.","rationale":"The reader's weakest assumption concerned benchmark representativeness, which is a real external-validity limitation. But the more immediate threat to the central claim is internal: the paper's own TCr definition in Eq. (1) contradicts the values used in Table I and in the Section V comparison. The empirical tables do appear to support the qualitative 20% claim if the intended TCr formula is the saved-fraction form behind Table I, and the LLaMA 4-bit MATH anomaly is explicitly acknowledged in the text, so I would not reject the paper. However, the published equation must be corrected and the matched-TCr comparisons re-verified before the headline claim can stand as stated. Since the reader's verdict is already CONDITIONAL, I recommend keeping that verdict and adding the TCr correction as an explicit condition. My disagreement with the reader's weakest_assumption is therefore about which concern is load-bearing, not about the overall verdict.","tokens_in":14068,"tokens_out":7408,"duration_ms":62983,"concrete_test":"Recompute TCr for 25% pruning with 4-bit quantization using Eq. (1) as printed: (1 − 4/16) × ((100 − 25)/100) = 0.75 × 0.75 = 0.5625. Compare this with Table I's entry of 81.25% and with pure 3-bit quantization, TCr = 1 − 3/16 = 0.8125. If the joint value does not equal 81.25%, then either Eq. (1) must be revised to TCr = 1 − (q/16) × ((100 − s)/100) or the Section V claim that the two configurations have the same TCr is unsupported. This single arithmetic check determines whether the headline comparison is matched.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section V is that 25% pruning with 4-bit quantization outperforms pure 3-bit quantization by about 20% in semantic retention 'at the same Theoretical Compression Rate (TCr).' This requires TCr(25% pruning + 4-bit) = TCr(3-bit) = 81.25%, which is exactly what Table I lists. However, Eq. (1) in Section III-A-1 defines joint TCr as (1 − q/16) × ((100 − s)/100). For q=4 and s=25 this evaluates to 0.75 × 0.75 = 0.5625, that is, 56.25%, not 81.25%. The table values correspond instead to the saved-fraction formula TCr = 1 − (q/16) × ((100 − s)/100), which gives 1 − 0.25 × 0.75 = 0.8125. Since TCr is the paper's stated device for making 'fair comparisons across setups,' this is not a cosmetic typo: as printed, the equation does not support the headline comparison, and any reader who applies Eq. (1) will find the two configurations are not matched at all. The paper must either correct Eq. (1) to match Table I or explicitly redefine TCr; otherwise the strongest claim is not internally consistent.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates joint pruning and quantization for LLMs. It defines Theoretical Compression Rate (TCr) to align different compression configurations at the same nominal size and introduces Semantic Retention Compression Rate (SrCr) as a trade-off metric. Using SparseGPT (pruning) and GPTQ (quantization) on LLaMA-3.1-8B and Mistral-7B-v0.3, with MMLU-Pro, BBH, and MATH benchmarks, the authors report that 25% unstructured pruning with 4-bit quantization outperforms pure 3-bit quantization by approximately 20% in semantic retention at the same 81.25% TCr, and that semi-structured patterns such as 2:8 show similar promise. The central claim is that a balanced joint configuration beats aggressive single-method compression at equal nominal model size.","tokens_in":14280,"tokens_out":8188,"duration_ms":71254,"significance":"The empirical comparison is a useful addition: the results in Table VII directly support the qualitative claim that 25% pruning plus 4-bit quantization retains more benchmark performance than 3-bit quantization alone, and full result tables are provided for both models. However, the theoretical support is not yet sound. Equation (1) is inconsistent with Table I, the numerical claim of 'approximately 20%' is not what the tables show (observed average gain is roughly 30%), the validation of the error approximation compares the wrong quantities, and the SrCr metric embeds the preferences it is used to discover. These issues do not necessarily invalidate the empirical finding, but they must be repaired before the paper's stated claims can be accepted.","major_comments":[{"comment":"Equation (1) as printed gives TCr = (1 − q/16)·((100−s)/100) for joint compression. For q=4 and s=25 this evaluates to 56.25%, whereas Table I lists 81.25%. The table values are consistent with TCr_joint = 1 − (q/16)·((100−s)/100). Because the headline comparison in Section V requires TCr(25% pruning + 4-bit) = TCr(3-bit) = 81.25%, Eq. (1) must be corrected to match the table; otherwise the two configurations are not matched at the stated TCr.","section":"III-A-1, Eq. (1); Table I"},{"comment":"The abstract and Section V claim that 25% pruning + 4-bit quantization outperforms 3-bit quantization by approximately 20% in semantic retention at the same TCr. Reading directly from Table VII, the mean score of LLaMA-3.1-8B increases from 23.1 (3-bit) to 31.5 (25%+4-bit), a relative gain of 36.4%, and for Mistral-7B-v0.3 from 24.2 to 30.2, a gain of 24.8%. The average of the two models is about 30.6%, not roughly 20%. The 'approximately 20%' claim should be corrected or the computation behind it should be shown.","section":"Section V; Table VII"},{"comment":"The validation of the error decomposition in Eqs. (7)-(9) is not carried out on the quantities those equations describe. Equation (9) relates the difference in column-wise quantization error norms (E_GPTQ − E_simple) to the accumulated update norm ||δ_A,j||^2, but Fig. 4 and the accompanying text compare benchmark accuracy scores (MMLU-Pro, BBH, MATH) between GPTQ and NF4/LLM.int8(). Accuracy differences do not directly measure these error norms, so the conclusion that '||δ_A,j||^2 remains well-bounded' is not supported by the plotted evidence. The authors should either plot the actual error norms or supply a formal argument connecting the performance gaps to the error-norm bound.","section":"III-B-3, Fig. 4"},{"comment":"The SrCr metric is constructed with explicitly chosen functional forms—square-root in p, logarithmic in q, and normalization denominator 4—whose stated purpose is to match empirical observations and provide resolution in the practically-viable range. Using this metric in Section IV-B-3 to select 25% pruning + 4-bit quantization as the optimum therefore partly encodes the answer in the metric. The paper should include a sensitivity analysis over the metric's free parameters (exponent on p, normalization constant) and show that the ranking of configurations is robust, or else moderate the claim that SrCr is a principled basis for identifying optimal configurations.","section":"III-C-2, Eqs. (13)-(15)"}],"minor_comments":[{"comment":"Point 2 states that the 25% + 4-bit configuration achieves 'around 20% higher semantic retention across all evaluation metrics,' but Table VII shows per-task relative gains ranging from about 15% (Mistral MMLU-Pro) to 150% (LLaMA MATH); please report the per-task numbers or use a range.","section":"IV-B-3"},{"comment":"Because SrCr_j multiplies sqrt(p) by -log2(q/16)/4, it is exactly zero for any configuration with p=0 or q=16, so the metric cannot rank joint configurations against pruning-only or quantization-only baselines; this limitation should be stated explicitly.","section":"III-C-2, Eq. (15)"},{"comment":"The three-line piecewise definition of TCr in Eq. (1) is typeset without explicit multiplication signs; adding parentheses and an explicit product would prevent the ambiguity that contributes to the inconsistency noted in Major Comment 1.","section":"III-A-1, Eq. (1)"},{"comment":"The exclusion of IFEval, MuSR, and GPQA because the baseline models perform poorly on them narrows the meaning of 'semantic retention' to the three selected benchmarks; the abstract and conclusion should qualify the claim accordingly.","section":"IV-A-1"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest about its limitations and provides reproducible tables, which I credit. However, the TCr equation/table mismatch and the inaccurate 20% number are basic consistency issues that need to be resolved before the manuscript can be considered for publication. The contribution as a journal paper depends on whether the authors can supply the missing validation and robustness analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my honest read. The headline result—that a moderate joint configuration (25% unstructured pruning + 4-bit GPTQ) preserves more capability than pure 3-bit quantization at the same theoretical compression rate—is directly supported by Tables V and VII on both LLaMA-3.1-8B and Mistral-7B-v0.3. That is a useful, practical finding: a cheap recipe for edge deployment. The paper is transparent about its benchmarks and gives detailed per-task numbers, which I appreciate.\n\nWhat's actually new is limited. The idea of combining sparsity and low-bit quantization is already in SpQR, SqueezeLLM, and the joint sparsification-quantization literature they cite; the paper acknowledges this. The genuinely new pieces are the sequential-approximation analysis and the SrCr metric. Neither is fully convincing. Equation (7) is an approximation with no derivation; the validation in Figure 4 plots benchmark scores, not the error norms the theory is about. SrCr is explicitly constructed to 'provide higher resolution in the practically-viable pruning range' and to 'match empirical observations,' so using it later to declare 25% + 4-bit optimal is circular. It would be more honest to present SrCr as a heuristic ranking tool, not a principled metric.\n\nThe soft spots are real but not devastating. The benchmark set (MMLU-Pro, BBH, MATH) is narrow, and they exclude IFEval, MuSR, and GPQA because baselines score too low; that is disclosed but limits generalization. The LLaMA 4-bit MATH score of 0.2 versus 3-bit 4.2 is anomalous and unexplained—they mention it as 'unexpected' but never dig in. No code is released, which is a minor frustration for reproducibility.\n\nOne formal concern that I checked: the stress-test note claims Eq. (1) gives 56.25% for 25% pruning + 4-bit, contradicting Table I's 81.25%. That seems like a misreading. The intended formula is savings = 1 − (q/16)(100−s)/100, which gives 0.8125; the typesetting is ambiguous, but the table and the quantization-only values confirm this definition. So that particular inconsistency is not real.\n\nWho is this for? Practitioners deciding how to compress a 7-8B model, and people working on compression evaluation metrics. It is not a breakthrough, but it is a solid experimental recipe with enough substance to deserve peer review. I would recommend accepting it for review, with the expectation that the authors tighten the theory, re-frame SrCr, explain the MATH anomaly, and release code.","headline":"A practical configuration recommendation with a clean empirical core, wrapped in a theory and metric that are more heuristic than principled; the reported 20% advantage holds on the data.","tokens_in":14923,"tokens_out":7155,"would_cite":false,"duration_ms":58725,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that pruning 25% of a model's weights before 4-bit quantization preserves about 20% more benchmark performance than pure 3-bit quantization at the same theoretical compression rate.","keywords":["joint compression","pruning","quantization","semantic retention","theoretical compression rate","semantic retention compression rate","large language models","SparseGPT"],"falsifier":"Compare 25% pruning plus 4-bit quantization against pure 3-bit quantization on held-out benchmarks the paper excluded, such as instruction-following, multistep soft reasoning, or graduate-level QA tasks, using the same two models and compression pipeline. If the roughly 20% retention advantage is absent or reversed at the same TCr, the claim is limited to the original three-task evaluation rather than semantic retention in general.","tokens_in":13778,"feed_emoji":"🧠","tokens_out":6600,"duration_ms":52821,"temperature":0.7,"pith_summary":"This paper asks whether pruning and quantization can be combined to compress large language models beyond what either method achieves alone. It argues that applying SparseGPT pruning first and GPTQ quantization second can approximate true joint compression, and uses a new metric, the Semantic Retention Compression Rate (SrCr), to compare configurations at equal theoretical compression. On LLaMA-3.1-8B and Mistral-7B-v0.3, the paper reports that 25% pruning followed by 4-bit quantization retains about 20% more benchmark performance than pure 3-bit quantization at the same theoretical compression rate. The finding matters because it offers a practical recipe for shrinking models without giving up as much reasoning ability, and because it suggests future work should optimize pruning and quantization jointly rather than pushing one method alone.","feed_headline":"Prune-then-quantize beats 3-bit alone by 20%","feed_subtitle":"A 25% pruning plus 4-bit recipe keeps more reasoning skill than pushing the same model to 3-bit weights.","key_machinery":"Three pieces carry the argument. First, the Theoretical Compression Rate, $TCr = (1 - q/16)(100 - s)/100$ for joint configurations, with pruning-only and quantization-only as special cases, puts different recipes on one comparable scale. Second, the Semantic Retention Compression Rate, $SrCr_j = (-\\log_2(q/16)/4) \\cdot \\sqrt{p} \\cdot Sr_j$, combines sparsity, bit-width, and measured retention into a bounded $[0,1]$ score used to pick optimal configurations. Third, a sequential approximation argument: by decomposing GPTQ's column-wise error into immediate quantization error plus accumulated Hessian-based updates, the paper argues that quantizing all weights after pruning (Case A) approximates quantizing only non-pruned weights (Case B) at low sparsity, and validates this empirically by comparing GPTQ with simpler quantizers (NF4 and LLM.int8()) before and after pruning.","core_discovery":"The central claim is that a balanced sequential combination of pruning and quantization outperforms quantization alone at the same Theoretical Compression Rate (TCr). Concretely, SparseGPT pruning at 25% sparsity followed by GPTQ 4-bit quantization consistently beats GPTQ-only 3-bit quantization on MMLU-Pro, BBH, and MATH for both tested 7-8B models, with approximately 20% higher semantic retention. The paper also claims this advantage extends qualitatively to semi-structured patterns: 2:8 and 1:4 patterns combined with 4-bit quantization outperform pure 3-bit quantization, and at higher rates 33.333% pruning with 3-bit quantization retains meaningful capability where pure 2-bit quantization collapses. Quantization alone is the stronger single method, so the paper treats it as the baseline; joint compression is presented as superior to pushing quantization to lower bit-widths.","pith_inferences":["If the 20% advantage reproduces on other task suites, it implies that compression recipes should spend a small pruning budget before dropping bit-width instead of pushing quantization alone.","The sequential approximation argument leaves a concrete open prediction: a true joint optimizer that coordinates pruning masks with quantization error could beat the sequential recipe by an amount the paper estimates only indirectly.","Because only 7-8B models are tested, the optimal joint configuration for larger or smaller models is unknown; the paper itself says the SrCr formula may need adjustment as information density changes.","The per-task tables suggest the 20% headline is an average, not uniform behavior; MATH is consistently the most fragile task, so deployment on math-heavy workloads should expect smaller joint-compression gains."],"forward_implications":["At 81.25% TCr, 25% pruning plus 4-bit quantization is the recommended configuration for both models, delivering roughly 20% higher semantic retention than pure 3-bit quantization.","Aggressive pruning with light quantization (50% pruning plus 8-bit) underperforms pure 4-bit quantization at the same 75% TCr, so balanced joint configurations matter rather than any combination.","Semi-structured patterns such as 2:8 retain most of the joint-compression benefit and are candidates for hardware acceleration.","Mistral-7B-v0.3 tolerates aggressive compression better than LLaMA-3.1-8B, and MATH is consistently the most compression-sensitive benchmark.","The SrCr metric gives future compression studies a bounded, interpretable way to compare pruning-quantization configurations on a single scale."],"supporting_citations":[{"why":"Supplies GPTQ, the quantization algorithm used for quantization-only and joint compression configurations.","marker":"[4]"},{"why":"Supplies SparseGPT, the pruning algorithm applied before quantization in joint configurations.","marker":"[8]"},{"why":"Provides LLM.int8(), one of two simpler quantizers used to validate the error-approximation analysis.","marker":"[13]"},{"why":"Provides NF4 quantization, the other simpler quantizer used to validate the error-approximation analysis.","marker":"[24]"},{"why":"Defines LLaMA-3.1-8B, one of the two models whose retention is measured.","marker":"[25]"},{"why":"Defines Mistral-7B-v0.3, the second model whose retention is measured.","marker":"[26]"},{"why":"One of the three benchmarks used to measure semantic retention.","marker":"[30]"},{"why":"One of the three benchmarks used to measure semantic retention.","marker":"[31]"},{"why":"One of the three benchmarks used to measure semantic retention.","marker":"[32]"}],"fun_headline_variants":["Prune plus 4-bit tops 3-bit by 20%","Combined pruning and quantization: 20% better than either alone","Two-method compression wins: 20% more semantic retention","Pruning and quantization together beat pure quantization by 20%","Semantic retention and extreme compression: both achievable, 20% better"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim stands on treating MMLU-Pro, BBH, and MATH as a representative sample of semantic retention; the paper's own benchmark screen excluded tasks where the baseline models were weak, so the 20% figure has only been measured on tasks the models could already handle.","fun_headline_variants_meta":{"raw":{"variants":["Prune plus 4-bit tops 3-bit by 20%","Combined pruning and quantization: 20% better than either alone","Two-method compression wins: 20% more semantic retention","Pruning and quantization together beat pure quantization by 20%","Semantic retention and extreme compression: both achievable, 20% better"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000315,"raw_usage":{"total_tokens":1744,"prompt_tokens":861,"completion_tokens":883,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":477,"completion_tokens_details":{"reasoning_tokens":792}},"tokens_in":477,"tokens_out":883,"duration_ms":7781,"temperature":1.0,"reasoning_tokens":792,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:20:15.877161+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare 25% pruning plus 4-bit quantization against pure 3-bit quantization on held-out benchmarks the paper excluded, such as instruction-following, multistep soft reasoning, or graduate-level QA tasks, using the same two models and compression pipeline. If the roughly 20% retention advantage is absent or reversed at the same TCr, the claim is limited to the original three-task evaluation rather than semantic retention in general.","supporting_citations":[{"cited_title":"Measuring mathematical problem solving with the MATH dataset","cited_arxiv_id":null,"evidence_quote":"One of the three benchmarks used to measure semantic retention."},{"cited_title":"GPTQ: Accurate post-training quantization for generative pre-trained transformers","cited_arxiv_id":null,"evidence_quote":"Supplies GPTQ, the quantization algorithm used for quantization-only and joint compression configurations."},{"cited_title":"SparseGPT: Massive language models can be accurately pruned in one-shot","cited_arxiv_id":null,"evidence_quote":"Supplies SparseGPT, the pruning algorithm applied before quantization in joint configurations."},{"cited_title":"LLM.int8(): 8-bit matrix multiplication for transformers at scale","cited_arxiv_id":null,"evidence_quote":"Provides LLM.int8(), one of two simpler quantizers used to validate the error-approximation analysis."},{"cited_title":"QLoRA: Efficient finetuning of quantized LLMs","cited_arxiv_id":null,"evidence_quote":"Provides NF4 quantization, the other simpler quantizer used to validate the error-approximation analysis."},{"cited_title":"MMLU-Pro: A more robust and challenging multi- task language understanding benchmark","cited_arxiv_id":null,"evidence_quote":"One of the three benchmarks used to measure semantic retention."},{"cited_title":"Challenging BIG-Bench tasks and whether chain- of-thought can solve them","cited_arxiv_id":null,"evidence_quote":"One of the three benchmarks used to measure semantic retention."}],"review_version":1}