{"id":"029cef5e-0593-4f6d-a197-53071367e726","arxiv_id":"2602.09130","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Compressed LLMs retain factual knowledge but degrade in multi-step reasoning, multilingual, and instruction-following, while preserved performance does not imply preserved reliability.","lead":"UniComp is a unified benchmark that compares pruning, quantization, and distillation of LLMs across performance, reliability, and efficiency. It finds that compressed models keep factual knowledge but lose reasoning, multilingual, and instruction-following ability, and that keeping performance does not mean keeping reliability.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Potential calibration/evaluation overlap behind the headline +50% reasoning gain: if GSM8K/MATH calibration examples are not disjoint from GSM8K/MATH-500 eval, the reasoning-aware calibration result is mask-fitting, not a general property.","rationale":"I focused on the calibration/evaluation disjointness because it is the only assumption that, if false, would invalidate a headline quantitative claim rather than merely qualify it. The reader's weakest assumption matches this; I agree. The knowledge-bias finding itself is supported by Table 1 default-calibration results and is less exposed, but the paper uses the calibration experiment as the causal explanation of the bias; overlap would sever that link. I considered the unvalidated GPT-4o-mini judge and the missing error bars on reliability, but those concerns are limitations that can be addressed by robustness checks; the split-provenance issue is a potential correctness bug in a reported 50% improvement. Since the paper currently reports only a GitHub link and the review cannot verify code, the condition is verification of split provenance. If the split is clean, the paper should remain CONDITIONAL primarily for reproducibility/error-bar reasons; if the split overlaps, the headline claims would need substantial revision. No verdict shift from the Reader's CONDITIONAL is needed at this stage.","tokens_in":28081,"tokens_out":5869,"duration_ms":51552,"concrete_test":"Download the UniComp repository and inspect the calibration set construction for Table 10. Determine the exact source splits of the GSM8K/MATH/ARC-c calibration examples and whether MATH-500 is a subset of the MATH split used. Then rerun SparseGPT 50% on LLaMA-3.1-8B with a verification calibration set drawn strictly from GSM8K train, MATH train, and ARC-c train, same size and prompt format as the paper, and evaluate on the official GSM8K/MATH-500/GPQA-D test sets. If GSM8K stays near 55% and MATH-500 near 12%, the concern is resolved; if it reverts toward 36.9%/8.4% (the C4-calibrated rows), claim (iii) and the causal interpretation in Section 5.4 collapse.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The single most load-bearing assumption is the unstated disjointness of the reasoning-aware calibration set from the reasoning evaluation sets. Section 5.4/Appendix C.2 describe a 'reasoning-centric calibration set' built from MATH, GSM8K, and ARC-c, evaluated with GSM8K, MATH-500, and GPQA-D (Table 10). For the advertised +50% relative GSM8K improvement (36.92 to 55.04 for LLaMA-3.1-8B SparseGPT), the calibration examples must come from a split different from the GSM8K test split used for evaluation; likewise, MATH calibration must be disjoint from MATH-500. The manuscript never states the split provenance; 'equal training samples of GSM8K, MATH and ARC-c' is ambiguous—'training' may mean the train split, but this is never made explicit. If either overlap exists, the improvement is an artifact of fitting the pruning mask to the benchmark, and the paper's argument that calibration data cause the knowledge bias (Section 5.4) loses its causal support. Note this does not invalidate the Table 1 default-calibration knowledge-bias finding, but it does invalidate a headline claim and the interpretation that task-specific calibration 'directly supports' the bias.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"UniComp proposes a unified framework for evaluating LLM compression along performance, reliability, and efficiency. The paper compares six compression techniques (SparseGPT, Wanda, GPTQ, AWQ, Minitron, Low-Rank Clone) across 40+ datasets on LLaMA-3.1-8B, Qwen-2.5-7B, and additional models. It reports three headline findings: (i) a consistent knowledge bias, with factual recall largely preserved while reasoning, multilingual, and instruction-following degrade; (ii) a performance–reliability decoupling; and (iii) up to 50% relative reasoning improvement from reasoning-aware calibration for pruned models. The paper also provides hardware-aware efficiency analysis, concluding that quantization offers the best overall trade-off.","tokens_in":28412,"tokens_out":6385,"duration_ms":63432,"significance":"If the findings hold, UniComp is a valuable contribution: it broadens compression evaluation beyond knowledge-centric benchmarks and explicitly adds reliability and efficiency axes. The cross-model consistency of the knowledge-bias pattern and the breadth of benchmarks are strengths, as is the effort to evaluate distillation as a compression method under a common protocol. However, the headline calibration result and the reliability decoupling claim currently rest on insufficiently specified experimental details and unvalidated measurements. With those resolved, the framework would be a useful resource for the compression community.","major_comments":[{"comment":"The reasoning-aware calibration experiment does not state the split provenance of the calibration set. The text says 'equal training samples of GSM8K, MATH and ARC-c' and evaluates on GSM8K, MATH-500, and GPQA-D. If the calibration set includes examples from the GSM8K test split or MATH-500, the reported +50% relative GSM8K improvement for LLaMA-3.1-8B SparseGPT (36.92 to 55.04) is an artifact of fitting the pruning mask to the evaluation benchmark, not a general property of reasoning-aware calibration. Similarly, ARC-c appears in both the calibration set and the knowledge evaluation in Table 10. Please specify the exact split used for each calibration source and verify disjointness from every evaluation set. The causal interpretation in Section 5.4 ('directly supports the presence of a knowledge bias') depends on this.","section":"Section 5.4, Table 10, Appendix C.2"},{"comment":"The performance–reliability decoupling is a central claim, but it rests on single-run point estimates with no variance or confidence intervals. Many reliability scores exceed 100 (e.g., Low-Rank Clone fairness 124.19 in Table 2; also Table 18), which is surprising given the stated [0,100] scaling in Section 3.2. The reliability judge (GPT-4o-mini) is unvalidated beyond an informal sanity check of ~30 instances. Apparent reliability 'improvements' over baseline could be judge noise or prompt sensitivity. Please provide multiple runs or bootstrap intervals, and report agreement between the LLM judge and human annotations or a validated judge. Without this, the decoupling claim is not yet supported at the strength stated.","section":"Section 5.2, Table 2, Appendix B.3"},{"comment":"There are internal inconsistencies in the reported knowledge scores. For LLaMA-3.1-8B SparseGPT 50%, Table 1 reports MMLU=48.33 and SK=82.7, while Table 8 reports MMLU=55.74 and SK=0.89 (i.e., 89.3 after scaling), and Table 10 reports the same default-calibration MMLU=55.74. The baseline MMLU also differs: Table 1 gives 61.38, Table 10 gives 63.09. The aggregate knowledge score cannot be reconciled across tables. Please audit the underlying runs and provide a single consistent set of results; the knowledge-bias claim is qualitative, but the specific magnitudes and aggregate scores should be reproducible from the reported tables.","section":"Table 1 vs. Table 8 and Table 10"},{"comment":"The efficiency conclusion that distillation has 'prohibitive compute cost' relies on estimated Minitron training times (120h and 140h). Appendix A.4 gives a plausible range of 2.3–7.5 days (about 55–180 hours) from throughput assumptions, yet Table 3 reports point estimates without sensitivity. The compute-cost score SCC is then normalized against these estimates, which affects Figure 2 and the relative ranking of distillation versus pruning/quantization. Please report the range or a sensitivity analysis, or use directly measured training time from the original Minitron release if available.","section":"Table 3, Appendix A.4"}],"minor_comments":[{"comment":"Model family names are inconsistent: 'LLaMA' appears in most places, but Table 1 and Appendix tables use 'LLAMA'; Table 17 uses 'LLaMA-3-8B' instead of LLaMA-3.1-8B. Please unify naming.","section":"Throughout"},{"comment":"The stated scaling 'All scores are scaled to [0,100]' conflicts with reliability scores above 100. Please clarify whether the scaling applies only to performance scores, or whether values above 100 are intentional (e.g., relative retention without capping).","section":"Section 3.2 / Table 2"},{"comment":"The comparison between knowledge and reasoning mixes task formats: knowledge benchmarks are multiple-choice, while GSM8K/MATH require free-form generation. The paper acknowledges this for GPQA-D, but the knowledge-bias interpretation could be strengthened by adding at least one generation-based knowledge task or one multiple-choice reasoning task to separate format sensitivity from capability-specific degradation.","section":"Section 5.1 / Table 1"},{"comment":"Table 10 would benefit from a column indicating the exact number of calibration samples per source and whether they were drawn from train or test splits. This is already requested in the major comment, but it also affects reproducibility of the calibration experiment.","section":"Appendix C.2 / Table 10"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful scope and the default-calibration knowledge-bias finding appears reasonably robust across model families. However, the calibration split ambiguity directly affects a headline claim, and the table inconsistencies (e.g., SparseGPT MMLU 48.33 vs 55.74) suggest the numerical appendix needs a careful audit. I would ask the authors to provide exact split specifications, rerun or clarify the disputed numbers, and add uncertainty/validation for the reliability judge. These are fixable within the manuscript's scope, so I am not recommending rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you work on LLM compression. It does something useful: unified three-axis comparison of pruning, quantization, and distillation across 40 datasets, with reliability added. The knowledge-bias finding is consistent across model families and matches prior intuitions. That part is solid. What's new: the framework, the explicit inclusion of distillation, the reliability dimension, and the calibration experiment. The efficiency analysis is sensible, though the Minitron training times are admittedly estimates. Soft spots: (1) The headline +50% GSM8K gain from reasoning-aware calibration depends on the calibration set being disjoint from test sets. The text says 'equal training samples of GSM8K, MATH and ARC-c' which strongly implies train splits, but it never explicitly says so. That's one sentence to fix. If it's actually train, the result holds. (2) The reliability decoupling rests on single-run point estimates with no error bars; several scores over 100 are plausible but we can't judge significance. The GPT-4o-mini judge gets only a 30-instance informal sanity check, which is thin. These make the decoupling suggestive, not definitive. (3) Table 1 lists SparseGPT-50% MMLU for LLaMA-3.1-8B as 48.33 while Table 8 and Table 10 list 55.74. That looks like a typo but needs correction. The calibration overlap concern raised elsewhere doesn't land as a fatal flaw; the provenance is probably fine, but should be stated. Overall this is a serious evaluation paper, not a benchmark dump. It deserves a real peer review. I'd send it out with requests to clarify splits, add error bars or variance discussion, validate the judge on a small human sample, and fix the table discrepancy.","headline":"A solid evaluation framework with a consistent knowledge-bias finding; the calibration split should be clarified and the reliability decoupling needs error bars, but it deserves review.","tokens_in":703,"tokens_out":1695,"would_cite":true,"duration_ms":45678,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Compressing LLMs preserves factual recall but systematically erodes reasoning, multilingual ability, and instruction-following.","keywords":["model compression","pruning","quantization","knowledge distillation","knowledge bias","LLM reliability","calibration data","reasoning degradation"],"falsifier":"Run the pruning experiments with a calibration set that provably excludes every evaluation example—for instance, by holding out entire problem sets and using only disjoint splits—and check whether the GSM8K and MATH-500 gains persist. Alternatively, calibrate on random problems of the same format and see if the gain vanishes, which would indicate the improvement is specific to matching the test distribution.","tokens_in":28006,"feed_emoji":"🧠","tokens_out":6686,"duration_ms":66186,"temperature":0.7,"pith_summary":"This paper tries to establish a consistent 'knowledge bias' in large language model compression: pruning, quantization, and distillation all tend to preserve factual recall and commonsense while substantially degrading multi-step reasoning, multilingual capability, and instruction following. It introduces UniComp, a three-dimensional evaluation framework (performance, reliability, efficiency) applied to six compression techniques across more than 40 datasets. A second central claim is that compression decouples performance from reliability: a model that retains its task scores does not necessarily retain its truthfulness, safety, fairness, or privacy behavior. A third claim is that the choice of calibration data drives the bias: a reasoning-aware calibration set built from math and science problems improves pruned-model reasoning by up to 50% relative, without hurting knowledge benchmarks. If true, this means knowledge-centric compression benchmarks overstate how much of a model's capability survives compression, and that calibration data deserve the same scrutiny as the compression method itself.","feed_headline":"Compression preserves facts, erodes reasoning and reliability","feed_subtitle":"Across 40 datasets, facts survive compression while reasoning and reliability do not.","key_machinery":"The carrying mechanism is the UniComp evaluation framework: a standardized protocol that normalizes each compressed model's scores against its base model across performance, reliability, and efficiency using 13 aggregate metrics. The performance track converts knowledge, multilingual, reasoning, and instruction-following benchmarks into retained-performance ratios; the reliability track covers truthfulness, safety, fairness, robustness, privacy, and ethics; the efficiency track measures runtime acceleration, inference efficiency, and compute cost. The mechanism behind the calibration finding is the reasoning-aware calibration set—constructed from MATH, GSM8K, and ARC-c problems—which is pass","core_discovery":"Across pruning, quantization, and distillation applied to LLaMA-3.1-8B, Qwen-2.5-7B, and other architectures, the paper finds that factual recall and commonsense multiple-choice tasks are the least affected by compression, while chain-of-thought reasoning benchmarks such as GSM8K and MATH-500 fall the most. Models that retain 80–100% of knowledge scores often retain only 20–60% of reasoning scores; the gap widens with aggressive pruning and smaller models. Reliability scores across truthfulness, safety, fairness, robustness, privacy, and ethics do not track task performance, and some compressed models actually beat the base model on reliability. Replacing the default calibration corpus with","pith_inferences":["If the knowledge bias is a general property of existing compression techniques, then compression research should shift its optimization target from perplexity and MMLU to reasoning-preserving objectives; a testable next step is calibrating pruning masks on synthetic chain-of-thought data rather than static problem sets.","The performance–reliability decoupling implies that compressed models should undergo separate safety and privacy audits before deployment; the paper's own admission that reliability scoring relies on a single LLM judge suggests these audits should be re-run with human raters.","The fact that AWQ outperforms GPTQ on multilingual benchmarks while performing similarly in English points to an untested hypothesis: quantization that preserves activation outliers in English-centric calibration may be misaligned with low-resource languages. A direct test would compare the two on a broader multilingual suite with per-language calibration data.","Reasoning-aware calibration might be combined with reliability-aware calibration: since fairness and robustness scores also vary with compression, a calibration set that mixes reasoning and safety-critical examples could yield a single pruning recipe that preserves both."],"forward_implications":["Knowledge-centric compression benchmarks—such as MMLU, ARC, and HellaSwag—systematically overstate the health of a compressed model; a model can score near-baseline on those while losing most of its reasoning ability.","Deployers who use compressed models for multi-step reasoning or agentic tasks should expect materially worse performance than single-score compression metrics suggest.","Quantization is not a drop-in replacement for dense models: it preserves most performance but still shows non-trivial drops on challenging reasoning tasks, and multilingual behavior varies by method.","Reliability must be measured independently of performance; a compressed model that retains task accuracy may have degraded privacy, safety, or fairness scores.","Calibration data is a load-bearing choice in pruning: switching from generic text to task-relevant problems can recover a large fraction of lost reasoning at negligible extra cost."],"fun_headline_variants":["Compression keeps facts, kills reasoning","LLM compression: facts stay, thought fades","Compression erodes reasoning more than facts","In LLMs, compression preserves recall, not reasoning"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The reasoning-aware calibration set built from MATH, GSM8K, and ARC-c is assumed to be disjoint from the evaluation sets MATH-500, GSM8K, and GPQA-D; if any of those examples overlap, the reported up-to-50% reasoning improvement is an artifact of fitting the pruning mask to the test benchmark.","fun_headline_variants_meta":{"raw":{"variants":["Compression keeps facts, kills reasoning","LLM compression: facts stay, thought fades","Compression erodes reasoning more than facts","In LLMs, compression preserves recall, not reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000172,"raw_usage":{"total_tokens":1081,"prompt_tokens":681,"completion_tokens":400,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":425,"completion_tokens_details":{"reasoning_tokens":342}},"tokens_in":425,"tokens_out":400,"duration_ms":5283,"temperature":1.0,"reasoning_tokens":342,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T03:01:45.049075+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the pruning experiments with a calibration set that provably excludes every evaluation example—for instance, by holding out entire problem sets and using only disjoint splits—and check whether the GSM8K and MATH-500 gains persist. Alternatively, calibrate on random problems of the same format and see if the gain vanishes, which would indicate the improvement is specific to matching the test distribution.","supporting_citations":[],"review_version":2}