{"id":"342cd3cf-8732-4d12-a3a8-9aed26179a3b","arxiv_id":"2412.15921","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Flab-Pruner applies unified vocabulary, layer, and FFN pruning with a KL objective to Code LLMs, cutting 22% of parameters while retaining about 97% of code-generation performance and recovering the rest via LoRA post-training.","lead":"This paper introduces Flab-Pruner, a method that combines vocabulary, layer, and feed-forward network pruning to shrink code-generating language models by about 22% of their parameters while keeping most of their performance. It also introduces a data-replacement post-training step that helps the pruned models recover, and reports savings in memory, compute, and carbon emissions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 97% retention headline is the code-generation average, not the overall result; in-distribution CodeHarmony scores inflate it, while held-out OpenEval and BigCodeBench retention is roughly 83-93%.","rationale":"The paper has real value: Flab-Pruner demonstrates that combining vocabulary, layer, and FFN pruning with a KL-divergence objective and LoRA-based recovery is a viable compression route for Code LLMs, and the BigCodeBench and quantization results provide useful independent evidence. However, the central advertised quantity is not robust. The reader's conditional verdict already notes that the 97% figure rests on the code-generation task average and lacks error bars; the analysis above makes this concrete by showing that the headline number is a favorable subset that is further inflated by in-distribution CodeHarmony evaluation. Because the Abstract, Introduction, and Conclusion state the 97% retention claim without the task/benchmark scope, the paper as written overstates its central contribution. The correct fix is not to discard the method but to report retention separately for each task group and for held-out benchmarks, disclose the overlap between CodeHarmony-based selection and CodeHarmony-based evaluation, and then frame the green-software claims accordingly. With those revisions, the paper's contribution remains meaningful, so the appropriate verdict is conditional rather than reject.","tokens_in":25428,"tokens_out":5683,"duration_ms":51506,"concrete_test":"Recompute the no-post-training retention ratios from Table 2 in three ways: (i) across all nine task columns per model; (ii) per task group (code generation, CoT generation, output prediction); and (iii) for held-out benchmarks only, excluding CodeHarmony and including HumanEval, OpenEval, Crux-O, and BigCodeBench from Section 6.1. Report the resulting percentages alongside the current 97% claim. If the held-out-only aggregate falls below roughly 93%, or if per-task-group retention varies from about 90% to 97%, the headline claim should be revised to state the task and benchmark scope explicitly.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that 'by pruning about 22% of the parameters, the pruned model retains around 97% of the original model's performance' is stated without qualification in the Abstract, Introduction, and Conclusion. The paper's own Table 2 does not support this as an aggregate claim. Recomputing the no-post-training columns: for code generation the retention is about 98% on CodeQwen (58.90/60.12), 96% on NxCode (58.13/60.37), and 96% on CodeSlerp (58.34/60.71), hence roughly the advertised 97%. For CoT generation, retention is about 89%, 91%, and 92% on the three models; for output prediction it is about 89%, 90%, and 90%. Within code generation, the held-out OpenEval benchmark shows retention of only 89.3%, 86.5%, and 83.1% respectively, while the authors' CodeHarmony benchmark, used both for pruning selection and post-training, shows no-post-training retention above 103% on all three models. BigCodeBench in Section 6.1 reports 91-93% retention without post-training. Thus the 97% claim is a favorable subset of the measurements, and the apparent strength of the method depends on the overlap between the pruning/post-training corpus and the evaluation benchmark. This directly weakens the headline contribution.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Flab-Pruner, a unified structural pruning method for generative Code LLMs that combines vocabulary pruning, layer pruning, and FFN pruning, all guided by a KL-divergence objective intended to preserve the original model's output token distribution. A code-specific post-training strategy replaces training outputs with original-model-generated code that passes tests, then fine-tunes the pruned model with LoRA. The method is evaluated on three 7B-scale Code LLMs (CodeQwen-1.5, NxCode, CodeSlerp) across code generation (HumanEval, OpenEval, CodeHarmony), chain-of-thought generation, and output prediction (Crux-O), with additional efficiency, robustness, quantization-compatibility, and ablation analyses. The paper reports that pruning about 22% of parameters retains around 97% of original performance, with comparable or better performance after post-training.","tokens_in":25744,"tokens_out":7679,"duration_ms":62941,"significance":"If substantiated, Flab-Pruner is a practically useful contribution to green AI for code: it is one of the few unified structural pruning approaches for generative Code LLMs, ships public pruned models and the CodeHarmony dataset, and demonstrates orthogonal combination with quantization. The efficiency and robustness analyses are welcome additions to the literature. However, the headline 97% retention claim is not supported by the full results when all three tasks are considered, and several technical details of the pruning objective and the evaluation protocol need clarification before the claims can be accepted.","major_comments":[{"comment":"The claim that Flab-Pruner 'retains 97% of the original performance' after pruning 22% of parameters is supported only by the code-generation Pass@1 average in Table 2. Recomputing from Table 2, for CodeQwen the no-post-training retention for CoT generation is 29.37/32.97 = 89.1%, and for output prediction it is 51.17/57.28 = 89.3%; analogous values hold for NxCode and CodeSlerp. The 97% figure is an artifact of averaging HumanEval, OpenEval, and CodeHarmony for the code-generation task only, where the in-distribution CodeHarmony scores actually increase after pruning. The abstract, introduction, and conclusion should either qualify the claim as 'retains 97% of code-generation performance' or report a task-averaged retention (approximately 92-93%), and the conclusion should not present the 97% figure as the overall result.","section":"Abstract, Section 5.1 Summary of RQ1, Conclusion"},{"comment":"The KL-divergence pruning objective in Eq. (6) is defined as a sum over the full vocabulary V. After vocabulary pruning (Section 3.2) removes a large fraction of tokens, the pruned model assigns zero probability to those removed tokens, so D_KL(P_original || P_pruned) would be infinite unless the objective is restricted to the surviving vocabulary and the original distribution is renormalized accordingly. The paper never specifies how the KL divergence is computed for a pruned model, yet Algorithm 2 (layer pruning) explicitly relies on kl_divergence(P_pruned, P_original). Please state the exact computation, including any renormalization over V' or approximation used, and explain whether the pruning objective is applied before or after vocabulary pruning.","section":"Section 3.1, Eq. (6); Section 3.2"},{"comment":"CodeHarmony is used for vocabulary token collection (Section 3.2), for the KL-based layer/FFN pruning selection (Algorithm 2), for post-training (Section 3.5), and also as one of the evaluation benchmarks (Table 2). This circularity inflates the reported retention: the paper's own numbers show that without post-training, CodeHarmony Pass@1 exceeds the dense model (e.g., 64.05 vs. 60.78 for CodeQwen) while the held-out OpenEval code-generation retention is only 89.3%, 86.5%, and 83.1% on the three models. Section 6.2 further shows that adding FFN pruning to vocabulary+layer pruning degrades OpenEval from 42.13 to 36.52 while improving CodeHarmony from 64.92 to 66.63, which is a sign of overfitting to the pruning/training corpus. The authors should report held-out retention as the primary evidence for generalization and state clearly how much of the claimed retention depends on the CodeHarmony distribution.","section":"Section 4.3, Section 5.1 Table 2, Section 6.2 Table 9"},{"comment":"All reported results are from single runs; no random seeds or variance estimates are provided for the LoRA post-training or for the stochastic FFN heuristics (e.g., Random Sampling in Section 3.4). Since the claim of 'same or even better performance after post-training' rests on small differences (e.g., code-generation average 61.97 vs. 60.12 for CodeQwen), the absence of variability analysis makes it impossible to assess whether these differences are meaningful. Please provide at least 3-5 independent post-training runs with standard deviations, or justify why the results are deterministic and the LoRA optimization introduces no variance.","section":"Section 4.5, Table 2, Algorithm 3"}],"minor_comments":[{"comment":"The vocabulary pruning threshold tau in Eq. (7) is never specified; the paper only reports the resulting vocabulary size (17,176 from 92,416). Please state how tau was chosen (e.g., a frequency cutoff or coverage percentage) and how sensitive the results are to this hyperparameter.","section":"Section 4.5"},{"comment":"The CodeHarmony dataset's construction is described at a high level (regular-expression extraction, test-case generation with GPT-4o/Gemini), but the exact filtering criteria, the number of functions before/after filtering, and the prompt templates used to generate test cases and CoT are not given. Please provide these details or a clear pointer to the released dataset's documentation.","section":"Section 4.3"},{"comment":"The break-even calculation in Eq. (8) is unclear: it assumes a 'frequency of calculation' of 1 time/second and multiplies 6 hours by 3600 by 7.04T FLOPs, but the 7.04T value is the per-inference FLOPs, not a throughput in FLOPs per second. Please clarify the throughput assumption and re-derive the break-even estimate with proper units.","section":"Section 5.2, Eq. (8)"},{"comment":"The robustness analysis text says that after post-training the pruned models perform 'even better than the dense model under certain perturbations,' but does not mention the cases where post-trained pruned models are clearly worse (e.g., EvoEval 'Combine': 18 vs. 27 for CodeQwen, 19 vs. 25 for NxCode). Please add a sentence acknowledging these decreases to avoid overstating robustness.","section":"Section 5.3, Table 7"},{"comment":"There are numerous typos and formatting issues: 'generation' for 'generation', 'evaluating the HumanEval' (Section 5.2), 'producesdemand' (Section 1), missing spaces around 'Flab-Pruner' in several places, and a garbled character in Eq. (3) for the vocabulary size. Please run a careful proofreading pass.","section":"Throughout"},{"comment":"Several references are incomplete or inconsistent (e.g., [67] is listed as 'arXiv preprint' without a year or venue in one entry, and some entries lack page numbers). Please unify the reference style and verify that all URLs and bibliographic details are complete.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The central pruning methodology is plausible and the experimental scope is substantial, but the headline claim (97% retention) is overstated and the evaluation protocol risks circularity because CodeHarmony serves as the pruning corpus, the post-training corpus, and one of the evaluation benchmarks. The KL objective is also underspecified after vocabulary pruning. These are fixable with rewording, additional clarification, and supplementary experiments on held-out benchmarks, so I do not recommend rejection. The comparison against baselines is also not entirely fair, as Flab-Pruner combines three pruning components while the baselines are single-component layer-pruning methods; the 0% scores for UIDL and Linearity on most benchmarks deserve scrutiny to rule out implementation issues. The public release of models and data is a strong point in favor of publication after the concerns are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth a serious look. Flab-Pruner combines vocabulary, layer, and FFN pruning into one pipeline with a KL-divergence objective aimed at generative code models, plus a data-replacement post-training trick and a new CodeHarmony dataset. That combination is genuinely new, and the evaluation is broad: three 7B-scale models, code generation, CoT, output prediction, efficiency, robustness, and an ablation study. The efficiency analysis with quantized variants is a nice practical touch, and the cost amortization estimate is a useful framing for deployment decisions.\n\nThe main caveat is the headline. The claim that the pruned model retains 97% of original performance is true only for the code-generation Pass@1 average in Table 2. On chain-of-thought and output prediction, retention is around 89-92%. On OpenEval, a held-out code-generation benchmark, it drops to 83-89%. The CodeHarmony benchmark, which is used both for pruning selection and post-training, shows retention above 100% without post-training, so the headline average is inflated by in-domain overlap. The stress-test note is correct on this point, and the paper itself acknowledges the effect implicitly in Section 6.2, where all pruning strategies improve CodeHarmony scores while hurting HumanEval/OpenEval.\n\nOther soft spots are real but smaller. All results are single runs with no error bars, and with HumanEval's 164 samples, a few correct or incorrect cases move Pass@1 by a point or two. The pruning pipeline code is not released, only models and datasets, which makes replication slower. Equation 6 sums the KL over the original vocabulary even after vocabulary pruning; the authors should clarify whether the sum is over the pruned vocabulary or the pruned model's renormalized distribution. None of these sink the central conclusion, which is that 22% structural pruning with a KL objective plus LoRA recovery preserves most code-generation capability.\n\nBottom line: this is an incremental but practical contribution for anyone working on compressing code LLMs or on green software engineering. It deserves a serious referee, but the authors should be asked to qualify the 97% claim, add multiple runs or error bars, and release the pruning code before publication.","headline":"Useful pruning pipeline for code LLMs, but the 97% retention headline is a favorable subset; still deserves a serious referee.","tokens_in":651,"tokens_out":1208,"would_cite":true,"duration_ms":27381,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A three-part structural prune can shrink a 7B code LLM by 22% while keeping 97% of its performance.","keywords":["Large Language Models","Code Intelligence","Structural Pruning","Vocabulary Pruning","Layer Pruning","FFN Pruning","Post-Training","Green Software Engineering"],"falsifier":"Run Flab-Pruner exactly as described, then evaluate the pruned, unrecovered model on a Python benchmark built from libraries and identifiers that do not occur in CodeHarmony; if Pass@1 retention falls substantially below the reported ~97% average (e.g., below 90%), the corpus-representativeness assumption is violated. A simpler check is to compare the pruned tokenizer's vocabulary coverage on a held-out corpus of modern Python code versus CodeHarmony.","tokens_in":25240,"feed_emoji":"✂️","tokens_out":6239,"duration_ms":46258,"temperature":0.7,"pith_summary":"The paper argues that generative code LLMs can be made substantially smaller without losing their abilities, by pruning vocabulary, layers, and feed-forward neurons together rather than pruning any single component. It introduces Flab-Pruner, which removes roughly 22% of a model's parameters while retaining about 97% of the original performance, and then a code-specific post-training step that typically restores performance to the level of the original dense model. The motivation is practical and environmental: smaller models use less GPU memory, fewer FLOPs, less energy, and produce less CO2, making them easier to deploy in resource-constrained settings. The paper evaluates this on three 7B Code LLMs and three generative coding tasks, including code generation, chain-of-thought reasoning, and output prediction.","feed_headline":"Pruning 22% of a code LLM keeps 97% of its power","feed_subtitle":"A unified vocabulary-layer-FFN prune plus code-specific retraining cuts GPU cost and CO2 without losing capability.","key_machinery":"The load-bearing object is the KL divergence between the original model's output distribution and the pruned model's output distribution, defined as $\\mathcal{L}_{\\mathrm{prune}} = \\min_{\\theta_p} D_{\\mathrm{KL}}(P(y|X;\\theta_o) \\| P(y|X;\\theta_p))$. This single objective ties together three complementary pruning operations: vocabulary pruning keeps only tokens that appear in the CodeHarmony corpus, layer pruning iteratively removes the layer whose deletion least increases KL divergence, and FFN pruning applies one of four heuristic masks (top-K, bottom-K, middle-K, random) to keep the strongest neurons. The third component is a performance recovery step that replaces training outputs with the original model's own code when that code passes test cases, so the pruned model is trained on high-quality, semantically correct examples.","core_discovery":"On the paper's own terms, the central discovery is that a unified, KL-divergence-driven structural pruning pipeline can compress Code LLMs by 22% of parameters while keeping about 97% of the original model's performance, and that after a code-specific instruction-tuning recovery step the pruned model matches or slightly exceeds the dense model on code generation, CoT generation, and output prediction. The method treats the model's token-level output distribution as the pruning objective, removing vocabulary entries absent from a curated code corpus, iteratively deleting the least harmful transformer layer by measuring KL divergence on correctly generated samples, and trimming FFN neurons through lightweight heuristic masks. The authors report that single-component baselines collapse to zero performance on these tasks, while the combination of all three components degrades gracefully and post-training recovers the gap.","pith_inferences":["A testable extension the paper does not explore is measuring how much of the 97% retention depends on the Python-centric CodeHarmony corpus by running the same pipeline on multilingual code benchmarks.","The close parity between the heuristic FFN masks and the Fisher Information Matrix baseline suggests that simple positional heuristics may capture most of the neuron-importance signal for code generation, which could simplify future pruning pipelines even further if confirmed on other models.","The reported robustness results under ReCode and EvoEval perturbations suggest that the post-training step, not the pruning itself, is what re-establishes performance under distributional shifts; a direct comparison of pruned-without-PT versus pruned-with-PT on each perturbation type would test this."],"forward_implications":["Pruned models at BF16 need about 10.72 GB of GPU memory instead of 13.55 GB, letting them run on 12 GB cards that the dense model cannot fit on.","At INT4 precision the pruned model fits in roughly 3 GB, enabling deployment on edge GPU hardware while retaining most benchmark performance.","The one-time pruning and recovery cost of about 6 hours is amortized after roughly 108,000 inference runs, after which every call saves about 20% of FLOPs.","Structured pruning is orthogonal to quantization: combining Flab-Pruner with FP8 or INT4 gives compounding efficiency gains without compounding performance loss.","On the harder BigCodeBench suite, pruned models keep about 91-93% of dense performance before post-training and match or slightly exceed it after post-training."],"supporting_citations":[{"why":"Provides the ShortGPT layer-pruning baseline that the authors show collapses on HumanEval, motivating the unified KL objective.","marker":"[45]"},{"why":"Supplies the LLM-Pruner baseline and the Fisher Information Matrix neuron importance approach that Flab-Pruner's heuristics are compared against.","marker":"[43]"},{"why":"Defines HumanEval, the primary code generation benchmark and the test case for the Pass@1 metric.","marker":"[4]"},{"why":"Introduces the CRUX output prediction benchmark used to assess code understanding.","marker":"[18]"},{"why":"Contributes the Evol dataset, one of the two main sources for constructing the CodeHarmony corpus.","marker":"[42]"},{"why":"Provides the OSS dataset used alongside Evol to build CodeHarmony.","marker":"[65]"},{"why":"Supplies the HumanEval-CoT and OpenEval-CoT tasks and the chain-of-thought construction approach adopted for CodeHarmony.","marker":"[71]"},{"why":"Defines the ReCode perturbation suite used to test the pruned models' robustness.","marker":"[61]"},{"why":"Provides the EvoEval benchmark with semantic perturbations used for additional robustness evaluation.","marker":"[67]"},{"why":"Introduces BigCodeBench, the harder tool-composition benchmark that corroborates the retention results.","marker":"[82]"}],"fun_headline_variants":["Cut 22% of code LLM, keep 97% performance","Unified pruning trims code LLM by 22%, retains 97%","Green code LLM: prune 22%, keep 97% ability","Flab-Pruner: 22% smaller code LLM, 97% performance","Prune 22% of code LLM, preserve 97% capability"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole result depends on the CodeHarmony corpus being representative of deployment code: tokens absent from it are deleted permanently, and KL divergence measured on its samples is assumed to transfer to held-out tasks such as HumanEval and OpenEval.","fun_headline_variants_meta":{"raw":{"variants":["Cut 22% of code LLM, keep 97% performance","Unified pruning trims code LLM by 22%, retains 97%","Green code LLM: prune 22%, keep 97% ability","Flab-Pruner: 22% smaller code LLM, 97% performance","Prune 22% of code LLM, preserve 97% capability"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000647,"raw_usage":{"total_tokens":2972,"prompt_tokens":944,"completion_tokens":2028,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":1925}},"tokens_in":560,"tokens_out":2028,"duration_ms":12747,"temperature":1.0,"reasoning_tokens":1925,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:57:32.554738+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Flab-Pruner exactly as described, then evaluate the pruned, unrecovered model on a Python benchmark built from libraries and identifiers that do not occur in CodeHarmony; if Pass@1 retention falls substantially below the reported ~97% average (e.g., below 90%), the corpus-representativeness assumption is violated. A simpler check is to compare the pruned tokenizer's vocabulary coverage on a held-out corpus of modern Python code versus CodeHarmony.","supporting_citations":[{"cited_title":"Llm-pruner:Onthestructuralpruningoflargelanguagemodels","cited_arxiv_id":null,"evidence_quote":"Supplies the LLM-Pruner baseline and the Fisher Information Matrix neuron importance approach that Flab-Pruner's heuristics are compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the CRUX output prediction benchmark used to assess code understanding."},{"cited_title":"Wizardcoder:Empoweringcodelargelanguage models with evol-instruct, in: The Twelfth International Conference on Learning Representations","cited_arxiv_id":null,"evidence_quote":"Contributes the Evol dataset, one of the two main sources for constructing the CodeHarmony corpus."},{"cited_title":"Magicoder: Empowering code generation with oss-instruct, in: Forty-first International Conference on Machine Learning","cited_arxiv_id":null,"evidence_quote":"Provides the OSS dataset used alongside Evol to build CodeHarmony."},{"cited_title":"Chain-of-thought in neural code generation: From and for lightweight language models","cited_arxiv_id":null,"evidence_quote":"Supplies the HumanEval-CoT and OpenEval-CoT tasks and the chain-of-thought construction approach adopted for CodeHarmony."},{"cited_title":"Recode:Robustnessevaluation of code generation models, in: The 61st Annual Meeting Of The Association For Computational Linguistics","cited_arxiv_id":null,"evidence_quote":"Defines the ReCode perturbation suite used to test the pruned models' robustness."},{"cited_title":"Top leaderboard ranking = top coding proficiency, always? evoeval: Evolving coding benchmarks via llm","cited_arxiv_id":null,"evidence_quote":"Provides the EvoEval benchmark with semantic perturbations used for additional robustness evaluation."}],"review_version":1}