{"id":"31704bf8-7a93-48c2-90b6-33091ea6ee48","arxiv_id":"2412.13737","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"On CodeBERT, quantization best preserves effectiveness while cutting size, distillation best improves latency, and pruning only pays off in specific CPU configurations.","lead":"Three compression methods, knowledge distillation, quantization, and pruning, were tested on CodeBERT for vulnerability detection, code summarization, and code search. The best method depends on the task and the hardware, with quantization shrinking models while keeping accuracy and distillation speeding up inference but sometimes hurting quality.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The knowledge-distillation arm substitutes DistilBERT, a different pre-trained model, for a student actually distilled from CodeBERT, so the paper's KD-specific effectiveness conclusions do not follow from the measurements.","rationale":"The reader's weakest_assumption correctly identifies the KD confound, and this stress-test agrees that it is the most load-bearing concern. The paper's headline finding that compression impact varies by task and method remains credible because the pruning and quantization arms directly transform the fine-tuned CodeBERT models and report large, internally consistent differences. However, the KD arm is the only arm that does not start from the fine-tuned CodeBERT checkpoint, so any KD-specific conclusion about effectiveness loss in generation and search, or about KD being the only strategy improving both time and size, is not established by the data as presented. The concrete test of actually distilling from CodeBERT would settle whether the observed losses are due to knowledge distillation or to the choice of DistilBERT as an unrelated baseline. Because this is an addressable methodological gap rather than a demonstration that the overall empirical comparison is wrong, the reader's conditional verdict remains appropriate; no change to the verdict is needed, though the revision should explicitly re-scope or re-run the KD arm.","tokens_in":18827,"tokens_out":2299,"duration_ms":23922,"concrete_test":"Re-run the KD arm using a student actually distilled from the fine-tuned CodeBERT teacher, for example by fine-tuning a student initialized from DistilBERT with the teacher's logits (or representations) on the same training splits for vulnerability detection, code summarization, and code search. Compare the resulting effectiveness deltas to the current 'Know. Distil.' rows in Table III; if the deltas differ materially (e.g., by more than 10 percentage points for BLEU, SIDE, or MRR), then the current KD-specific conclusions should be re-labeled as comparisons to a different pretrained model and revised accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-B1 states that the authors 'opted not to retrain a distilled model ourselves' and instead fine-tuned DistilBERT for each task. This makes the 'Know. Distil.' rows in Table III comparisons between CodeBERT and an unrelated pretrained model, not measurements of knowledge distillation applied to CodeBERT. DistilBERT differs from CodeBERT in pretraining corpus (natural language versus code), architecture depth, and training objective; it is a distilled version of BERT, not of CodeBERT. Consequently, the large effectiveness losses attributed to knowledge distillation in code summarization (BLEU -42.3%, SIDE -70.6%) and code search (MRR -52.4%) may simply reflect the gap between a code-specialized teacher and a general-domain small model, rather than the effect of the compression strategy. The Section VI external-validity caveat that results 'are specific to the DistilBERT LM' is not sufficient: the issue is not only limited generality, but that the arm does not operationalize knowledge distillation on the target model at all. The central claim about task- and method-dependent impact still has support from the pruning and quantization arms, but the KD-specific conclusions in Section V-A1 and the practitioner guidance built on them are load-bearing and are not supported by the experimental design.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper empirically evaluates three compression strategies—knowledge distillation, quantization, and pruning—applied to CodeBERT fine-tuned for vulnerability detection, code summarization, and code search. It measures inference time on CPU and GPU with bootstrap confidence intervals, model size, and task-specific effectiveness metrics (Accuracy/F1/MCC, BLEU/BERTScore/SIDE, MRR variants). The results show that quantization preserves effectiveness with large size reductions, pruning is generally ineffective or harmful, and knowledge distillation reduces both time and size but with substantial effectiveness losses for generation and search. The authors provide practitioner guidance and a replication package.","tokens_in":19005,"tokens_out":6444,"duration_ms":53117,"significance":"If the results are taken at face value, the study provides a useful map of compression trade-offs for a widely used code model, with rigorous timing methodology and external datasets. The strongest contribution is the comparative, task-heterogeneous design: no universally best strategy exists, and the paper quantifies hardware-dependent behavior. The pruning and quantization arms are clean and reproducible. However, the knowledge-distillation arm does not measure distillation on CodeBERT, which limits the significance of the KD-specific findings.","major_comments":[{"comment":"The knowledge distillation arm replaces CodeBERT with DistilBERT, a general-domain distilled BERT model, rather than a student distilled from CodeBERT. This operationalizes a different research question: fine-tuning a smaller pretrained model vs. compressing CodeBERT via distillation. As a result, the KD rows in Table III and the KD-specific conclusions in Sections IV-A1, IV-B3, IV-C3, and V-A1 are not supported as measurements of knowledge distillation applied to CodeBERT. The large effectiveness losses attributed to KD may simply reflect the gap between a code-specialized teacher and a general-domain student. The Section VI caveat that results are specific to DistilBERT does not fix the internal validity problem; the arm should be either re-run with a student distilled from CodeBERT (e.g., using the teacher's logits) or explicitly relabeled as a cross-architecture comparison, with KD conclusions removed from the abstract and practitioner guidance.","section":"Section III-B1, Table III"},{"comment":"Effectiveness metrics are reported as single point estimates without confidence intervals or significance tests, yet the paper states that int8 and float8 quantization provide \"no significant change\" and uses fine-grained differences (e.g., -0.2% vs -6.3% MRR) to rank strategies. Given the paper's otherwise careful uncertainty quantification for inference time, the absence of variance for effectiveness undermines the strength of claims about small differences. The authors should either add bootstrap CIs or equivalent for effectiveness metrics, or temper language such as \"no significant change\" and the fine-grained ordering of quantization configurations.","section":"Section III-D, Section IV-C3"}],"minor_comments":[{"comment":"The text reads \"From Table IIIc\" but should be \"From Table IIIb\" because it is discussing code summarization results.","section":"Section IV-B3"},{"comment":"The text reports KD's MRR loss as -52.1%, but Table IIIc lists -52.4% for MRR; the value -52.1% corresponds to MRR@5.","section":"Section IV-C3"},{"comment":"The BERT architecture is cited as reference [17], which is the OpenAI Codex paper; add a proper citation to Devlin et al. (BERT) and renumber the references accordingly.","section":"Section II-A"},{"comment":"The external-validity caveat \"results concerning Knowledge Distillation are specific to the DistilBERT LM\" should be strengthened to explicitly acknowledge that the arm does not implement KD on CodeBERT; see major comment 1.","section":"Section VI"},{"comment":"The description of the CPU timing measurement is vague (\"we use the time Python function\"); specify which function from the time module (e.g., time.perf_counter) and whether the measurement covers the full batch processing loop.","section":"Section III-C1"},{"comment":"The claim that knowledge distillation improves all efficiency metrics for code summarization overlooks that the GPU inference time reduction (-2.2%) is not statistically significant, as marked by the asterisk in Table IIIb.","section":"Answer to RQ 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits the journal's scope and the replication package is a strength. The pruning and quantization arms are solid, but the KD arm requires substantial rework: it currently does not measure knowledge distillation applied to CodeBERT. If a proper distillation-on-CodeBERT experiment cannot be added in a reasonable revision, the KD-specific effectiveness claims should be removed from the abstract and guidance, and the DistilBERT comparison should be repositioned as a secondary cross-architecture analysis. Adding uncertainty quantification for effectiveness metrics would also strengthen the paper's claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, it's a solid, carefully measured empirical comparison of pruning and quantization on CodeBERT across three SE tasks. Second, its knowledge-distillation arm is not a test of knowledge distillation at all—it fine-tunes DistilBERT, a different pre-trained model, and compares it to CodeBERT. The KD-specific conclusions in the paper do not follow from the experiments.\n\nWhat's new is the unified comparison: three compression families, three task classes (classification, generation, search), on a single architecture, with a public replication package. The timing methodology is a real strength—bootstrap confidence intervals following Kalibera and Jones, CPU and GPU environments, warm-ups, synchronization. The use of multiple effectiveness metrics (MCC, BLEU/BERTScore/SIDE, MRR variants) is thoughtful. The pruning and quantization arms are clean and produce interesting, non-obvious findings: quantization often slows inference dramatically on GPU, and moderate pruning can help or hurt depending on task and hardware. That alone supports the paper's central claim that compression impact is task- and method-dependent.\n\nThe soft spot is the KD arm. Section III-B1 says they \"opted not to retrain a distilled model ourselves\" and used DistilBERT. That means the KD rows in Table III compare CodeBERT to a general-domain, shallower BERT, not to a student distilled from CodeBERT. The large effectiveness losses for summarization and search (BLEU -42%, MRR -52%) likely reflect the gap between a code-specialized model and a general text model, not the effect of distillation. The external-validity caveat in Section VI doesn't fix this—it's not just limited generality, it's a different operationalization of the strategy. Re-labeling the rows as \"DistilBERT\" or re-running with a true CodeBERT-distilled student would address it. A second, minor issue: effectiveness metrics are reported as point estimates with no variance or significance assessment, so small differences (e.g., quantization int8 vs float8) may be noise. The significance rule for timing CIs is also ambiguously worded, though the CIs themselves are transparent. Neither of these undermines the pruning/quantization results.\n\nWho is this for? Practitioners choosing a compression strategy for CodeBERT-like models, and researchers wanting a baseline comparison. The central comparative claim about task-dependence is supported once you set the KD arm aside. It deserves a serious referee, but the revision must address the KD arm and add variance estimates for effectiveness. I'd send it to peer review with a request for major revision on the KD arm. If the authors fix that, the paper becomes a genuinely useful reference.","headline":"A solid, carefully measured comparison of pruning and quantization on CodeBERT, but the knowledge-distillation arm compares DistilBERT to CodeBERT and doesn't actually test distillation.","tokens_in":19547,"tokens_out":2253,"would_cite":true,"duration_ms":19518,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Compression strategies for CodeBERT have no one-size-fits-all winner; impact depends on task, hardware, and configuration.","keywords":["model compression","knowledge distillation","quantization","pruning","CodeBERT","vulnerability detection","code summarization","code search"],"falsifier":"Measure the MRR of a code-search model produced by actual knowledge distillation from the fine-tuned CodeBERT teacher; if its loss is much smaller than the reported -52.4% for DistilBERT, then the paper's distillation conclusions for search do not generalize.","tokens_in":18591,"feed_emoji":"🗜️","tokens_out":4148,"duration_ms":37922,"temperature":0.7,"pith_summary":"This paper asks whether the three standard ways of shrinking a language model—knowledge distillation, quantization, and pruning—behave the same way across software-engineering tasks. To find out, the authors fine-tune CodeBERT for vulnerability detection, code summarization, and code search, then compress each model and measure inference time, model size, and effectiveness. They find that no strategy is uniformly best: quantization preserves accuracy and cuts memory but often slows inference, knowledge distillation speeds up both CPU and GPU inference and shrinks the model but can cut effectiveness sharply on generation and search, and pruning only helps in specific CPU configurations. The practical payoff is a decision rule: pick the strategy by your bottleneck—size, latency, or correctness—and by the task and environment.","feed_headline":"No universal best compression strategy for CodeBERT tasks","feed_subtitle":"Study of three SE tasks: pick quantization for size, distillation for speed, pruning only with care.","key_machinery":"The experimental matrix that carries the argument: three fine-tuned CodeBERT task models (Devign for vulnerability detection, CodeSearchNet Java for summarization, CodeSearchNet Python for search), each compressed by knowledge distillation (via fine-tuned DistilBERT), post-training quantization (int4/int8/float8), and unstructured L1 global pruning (20/40/60%), then measured for CPU/GPU inference time with bootstrap confidence intervals, model size, and task-specific effectiveness metrics (MCC/F1/Accuracy, BLEU/BERTScore/SIDE, MRR/MRR@1/MRR@5). The comparisons are percentage changes relative to uncompressed CodeBERT, which makes the trade-offs directly readable.","core_discovery":"The central discovery is that compression outcomes for CodeBERT are task- and method-dependent, with no dominant strategy. On vulnerability detection, knowledge distillation is the only strategy that improves both inference time and model size (up to -47.7% GPU latency and -48.8% size) at a moderate MCC cost, while quantization cuts size up to -59.3% with almost no effectiveness loss but slows inference by up to +201.6% on GPU. On code summarization, quantization is the best overall trade-off (size -51.9%, CPU latency -27.2%, SIDE loss at most -0.2%), whereas knowledge distillation loses 70.6% SIDE. On code search, quantization again preserves effectiveness (MRR loss up to -6.3%) but slows inference, knowledge distillation gives the largest speedups (-84.7% CPU, -29.2% GPU) at a -52.4% MRR cost, and pruning never helps efficiency and can destroy effectiveness (MRR -99.6% at 60% pruning).","pith_inferences":["Our inference: the task-dependence may track the number of sequential forward passes—code search and summarization, which need multiple comparisons or decoding steps, show pruning and quantization slowdowns, suggesting latency effects are driven by hardware sparsity and low-precision kernel support rather than model capacity alone.","Our inference: an automatic strategy selector could be built on a small benchmark matrix like this one, choosing quantization for size-constrained deployments and distillation for latency-constrained ones, then validating the choice on the target task.","Our inference: the energy and carbon consequences of each strategy are not measured here, but if inference-time changes transfer to energy use, distillation's CPU speedups would make it the green choice for classification tasks, while quantization would be greener for generation tasks on CPU.","Our inference: the reported knowledge-distillation results likely understate what a true CodeBERT-distilled student would achieve, because DistilBERT is a different architecture; the effectiveness losses for generation and search may be a property of the student architecture, not of distillation itself."],"forward_implications":["Practitioners prioritizing model size with minimal effectiveness loss should choose quantization (int8/float8) across all three tasks.","Practitioners needing both CPU and GPU inference speedups should choose knowledge distillation, accepting that effectiveness may drop sharply outside classification.","Pruning is only advisable when a specific configuration and hardware combination has been tested; 60% pruning helps CPU classification but hurts summarization and search.","No tested strategy improves code-search inference time without a significant effectiveness cost; knowledge distillation is fastest but loses about 52% of MRR.","Choosing a compression strategy requires knowing the task and the deployment environment, not just the model family."],"supporting_citations":[{"why":"Supplies CodeBERT, the reference language model that is fine-tuned and then compressed for all three tasks.","marker":"[12]"},{"why":"Supplies the Devign dataset used for the vulnerability detection task.","marker":"[38]"},{"why":"Supplies the CodeSearchNet dataset used for both code summarization (Java) and code search (Python).","marker":"[39]"},{"why":"Supplies the CodeXGLUE pipeline, hyperparameters, and evaluation splits that the experiments reuse and extend.","marker":"[40]"},{"why":"Supplies DistilBERT, the model used to represent the knowledge-distillation strategy.","marker":"[41]"},{"why":"Supplies the pruning recipe (L1 norm on linear layers) and the expectation that effectiveness degrades once pruned weight share reaches about 40%.","marker":"[42]"},{"why":"Supplies the bootstrap confidence-interval method used to quantify inference-time changes with uncertainty.","marker":"[48]"},{"why":"Provides the prior knowledge-distillation approach for code models and the efficiency claims that this study extends to multiple tasks.","marker":"[9]"},{"why":"Provides the prior empirical study of quantization for code generation, which the authors use to explain and contextualize their quantization findings.","marker":"[11]"}],"fun_headline_variants":["No one-size-fits-all compression for CodeBERT","Quantize for size, distill for speed, prune with care","Pruning can cut CodeBERT search accuracy by 99%","Compression trade-offs in CodeBERT vary by task"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The knowledge-distillation results assume that fine-tuning DistilBERT on each task measures what distilling CodeBERT would achieve, because the paper did not retrain a distilled version of CodeBERT itself.","fun_headline_variants_meta":{"raw":{"variants":["No one-size-fits-all compression for CodeBERT","Quantize for size, distill for speed, prune with care","Pruning can cut CodeBERT search accuracy by 99%","Compression trade-offs in CodeBERT vary by task"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001957,"raw_usage":{"total_tokens":7636,"prompt_tokens":915,"completion_tokens":6721,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":6654}},"tokens_in":531,"tokens_out":6721,"duration_ms":44561,"temperature":1.0,"reasoning_tokens":6654,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:50:45.014954+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the MRR of a code-search model produced by actual knowledge distillation from the fine-tuned CodeBERT teacher; if its loss is much smaller than the reported -52.4% for DistilBERT, then the paper's distillation conclusions for search do not generalize.","supporting_citations":[{"cited_title":"CodeBERT: A pre-trained model for programming and natural languages,","cited_arxiv_id":null,"evidence_quote":"Supplies CodeBERT, the reference language model that is fine-tuned and then compressed for all three tasks."},{"cited_title":"Devign: Effective vulnerability identi- fication by learning comprehensive program semantics via graph neural networks,","cited_arxiv_id":null,"evidence_quote":"Supplies the Devign dataset used for the vulnerability detection task."},{"cited_title":"Rigorous benchmarking in reasonable time,","cited_arxiv_id":null,"evidence_quote":"Supplies the bootstrap confidence-interval method used to quantify inference-time changes with uncertainty."}],"review_version":1}