Pith. sign in

REVIEW 2 major objections 6 minor 65 references

On the Compression of Language Models for Code: An Empirical Study on CodeBERT

T0 review · 2 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Compression strategies for CodeBERT have no one-size-fits-all winner; impact depends on task, hardware, and configuration.

desk verdict A solid, carefully measured comparison of pruning and quantization on CodeBERT, but the knowledge-distillation arm compares DistilBERT to CodeBERT and doesn't actually test distillation. read the letter →

arxiv 2412.13737 v1 pith:VKK44NAS submitted 2024-12-18 cs.SE cs.AIcs.PF

Giordano d'Aloisio , Luca Traini , Federica Sarro , Antinisca Di Marco This is my paper · ORCID
classification cs.SEcs.AIcs.PF
keywords modelcompressionknowledgedistillationquantizationpruningCodeBERTvulnerabilitydetectionsummarizationsearch
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether the three standard ways of shrinking a language model—knowledge distillation, quantization, and pruning—behave the same way across software-engineering tasks. To find out, the authors fine-tune CodeBERT for vulnerability detection, code summarization, and code search, then compress each model and measure inference time, model size, and effectiveness. They find that no strategy is uniformly best: quantization preserves accuracy and cuts memory but often slows inference, knowledge distillation speeds up both CPU and GPU inference and shrinks the model but can cut effectiveness sharply on generation and search, and pruning only helps in specific CPU configurations. The practical payoff is a decision rule: pick the strategy by your bottleneck—size, latency, or correctness—and by the task and environment.

What carries the argument

The experimental matrix that carries the argument: three fine-tuned CodeBERT task models (Devign for vulnerability detection, CodeSearchNet Java for summarization, CodeSearchNet Python for search), each compressed by knowledge distillation (via fine-tuned DistilBERT), post-training quantization (int4/int8/float8), and unstructured L1 global pruning (20/40/60%), then measured for CPU/GPU inference time with bootstrap confidence intervals, model size, and task-specific effectiveness metrics (MCC/F1/Accuracy, BLEU/BERTScore/SIDE, MRR/MRR@1/MRR@5). The comparisons are percentage changes relative to uncompressed CodeBERT, which makes the trade-offs directly readable.

What would settle it

Measure the MRR of a code-search model produced by actual knowledge distillation from the fine-tuned CodeBERT teacher; if its loss is much smaller than the reported -52.4% for DistilBERT, then the paper's distillation conclusions for search do not generalize.

Watch

Extended reading notes

Core claim

The central discovery is that compression outcomes for CodeBERT are task- and method-dependent, with no dominant strategy. On vulnerability detection, knowledge distillation is the only strategy that improves both inference time and model size (up to -47.7% GPU latency and -48.8% size) at a moderate MCC cost, while quantization cuts size up to -59.3% with almost no effectiveness loss but slows inference by up to +201.6% on GPU. On code summarization, quantization is the best overall trade-off (size -51.9%, CPU latency -27.2%, SIDE loss at most -0.2%), whereas knowledge distillation loses 70.6% SIDE. On code search, quantization again preserves effectiveness (MRR loss up to -6.3%) but slows inference, knowledge distillation gives the largest speedups (-84.7% CPU, -29.2% GPU) at a -52.4% MRR cost, and pruning never helps efficiency and can destroy effectiveness (MRR -99.6% at 60% pruning).

Load-bearing premise

The knowledge-distillation results assume that fine-tuning DistilBERT on each task measures what distilling CodeBERT would achieve, because the paper did not retrain a distilled version of CodeBERT itself.

Editorial extensions

If this is right

  • Practitioners prioritizing model size with minimal effectiveness loss should choose quantization (int8/float8) across all three tasks.
  • Practitioners needing both CPU and GPU inference speedups should choose knowledge distillation, accepting that effectiveness may drop sharply outside classification.
  • Pruning is only advisable when a specific configuration and hardware combination has been tested; 60% pruning helps CPU classification but hurts summarization and search.
  • No tested strategy improves code-search inference time without a significant effectiveness cost; knowledge distillation is fastest but loses about 52% of MRR.
  • Choosing a compression strategy requires knowing the task and the deployment environment, not just the model family.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Our inference: the task-dependence may track the number of sequential forward passes—code search and summarization, which need multiple comparisons or decoding steps, show pruning and quantization slowdowns, suggesting latency effects are driven by hardware sparsity and low-precision kernel support rather than model capacity alone.
  • Our inference: an automatic strategy selector could be built on a small benchmark matrix like this one, choosing quantization for size-constrained deployments and distillation for latency-constrained ones, then validating the choice on the target task.
  • Our inference: the energy and carbon consequences of each strategy are not measured here, but if inference-time changes transfer to energy use, distillation's CPU speedups would make it the green choice for classification tasks, while quantization would be greener for generation tasks on CPU.
  • Our inference: the reported knowledge-distillation results likely understate what a true CodeBERT-distilled student would achieve, because DistilBERT is a different architecture; the effectiveness losses for generation and search may be a property of the student architecture, not of distillation itself.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper empirically evaluates three compression strategies—knowledge distillation, quantization, and pruning—applied to CodeBERT fine-tuned for vulnerability detection, code summarization, and code search. It measures inference time on CPU and GPU with bootstrap confidence intervals, model size, and task-specific effectiveness metrics (Accuracy/F1/MCC, BLEU/BERTScore/SIDE, MRR variants). The results show that quantization preserves effectiveness with large size reductions, pruning is generally ineffective or harmful, and knowledge distillation reduces both time and size but with substantial effectiveness losses for generation and search. The authors provide practitioner guidance and a replication package.

Significance. If the results are taken at face value, the study provides a useful map of compression trade-offs for a widely used code model, with rigorous timing methodology and external datasets. The strongest contribution is the comparative, task-heterogeneous design: no universally best strategy exists, and the paper quantifies hardware-dependent behavior. The pruning and quantization arms are clean and reproducible. However, the knowledge-distillation arm does not measure distillation on CodeBERT, which limits the significance of the KD-specific findings.

major comments (2)
  1. [Section III-B1, Table III] The knowledge distillation arm replaces CodeBERT with DistilBERT, a general-domain distilled BERT model, rather than a student distilled from CodeBERT. This operationalizes a different research question: fine-tuning a smaller pretrained model vs. compressing CodeBERT via distillation. As a result, the KD rows in Table III and the KD-specific conclusions in Sections IV-A1, IV-B3, IV-C3, and V-A1 are not supported as measurements of knowledge distillation applied to CodeBERT. The large effectiveness losses attributed to KD may simply reflect the gap between a code-specialized teacher and a general-domain student. The Section VI caveat that results are specific to DistilBERT does not fix the internal validity problem; the arm should be either re-run with a student distilled from CodeBERT (e.g., using the teacher's logits) or explicitly relabeled as a cross-architecture comparison, with KD conclusions removed from the abstract and practitioner guidance.
  2. [Section III-D, Section IV-C3] Effectiveness metrics are reported as single point estimates without confidence intervals or significance tests, yet the paper states that int8 and float8 quantization provide "no significant change" and uses fine-grained differences (e.g., -0.2% vs -6.3% MRR) to rank strategies. Given the paper's otherwise careful uncertainty quantification for inference time, the absence of variance for effectiveness undermines the strength of claims about small differences. The authors should either add bootstrap CIs or equivalent for effectiveness metrics, or temper language such as "no significant change" and the fine-grained ordering of quantization configurations.
minor comments (6)
  1. [Section IV-B3] The text reads "From Table IIIc" but should be "From Table IIIb" because it is discussing code summarization results.
  2. [Section IV-C3] The text reports KD's MRR loss as -52.1%, but Table IIIc lists -52.4% for MRR; the value -52.1% corresponds to MRR@5.
  3. [Section II-A] The BERT architecture is cited as reference [17], which is the OpenAI Codex paper; add a proper citation to Devlin et al. (BERT) and renumber the references accordingly.
  4. [Section VI] The external-validity caveat "results concerning Knowledge Distillation are specific to the DistilBERT LM" should be strengthened to explicitly acknowledge that the arm does not implement KD on CodeBERT; see major comment 1.
  5. [Section III-C1] The description of the CPU timing measurement is vague ("we use the time Python function"); specify which function from the time module (e.g., time.perf_counter) and whether the measurement covers the full batch processing loop.
  6. [Answer to RQ 2] The claim that knowledge distillation improves all efficiency metrics for code summarization overlooks that the GPU inference time reduction (-2.2%) is not statistically significant, as marked by the asterisk in Table IIIb.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the study's conclusions rest on measured benchmark comparisons, and the DistilBERT substitution is a validity limitation, not a circular reduction.

full rationale

The paper is an empirical study, not a derivation. Its load-bearing claims, namely that compression impact varies by task and method, are supported by measured comparisons on external benchmarks (Devign and CodeSearchNet under the CodeXGLUE pipelines) using standard effectiveness metrics (Accuracy, F1, MCC, BLEU, BERTScore, SIDE, MRR) and efficiency measurements with bootstrapped confidence intervals. Quantization and pruning are applied directly to the fine-tuned CodeBERT checkpoints and then measured; no fitted parameter is renamed as a prediction, and no conclusion is assumed in the experimental setup. The knowledge-distillation arm uses DistilBERT fine-tuned per task instead of a student actually distilled from CodeBERT, as the paper states in Section III-B1 ('we opted not to retrain a distilled model ourselves'). This is a genuine construct-validity limitation: the observed KD effectiveness losses may conflate the compression strategy with a different pretrained architecture. However, it is not circularity in the sense defined here, because DistilBERT is not fitted to the reported outcome and the paper explicitly flags the limitation in Section VI. The only self-citations are to prior benchmarking and methodology work and to the replication package; none is load-bearing or used to forbid alternatives. Therefore no step in the paper's argument reduces to its own inputs.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The empirical design introduces no invented entities and fits no parameters to the reported outcomes. Its conclusions rest on domain assumptions about the representativeness of DistilBERT, CodeXGLUE baselines, single-run effectiveness estimates, post-training quantization, and the measurement environment.

free parameters (3)
  • Pruning ratios = 0.2, 0.4, 0.6 (fraction of weights removed)
    Chosen by hand following prior work; conclusions about pruning depend on these exact levels.
  • Quantization bit widths = int4, int8, float8
    Selected because optimum-quanto supported them at experiment time; post-training results may differ with other precisions or quantization-aware training.
  • Fine-tuning hyperparameters = Epochs 5/10/2; learning rates 2e-5/5e-5/2e-5
    Inherited from CodeXGLUE rather than fitted; they set the baseline effectiveness that all compression comparisons reference.
assumptions (5)
  • domain assumption DistilBERT is a valid operationalization of knowledge distillation for CodeBERT.
    Section III-B1: the authors use pre-trained DistilBERT fine-tuned per task rather than distilling CodeBERT; conclusions about KD assume this substitution is valid.
  • domain assumption CodeXGLUE pipeline, splits, and hyperparameters provide a correct baseline.
    Section III-A: the pipeline is reused for all tasks; effectiveness comparisons assume these baselines are valid.
  • domain assumption Single-run effectiveness point estimates are sufficient to compare strategies.
    Section IV Tables IIIa-IIIc: one value per metric, no repeated runs or variance, yet small differences are interpreted.
  • domain assumption Post-training quantization with optimum-quanto and validation-set calibration represents quantization.
    Section III-B2: only post-training quantization is tested; conclusions may not extend to quantization-aware training.
  • domain assumption Inference-time behavior on the used HPC cluster generalizes to other hardware.
    Section III-C1 and Section VI: measurements come from one cluster; the practical recommendations assume similar behavior elsewhere.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On the Compression of Language Models for Code: An Empirical Study on CodeBERT." pith.science (2026). https://pith.science/paper/VKK44NAS

@misc{pith2026241213737,
  author       = {Pith},
  title        = {Pith review of: On the Compression of Language Models for Code: An Empirical Study on CodeBERT},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VKK44NAS}},
  note         = {Machine review of arXiv:2412.13737}
}
read the original abstract

Language models have proven successful across a wide range of software engineering tasks, but their significant computational costs often hinder their practical adoption. To address this challenge, researchers have begun applying various compression strategies to improve the efficiency of language models for code. These strategies aim to optimize inference latency and memory usage, though often at the cost of reduced model effectiveness. However, there is still a significant gap in understanding how these strategies influence the efficiency and effectiveness of language models for code. Here, we empirically investigate the impact of three well-known compression strategies -- knowledge distillation, quantization, and pruning -- across three different classes of software engineering tasks: vulnerability detection, code summarization, and code search. Our findings reveal that the impact of these strategies varies greatly depending on the task and the specific compression method employed. Practitioners and researchers can use these insights to make informed decisions when selecting the most appropriate compression strategy, balancing both efficiency and effectiveness based on their specific needs.

Figures

Figures reproduced from arXiv: 2412.13737 by the authors.

Figure 1
Figure 1. Experimental Methodology TABLE I EVALUATION METRICS Task Task Metrics Category Effectiveness Efficiency RQ1 Vulnerability Detection Code-Code Classification Accuracy [31] F1 Score [32] MCC [33] RQ2 Code Summarization Code-Text Generation Bleu [34] BERTScore [35] SIDE [36] Inf. Time (sec.) Model Size (MB) RQ3 Code Search Text-Code Search MRR [37] MRR@1 [37] MRR@5 [37] RQ1 How do compression strategies impact the effi… view at source ↗
Figure 2
Figure 2. Trade-off between effectiveness (y-axis) and efficiency (x-axis) metrics for each of the SE tasks. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

65 extracted references · 28 canonical work pages

  1. [1]

    Vulnerability detection with code language models: How far are we?

    Y . Ding, Y . Fu, O. Ibrahim et al. , “Vulnerability detection with code language models: How far are we?” 2024. [Online]. Available: https://arxiv.org/abs/2403.18624

    arXiv 2024

  2. [2]

    When neural code completion models size up the situation: Attaining cheaper and faster completion through dynamic model inference,

    Z. Sun, X. Du, F. Song et al. , “When neural code completion models size up the situation: Attaining cheaper and faster completion through dynamic model inference,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , ser. ICSE ’24. New York, NY , USA: Association for Computing Machinery, 2024. [Online]. Available: https:...

    arXiv 2024

  3. [3]

    Code search is all you need? improving code suggestions with code search,

    J. Chen, X. Hu, Z. Li et al. , “Code search is all you need? improving code suggestions with code search,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , ser. ICSE ’24. New York, NY , USA: Association for Computing Machinery, 2024. [Online]. Available: https://doi.org/10.1145/3597503.3639085

    arXiv 2024

  4. [4]

    Large language models for software engineering: A systematic literature review,

    X. Hou, Y . Zhao, Y . Liu et al. , “Large language models for software engineering: A systematic literature review,” 2024. [Online]. Available: https://arxiv.org/abs/2308.10620

    arXiv 2024

  5. [5]

    Green ai,

    R. Schwartz, J. Dodge, N. A. Smith et al., “Green ai,” 2019. [Online]. Available: https://arxiv.org/abs/1907.10597

    arXiv 2019

  6. [6]

    Distilling the knowledge in a neural network,

    G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” in NIPS Deep Learning and Representation Learning Workshop, 2015. [Online]. Available: http://arxiv.org/abs/1503.02531

    arXiv 2015

  7. [7]

    Q8bert: Quantized 8bit bert,

    O. Zafrir, G. Boudoukh, P. Izsak et al. , “Q8bert: Quantized 8bit bert,” in 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing - NeurIPS Edition (EMC2-NIPS) . Los Alamitos, CA, USA: IEEE Computer Society, dec 2019, pp. 36–

    work page 2019

  8. [8]

    Movement pruning: adaptive sparsity by fine-tuning,

    V . Sanh, T. Wolf, and A. M. Rush, “Movement pruning: adaptive sparsity by fine-tuning,” in Proceedings of the 34th International Conference on Neural Information Processing Systems, ser. NIPS ’20. Red Hook, NY , USA: Curran Associates Inc., 2020

    2020

Show all 65 references
  1. [9]

    Compressing pre-trained models of code into 3 mb,

    J. Shi, Z. Yang, B. Xu et al. , “Compressing pre-trained models of code into 3 mb,” in Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering , ser. ASE ’22. New York, NY , USA: Association for Computing Machinery, 2023. [Online]. Available...

    2023

  2. [10]

    Greening large language models of code,

    J. Shi, Z. Yang, H. J. Kang et al., “Greening large language models of code,” in Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Society , ser. ICSE-SEIS’24. New York, NY , USA: Association for Computing Machinery, 2024, p. 142...

    2024

  3. [11]

    Towards greener yet powerful code generation via quantization: An empirical study,

    X. Wei, S. K. Gonugondla, S. Wang et al. , “Towards greener yet powerful code generation via quantization: An empirical study,” in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/FSE...

    2023

  4. [12]

    CodeBERT: A pre-trained model for programming and natural languages,

    Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou, “CodeBERT: A pre-trained model for programming and natural languages,” in Findings of the Association for Computational Linguistics: EMNLP 2020, T. Cohn, Y . He, and Y . Liu, E...

    2020

  5. [13]

    On the compression of language models for code: An empirical study on codebert,

    G. d’Aloisio, L. Traini, F. Sarro, and A. Di Marco, “On the compression of language models for code: An empirical study on codebert,” Dec

  6. [14]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar et al. , “Attention is all you need,” in Advances in Neural Information Processing Systems , I. Guyon, U. V . Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017. [Online]. A...

    2017

  7. [15]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child et al., “Language models are unsupervised multitask learners,” OpenAI blog, vol. 1, no. 8, p. 9, 2019

    2019

  8. [16]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel, N. Shazeer, A. Roberts et al. , “Exploring the limits of transfer learning with a unified text-to-text transformer,” 2023. [Online]. Available: https://arxiv.org/abs/1910.10683

    2023 arXiv

  9. [17]

    Openai codex,

    OpenAI, “Openai codex,” 2019. [Online]. Available: https://arxiv.org/ abs/1810.04805

    2019 arXiv

  10. [18]

    CodeT5+: Open code large language models for code understanding and generation,

    Y . Wang, H. Le, A. Gotmare, N. Bui, J. Li, and S. Hoi, “CodeT5+: Open code large language models for code understanding and generation,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , H. Bouamor, J. Pino, and K. Bali, Eds. Singapor...

    2023

  11. [19]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun et al. , “Evaluating large language models trained on code,” 2021. [Online]. Available: https://arxiv.org/abs/2107. 03374

    2021

  12. [20]

    Assemble foundation models for automatic code summarization,

    J. Gu, P. Salza, and H. C. Gall, “Assemble foundation models for automatic code summarization,” in 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) . Los Alamitos, CA, USA: IEEE Computer Society, mar 2022, pp. 935–

    2022

  13. [21]

    Assessing generalizability of codebert,

    X. Zhou, D. Han, and D. Lo, “Assessing generalizability of codebert,” in 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME), 2021, pp. 425–436

    2021

  14. [23]

    Optimal brain damage,

    Y . LeCun, J. Denker, and S. Solla, “Optimal brain damage,” in Advances in Neural Information Processing Systems, D. Touretzky, Ed., vol. 2. Morgan-Kaufmann, 1989. [Online]. Available: https://proceedings.neurips.cc/paper files/paper/ 1989/file/6c9882bbac1c7093bd25041881277658...

    1989

  15. [24]

    Pruning filters for efficient convnets,

    H. Li, A. Kadav, I. Durdanovic et al. , “Pruning filters for efficient convnets,” in International Conference on Learning Representations ,

  16. [25]

    Dynamic network surgery for efficient dnns,

    Y . Guo, A. Yao, and Y . Chen, “Dynamic network surgery for efficient dnns,” 2016. [Online]. Available: https://arxiv.org/abs/1608.04493

    2016 arXiv

  17. [26]

    Pruning convolutional neural networks for resource efficient inference,

    P. Molchanov, S. Tyree, T. Karras et al., “Pruning convolutional neural networks for resource efficient inference,” 2017. [Online]. Available: https://arxiv.org/abs/1611.06440

    2017 arXiv

  18. [27]

    The state of sparsity in deep neural networks,

    T. Gale, E. Elsen, and S. Hooker, “The state of sparsity in deep neural networks,” 2019. [Online]. Available: https://arxiv.org/abs/1902.09574

    2019 arXiv

  19. [28]

    Graphcodebert: Pre-training code representations with data flow,

    D. Guo, S. Ren, S. Lu et al. , “Graphcodebert: Pre-training code representations with data flow,” 2021. [Online]. Available: https://arxiv.org/abs/2009.08366

    2021 arXiv

  20. [29]

    Software defect prediction via transformer,

    Q. Zhang and B. Wu, “Software defect prediction via transformer,” in 2020 IEEE 4th Information Technology, Networking, Electronic and Automation Control Conference (ITNEC), vol. 1. IEEE, 2020, pp. 874– 879

    2020

  21. [30]

    Transˆ 3: A transformer-based framework for unifying code summarization and code search,

    W. Wang, Y . Zhang, Z. Zeng et al. , “Transˆ 3: A transformer-based framework for unifying code summarization and code search,” arXiv preprint arXiv:2003.03238, 2020

    2003 arXiv

  22. [31]

    A coefficient of agreement as a measure of thematic classification accuracy

    G. Rosenfield and K. Fitzpatrick-Lins, “A coefficient of agreement as a measure of thematic classification accuracy.” Photogrammetric Engineering and Remote Sensing , vol. 52, no. 2, pp. 223–227, 1986. [Online]. Available: http://pubs.er.usgs.gov/publication/70014667

    1986

  23. [32]

    Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool,

    A. A. Taha and A. Hanbury, “Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool,” BMC Medical Imaging, vol. 15, no. 1, p. 29, Aug. 2015. [Online]. Available: https://doi.org/10.1186/s12880-015-0068-x

    2015 doi

  24. [33]

    The advantages of the matthews correlation coefficient (mcc) over f1 score and accuracy in binary classification evaluation,

    D. Chicco and G. Jurman, “The advantages of the matthews correlation coefficient (mcc) over f1 score and accuracy in binary classification evaluation,” BMC genomics, vol. 21, pp. 1–13, 2020

    2020

  25. [34]

    Bleu: a Method for Automatic Evaluation of Machine Translation,

    K. Papineni, S. Roukos, T. Ward et al., “Bleu: a Method for Automatic Evaluation of Machine Translation,” in Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics , P. Isabelle, E. Charniak, and D. Lin, Eds. Philadelphia, Pennsylvania, USA: As...

    2002

  26. [35]

    Bertscore: Evaluating text generation with bert,

    T. Zhang, V . Kishore, F. Wuet al., “Bertscore: Evaluating text generation with bert,” arXiv preprint arXiv:1904.09675 , 2019

    1904 arXiv

  27. [36]

    Evaluating Code Summarization Techniques: A New Metric and an Empirical Characterization,

    A. Mastropaolo, M. Ciniselli, M. Di Penta et al. , “Evaluating Code Summarization Techniques: A New Metric and an Empirical Characterization,” Dec. 2023. [Online]. Available: https://arxiv.org/abs/ 2312.15475v1

    2023 arXiv

  28. [37]

    Expected reciprocal rank for graded relevance,

    O. Chapelle, D. Metlzer, Y . Zhang et al. , “Expected reciprocal rank for graded relevance,” in Proceedings of the 18th ACM Conference on Information and Knowledge Management , ser. CIKM ’09. New York, NY , USA: Association for Computing Machinery, 2009, p. 621–630. [Online]. ...

    2009

  29. [38]

    Devign: Effective vulnerability identi- fication by learning comprehensive program semantics via graph neural networks,

    Y . Zhou, S. Liu, J. Siow et al., “Devign: Effective vulnerability identi- fication by learning comprehensive program semantics via graph neural networks,” in Advances in Neural Information Processing Systems, 2019, pp. 10 197–10 207

    2019

  30. [39]

    Available: https://doi.ieeecomputersociety.org/10.1109/ EMC2-NIPS53020.2019.00016

    [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/ EMC2-NIPS53020.2019.00016

    2019

  31. [40]

    Codexglue: A machine learning benchmark dataset for code understanding and generation,

    S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. B. Clement, D. Drain, D. Jiang, D. Tang, G. Li, L. Zhou, L. Shou, L. Zhou, M. Tufano, M. Gong, M. Zhou, N. Duan, N. Sundaresan, S. K. Deng, S. Fu, and S. Liu, “Codexglue: A machine learning benchmark dataset for c...

    2021 arXiv

  32. [41]

    Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,

    V . Sanh, L. Debut, J. Chaumond et al. , “Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,” 2020. [Online]. Available: https://arxiv.org/abs/1910.01108

    2020 arXiv

  33. [42]

    Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning,

    M. A. Gordon, K. Duh, and N. Andrews, “Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning,” May 2020, arXiv:2002.08307 [cs]. [Online]. Available: http://arxiv.org/abs/ 2002.08307

    2020 arXiv

  34. [43]

    Codesearchnet challenge: Evaluat- ing the state of semantic code search,

    H. Husain, H.-H. Wu, T. Gazit et al., “Codesearchnet challenge: Evaluat- ing the state of semantic code search,” arXiv preprint arXiv:1909.09436, 2019

    1909 arXiv

  35. [44]

    Ai-driven java performance testing: Balancing result quality with testing time,

    L. Traini, F. Di Menna, and V . Cortellessa, “Ai-driven java performance testing: Balancing result quality with testing time,” in Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ser. ASE ’24. New York, NY , USA: Association for Comp...

    2024

  36. [45]

    Automated generation and evaluation of jmh microbenchmark suites from unit tests,

    M. Jangali, Y . Tang, N. Alexandersson et al. , “Automated generation and evaluation of jmh microbenchmark suites from unit tests,” IEEE Transactions on Software Engineering , vol. 49, no. 4, pp. 1704–1725, 2023

    2023

  37. [46]

    Faster or slower? performance mystery of python idioms unveiled with empirical evidence,

    Z. Zhang, Z. Xing, X. Xia et al., “Faster or slower? performance mystery of python idioms unveiled with empirical evidence,” in Proceedings of the 45th International Conference on Software Engineering , ser. ICSE ’23. IEEE Press, 2023, p. 1495–1507. [Online]. Available: https:...

    2023

  38. [47]

    Pruning filters with l1-norm and capped l1-norm for cnn compression,

    A. Kumar, A. M. Shaikh, Y . Li et al., “Pruning filters with l1-norm and capped l1-norm for cnn compression,” Applied Intelligence, vol. 51, pp. 1152–1160, 2021

    2021

  39. [48]

    Rigorous benchmarking in reasonable time,

    T. Kalibera and R. Jones, “Rigorous benchmarking in reasonable time,” in Proceedings of the 2013 International Symposium on Memory Management, ser. ISMM ’13. New York, NY , USA: Association for Computing Machinery, 2013, p. 63–74. [Online]. Available: https://doi.org/10.1145/2...

    2013

  40. [49]

    Quantifying performance changes with effect size confidence intervals,

    ——, “Quantifying performance changes with effect size confidence intervals,” University of Kent, Technical Report 4–12, June 2012. [Online]. Available: http://www.cs.kent.ac.uk/pubs/2012/3233

    2012

  41. [50]

    Wilcoxon signed-rank test,

    R. F. Woolson, “Wilcoxon signed-rank test,” Encyclopedia of Biostatis- tics, vol. 8, 2005

    2005

  42. [51]

    Dynamically reconfiguring software microbenchmarks: Reducing execution time without sacrificing result quality,

    C. Laaber, S. W ¨ursten, H. C. Gall et al. , “Dynamically reconfiguring software microbenchmarks: Reducing execution time without sacrificing result quality,” in Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundati...

    2020

  43. [52]

    Searching for a needle in a haystack: Predicting security vulnerabilities for windows vista,

    T. Zimmermann, N. Nagappan, and L. Williams, “Searching for a needle in a haystack: Predicting security vulnerabilities for windows vista,” in Proceedings of the 3rd International Conference on Software Testing, Verification and Validation (ICST). IEEE, April 2010, most Influe...

    2010

  44. [53]

    The importance of accounting for real-world labelling when predicting software vulnerabilities,

    M. Jimenez, R. Rwemalika, M. Papadakis et al. , “The importance of accounting for real-world labelling when predicting software vulnerabilities,” in Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of S...

    2019

  45. [54]

    On the use of evaluation measures for defect prediction studies,

    R. Moussa and F. Sarro, “On the use of evaluation measures for defect prediction studies,” in Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis . New York, NY , USA: Association for Computing Machinery, 2022. [Online]. Available: http...

    2022

  46. [55]

    How software refactoring impacts execution time,

    L. Traini, D. Di Pompeo, M. Tucci et al. , “How software refactoring impacts execution time,” ACM Trans. Softw. Eng. Methodol. , vol. 31, no. 2, dec 2021. [Online]. Available: https://doi.org/10.1145/3485136

    2021 doi

  47. [56]

    Survey of code search based on deep learning,

    Y . Xie, J. Lin, H. Dong et al. , “Survey of code search based on deep learning,” ACM Trans. Softw. Eng. Methodol. , vol. 33, no. 2, Dec

  48. [57]

    A survey of quantization methods for efficient neural network inference,

    A. Gholami, S. Kim, Z. Dong et al., “A survey of quantization methods for efficient neural network inference,” in Low-Power Computer Vision. Chapman and Hall/CRC, 2022, pp. 291–326

    2022

  49. [58]

    Up or down? adaptive rounding for post-training quantization,

    M. Nagel, R. A. Amjad, M. van Baalen et al., “Up or down? adaptive rounding for post-training quantization,” 2020. [Online]. Available: https://arxiv.org/abs/2004.10568

    2020 arXiv

  50. [59]

    Source Code Summarization in the Era of Large Language Models,

    W. Sun, Y . Miao, Y . Li et al. , “Source Code Summarization in the Era of Large Language Models,” Jul. 2024, arXiv:2407.07959 [cs]. [Online]. Available: http://arxiv.org/abs/2407.07959

    2024 arXiv

  51. [60]

    Rethinking the value of network pruning,

    Z. Liu, M. Sun, T. Zhou et al. , “Rethinking the value of network pruning,” 2019. [Online]. Available: https://arxiv.org/abs/1810.05270

    2019 arXiv

  52. [61]

    Taming performance variability,

    A. Maricq, D. Duplyakin, I. Jimenez et al. , “Taming performance variability,” in 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18) . Carlsbad, CA: USENIX Association, Oct. 2018, pp. 409–425. [Online]. Available: https: //www.usenix.org/conference/...

    2018

  53. [64]

    Compressing large-scale transformer- based models: A case study on bert,

    P. Ganesh, Y . Chen, X. Louet al., “Compressing large-scale transformer- based models: A case study on bert,” Transactions of the Association for Computational Linguistics , vol. 9, pp. 1061–1080, 2021

    2021

  54. [946]

    Available: https://doi.ieeecomputersociety.org/10.1109/ SANER53432.2022.00112

    [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/ SANER53432.2022.00112

    2022

  55. [2017]

    Available: https://openreview.net/forum?id=rJqFGTslg

    [Online]. Available: https://openreview.net/forum?id=rJqFGTslg

  56. [2023]

    Available: https://doi.org/10.1145/3628161

    [Online]. Available: https://doi.org/10.1145/3628161

    doi

  57. [2024]

    Available: https://doi.org/10.5281/zenodo.14357478

    [Online]. Available: https://doi.org/10.5281/zenodo.14357478

    doi

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.