Pith. sign in

REVIEW 4 major objections 4 minor 70 references

This paper claims that the best way to shrink code tokens for summarization depends heavily on the programming language, with ASTs winning in Java, function signatures winning in Python, and which tokens you keep mattering more than how man

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 07:31 UTC pith:TGHOI4DE

load-bearing objection Substantial two-language study with real artifacts, but the central cross-language table contradicts the per-language Java results; fix the Java numbers before anyone leans on the headline claim. the 4 major comments →

arxiv 2601.20147 v3 pith:TGHOI4DE submitted 2026-01-28 cs.SE

Not All Tokens Matter: Data-Centric Optimization for Efficient Code Summarization

Saima Afrin , Zaiyu Cheng , Tushar Sharma , Alexander Serebrenik , Massimiliano Di Penta , Antonio Mastropaolo This is my paper
classification cs.SE
keywords code summarizationtoken reductionabstract syntax treefunction signatureCrystalBLEUdata-centric optimizationSIDE metricPython benchmark
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that token-level data optimization for code summarization is not language-agnostic. It compares three reduction techniques—AST-based representation, function-signature extraction, and CrystalBLEU-guided pruning—on Java and Python, with and without semantic filtering. The central claim is that the optimal strategy flips between languages: ASTs give large performance gains in Java but severe degradation in Python, while function signatures do the opposite. The authors also argue that aggressive token reduction can maintain or even improve summary quality when the right representation is chosen, and that semantic filtering enables even more compression without loss. If correct, practitioners cannot assume a single token-reduction recipe works across languages.

Core claim

The paper's central discovery is a language-dependent reversal in token-reduction effectiveness. In Java, AST-based optimization improves BLEU by about 37% over the baseline while cutting tokens by 56–73%, but the same AST approach in Python degrades performance by up to 49%. Conversely, function-signature extraction performs poorly in Java but is optimal in Python, achieving 83% token reduction with negligible quality loss or even improvement on a harder benchmark. CrystalBLEU-guided pruning provides cross-language robustness, consistently achieving 60–72% token reduction with modest quality trade-offs. The authors interpret these results as evidence that the semantic weight carried by stru

What carries the argument

The paper's central objects are three token-level transformations: abstract syntax tree (AST) token sequences, function signatures (method name, return type, parameters), and CrystalBLEU-guided pruning that removes trivially shared n-grams. These are applied to the training input of a CodeT5+ model, either standalone or cascaded after semantic filtering with the SIDE/SIDEpy coherence metric. The work also introduces SIDEpy, a Python-retrained version of the SIDE semantic alignment metric, and PyBench, a 500-item human-annotated Python evaluation benchmark. The machinery works by showing how each transformation alters the token distribution (measured via Shannon entropy reduction and token re

Load-bearing premise

The Java-versus-Python difference is caused by language semantics rather than by the particular parsers used to build ASTs and signatures, or by the proxy metrics chosen to measure summary quality.

What would settle it

Train the same code summarization models using the same token-reduction pipeline but with a single shared AST extractor applied to both Java and Python (or with AST and signature representations swapped between the two parsers); if the performance reversal disappears, the 'language-dependence' claim is actually parser-dependence.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If correct, code summarization systems should be tuned per language: use ASTs for Java-like verbose languages and function signatures for Python-like concise languages.
  • Semantic filtering of code-comment pairs before token reduction can enable aggressive compression without quality loss, reducing training compute and energy costs.
  • Large token reductions (up to 83%) can be applied without hurting summary quality when the retained tokens carry the right semantic information—so data optimization should focus on token selection rather than raw volume.
  • CrystalBLEU-guided pruning offers a safe, language-robust default when the optimal strategy is unknown.
  • The SIDEpy metric and PyBench benchmark provide tools for evaluating Python code summarization in a way that better aligns with human judgments.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct extension is that the language-dependence likely extends beyond Java/Python to other languages based on syntax verbosity and naming conventions; for example, C# or Kotlin may behave more like Java, while Ruby or JavaScript (concise, expressive names) may behave more like Python.
  • The optimal representation may also depend on the downstream task, not just the language: structure-preserving ASTs may favor structural tasks like bug detection, while identifier-preserving signatures may favor intent-inference tasks.
  • The finding suggests a testable adaptive scheme: dynamically choose the token representation per example based on language and code complexity, rather than applying one strategy corpus-wide.
  • The parser-choice confounder—Java's AST from java.lang, Python's from the ast module—means the 'language' effect could partly be a parser artifact; replicating with a unified parser would isolate the true source.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper investigates token-level data optimization for code summarization, comparing AST-based representations, function-signature-based inputs, and CrystalBLEU-guided pruning, optionally cascaded with SIDE-based semantic filtering. Experiments use CodeT5+ (220M) fine-tuned on Funcom Java and Python variants and evaluated on CoderEval, Mastropaolo et al.'s Java benchmark, and a newly introduced PyBench benchmark. The paper claims that the optimal token-reduction strategy is language-dependent: AST gives 37% improvement in Java while degrading in Python, Function Signatures excel in Python but not Java, and CrystalBLEU is robust across languages. It also introduces SIDEpy, a Python-specific semantic alignment metric, and PyBench, a 500-item human-annotated Python benchmark.

Significance. If the central claims held, the paper would make a valuable contribution to data-centric optimization for code summarization, with practical implications for reducing training cost. The paper has notable strengths: it ships a replication package; it validates SIDEpy against human judgments with logistic regression; it applies Holm-Bonferroni corrections and reports effect sizes; and Section 10 explicitly acknowledges multiple internal-validity threats. These strengths, however, are currently undermined by a numerically inconsistent central comparison (Tables 3/4 vs. Table 9) and by unresolved confounds that the paper itself identifies.

major comments (4)
  1. [Table 9 vs. Tables 3/4; abstract; Section 7] The central cross-language claim is unsupported by the reported Java CoderEval numbers. Table 3 gives Original FuncomJava CoderEval BLEU = 6.32 and AST standalone = 8.67, which is a 37.2% improvement and matches the abstract's "37%" claim. Table 9, however, lists the same Java baseline as 12.41 and AST standalone as 9.16, which is a 26% degradation. Likewise, Table 4's SIDE baseline is 6.32 and AST SIDE-optimized is 7.04, while Table 9 gives 12.13 and 11.81. ROUGE-L and METEOR also differ (e.g., Java baseline ROUGE-L 34.09 in Table 3 vs. 36.94 in Table 9; METEOR 28.64 vs. 35.35). This is not a minor typo: the 37% improvement claim and the language-dependence conclusion rest on these numbers. The authors must rerun/report the Java CoderEval evaluation consistently and recompute all percentages, effect sizes, and significance tests accordingly.
  2. [Section 10 (Internal validity); Sections 5.1 and 7.1] The paper acknowledges that Java ASTs are produced by JavaLang while Python ASTs are produced by Python's ast module, and that parser choices may introduce language-specific biases. This is precisely the confound that threatens the headline "language-dependent" conclusion: the observed Java/Python difference could be parser-dependent rather than language-dependent. The acknowledgment alone is insufficient. A concrete control is needed, e.g., using the same parser framework for both languages, or comparing within one language across two parser implementations to estimate parser-induced variation. Without such evidence, the central claim that language semantics, rather than tooling differences, drive the reversal remains a hypothesis.
  3. [Section 7.2.1 and 7.2.2] PyBench is described inconsistently: Section 7.2.1 states the benchmark "consists of 500 human-annotated instances," while the following paragraph says "This curation process resulted in a final evaluation set of 230 high-quality Python code-summary pairs." Since PyBench is a newly introduced benchmark and is used as an evaluation set in Tables 7/8 and in the significance tests of Table 10, the number of instances must be clarified and the reported scores must be tied to the correct dataset. As written, a reader cannot tell which PyBench was evaluated.
  4. [Section 10 (Internal validity); Section 5.1] The paper states that "The construction of the CrystalBLEU frequent n-gram list from the training split introduces potential circularity; any leakage from validation/test into that list would bias results." This is a load-bearing threat for the CrystalBLEU results, which are used to support the cross-language robustness claim. The manuscript should specify exactly how the frequent n-gram list was computed, what data splits were used, and what leakage checks were performed. Currently the risk is named but not resolved.
minor comments (4)
  1. [Section 7.1] Typo: "unoptimized FncomPython dataset" should be "FuncomPython."
  2. [Section 5.3.3 and 7.2.4] In Section 7.2.4 the text refers to "standalone token-optimization results in Table 8," but the standalone Python results are in Table 7; the cascaded results are in Table 8. The table references in the Python section are swapped or otherwise inconsistent.
  3. [Section 6.2] Table 6's caption says "Rows with statistically significant predictors (p < 0.05) are shaded" and the text says "shaded in black," but the in-text description of the shading could be clearer and should be consistent with the actual rendered table.
  4. [Section 2.2] The sentence "Moreover, by replicating the original experimental setup and releasing our implementation publicly available on GitHub [18]," is grammatically incomplete and should be finished or merged with the following paragraph.

Circularity Check

0 steps flagged

No significant circularity; the token-reduction claims are empirical comparisons on external benchmarks, and the paper's self-identified 'potential circularity' is an unsubstantiated leakage concern, not a demonstrated derivation loop.

full rationale

The paper's claims (AST helps Java, Function Signatures help Python, CrystalBLEU is robust, etc.) are empirical results from fine-tuning CodeT5+ on transformed versions of Funcom and evaluating on external or independently curated benchmarks (CoderEval, Mastropaolo et al., Crupi et al., PyBench). No equation in the paper derives a prediction from a fitted parameter; token retention, entropy reduction, BLEU, ROUGE-L, and METEOR are computed as descriptive statistics of the actual outputs. The only passage that explicitly mentions circularity is in Section 10 (internal validity): 'The construction of the CrystalBLEU frequent n-gram list from the training split introduces potential circularity; any leakage from validation/test into that list would bias results.' This is a caveat about possible data leakage, not a demonstrated reduction; building a pruning n-gram list from the training split is a legitimate preprocessing step, and the paper does not claim to predict the test split from it. The paper does rely on prior work co-authored by one of the present authors (SIDE/Vitale et al., Crupi et al., Mastropaolo et al.), but those are external, human-validated benchmarks and metrics, not unverified self-assertions used as the sole justification for the central claims. For example, SIDEpy is evaluated against human adequacy judgments in a regression with odds ratios, and the pattern of Java/Python differences is cross-checked on two benchmarks per language. There is a serious correctness concern outside circularity: Table 9 reports Java baseline BLEU 12.41 and AST standalone 9.16, whereas Tables 3/4 report baseline 6.32 and AST standalone 8.67; the abstract's '37% improvement' cannot be reconciled with both. That inconsistency undermines confidence in the headline comparison, but it is not a circularity—it is an internal numerical conflict that would need re-running the Java CoderEval evaluation. Accordingly, the circularity score is 0; the correctness risk should be handled separately.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 2 invented entities

The central claims rest on measurement choices rather than mathematical axioms. Summary quality is read from BLEU/ROUGE/METEOR; efficiency from token retention and Shannon entropy; the model/training setup is one 220M CodeT5+ configuration; and the cross-language comparison assumes JavaLang and Python ast parser outputs are comparable. SIDEpy and PyBench have external handles (human-judgment correlation, released data) but were partly built by the authors.

free parameters (3)
  • SIDE/SIDEpy semantic filtering threshold = 0.9
    Chosen to match Vitale et al.'s Pareto-optimal cutoff; retains roughly 49% of Java pairs and 31.7% of Python pairs. This directly shapes the cascaded conditions and the final training sets.
  • CrystalBLEU frequent n-gram pruning criterion = not reported
    The rule that decides which n-grams are 'trivially shared' and pruned is not specified. It controls the reported 72.23% and 60.76% token reductions and is central to the CrystalBLEU condition.
  • Training hyperparameters = 20 epochs, batch 16, LR 2e-5, patience 5, max input 512
    Standard settings adopted from prior work; they are identical across conditions but no seed variation is reported, so training variance is not quantified.
axioms (5)
  • domain assumption BLEU, ROUGE-L, and METEOR are valid proxies for code summary quality.
    These metrics are the primary outcome measures; the paper itself lists this as a construct-validity threat in Section 10.
  • domain assumption Token retention and Shannon entropy are valid proxies for computational efficiency.
    All efficiency claims are based on these proxies; no wall-clock time, FLOPs, or energy data are reported, as conceded in Section 10.
  • domain assumption CodeT5+-220M fine-tuned on Funcom is representative of code summarization training.
    All conclusions rest on one model size and one dataset family; external-validity threats in Section 10 acknowledge that larger or different models may behave differently.
  • domain assumption JavaLang and Python ast parsers produce comparably faithful AST/signature transformations.
    The cross-language comparison assumes parser differences are not the cause of the observed rank reversals; Section 10 lists parser corner cases as an internal-validity threat.
  • domain assumption SIDEpy contrastive embeddings capture genuine code-summary semantic alignment.
    SIDEpy is validated only on the Crupi et al. human-judgment set, and Section 10 acknowledges its embedding space may encode dataset-specific regularities.
invented entities (2)
  • SIDEpy independent evidence
    purpose: Python-specific semantic alignment metric for filtering and evaluating code-summary pairs.
    Evaluated against 1,163 human-judged code summaries (Crupi et al.), where it achieves an odds ratio of 1.977 for content adequacy, and the implementation is promised on GitHub.
  • PyBench independent evidence
    purpose: 500-example human-annotated Python benchmark for code summarization evaluation.
    Constructed from post-August-2024 GitHub repositories with manual ratings on three quality dimensions and released with the replication package; the summaries are generated by GPT-4o-mini, so the ratings target model-generated text.

pith-pipeline@v1.3.0-alltime-deepseek · 31061 in / 16148 out tokens · 180592 ms · 2026-08-03T07:31:18.559402+00:00 · methodology

0 comments
read the original abstract

The rapid advancement of Large Language Models (LLMs) has revolutionized software engineering automation, particularly in automated code summarization, which enhances program comprehension and supports development activities. However, training LLMs for code summarization remains computationally expensive, with performance deteriorating on longer inputs-challenges that intensify when handling millions of code-comment pairs. We investigate strategic data optimization through targeted token reduction to minimize computational overhead while maintaining summary quality. We compare three token-level reduction techniques-(i) Abstract Syntax Tree (AST) representations, (ii) Function Signatures, and (iii) CrystalBLEU-guided pruning-combined with semantic filtering, evaluating them on Java and Python in standalone and cascaded reduction settings. Our findings reveal highly language-dependent optimal strategies: AST-based optimization achieves 37% performance improvements in Java with 56-73% token reduction but shows up to 49% degradation in Python. Conversely, Function Signatures perform poorly in Java but optimally in Python, achieving 83% token reduction while maintaining quality. CrystalBLEU demonstrates cross-language robustness with 60-72% reduction. These results challenge assumptions about cross-language transferability, demonstrating that which tokens are kept matters more than how many are removed, making language-aware token curation essential for efficient code summarization.

Figures

Figures reproduced from arXiv: 2601.20147 by Alexander Serebrenik, Antonio Mastropaolo, Massimiliano Di Penta, Saima Afrin, Tushar Sharma, Zaiyu Cheng.

Figure 1
Figure 1. Figure 1: Different Token Representation Techniques in Token-based Optimization and their computed Shan [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Qualitative comparison of token optimization strategies on Python code summarization. Function [PITH_FULL_IMAGE:figures/full_fig_p024_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative comparison of token optimization strategies on Java code summarization. Full code tokens [PITH_FULL_IMAGE:figures/full_fig_p025_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

70 extracted references · 19 linked inside Pith

  1. [1]

    Gpt-4technicalreport

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, JankoAltenschmidt,SamAltman,ShyamalAnadkat,etal.2023. Gpt-4technicalreport. arXiv preprint arXiv:2303.08774 (2023)

    Pith/arXiv arXiv 2023

  2. [2]

    Toufique Ahmed, Kunal Suresh Pai, Premkumar Devanbu, and Earl Barr. 2024. Automatic semantic augmentation of language model prompts (for code summarization). InProceedings of the IEEE/ACM 46th international conference on software engineering. 1–13

    2024

  3. [3]

    Tamás Aladics, Péter Hegedűs, and Rudolf Ferenc. 2022. An AST-based code change representation and its perfor- mance in just-in-time vulnerability prediction. InInternational Conference on Software Technologies . Springer, 169– 186

    2022

  4. [4]

    Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. 2018. code2seq: Generating sequences from structured repre- sentations of code.arXiv preprint arXiv:1808.01400 (2018)

    Pith/arXiv arXiv 2018

  5. [5]

    Menaka Pushpa Arthur. 2020. Automatic source code documentation using code summarization technique of NLP. Procedia Computer Science 171 (2020), 2522–2531

    2020

  6. [6]

    Devanbu, and Abraham Bernstein

    Adrian Bachmann, Christian Bird, Foyzur Rahman, Premkumar T. Devanbu, and Abraham Bernstein. 2010. The miss- ing links: bugs and bug-fix commits. InProceedings of the 18th ACM SIGSOFT International Symposium on Foundations of Software Engineering, 2010, Santa Fe, NM, USA, November 7-11, 2010 . ACM, 97–106

    2010

  7. [7]

    Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An Automatic Metric for MT Evaluation with Improved Cor- relation with Human Judgments. InWorkshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, ACL. 65–72

    2005

  8. [8]

    Daniele Bifolco, Pietro Cassieri, Giuseppe Scanniello, Massimiliano Di Penta, and Fiorella Zampetti. 2025. Do LLMs provide links to code similar to what they generate? A Study with Gemini and Bing CoPilot. In2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR) . IEEE, 223–235

    2025

  9. [9]

    Christian Bird, Adrian Bachmann, Eirik Aune, John Duffy, Abraham Bernstein, Vladimir Filkov, and Premkumar T. Devanbu. 2009. Fair and balanced?: bias in bug-fix datasets. InProceedings of the 7th joint meeting of the European Soft- ware Engineering Conference and the ACM SIGSOFT International Symposium on Foundations of Software Engineering, 2009, Amsterd...

    2009

  10. [10]

    Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo- Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder.arXiv preprint arXiv:1803.11175 (2018)

    Pith/arXiv arXiv 2018

  11. [11]

    Matteo Ciniselli, Alberto Martin-Lopez, and Gabriele Bavota. 2023. On the Generalizability of Deep Learning-based Code Completion Across Programming Language Versions. In32nd International Conference on Program Comprehen- sion, ICPC 2023. in press

    2023

  12. [12]

    Matteo Ciniselli, Alberto Martin-Lopez, and Gabriele Bavota. 2024. On the generalizability of deep learning-based codecompletionacrossprogramminglanguageversions.In Proceedings of the 32nd IEEE/ACM International Conference on Program Comprehension. 99–111

    2024

  13. [13]

    Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data.arXiv preprint arXiv:1705.02364 (2017)

    Pith/arXiv arXiv 2017

  14. [14]

    Giuseppe Crupi, Rosalia Tufano, Alejandro Velasco, Antonio Mastropaolo, Denys Poshyvanyk, and Gabriele Bavota

  15. [15]

    Jacob Devlin. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805 (2018)

    Pith/arXiv arXiv 2018

  16. [16]

    Xi Ding, Rui Peng, Xiangping Chen, Yuan Huang, Jing Bian, and Zibin Zheng. 2024. Do code summarization models processtoomuchinformation?functionsignaturemaybeallthatisneeded. ACM Transactions on Software Engineering and Methodology 33, 6 (2024), 1–35

    2024

  17. [17]

    Aryaz Eghbali and Michael Pradel. 2022. CrystalBLEU: precisely and efficiently measuring the similarity of code. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering . 1–12

    2022

  18. [18]

    gitpromptproject. 2025. Replication Package: Data‑Centric Optimization in Code Summarization Dataset. https: //github.com/gitpromptproject/optimizing-data-for-Code-Summarization.git . Accessed: 2025-09-03

    2025

  19. [19]

    Xiaodong Gu, Meng Chen, Yalan Lin, Yuhan Hu, Hongyu Zhang, Chengcheng Wan, Zhao Wei, Yong Xu, and Juhong Wang. 2025. On the effectiveness of large language models in domain-specific code generation.ACM Transactions on Software Engineering and Methodology 34, 3 (2025), 1–22

    2025

  20. [20]

    Kim Herzig, Sascha Just, and Andreas Zeller. 2013. It’s not a bug, it’s a feature: how misclassification impacts bug prediction. In35th International Conference on Software Engineering, ICSE ’13, San Francisco, CA, USA, May 18-26, 2013. IEEE Computer Society, 392–401

    2013

  21. [21]

    Sture Holm. 1979. A simple sequentially rejective multiple test procedure.Scandinavian Journal of Statistics 6 (1979), 65–70. , Vol. 1, No. 1, Article . Publication date: February 2026. 30 Saima Afrin, Zaiyu Cheng, Tushar Sharma, Alexander Serebrenik, Massimiliano Di Penta, and Antonio Mastropaolo

    1979

  22. [22]

    Alexander LeClair, Sakib Haque, Lingfei Wu, and Collin McMillan. 2020. Improved code summarization via a graph neural network. In28th IEEE/ACM International Conference on Program Comprehension, ICPC . 184–195

    2020

  23. [23]

    Alexander LeClair, Siyuan Jiang, and Collin McMillan. 2019. A neural model for generating natural language sum- maries of program subroutines. In41st IEEE/ACM International Conference on Software Engineering, ICSE . 795–806

    2019

  24. [24]

    Vladimir I Levenshtein et al. 1966. Binary codes capable of correcting deletions, insertions, and reversals.Soviet physics doklady 10, 8 (1966), 707–710

    1966

  25. [25]

    Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. InText summarization branches out. 74–81

    2004

  26. [26]

    Bohong Liu, Tao Wang, Xunhui Zhang, Qiang Fan, Gang Yin, and Jinsheng Deng. 2019. A neural-network based code summarization approach by using source code and its call dependencies. InProceedings of the 11th Asia-Pacific Symposium on Internetware. 1–10

    2019

  27. [27]

    Cristina Lopes et al. 2010. Sourcerer: An Infrastructure for Large‐scale Collection and Analysis of Open‐source Code. W ASDeTT Workshop, University of California, Irvine(2010)

    2010

  28. [28]

    Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)

    Pith/arXiv arXiv 2017

  29. [29]

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. 2024. Starcoder 2 and the stack v2: The next generation.arXiv preprint arXiv:2402.19173 (2024)

    Pith/arXiv arXiv 2024

  30. [30]

    Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664 (2021)

    Pith/arXiv arXiv 2021

  31. [31]

    Antonio Mastropaolo, Emad Aghajani, Luca Pascarella, and Gabriele Bavota. 2021. An empirical study on code com- ment completion. In2021 IEEE International Conference on Software Maintenance and Evolution (ICSME) . IEEE, 159– 170

    2021

  32. [32]

    Antonio Mastropaolo, Matteo Ciniselli, Massimiliano Di Penta, and Gabriele Bavota. 2024. Evaluating Code Summa- rization Techniques: A New Metric and an Empirical Characterization. InProceedings of the IEEE/ACM 46th Interna- tional Conference on Software Engineering. 1–13

    2024

  33. [33]

    Antonio Mastropaolo, Matteo Ciniselli, Luca Pascarella, Rosalia Tufano, Emad Aghajani, and Gabriele Bavota. 2024. Towardssummarizingcodesnippetsusingpre-trainedtransformers.In Proceedings of the 32nd IEEE/ACM International Conference on Program Comprehension. 1–12

    2024

  34. [34]

    Antonio Mastropaolo, Nathan Cooper, David Nader Palacio, Simone Scalabrino, Denys Poshyvanyk, Rocco Oliveto, and Gabriele Bavota. 2022. Using transfer learning for code-related tasks.IEEE Transactions on Software Engineering 49, 4 (2022), 1580–1598

    2022

  35. [35]

    Antonio Mastropaolo, Massimiliano Di Penta, and Gabriele Bavota. 2023. Towards Automatically Addressing Self- Admitted Technical Debt: How Far Are We?. In2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 585–597

    2023

  36. [36]

    Antonio Mastropaolo, Vittoria Nardone, Gabriele Bavota, and Massimiliano Di Penta. 2024. How the training proce- dure impacts the performance of deep learning-based vulnerability patching. InProceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering . 150–159

    2024

  37. [37]

    Antonio Mastropaolo, Luca Pascarella, Emanuela Guglielmi, Matteo Ciniselli, Simone Scalabrino, Rocco Oliveto, and Gabriele Bavota. 2023. On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilot. In 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023 . IEEE, 2149–2160

    2023

  38. [38]

    Antonio Mastropaolo and Denys Poshyvanyk. 2025. A Path Less Traveled: Reimagining Software Engineering Au- tomation via a Neurosymbolic Paradigm. InProceedings of the 33rd ACM International Conference on the Foundations of Software Engineering. 1358–1362

    2025

  39. [39]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation. In40th Annual Meeting on Association for Computational Linguistics, ACL . 311–318

    2002

  40. [40]

    JeffreyPennington, RichardSocher,andChristopher DManning. 2014. Glove:Global vectorsfor wordrepresentation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) . 1532–1543

    2014

  41. [41]

    Huy Nhat Phan, Hoang Nhat Phan, Tien N Nguyen, and Nghi DQ Bui. 2024. Repohyper: Better context retrieval is all you need for repository-level code completion.CoRR (2024)

    2024

  42. [42]

    Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In10th Workshop on Statistical Machine Translation, WMT. 392–395

    2015

  43. [43]

    RuchirPuri,DavidSKung,GeertJanssen,WeiZhang,GiacomoDomeniconi,VladimirZolotov,JulianDolby,JieChen, Mihir Choudhury, Lindsey Decker, et al. 2021. Codenet: A large-scale ai for code dataset for learning a diversity of coding tasks. arXiv preprint arXiv:2105.12655 (2021). , Vol. 1, No. 1, Article . Publication date: February 2026. Not All Tokens Matter: Data...

    Pith/arXiv arXiv 2021

  44. [44]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research 21, 140 (2020), 1–67

    2020

  45. [45]

    Juan Ramos et al. 2003. Using tf-idf to determine word relevance in document queries. InProceedings of the first instructional conference on machine learning, Vol. 242. New Jersey, USA, 29–48

    2003

  46. [46]

    Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks.arXiv preprint arXiv:1908.10084 (2019)

    Pith/arXiv arXiv 2019

  47. [47]

    Devjeet Roy, Sarah Fakhoury, and Venera Arnaoudova. 2021. Reassessing automatic evaluation metrics for code summarization tasks. InProceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering . 1105–1116

    2021

  48. [48]

    Claude E Shannon. 1948. A mathematical theory of communication.The Bell system technical journal 27, 3 (1948), 379–423

    1948

  49. [49]

    Ke Shi, Yang Lu, Jingfei Chang, and Zhen Wei. 2020. PathPair2Vec: An AST path pair-based code representation method for defect prediction.Journal of Computer Languages 59 (2020), 100979

    2020

  50. [50]

    Lin Shi, Fangwen Mu, Xiao Chen, Song Wang, Junjie Wang, Ye Yang, Ge Li, Xin Xia, and Qing Wang. 2022. Are we buildingontherock?ontheimportanceofdatapreprocessingforcodesummarization.In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering . 107–119

    2022

  51. [51]

    Saleh Soltan, Shankar Ananthakrishnan, Jack FitzGerald, Rahul Gupta, Wael Hamza, Haidar Khan, Charith Peris, Stephen Rawls, Andy Rosenbaum, Anna Rumshisky, et al. 2022. Alexatm 20b: Few-shot learning using a large-scale multilingual seq2seq model.arXiv preprint arXiv:2208.01448 (2022)

    Pith/arXiv arXiv 2022

  52. [52]

    Daniela Steidl, Benjamin Hummel, and Elmar Juergens. 2013. Quality analysis of source code comments. In2013 21st International Conference on Program Comprehension (ICPC) . 83–92

    2013

  53. [53]

    Chia-Yi Su and Collin McMillan. 2024. Semantic similarity loss for neural source code summarization.Journal of Software: Evolution and Process 36, 11 (2024), e2706

    2024

  54. [54]

    WeisongSun,YunMiao,YuekangLi,HongyuZhang,ChunrongFang,YiLiu,GeleiDeng,YangLiu,andZhenyuChen

  55. [55]

    Xiangru Tang, Bill Qian, Rick Gao, Jiakang Chen, Xinyun Chen, and Mark Gerstein. 2023. Biocoder: A benchmark for bioinformatics code generation with contextual pragmatic knowledge. (2023)

    2023

  56. [56]

    YiTay,MostafaDehghani,VinhQTran,XavierGarcia,JasonWei,XuezhiWang,HyungWonChung,SiamakShakeri, Dara Bahri, Tal Schuster, et al. 2022. Ul2: Unifying language learning paradigms.arXiv preprint arXiv:2205.05131 (2022)

    Pith/arXiv arXiv 2022

  57. [57]

    Chris Thunes. 2020. javalang: Pure Python Java parser and tools.https://github.com/c2nes/javalang. Version 0.13.0

    2020

  58. [58]

    Alejandro Velasco, Aya Garryyeva, David N Palacio, Antonio Mastropaolo, and Denys Poshyvanyk. 2025. Toward neurosymbolic program comprehension.arXiv preprint arXiv:2502.01806 (2025)

    Pith/arXiv arXiv 2025

  59. [59]

    Optimizing Datasets for Code Summarization: Is Code-Comment Coherence Enough?arXiv preprint arXiv:2502.07611 (2025)

    AntonioVitale,AntonioMastropaolo,RoccoOliveto,MassimilianoDiPenta,andSimoneScalabrino.2025. Optimizing Datasets for Code Summarization: Is Code-Comment Coherence Enough?arXiv preprint arXiv:2502.07611 (2025)

    Pith/arXiv arXiv 2025

  60. [60]

    Anempiricalstudyonnoisylabellearn- ing for program understanding

    WenhanWang,YanzhouLi,AnranLi,JianZhang,WeiMa,andYangLiu.2024. Anempiricalstudyonnoisylabellearn- ing for program understanding. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering. 1–12

    2024

  61. [61]

    Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi DQ Bui, Junnan Li, and Steven CH Hoi. 2023. Codet5+: Open code large language models for code understanding and generation.arXiv preprint arXiv:2305.07922 (2023)

    Pith/arXiv arXiv 2023

  62. [62]

    Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. 2021. CodeT5: Identifier-aware unified pre-trained encoder- decoder models for code understanding and generation.arXiv preprint arXiv:2109.00859 (2021)

    Pith/arXiv arXiv 2021

  63. [63]

    Guang Yang, Yu Zhou, Wei Cheng, Xiangyu Zhang, Xiang Chen, Terry Yue Zhuo, Ke Liu, Xin Zhou, David Lo, and Taolue Chen. 2024. Less is more: Docstring compression in code generation.arXiv preprint arXiv:2410.22793 (2024)

    Pith/arXiv arXiv 2024

  64. [64]

    Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Qianxiang Wang, and Tao Xie. 2024. Codereval: A benchmark of pragmatic code generation with generative pre-trained models. InProceedings of the 46th IEEE/ACM International Conference on Software Engineering . 1–12

    2024

  65. [65]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert.arXiv preprint arXiv:1904.09675 (2019)

    Pith/arXiv arXiv 2019

  66. [66]

    Xuejun Zhang, Xia Hou, Xiuming Qiao, and Wenfeng Song. 2024. A review of automatic source code summarization. Empirical Software Engineering 29, 6 (2024), 162

    2024

  67. [67]

    Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. 2020. Graph neural networks: A review of methods and applications.AI open 1 (2020), 57–81

    2020

  68. [68]

    Yu Zhou, Juanjuan Shen, Xiaoqing Zhang, Wenhua Yang, Tingting Han, and Taolue Chen. 2022. Automatic source code summarization with graph attention networks.Journal of Systems and Software 188 (2022), 111257. , Vol. 1, No. 1, Article . Publication date: February 2026

    2022

  69. [2024]

    Source Code Summarization in the Era of Large Language Models.arXiv preprint arXiv:2407.07959 (2024)

    Pith/arXiv arXiv 2024

  70. [2025]

    IEEE Transactions on Software Engineering (2025)

    OntheEffectivenessofLLM-as-a-judgeforCodeGenerationandSummarization. IEEE Transactions on Software Engineering (2025)

    2025