Pith. sign in

REVIEW 4 major objections 6 minor 54 references

Simple and Effective Baselines for Code Summarisation Evaluation

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that directly prompting a large language model to rate a code summary—showing it the code and optionally a reference—produces scores that correlate with human judgments as well as the best existing metrics, and that the…

desk verdict Useful reference-free LLM-as-judge baselines for code summarization, honestly reported, but the abstract's 'consistently better' outruns the significance tests and the prompt-tuning confound. read the letter →

arxiv 2505.19392 v1 pith:DXVGJA63 submitted 2025-05-26 cs.CL cs.AIcs.SE

Jade Robinson , Jonathan K. Kummerfeld This is my paper
classification cs.CLcs.AIcs.SE
keywords codesummarisationevaluationmetricsLLM-as-judgereference-freeSpearmancorrelationembeddingsimilarityhumann-gram
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a simple evaluation baseline for code summarisation: instead of comparing generated summaries to references with n-gram or embedding similarity, just ask a large language model to rate the summary, giving it the code (and optionally the reference). On two human-judged datasets, the best LLM-based metric reaches a Spearman correlation of 0.47 with overall quality scores, slightly ahead of the strongest embedding baseline (0.43), and the reference-free variant performs just as well as the reference-based one. If correct, the payoff is a cheap, consistent evaluation signal that does not need a human-written reference, which could be used to flag low-quality documentation in real code bases or to guide summary generation. The authors recommend pairing it with embedding-based methods to hedge against the risk that an LLM favors its own output.

What carries the argument

The central object is the Ask-LLM-Directly prompt, a composition of a professional-software-engineer role statement, an evaluation statement such as 'Independent of other factors, I feel the new summary is accurate', the reference summary (omitted in the no-ref variant), the function's source code, and a chain-of-thought instruction to show evaluation steps before answering on a four-point agree/disagree scale. The prompt's decisive feature is that the LLM sees the code, so its rating can reflect whether the summary is true to the function rather than merely similar to a reference. Scores are the LLM's discrete ratings, and the evaluation measures their Spearman rank correlation with mean human ratings, following Haque et al.'s methodology, with significance assessed by permutation tests adapted from Deutsch et al. (2021b).

What would settle it

Re-run the Ask-LLM metric with prompts tuned independently for each model (Claude, GPT-4o, OLMo-2) on the Roy et al. Overall Score data; if ask-claude no longer beats voyage-code-3 and the other LLMs, the reported advantage is a prompt-tuning artifact rather than a property of directly asking LLMs.

Watch

Extended reading notes

Core claim

The paper's central claim is that directly prompting an LLM to give an overall score to a code summary, conditioning on the code itself and optionally on a human-written reference, produces scores that correlate with human judgments at least as well as the best existing automatic metrics, and that dropping the reference does not measurably hurt. On Roy et al.'s Overall Score, ask-claude achieves 0.47 Spearman, above voyage-code-3's 0.43, while ask-claude-no-ref reaches 0.46; the difference between the reference-based and reference-free variants is not statistically significant. On Haque et al.'s Similarity, embeddings remain better, but the ask-LLM metrics sit in the same range as n-gram metrics. The paper also reports that the approach handles low-quality reference summaries gracefully, because it reads the code rather than relying on similarity to a possibly bad reference.

Load-bearing premise

The central claim assumes the human ratings are a valid measure of summary quality; a quieter fragility is that the prompt was tuned on the same model that reports the best score, which could inflate ask-claude's correlation.

Editorial extensions

If this is right

  • Reference-free evaluation becomes practical: documentation in a code base can be scored without a gold reference, enabling large-scale flagging of low-quality comments.
  • Because ask-claude and voyage-code-3 are not statistically distinguishable on overall quality, either can serve as the primary automatic metric, and the paper recommends using the two together during development before final human evaluation.
  • The approach is stable across Java and Python, where BLEU-A and SIDE degrade on Python, suggesting it may transfer to other languages better than trained or n-gram baselines.
  • A caution follows from the paper's own analysis: an LLM may inflate scores for summaries produced by the same model, with Claude giving its own output the highest rating in 92.7% of cases, so single-metric reliance is risky.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A neutral-prompt replication is the natural next experiment: tune the prompt separately for GPT-4o and OLMo-2 and re-run the Roy et al. comparison; if ask-claude's edge shrinks, the reported model ranking is a tuning artifact rather than a property of the method.
  • The reference-free variant's stability suggests an unexplored use: scoring documentation quality in repositories that never had gold summaries, which the paper mentions but does not measure.
  • Because the LLM metric fixates on a holistic adequacy impression and resists redirection to conciseness or fluency, a composite score will likely require separate probes per dimension rather than one prompt with different instructions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a new family of automatic metrics for code summarisation evaluation: directly prompting an LLM to rate a generated summary given the code, with an optional reference summary. The authors compare ask-LLM (Claude, GPT, OLMo) and ask-LLM-no-ref variants against n-gram metrics (BLEU-A, METEOR, ROUGE-L), a trained metric (SIDE), and embedding-based metrics (SentenceBERT, gte-base-en, voyage-code-3), measuring Spearman correlation with human ratings from Roy et al. (2021) Overall Score and Haque et al. (2022) Similarity, plus additional quality-dimension datasets (Gao et al., 2023; Su et al., 2024). They report that ask-claude achieves the highest Overall Score correlation (0.47), that the reference-free variant is statistically indistinguishable from the reference-based variant, and that a combination with embedding-based metrics is recommended to mitigate LLM self-preference bias.

Significance. The paper's strength is its breadth: it evaluates a wide range of baselines, uses permutation tests with Bonferroni correction, reports p-values for key comparisons, and documents costs and biases. The reference-free variant is genuinely novel for code summarisation and could be practically useful for screening documentation quality. The authors also release reformatted versions of the datasets, which is a useful community resource. However, the headline claim that the method is 'consistently better than prior metrics' is not supported by the reported significance tests or by the appendix results on the Su et al. dataset, and the prompt-selection procedure confounds the main comparison. With more careful claim framing and a prompt-neutral or per-model prompt evaluation, this would be a solid empirical contribution.

major comments (4)
  1. [Section 7 (Conclusion); Tables 1, 6, 7] The unqualified claim in the Conclusion, 'Our approach is consistently better than prior metrics,' is contradicted by results elsewhere in the manuscript. On Haque et al. Similarity (Table 1), ask-claude-no-ref (0.61) is significantly worse than voyage-code-3 (0.81), with p=0.0001 in Table 7. On Su et al. Informativeness (Table 6), ask-claude (0.23) and ask-claude-no-ref (0.28) are below SIDE (0.45), gte-base-en (0.35), and voyage-code-3 (0.34). Even on Roy et al. Overall Score, the best ask-LLM result (ask-claude, 0.47) is not statistically significantly different from voyage-code-3 (0.43), with p=0.1035 in Table 7. The abstract's 'as good or better' is also too strong; the evidence supports at most 'comparable on Overall Score, with no significant advantage over the best embedding metric.'
  2. [Section 6 (Different LLMs may perform differently); Appendix E.1] The prompt used for ask-claude was selected by testing many variants on Claude against the Gao et al. training set (Appendix E.1, Table 9, 'Final Method'), and the same prompt was then applied to ask-gpt and ask-OLMo. As the paper acknowledges in Section 6, this makes the comparison between LLM evaluators unfair. More importantly, it confounds the central comparison against fixed embedding baselines: ask-claude's numerical advantage over voyage-code-3 (0.47 vs 0.43 on Overall Score) could reflect the prompt's adaptation to Claude rather than a general property of the ask-LLM approach. To support the claim that asking an LLM directly is at least as good as the best embeddings, please report results with a prompt that was not tuned on the evaluated model (e.g., a fixed neutral prompt applied uniformly, or per-model prompt development done on a held-out split and then evaluated on the test data). Without such an experiment, the method-level comparison remains biased.
  3. [Section 6 (ask-LLM-no-ref is just as effective); Tables 1 and 7] The reference-free variant is a core contribution, but the evidence for 'ask-LLM-no-ref is just as effective' is limited to one model under significance testing. In Table 7, the only p-values reported for the with/without-reference comparison are for Claude (ask-claude vs ask-claude-no-ref: p=0.6779 on Overall Score, p=0.3585 on Similarity). For ask-OLMo, the correlations in Table 1 (0.35 vs 0.36) are reported without a significance test, and ask-gpt-no-ref is missing from Table 1 entirely. Please either provide significance tests (or at least bootstrap confidence intervals) for all models with a no-ref variant, or restrict the claim to Claude.
  4. [Section 6.1 (Model Sensitivity); Appendix F] The documented self-preference is large: in Appendix F, Claude gives its own generated summaries the highest possible rating in 92.7% of cases. This directly bears on the claimed usefulness of the reference-free variant for evaluating summaries in a code base (Section 4), since in that setting the evaluator LLM is likely to be the same as the generator. The paper does not provide any human ratings for Claude-generated summaries, so it is unknown whether ask-claude-no-ref's correlation with human judgments persists when the evaluator and generator are the same model. Please either collect human ratings for same-model summaries and measure correlation, or explicitly scope the reference-free claim to cross-model evaluation and present the combination with embedding metrics as a required condition rather than a precaution.
minor comments (6)
  1. [Throughout] There are several typos: 'Automatic metrics address are cheap and consistent' in Section 1 should be 'Automatic metrics are cheap and consistent'; 'langauges' in the Limitations section; 'valuated' in Section 6.1; 'enabler' in the Conclusion.
  2. [References] The reference list contains duplicate entries: 'Nguyen et al., 2024a' and '2024b' both cite the same paper 'HierarchyNet: Learning to summarize source code with heterogeneous representations'; please merge them.
  3. [Appendix B, Table 7] Table 7 is difficult to read because the column headers span two system-pair comparisons and the layout does not clearly align each p-value with the compared metrics; please restructure the table so each row names the two metrics being compared explicitly.
  4. [Appendix A] Appendix A states that 'the Ask-LLM-Directly approaches with Claude consistently perform better than existing metrics' before presenting the adequacy/accuracy tables; this wording should be scoped to those tables, since the Su et al. informativeness results in Table 6 show SIDE outperforming all ask-LLM variants, and the appendix itself notes that significance tests were not run for these comparisons.
  5. [Section 5.1] The code-summary matching process is described as 'somewhat involved'; please add a short validation description (e.g., number of manual checks, matching accuracy, handling of ambiguous cases) to support the reliability of the released datasets.
  6. [Table 1] The note that new metrics are in italics is not visible in the submitted text; please ensure the formatting appears in the camera-ready version or use a symbol/footnote instead.

Circularity Check

1 steps flagged · score 3.0 of 10

Minor in-sample prompt selection affects the Gao adequacy claim; the central Roy et al. result is independent.

  1. fitted input called prediction [Appendix E.1 / Table 10, Table 4, Section 5.1]
    "Table 10 is headed 'Variants tested for Ask-Claude: Spearman's Correlation with Adequacy, Conciseness and Fluency on Gao et al. training dataset' and the 'Final Method' row reads '0.64 0.32 0.43'. Section 5.1 states: 'To avoid overfitting, during development we used a subset of the data. For the final results we used all of the data with 10-fold cross-validation.'"

    The 'Final Method' prompt (Figure 3) was selected from the Table 10 variants by taking the highest Adequacy correlation (0.64) on the Gao et al. training set. The same prompt then appears in Table 4 as ask-claude with Gao et al. Adequacy 0.61 (Java) and 0.62 (Python), and Appendix A uses this to conclude that 'the Ask-LLM-Directly approaches with Claude consistently perform better than existing metrics.' Since the final all-data evaluation includes the development subset used for prompt selection, the reported Gao Adequacy numbers are in-sample fitted values, not independent predictions. The central Overall Score result (Roy et al.) was not used for prompt selection, so this circularity is limited to the Gao-based adequacy claim.

full rationale

The paper's headline result is an LLM-as-judge correlation with external human ratings, so there is no definitional equivalence between the metric and the evaluation target. No self-citation chain or imported uniqueness theorem is load-bearing. The only concrete reduction is in the appendix: the 'Final Method' prompt was chosen on the Gao et al. training data, and the Gao adequacy correlations in Table 4 are reported from the same dataset, making that specific comparison partly fitted. The paper also discloses that the prompt was tuned on Claude, which is a cross-model confound rather than a circular step; the Roy et al. Overall Score comparison (ask-claude 0.47 vs voyage-code-3 0.43, p=0.1035) is not significant, so 'consistently better' overstates the evidence, but overstatement is not circularity. Overall, the central claim has independent empirical content, and the circularity is confined to a secondary appendix result.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

No fitted parameters, new axioms, or invented entities. The paper is an empirical evaluation of an existing technique applied to a new domain. The core assumptions are domain assumptions: human ratings are treated as ground truth, LLM outputs are stochastic but representative, and the datasets measure the relevant notion of quality.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Simple and Effective Baselines for Code Summarisation Evaluation." pith.science (2026). https://pith.science/paper/DXVGJA63

@misc{pith2026250519392,
  author       = {Pith},
  title        = {Pith review of: Simple and Effective Baselines for Code Summarisation Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DXVGJA63}},
  note         = {Machine review of arXiv:2505.19392}
}
read the original abstract

Code documentation is useful, but writing it is time-consuming. Different techniques for generating code summaries have emerged, but comparing them is difficult because human evaluation is expensive and automatic metrics are unreliable. In this paper, we introduce a simple new baseline in which we ask an LLM to give an overall score to a summary. Unlike n-gram and embedding-based baselines, our approach is able to consider the code when giving a score. This allows us to also make a variant that does not consider the reference summary at all, which could be used for other tasks, e.g., to evaluate the quality of documentation in code bases. We find that our method is as good or better than prior metrics, though we recommend using it in conjunction with embedding-based methods to avoid the risk of LLM-specific bias.

Figures

Figures reproduced from arXiv: 2505.19392 by the authors.

Figure 1
Figure 1. Correlation with Adequacy by Reference Quality on the Haque et al. dataset all of the references were individually written and validated as part of another study. Here we see that SIDE actually outperforms all other metrics, but the other commonly used metrics such as the n-gram metrics and the SentenceBERT embeddings-based metric perform particularly poorly, with METEOR even giving a negative correlation. The new e… view at source ↗
Figure 2
Figure 2. Example from Roy et al. (Note: this is a particularly short example) D Dataset Statistics [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Ask LLM Directly Final Prompt • JAM-FT (Su et al., 2024) Fine-tuned version of JAM (Su et al., 2023) based on the outputs of GEMINI-CONTEXT. E Other Variations E.1 Prompt Variations [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Question Answering Prompt for Question Generation Step E.2.2 Variants We tested many different variants on the Gao et al. dataset, varying seven different aspects of the pro￾cess. The combinations tested are provided in Ta￾ble 12, and the results are in [PITH_FULL_IMA…
Figure 5
Figure 5. Figure 5: Prompt given to Claude for Summary Genera [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 6
Figure 6. Figure 6: Ask-Claude scores for summaries generated [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Relative Rankings of Each Model by Metric [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

54 extracted references · 16 canonical work pages

  1. [1]

    Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2020. https://doi.org/10.18653/v1/2020.acl-main.449 A transformer-based approach for source code summarization . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4998--5007, Online. Association for Computational Linguistics

    doi:10.18653/v1/2020.acl-main.449 2020

  2. [2]

    Toufique Ahmed, Kunal Suresh Pai, Premkumar Devanbu, and Earl Barr. 2024. https://doi.org/10.1145/3597503.3639183 Automatic semantic augmentation of language model prompts (for code summarization) . In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ICSE '24, New York, NY, USA. Association for Computing Machinery

    arXiv 2024

  3. [3]

    Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. 2019. https://openreview.net/forum?id=H1gKYo09tX code2seq: Generating sequences from structured representations of code . In International Conference on Learning Representations

    work page 2019

  4. [4]

    David Anugraha, Garry Kuwanto, Lucky Susanto, Derry Tanti Wijaya, and Genta Winata. 2024. https://doi.org/10.18653/v1/2024.wmt-1.32 M eta M etrics- MT : Tuning meta-metrics for machine translation via human preference calibration . In Proceedings of the Ninth Conference on Machine Translation, pages 459--469, Miami, Florida, USA. Association for Computati...

    doi:10.18653/v1/2024.wmt-1.32 2024

  5. [5]

    Satanjeev Banerjee and Alon Lavie. 2005. https://aclanthology.org/W05-0909 METEOR : An automatic metric for MT evaluation with improved correlation with human judgments . In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization , pages 65--72, Ann Arbor, Michigan. Association for Comput...

    2005

  6. [6]

    Aakash Bansal, Robert Wallace, Zachary Karas, Ningzhi Tang, Yu Huang, Toby Jia-Jun Li, and Collin McMillan. 2024. https://arxiv.org/abs/2405.18573 Programmer visual attention during context-aware code summarization . Preprint, arXiv:2405.18573

    work page Pith review arXiv 2024

  7. [7]

    Yufan Cai, Yun Lin, Chenyan Liu, Jinglian Wu, Yifan Zhang, Yiming Liu, Yeyun Gong, and Jin Song Dong. 2024. https://proceedings.neurips.cc/paper_files/paper/2023/hash/b16e6de5fbbdcb2df237aa66b302bc17-Abstract-Conference.html On-the-fly adapting code summarization on trainable cost-effective language models . In Proceedings of the 37th International Confer...

    work page 2024

  8. [8]

    de Souza, Nicolas Anquetil, and Káthia M

    Sergio Cozzetti B. de Souza, Nicolas Anquetil, and Káthia M. de Oliveira. 2006. https://doi.org/10.1007/BF03194494 Which documentation for software maintenance? Journal of the Brazilian Computer Society, 12(3):31--44

    doi:10.1007/bf03194494 2006

Show all 54 references
  1. [9]

    Daniel Deutsch, Tania Bedrax-Weiss, and Dan Roth. 2021 a . https://doi.org/10.1162/tacl_a_00397 Towards question-answering as an automatic metric for evaluating the content quality of a summary . Transactions of the Association for Computational Linguistics, 9:774--789

    2021 doi

  2. [10]

    Daniel Deutsch, Rotem Dror, and Dan Roth. 2021 b . https://doi.org/10.1162/tacl_a_00417 A statistical analysis of summarization evaluation metrics using resampling methods . Transactions of the Association for Computational Linguistics, 9:1132--1146

    2021 doi

  3. [11]

    Daniel Deutsch, Rotem Dror, and Dan Roth. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.753 On the limitations of reference-free evaluations of generated text . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10960--10977, A...

    2022 doi

  4. [12]

    Alexander Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. https://doi.org/10.18653/v1/2022.naacl-main.187 QAF act E val: Improved QA -based factual consistency evaluation for summarization . In Proceedings of the 2022 Conference of the North American Chapter of th...

    2022 doi

  5. [13]

    Markus Freitag, Nitika Mathur, Daniel Deutsch, Chi-Kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Frederic Blain, Tom Kocmi, Jiayi Wang, David Ifeoluwa Adelani, Marianna Buchicchio, Chrysoula Zerva, and Alon Lavie. 2024. https://doi.org/10.18653/v1/2024.wmt-1.2 Ar...

    2024 doi

  6. [14]

    Shuzheng Gao, Cuiyun Gao, Yulan He, Jichuan Zeng, Lunyiu Nie, Xin Xia, and Michael Lyu. 2023. https://doi.org/10.1145/3522674 Code structure–guided transformer for source code summarization . ACM Trans. Softw. Eng. Methodol., 32(1):Article 23

    2023 doi

  7. [15]

    Sakib Haque, Zachary Eberhart, Aakash Bansal, and Collin McMillan. 2022. https://doi.org/10.1145/3524610.3527909 Semantic similarity metrics for evaluating source code summarization . In Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension, page 3...

    2022

  8. [16]

    Sakib Haque, Alexander LeClair, Lingfei Wu, and Collin McMillan. 2020. https://doi.org/10.1145/3379597.3387449 Improved automatic summarization of subroutines via attention to file context . In Proceedings of the 17th International Conference on Mining Software Repositories, M...

    2020

  9. [17]

    Hellendoorn, Charles Sutton, Rishabh Singh, Petros Maniatis, and David Bieber

    Vincent J. Hellendoorn, Charles Sutton, Rishabh Singh, Petros Maniatis, and David Bieber. 2020. https://api.semanticscholar.org/CorpusID:213352113 Global relational models of source code . In International Conference on Learning Representations

    2020

  10. [18]

    X. Hu, G. Li, X. Xia, D. Lo, and Z. Jin. 2018 a . https://xin-xia.github.io/publication/icpc182.pdf Deep code comment generation . In 2018 IEEE/ACM 26th International Conference on Program Comprehension (ICPC), pages 200--20010

    2018

  11. [19]

    Xing Hu, Ge Li, Xin Xia, David Lo, Shuai Lu, and Zhi Jin. 2018 b . https://doi.org/10.24963/ijcai.2018/314 Summarizing source code with transferred api knowledge . In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pages 2...

    2018 doi

  12. [20]

    Xu Huang, Zhirui Zhang, Xiang Geng, Yichao Du, Jiajun Chen, and Shujian Huang. 2024. https://doi.org/10.18653/v1/2024.findings-acl.211 Lost in the source language: How large language models evaluate the quality of machine translation . In Findings of the Association for Comput...

    2024 doi

  13. [21]

    Tom Kocmi and Christian Federmann. 2023. https://aclanthology.org/2023.eamt-1.19 Large language models are state-of-the-art evaluators of translation quality . In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 193--203, Tam...

    2023

  14. [22]

    Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. 2024. https://aclanthology.org/2024.findings-acl.29 Benchmarking cognitive biases in large language models as evaluators . In Findings of the Association for Computational Linguistics ACL 2024...

    2024

  15. [24]

    Alexander LeClair, Siyuan Jiang, and Collin McMillan. 2019. https://doi.org/10.1109/icse.2019.00087 A neural model for generating natural language summaries of program subroutines . In Proceedings of the 41st International Conference on Software Engineering, page 795–806. IEEE Press

    2019

  16. [25]

    Zongjie Li, Chaozheng Wang, Pingchuan Ma, Chaowei Liu, Shuai Wang, Daoyuan Wu, Cuiyun Gao, and Yang Liu. 2024. https://doi.org/10.1145/3597503.3639091 On extracting specialized code abilities from large language models: A feasibility study . In Proceedings of the IEEE/ACM 46th...

    2024

  17. [26]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013 ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  18. [27]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.153 G - E val: NLG evaluation using GPT -4 with better human alignment . In Proceedings of the 2023 Conference on Empirical Methods in Natural Langua...

    2023 doi

  19. [28]

    Yingjie Mao, Xiao Li, Zongwei Li, and Wenkai Li. 2024. https://api.semanticscholar.org/CorpusID:267523444 Automated smart contract summarization via LLM s . ArXiv, abs/2402.04863

    2024 arXiv

  20. [29]

    Mastropaolo, M

    A. Mastropaolo, M. Ciniselli, M. Di Penta, and G. Bavota. 2024. http://doi.ieeecomputersociety.org/ Evaluating code summarization techniques: A new metric and an empirical characterization . In 2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE), pages 1...

    2024

  21. [30]

    Vishal Misra, Jakku Sai Krupa Reddy, and Sridhar Chimalakonda. 2020. https://doi.org/10.1145/3341105.3374009 Is there a correlation between code comments and issues? A n exploratory study . In SAC '20: Proceedings of the 35th Annual ACM Symposium on Applied Computing

    2020

  22. [31]

    Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. https://doi.org/10.18653/v1/2023.eacl-main.148 MTEB : Massive text embedding benchmark . In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages...

    2023 doi

  23. [32]

    Minh Nguyen, Nghi Bui, Truong Son Hy, Long Tran-Thanh, and Tien Nguyen. 2024 a . https://aclanthology.org/2024.findings-eacl.156 H ierarchy N et: Learning to summarize source code with heterogeneous representations . In Findings of the Association for Computational Linguistics...

    2024

  24. [33]

    Minh Nguyen, Nghi Bui, Truong Son Hy, Long Tran-Thanh, and Tien Nguyen. 2024 b . https://aclanthology.org/2024.findings-eacl.156 H ierarchy N et: Learning to summarize source code with heterogeneous representations . In Findings of the Association for Computational Linguistics...

    2024

  25. [34]

    Xinglu Pan, Chenxiao Liu, Yanzhen Zou, Xianlin Zhao, and Bing Xie. 2024. https://doi.org/10.1109/COMPSAC61105.2024.00177 Context-focused prompt tuning pre-trained code models to improve code summarization . In 2024 IEEE 48th Annual Computers, Software, and Applications Confere...

    2024 arXiv

  26. [35]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 BLEU : a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311--3...

    2002

  27. [36]

    Shenbin Qian, Archchana Sindhujan, Minnie Kabra, Diptesh Kanojia, Constantin Orasan, Tharindu Ranasinghe, and Fred Blain. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.214 What do large language models need for machine translation evaluation? In Proceedings of the 2024 Con...

    2024 doi

  28. [37]

    Ricardo Rei, Ana C Farinha, Chrysoula Zerva, Daan van Stigt, Craig Stewart, Pedro Ramos, Taisiya Glushkova, Andr \'e F. T. Martins, and Alon Lavie. 2021. https://aclanthology.org/2021.wmt-1.111 Are references really needed? unbabel- IST 2021 submission for the metrics shared t...

    2021

  29. [38]

    Nils Reimers and Iryna Gurevych. 2019. https://doi.org/10.18653/v1/D19-1410 Sentence- BERT : Sentence embeddings using S iamese BERT -networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference...

    2019 doi

  30. [39]

    Devjeet Roy, Sarah Fakhoury, and Venera Arnaoudova. 2021. https://doi.org/10.1145/3468264.3468588 Reassessing automatic evaluation metrics for code summarization tasks . In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on t...

    2021

  31. [40]

    Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.529 Q uest E val: Summarization asks for fact-based evaluation . In Proceedings of the 2021 Conference o...

    2021 doi

  32. [41]

    Stapleton, Y

    S. Stapleton, Y. Gambhir, A. LeClair, Z. Eberhart, W. Weimer, K. Leach, and Y. Huang. 2020. https://doi.org/10.1145/3387904.3389258 A human study of comprehension and code summarization . In 2020 IEEE/ACM 28th International Conference on Program Comprehension (ICPC), pages 01--12

    2020

  33. [42]

    Chia-Yi Su, Aakash Bansal, Yu Huang, Toby Jia-Jun Li, and Collin McMillan. 2024. https://arxiv.org/abs/2408.09006 Context-aware code summary generation . Preprint, arXiv:2408.09006

    2024 arXiv

  34. [43]

    Chia-Yi Su, Aakash Bansal, Vijayanta Jain, Sepideh Ghanavati, and Collin McMillan. 2023. https://doi.org/10.1145/3611643.3613090 A language model of J ava methods with train/test deduplication . In Proceedings of the 31st ACM Joint European Software Engineering Conference and ...

    2023

  35. [44]

    Chia-Yi Su and Collin McMillan. 2024. https://doi.org/10.1007/s10515-024-00421-4 Distilled GPT for source code summarization . Automated Software Engineering, 31(1):22

    2024 doi

  36. [45]

    Weisong Sun, Chunrong Fang, Yuchen Chen, Quanjun Zhang, Guanhong Tao, Yudu You, Tingxu Han, Yifei Ge, Yuling Hu, Bin Luo, and Zhenyu Chen. 2024. https://doi.org/10.1145/3632742 An extractive-and-abstractive framework for source code summarization . ACM Trans. Softw. Eng. Metho...

    2024 doi

  37. [46]

    Gomez, ukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf Attention is all you need . In Proceedings o...

    2017

  38. [47]

    VoyageAI. 2024. voyage-code-3: more accurate code retrieval with lower dimensional, quantized embeddings. https://blog.voyageai.com/2024/12/04/voyage-code-3/#: :text=voyage Accessed: 2024-12-10

    2024

  39. [48]

    Yao Wan, Zhou Zhao, Min Yang, Guandong Xu, Haochao Ying, Jian Wu, and Philip S. Yu. 2018. https://doi.org/10.1145/3238147.3238206 Improving automatic source code summarization via deep reinforcement learning . In Proceedings of the 33rd ACM/IEEE International Conference on Aut...

    2018

  40. [49]

    Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. https://doi.org/10.18653/v1/2020.acl-main.450 Asking and answering questions to evaluate the factual consistency of summaries . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 500...

    2020 doi

  41. [50]

    Minghao Wu and Alham Fikri Aji. 2023. https://arxiv.org/abs/2307.03025 Style over substance: Evaluation biases for large language models . Preprint, arXiv:2307.03025

    2023 arXiv

  42. [51]

    Kun Xu, Lingfei Wu, Zhiguo Wang, Yansong Feng, Michael Witbrock, and Vadim Sheinin. 2018. https://arxiv.org/abs/1804.00823 Graph2seq: Graph to sequence learning with attention-based neural networks . Preprint, arXiv:1804.00823

    2018 arXiv

  43. [52]

    Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, Meishan Zhang, Wenjie Li, and Min Zhang. 2024. https://doi.org/10.48550/arXiv.2407.19669 m GTE : Generalized long-context text representation and reranking ...

    2024 doi arXiv

  44. [53]

    Junjie Zhao, Xiang Chen, Guang Yang, and Yiheng Shen. 2024. https://doi.org/10.1016/j.infsof.2024.107405 Automatic smart contract comment generation via large language models and in-context learning . Information and Software Technology, 168:107405

    2024

  45. [54]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2024. https://api.semanticscholar.org/CorpusID:259129398 Judging LLM -as-a-judge with MT -bench a...

    2024

  46. [55]

    Daniel Zügner, Tobias Kirschstein, Michele Catasta, Jure Leskovec, and Stephan Günnemann. 2021. https://iclr.cc/virtual/2021/poster/2838 Language-agnostic representation learning of source code from structure and context . In International Conference on Learning Representations

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.