Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Large Language Models are Qualified Benchmark Builders: Rebuilding Pre-Training Datasets for Advancing Code Intelligence Tasks

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that replacing human-written code comments with LLM-generated comments in pre-training data improves code models on natural-language tasks, and proposes two reference-free metrics to establish the comparison.

desk verdict The reference-free evaluation idea is the real contribution; the downstream RQ3 claim overstates what the controlled comparisons actually show. read the letter →

arxiv 2504.19444 v1 pith:VJYG3PL5 submitted 2025-04-28 cs.SE cs.CL

Kang Yang , Xinjun Mao , Shangwen Wang , Yanlin Wang , Tanghaoran Zhang , Bo Lin , Yihao Qin , Zhang Zhang
show 2 more authors
Yao Lu Kamal Al-Sabahi
This is my paper · ORCID
classification cs.SEcs.CL
keywords codecommentqualityreference-freeevaluationcode-commentinconsistencydetectionsemanticsearchpre-trainingdatasetrebuildingLLM-generatedcommentssummarizationintelligence
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the human-written comments used to pre-train code models are not the gold standard they are assumed to be, and that comments generated by large language models can serve as a better bridge between code and natural language. It introduces two reference-free evaluation tasks—flagging code-comment mismatches and using comments as search queries—to compare comment quality directly, and finds LLM-generated comments look more consistent with code than human comments. On that basis, it replaces the human comments in a widely used code-comment corpus with LLM-generated ones, re-trains a standard pre-trained code model, and reports gains on code summarization, natural-language-to-code generation, and code translation, while code refinement and clone detection are unaffected. The upshot is a challenge to the routine reliance on human reference comments and a recipe for rebuilding pre-training data with LLMs.

What carries the argument

Two auxiliary reference-free evaluation tasks carry the argument. The first, an inconsistency rate, uses a trained code-comment inconsistency detector to measure what fraction of code-comment pairs are semantically mismatched, with lower values indicating better comments. The second, a retrieval rank score, uses a semantic code-search model with each comment as a query and measures where the correct code snippet ranks among distractors, with higher values indicating better comments. Both are used to compare comment quality without treating human comments as the gold standard. The same comparison then justifies rebuilding the pre-training corpus: the paper generates over two million comments with one LLM and re-pre-trains a widely used encoder-decoder code model on the resulting pairs.

What would settle it

Rewrite a sample of human-written Javadoc comments into the same one-sentence, API-free summary format that the LLM prompt produces, re-run the inconsistency detector and semantic search on the style-matched comments, and check whether their scores on the two reference-free measures become comparable to LLM-generated comments. If they do, the claimed LLM superiority is a style artifact of the metrics rather than a semantic-quality difference.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the semantic link between code and natural language in pre-training data can be improved by discarding human-written comments and replacing them with one-sentence comments generated by a large language model. On two reference-free measures—how often a trained classifier labels a code-comment pair as inconsistent, and how well the comment retrieves its own code in a semantic search—LLM-generated comments are far less inconsistent and far more retrievable than the original human comments, with inconsistency rates below four percent versus roughly fifteen percent and retrieval scores up to about 0.97 versus 0.82. Re-pre-training an encoder-decoder code model on the rebuilt corpus improves downstream code summarization, natural-language-to-code generation, and code translation, while tasks driven by code structure alone, such as refinement and clone detection, show no meaningful change. The paper reads this as evidence that large language models are qualified builders of code-intelligence benchmarks and that the field's reliance on human reference comments should be reconsidered.

Load-bearing premise

The evaluation assumes that the two reference-free measures judge semantic consistency fairly rather than simply preferring the short, API-agnostic one-sentence style that LLMs output over the detailed Javadoc style that humans write.

Editorial extensions

If this is right

  • Pre-training corpora can be improved by replacing human-written comments with LLM-generated ones, since semantic consistency between code and natural language is the active ingredient.
  • Natural-language-heavy downstream tasks inherit the gain, while structure-only tasks do not, which demarcates where comment quality actually matters.
  • Reference-free metrics are necessary for comparing comment quality at all, because reference-based metrics assume the human reference is already the gold standard.
  • Rebuilding the dataset should be treated as a component of the model-building pipeline, on par with architecture and pre-training objective choices.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same rebuilding logic likely transfers to other code-natural-language paired resources, such as API documentation, notebooks, and code search queries, since the mechanism is semantic consistency rather than the specific corpus.
  • Editorial inference: a testable consequence is that the gains should be largest where pre-training comments are stale or noisy; if a corpus is already clean, LLM replacement may be neutral or even harmful, and the paper's null results on structure-only tasks already suggest a ceiling.
  • Editorial inference: style-matching experiments, where human comments are rewritten in the same one-sentence, API-free format the LLM prompt produces, would isolate whether the advantage is semantic or stylistic; if style is the driver, the conclusion should be reinterpreted as "LLM style suits the metrics" rather than "LLM comments are better."
  • Editorial inference: the inconsistency detector could be repurposed as a data-cleaning filter, flagging human comments before they ever enter a pre-training corpus.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper proposes two reference-free metrics—code-comment inconsistency detection (IncRate) and semantic code search (MRR)—to compare LLM-generated comments with human-written reference comments. The authors report that LLM-generated comments are more semantically consistent with code than human references, supported by automatic evaluation and a 357-sample human study, and then act on this finding by rebuilding CodeSearchNet with GPT-3.5-Turbo-generated comments (cgpt-CSN). They pre-train CodeT5 on this rebuilt corpus and evaluate it on code summarization, NL-code generation, code translation, code refinement, and clone detection, concluding that LLM-rebuilt pre-training data significantly improves NL-related code intelligence tasks and also enhances code translation.

Significance. If the central claim were established, the paper would make a useful contribution: it provides a large rebuilt pre-training corpus (about 2.07M PL-NL pairs), a re-implementation of CodeT5 pre-training, and a concrete proposal to replace human-written comments with LLM-generated ones in pre-training datasets. The human evaluation and the scale of the data-rebuilding effort are strengths. However, the downstream evaluation as currently reported does not isolate the effect of the rebuilt pre-training data: the largest gains come from also changing the fine-tuning data and the test references or inputs, and no variance or significance testing is reported. The reference-free metrics may also be format-biased in favor of one-sentence LLM comments. With additional controlled experiments and more cautious claims, this could become a solid empirical study; in its current form, the evidence is insufficient for the abstract's strong conclusion.

major comments (4)
  1. [Section IV-B, Table IV] The USE references for the cgpt-CSN-Sum rows are GPT-3.5-Turbo-generated comments, as the paper itself states. The model fine-tuned on GPT-3.5-Turbo comments is then scored against GPT-3.5-Turbo references, so the large USE increases (e.g., average 0.5941 to 0.7777 for CSN pre-training) largely reflect distribution matching rather than improved summarization quality. The pre-training-only comparison in the same table (CSN/CSN vs. cgpt-CSN/CSN) shows a much smaller effect (average USE 0.5941 vs. 0.6002). Please report the pre-training-only contrast on the original human-reference test set, or on both reference sets, and include variance or significance tests.
  2. [Section IV-B, Table V] The Concode NL inputs are replaced with GPT-3.5-Turbo-generated text to form cgpt-Concode, but the manuscript does not state whether the test split is also rebuilt. If the test inputs are rebuilt, the evaluation task changes: models are tested on GPT-paraphrased queries rather than original human NL queries, so the gain from 39.45 to 50.49 CodeBLEU cannot be attributed to the rebuilt pre-training corpus. The only pre-training-only contrast (CSN/Concode vs. cgpt-CSN/Concode) is +1.11 CodeBLEU and +0.10 EM, which is not shown to be significant. Please evaluate on the original Concode test set and explicitly state which splits were rebuilt, and report error bars or significance tests.
  3. [Section III-B (RQ2)] The two reference-free metrics may be systematically biased toward the LLM comment style. The CCID classifier (Section III-B1) was trained on commit-derived Javadoc labels from Panthaplackel et al., whereas the LLM comments are one-sentence prose generated with the prompt in Section III-C2; the classifier may flag structured Javadoc differently from one-sentence summaries. The MRR metric uses CodeBERT fine-tuned on CodeSearchNet human comments, so retrieval may favor query styles close to that distribution. Because RQ2 is the basis for rebuilding cgpt-CSN, please validate the two metrics on LLM-format comments or provide calibration evidence across comment formats.
  4. [Section IV-B, Tables VI-VIII] The code translation and negative-result claims are based on single runs without variance or significance testing. For instance, Java-to-Python EM changes from 2.57 to 2.60 (Table VI), and refinement results differ by less than 0.2 EM (Table VII), yet the text describes improvements as meaningful and the non-improvements as non-significant. Please add repeated-seed results or confidence intervals and either provide significance tests or substantially soften the significance language in the abstract and the RQ3 answer.
minor comments (5)
  1. [Table II and Section III-D] The metric name is inconsistent: the table header says "InRate" while Equation (1) and the surrounding text use "IncRate." Please unify the notation.
  2. [Section III-D3] There is a typo in the sentence "six LLMs achieve lower IncRate and higher MMR scores than all three DL baselines": MMR should be MRR.
  3. [Section IV-B, Clone Detection paragraph] The sentence "This none non-significant difference can be attributed..." is grammatically garbled; it should be rephrased.
  4. [Acknowledgments and Section III-C2] There are minor typos: "the the National Key Research and Development Program" in the footnote, and "tempreture" in the sampling-parameter description.
  5. [Section III-D2] In the reference-free metric analysis, "CSN performs the lowest MRR score" should presumably read "NCS performs the lowest MRR score."

Circularity Check

2 steps flagged · score 6.0 of 10

RQ3's headline NL-task gains are partly circular: code-summarization USE is scored against GPT-3.5-Turbo references and cgpt-Concode changes the test inputs, leaving only small pre-training-only effects.

  1. self definitional [Section IV-A2 (Code Intelligence Tasks and Metrics), code summarization paragraph; results in Table IV]
    "Note that the reference-based metric USE refers to the comment generated by GPT-3.5-Turbo, as it provides better consistency with code."

    The code summarization evaluation defines its USE reference as the GPT-3.5-Turbo comment. The rows that produce the large gains (CSN/cgpt-CSN-Sum and cgpt-CSN/cgpt-CSN-Sum in Table IV) fine-tune on exactly those GPT-3.5-Turbo comments, and the paper further states that the rebuilt CSN test set uses GPT-3.5-Turbo comments as the ground truth. The USE gap (0.7777 and 0.7822 versus 0.5941) therefore mostly measures the model's ability to emit GPT-3.5-Turbo-style text, not a benefit of the rebuilt pre-training corpus. MRR on the rebuilt test set has the same issue: the queries are GPT-generated comments and the winning models are trained on GPT-generated comments, so retrieval success reflects input-distribution matching.

  2. self definitional [Section IV-B Code Generation and Table V]
    "To investigate the impact of GPT-3.5-Turbo-generated comments in the fine-tuning stage, we also rebuild the Concode dataset by replacing the NL texts with GPT-3.5-Turbo-generated comments to form the updated inputs, as shown in Figure 3 and the updated version noted as cgpt-Concode."

    Table V's largest gains (CodeBLEU 39.45 to 50.49, EM 22.00 to 30.00) come from the cell where both pre-training data and fine-tuning inputs are GPT-3.5-Turbo text. The paper does not state whether the Concode test set was also rebuilt; if it was, the evaluation is on GPT-paraphrased NL queries rather than the original human NL task, and the gain is largely distribution matching between cgpt-Concode and cgpt-CSN. The controlled pre-training-only comparison (cgpt-CSN/Concode versus CSN/Concode) is only +1.11 CodeBLEU and +0.10 EM, and the paper itself attributes the larger part of the gain to the fine-tuning stage ('cgpt-Concode contributes more significant performance gains than pre-training').

full rationale

The first two research questions (RQ1/RQ2) are evaluated with external instruments: a CCID classifier trained on Panthaplackel et al.'s commit-pair data, CodeBERT code search, and an independent human evaluation. I find no circularity in that part of the paper. The circularity appears in RQ3's downstream demonstration. In code summarization, the USE ground truth is redefined as the GPT-3.5-Turbo comment, so the large USE gains of models fine-tuned on GPT-3.5-Turbo comments measure imitation of the GPT comment distribution rather than an improvement caused by the rebuilt pre-training corpus. In NL-code generation, the headline cell in Table V changes the fine-tuning inputs to the same GPT-3.5-Turbo text used in cgpt-CSN, while the pre-training-only contrast is only +1.11 CodeBLEU and +0.10 EM, so the 'significant' gain is largely a distribution-match artifact rather than an effect of the rebuilt pre-training data. The independent pre-training-only signals in Table IV (row 2), Table VI (code translation), and the small gain in Table V (cgpt-CSN with original Concode) give the paper some non-circular content, so the overall circularity is partial rather than total.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claims rest on proxy metrics and the commit-pair labeling rule rather than fitted constants. The reference-free metrics embed assumptions about what makes a comment consistent and about cross-format comparability; the LLM prompt imposes a one-sentence format on LLMs but not on human references. No new theoretical entities are introduced; cgpt-CSN is a dataset artifact, not a postulated entity.

free parameters (2)
  • LLM comment generation settings = max_tokens=30, temperature=1, top_p=1
    Chosen by hand in Section III-C2 to cap output length near human reference length; these choices affect the vocabulary and content of LLM comments and thus the quality comparison.
  • CodeT5 pre-training hyperparameters = batch size 48, peak LR 2e-5, 100 denoising epochs + 50 bimodal epochs
    Taken from CodeT5/CodeXGLUE defaults rather than tuned to downstream targets; they influence all downstream results but are not fitted to the reported test sets.
assumptions (4)
  • domain assumption A comment is high quality if a commit-trained CCID classifier does not flag it and if it retrieves its own code under CodeBERT.
    Section III-B1/2 formalizes IncRate and MRR as reference-free metrics; this equates quality with two proxy behaviors and assumes the proxies apply equally to human Javadoc and LLM one-sentence comments.
  • domain assumption Panthaplackel's commit-pair labeling rule (code changed without comment change implies consistency; comment change implies the old comment is inconsistent) yields valid labels.
    Section III-B1 cites Panthaplackel et al. [52]; this external labeling rule is the training signal for the CCID classifier and is not independently validated here.
  • domain assumption Differences in comment length, vocabulary size, and Javadoc structure do not bias the CCID and MRR comparisons.
    Table I shows systematic length and vocabulary differences between human references and LLM comments; the analysis treats these as irrelevant rather than as confounds.
  • ad hoc to paper A one-sentence natural-language description is the correct target format for comparing comment quality.
    The prompt in Section III-C2 ('write a one-sentence description') imposes that format on LLMs, while human references may be multi-sentence Javadoc with @param/@return tags, making the comparison asymmetric.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Large Language Models are Qualified Benchmark Builders: Rebuilding Pre-Training Datasets for Advancing Code Intelligence Tasks." pith.science (2026). https://pith.science/paper/VJYG3PL5

@misc{pith2026250419444,
  author       = {Pith},
  title        = {Pith review of: Large Language Models are Qualified Benchmark Builders: Rebuilding Pre-Training Datasets for Advancing Code Intelligence Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VJYG3PL5}},
  note         = {Machine review of arXiv:2504.19444}
}
read the original abstract

Pre-trained code models rely heavily on high-quality pre-training data, particularly human-written reference comments that bridge code and natural language. However, these comments often become outdated as software evolves, degrading model performance. Large language models (LLMs) excel at generating high-quality code comments. We investigate whether replacing human-written comments with LLM-generated ones improves pre-training datasets. Since standard metrics cannot assess reference comment quality, we propose two novel reference-free evaluation tasks: code-comment inconsistency detection and semantic code search. Results show that LLM-generated comments are more semantically consistent with code than human-written ones, as confirmed by manual evaluation. Leveraging this finding, we rebuild the CodeSearchNet dataset with LLM-generated comments and re-pre-train CodeT5. Evaluations demonstrate that models trained on LLM-enhanced data outperform those using original human comments in code summarization, generation, and translation tasks. This work validates rebuilding pre-training datasets with LLMs to advance code intelligence, challenging the traditional reliance on human reference comments.

Figures

Figures reproduced from arXiv: 2504.19444 by the authors.

Figure 1
Figure 1. Code comment inconsistency detection as a reference [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Code search as a reference-free evaluation metric. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. A Concode data sample rebuilt by GPT-3.5-Turbo. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PhoenixRepair: Rethinking Repair Strategy Exploration in Software Agents

    cs.AI 2026-07 conditional novelty 6.0 of 10

    A multi-agent repair framework that samples multiple edit locations and iteratively reflects on patch attempts reaches 76.0% Pass@1 on SWE-bench-Verified, up to a 7.8% relative gain over SWE-agent.

Reference graph

Works this paper leans on

84 extracted references · 50 canonical work pages · cited by 1 Pith paper

  1. [1]

    A survey of automatic generation of source code comments: Algorithms and techniques,

    X. Song, H. Sun, X. Wang, and J. Yan, “A survey of automatic generation of source code comments: Algorithms and techniques,” IEEE Access , vol. 7, pp. 111411–111428, 2019

    2019

  2. [2]

    Natural language to code: How far are we?,

    S. Wang, M. Geng, B. Lin, Z. Sun, M. Wen, Y . Liu, L. Li, T. F. Bissyand´e, and X. Mao, “Natural language to code: How far are we?,” in Proceedings of the 31st ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), ACM, 2023

    2023

  3. [3]

    Codefusion: A pre-trained diffusion model for code generation,

    M. Singh, J. Cambronero, S. Gulwani, V . Le, C. Negreanu, and G. Verbruggen, “Codefusion: A pre-trained diffusion model for code generation,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pp. 11697–11708, 2023

    2023

  4. [4]

    Two birds with one stone: Boosting code generation and code search via a generative adversarial network,

    S. Wang, B. Lin, Z. Sun, M. Wen, Y . Liu, Y . Lei, and X. Mao, “Two birds with one stone: Boosting code generation and code search via a generative adversarial network,” Proceedings of the ACM on Programming Languages, no. OOPSLA2, 2023

    2023

  5. [5]

    Deep code search,

    X. Gu, H. Zhang, and S. Kim, “Deep code search,” in Proceedings of the 40th ICSE , pp. 933–944, 2018

    work page 2018

  6. [6]

    On the importance of building high-quality training datasets for neural code search,

    Z. Sun, L. Li, Y . Liu, X. Du, and L. Li, “On the importance of building high-quality training datasets for neural code search,” in Proceedings of the 44th ICSE , pp. 1609–1620, 2022

    work page 2022

  7. [7]

    Fusing code searchers,

    S. Wang, M. Geng, B. Lin, Z. Sun, M. Wen, Y . Liu, L. Li, T. F. Bissyand´e, and X. Mao, “Fusing code searchers,” IEEE Transactions on Software Engineering , vol. 50, no. 7, pp. 1852–1866, 2024

    work page 2024

  8. [8]

    Deep learning application on code clone detection: A review of current knowledge,

    M. Lei, H. Li, J. Li, N. Aundhkar, and D.-K. Kim, “Deep learning application on code clone detection: A review of current knowledge,” Journal of Systems and Software , vol. 184, p. 111141, 2022

    2022

Show all 84 references
  1. [9]

    One size does not fit all: Multi-granularity patch generation for better automated program repair,

    B. Lin, S. Wang, M. Wen, L. Chen, and X. Mao, “One size does not fit all: Multi-granularity patch generation for better automated program repair,” in Proceedings of the 33rd ACM SIGSOFT International Sym- posium on Software Testing and Analysis , pp. 1554–1566, 2024

    2024

  2. [10]

    Context-aware code change embedding for better patch correctness assessment,

    B. Lin, S. Wang, M. Wen, and X. Mao, “Context-aware code change embedding for better patch correctness assessment,” ACM Trans. Softw. Eng. Methodol., vol. 31, may 2022

    2022

  3. [11]

    Codebert: A pre-trained model for programming and natural languages,

    Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, et al., “Codebert: A pre-trained model for programming and natural languages,” in Findings of the Association for Computational Linguistics: EMNLP 2020 , pp. 1536–1547, 2020

    2020

  4. [12]

    Graphcodebert: Pre-training code repre- sentations with data flow,

    D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, S. Liu, L. Zhou, N. Duan, A. Svyatkovskiy, S. Fu, et al., “Graphcodebert: Pre-training code repre- sentations with data flow,” arXiv preprint arXiv:2009.08366 , 2020

    2009 arXiv

  5. [13]

    Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,

    Y . Wang, W. Wang, S. Joty, and S. C. Hoi, “Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pp. 8696–8708, 2021

    2021

  6. [14]

    Unixcoder: Unified cross-modal pre-training for code representation,

    D. Guo, S. Lu, N. Duan, Y . Wang, M. Zhou, and J. Yin, “Unixcoder: Unified cross-modal pre-training for code representation,” in Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 7212–7225, 2022

    2022

  7. [15]

    Understanding code changes practically with small-scale language models,

    C. Li, Z. Xu, P. Di, D. Wang, Z. Li, and Q. Zheng, “Understanding code changes practically with small-scale language models,” in Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, pp. 216–228, 2024

    2024

  8. [16]

    Quality analysis of source code comments,

    D. Steidl, B. Hummel, and E. Juergens, “Quality analysis of source code comments,” in 2013 21st International Conference on Program Comprehension (ICPC), pp. 83–92, Ieee, 2013

    2013

  9. [17]

    Source code that talks: an exploration of eclipse task comments and their implication to repository mining,

    A. T. Ying, J. L. Wright, and S. Abrams, “Source code that talks: an exploration of eclipse task comments and their implication to repository mining,” ACM SIGSOFT software engineering notes , vol. 30, no. 4, pp. 1–5, 2005

    2005

  10. [18]

    A large-scale empirical study on code-comment inconsistencies,

    F. Wen, C. Nagy, G. Bavota, and M. Lanza, “A large-scale empirical study on code-comment inconsistencies,” in 2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC) , pp. 53– 64, IEEE, 2019

    2019

  11. [19]

    Are we building on the rock? on the importance of data preprocessing for code summarization,

    L. Shi, F. Mu, X. Chen, S. Wang, J. Wang, Y . Yang, G. Li, X. Xia, and Q. Wang, “Are we building on the rock? on the importance of data preprocessing for code summarization,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Fou...

    2022

  12. [20]

    Summarizing source code with transferred api knowledge,

    X. Hu, G. Li, X. Xia, D. Lo, S. Lu, and Z. Jin, “Summarizing source code with transferred api knowledge,” 27th International Joint Conference on Artificial Intelligence, IJCAI 2018, 27

    2018

  13. [21]

    Codesearchnet challenge: Evaluating the state of semantic code search,

    H. Husain, H.-H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt, “Codesearchnet challenge: Evaluating the state of semantic code search,” arXiv preprint arXiv:1909.09436 , 2019

    1909 arXiv

  14. [22]

    Code llama: Open foundation models for code,

    B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, T. Remez, J. Rapin, et al., “Code llama: Open foundation models for code,” arXiv preprint arXiv:2308.12950 , 2023

    2023 arXiv

  15. [23]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    2023 arXiv

  16. [24]

    Gpt-4o system card,

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al., “Gpt-4o system card,” arXiv preprint arXiv:2410.21276 , 2024

    2024 arXiv

  17. [25]

    Exploring the potential of chatgpt in automated code refinement: An empirical study,

    Q. Guo, J. Cao, X. Xie, S. Liu, X. Li, B. Chen, and X. Peng, “Exploring the potential of chatgpt in automated code refinement: An empirical study,” arXiv preprint arXiv:2309.08221 , 2023

    2023 arXiv

  18. [26]

    A large-scale survey on the usability of ai programming assistants: Successes and challenges,

    J. T. Liang, C. Yang, and B. A. Myers, “A large-scale survey on the usability of ai programming assistants: Successes and challenges,” in 2024 IEEE/ACM 46th International Conference on Software Engineer- ing (ICSE), pp. 605–617, IEEE Computer Society, 2024

    2024

  19. [27]

    Large language models are few-shot summarizers: Multi-intent comment generation via in-context learning,

    M. Geng, S. Wang, D. Dong, H. Wang, G. Li, Z. Jin, X. Mao, and X. Liao, “Large language models are few-shot summarizers: Multi-intent comment generation via in-context learning,” in Proceedings of the 46th IEEE/ACM ICSE, pp. 1–13, 2024

    2024

  20. [28]

    Deepseek-coder: When the large language model meets programming–the rise of code intelligence,

    D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y . Wu, Y . Li,et al., “Deepseek-coder: When the large language model meets programming–the rise of code intelligence,” arXiv preprint arXiv:2401.14196, 2024

    2024 arXiv

  21. [29]

    Starcoder 2 and the stack v2: The next generation,

    A. Lozhkov, R. Li, L. B. Allal, F. Cassano, J. Lamy-Poirier, N. Tazi, A. Tang, D. Pykhtar, J. Liu, Y . Wei, et al. , “Starcoder 2 and the stack v2: The next generation,” arXiv preprint arXiv:2402.19173 , 2024

    2024 arXiv

  22. [30]

    Umse: Unified multi-scenario summarization evaluation,

    S. Gao, Z. Yao, C. Tao, X. Chen, P. Ren, Z. Ren, and Z. Chen, “Umse: Unified multi-scenario summarization evaluation,” arXiv preprint arXiv:2305.16895, 2023

    2023 arXiv

  23. [31]

    Bertscore: Evaluating text generation with bert,

    T. Zhang, V . Kishore, F. Wu, K. Q. Weinberger, and Y . Artzi, “Bertscore: Evaluating text generation with bert,” arXiv preprint arXiv:1904.09675, 2019

    1904 arXiv

  24. [32]

    Semantic similarity metrics for evaluating source code summarization,

    S. Haque, Z. Eberhart, A. Bansal, and C. McMillan, “Semantic similarity metrics for evaluating source code summarization,” inProceedings of the 30th IEEE/ACM International Conference on Program Comprehension , pp. 36–47, 2022

    2022

  25. [33]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting of the Association for Computational Linguistics , pp. 311–318, 2002

    2002

  26. [34]

    Rouge: A package for automatic evaluation of summaries,

    C.-Y . Lin, “Rouge: A package for automatic evaluation of summaries,” in Text summarization branches out , pp. 74–81, 2004

    2004

  27. [35]

    Meteor: An automatic metric for mt evalua- tion with improved correlation with human judgments,

    S. Banerjee and A. Lavie, “Meteor: An automatic metric for mt evalua- tion with improved correlation with human judgments,” in Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization , pp. 65–72, 2005

    2005

  28. [36]

    Beyond bleu: Training neural machine translation with semantic similarity,

    J. Wieting, T. Berg-Kirkpatrick, K. Gimpel, and G. Neubig, “Beyond bleu: Training neural machine translation with semantic similarity,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4344–4355, 2019

    2019

  29. [37]

    A human study of comprehension and code summariza- tion,

    S. Stapleton, Y . Gambhir, A. LeClair, Z. Eberhart, W. Weimer, K. Leach, and Y . Huang, “A human study of comprehension and code summariza- tion,” in Proceedings of the 28th International Conference on Program Comprehension, pp. 2–13, 2020

    2020

  30. [38]

    Code to comment translation: A comparative study on model effective- ness & errors,

    J. Mahmud, F. Faisal, R. I. Arnob, A. Anastasopoulos, and K. Moran, “Code to comment translation: A comparative study on model effective- ness & errors,” arXiv preprint arXiv:2106.08415 , 2021

    2021 arXiv

  31. [39]

    Sentence-bert: Sentence embeddings using siamese bert-networks,

    N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Processing (EMNLP- IJCNLP), ...

    2019

  32. [40]

    Universal sentence encoder,

    D. Cer, Y . Yang, S.-y. Kong, N. Hua, N. Limtiaco, R. S. John, N. Constant, M. Guajardo-Cespedes, S. Yuan, C. Tar, et al., “Universal sentence encoder,” arXiv preprint arXiv:1803.11175 , 2018

    2018 arXiv

  33. [41]

    How do developers document database usages in source code?(n),

    M. Linares-V ´asquez, B. Li, C. Vendome, and D. Poshyvanyk, “How do developers document database usages in source code?(n),” in 2015 30th IEEE/ACM International Conference on Automated Software Engineer- ing (ASE), pp. 36–41, IEEE, 2015. 11

    2015

  34. [42]

    Learning and evaluating contextual embedding of source code,

    A. Kanade, P. Maniatis, G. Balakrishnan, and K. Shi, “Learning and evaluating contextual embedding of source code,” in International conference on machine learning , pp. 5110–5121, PMLR, 2020

    2020

  35. [43]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017

    2017

  36. [44]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. D. M.-W. C. Kenton and L. K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of NAACL-HLT, pp. 4171–4186, 2019

    2019

  37. [45]

    Roberta: A robustly optimized bert pretraining approach,

    Y . Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V . Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692 , 2019

    1907 arXiv

  38. [46]

    Electra: Pre- training text encoders as discriminators rather than generators,

    K. Clark, M.-T. Luong, Q. V . Le, and C. D. Manning, “Electra: Pre- training text encoders as discriminators rather than generators,” arXiv preprint arXiv:2003.10555, 2020

    2003 arXiv

  39. [47]

    Intellicode compose: Code generation using transformer,

    A. Svyatkovskiy, S. K. Deng, S. Fu, and N. Sundaresan, “Intellicode compose: Code generation using transformer,” inProceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 1433–1443, 2020

    2020

  40. [48]

    Codexglue: A machine learning benchmark dataset for code understanding and generation,

    S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang, et al., “Codexglue: A machine learning benchmark dataset for code understanding and generation,” arXiv preprint arXiv:2102.04664 , 2021

    2021 arXiv

  41. [49]

    Unified pre- training for program understanding and generation,

    W. Ahmad, S. Chakraborty, B. Ray, and K.-W. Chang, “Unified pre- training for program understanding and generation,” in Proceedings of the 2021 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies , pp. 2655–2...

    2021

  42. [50]

    Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehen- sion,

    M. Lewis, Y . Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V . Stoyanov, and L. Zettlemoyer, “Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehen- sion,” in Proceedings of the 58th Annual Meeting of the Associa...

    2020

  43. [51]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” The Journal of Machine Learning Research, vol. 21, no. 1, pp. 5485–5551, 2020

    2020

  44. [52]

    Deep just- in-time inconsistency detection between comments and source code,

    S. Panthaplackel, J. J. Li, M. Gligoric, and R. J. Mooney, “Deep just- in-time inconsistency detection between comments and source code,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 35, pp. 427–435, 2021

    2021

  45. [53]

    Code comment inconsistency detection with bert and longformer,

    T. Steiner and R. Zhang, “Code comment inconsistency detection with bert and longformer,” arXiv preprint arXiv:2207.14444 , 2022

    2022 arXiv

  46. [54]

    CCID Dadaset GoogleDrive

    “CCID Dadaset GoogleDrive.” https://drive.google.com/drive/folders/ 1heqEQGZHgO6gZzCjuQD1EYertN4SAYZ

  47. [55]

    Project Longformer4CCID

    “Project Longformer4CCID.” https://github.com/theo2023/ coco-bert-longformer, 2023

    2023

  48. [56]

    Codehow: Effective code search based on api understanding and extended boolean model (e),

    F. Lv, H. Zhang, J.-g. Lou, S. Wang, D. Zhang, and J. Zhao, “Codehow: Effective code search based on api understanding and extended boolean model (e),” in 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE) , pp. 260–270, IEEE, 2015

    2015

  49. [57]

    Learning to rank relevant files for bug reports using domain knowledge,

    X. Ye, R. Bunescu, and C. Liu, “Learning to rank relevant files for bug reports using domain knowledge,” in Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering, pp. 689–699, 2014

    2014

  50. [58]

    CodeBERT-CodeSearch

    “CodeBERT-CodeSearch.” https://github.com/microsoft/CodeBERT/ tree/master/CodeBERT/codesearch, 2021

    2021

  51. [59]

    A transformer- based approach for source code summarization,

    W. Ahmad, S. Chakraborty, B. Ray, and K.-W. Chang, “A transformer- based approach for source code summarization,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pp. 4998–5007, 2020

    2020

  52. [60]

    Code summarization with structure- induced transformer,

    H. Wu, H. Zhao, and M. Zhang, “Code summarization with structure- induced transformer,” in Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , pp. 1078–1090, 2021

    2021

  53. [61]

    Developer-intent driven code comment generation,

    F. Mu, X. Chen, L. Shi, S. Wang, and Q. Wang, “Developer-intent driven code comment generation,” arXiv preprint arXiv:2302.07055 , 2023

    2023 arXiv

  54. [62]

    Text-davinci-003

    “Text-davinci-003.” https://platform.openai.com/docs/models/ text-davinci-003, 2023

    2023

  55. [63]

    GPT-3.5-Turbo

    “GPT-3.5-Turbo.” https://platform.openai.com/docs/models/gpt-3-5, 2024

    2024

  56. [64]

    GPT-4-Turbo

    “GPT-4-Turbo.” https://platform.openai.com/docs/models/ gpt-4-and-gpt-4-turbo, 2024

    2024

  57. [65]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. , “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  58. [66]

    Code-Llama

    “Code-Llama.” https://llama.meta.com/code-llama, 2024

    2024

  59. [67]

    CodeBERT-CodeSearch

    “CodeBERT-CodeSearch.” https://deepseekcoder.github.io, 2024

    2024

  60. [68]

    Starcoder2

    “Starcoder2.” https://github.com/bigcode-project/starcoder2, 2024

    2024

  61. [69]

    Universal sentence encoder for English,

    D. Cer, Y . Yang, S.-y. Kong, N. Hua, N. Limtiaco, R. St. John, N. Constant, M. Guajardo-Cespedes, S. Yuan, C. Tar, B. Strope, and R. Kurzweil, “Universal sentence encoder for English,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Pro- cessing...

    2018

  62. [70]

    Reassessing automatic evaluation metrics for code summarization tasks,

    D. Roy, S. Fakhoury, and V . Arnaoudova, “Reassessing automatic evaluation metrics for code summarization tasks,” in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , pp. 1105– 1116, 2021

    2021

  63. [71]

    Singh and N

    R. Singh and N. S. Mangat, Elements of survey sampling , vol. 15. Springer Science & Business Media, 2013

    2013

  64. [72]

    Semantic similarity loss for neural source code summarization,

    C.-Y . Su and C. McMillan, “Semantic similarity loss for neural source code summarization,” arXiv preprint arXiv:2308.07429 , 2023

    2023 arXiv

  65. [73]

    Cct5: A code- change-oriented pre-trained model,

    B. Lin, S. Wang, Z. Liu, Y . Liu, X. Xia, and X. Mao, “Cct5: A code- change-oriented pre-trained model,” in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering , pp. 1509–1521, 2023

    2023

  66. [74]

    Mapping language to code in programmatic context,

    S. Iyer, I. Konstas, A. Cheung, and L. Zettlemoyer, “Mapping language to code in programmatic context,” in Proceedings of the 2018 Confer- ence on EMNLP , pp. 1643–1652, 2018

    2018

  67. [75]

    Codebleu: a method for automatic evaluation of code synthesis,

    S. Ren, D. Guo, S. Lu, L. Zhou, S. Liu, D. Tang, N. Sundaresan, M. Zhou, A. Blanco, and S. Ma, “Codebleu: a method for automatic evaluation of code synthesis,” arXiv preprint arXiv:2009.10297 , 2020

    2009 arXiv

  68. [76]

    Avatar: A parallel corpus for java-python program translation,

    W. U. Ahmad, M. G. R. Tushar, S. Chakraborty, and K.-W. Chang, “Avatar: A parallel corpus for java-python program translation,” arXiv preprint arXiv:2108.11590, 2021

    2021 arXiv

  69. [77]

    An empirical study on learning bug-fixing patches in the wild via neural machine translation,

    M. Tufano, C. Watson, G. Bavota, M. D. Penta, M. White, and D. Poshyvanyk, “An empirical study on learning bug-fixing patches in the wild via neural machine translation,” ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 28, no. 4, pp. 1–29, 2019

    2019

  70. [78]

    Detecting code clones with graph neural network and flow-augmented abstract syntax tree,

    W. Wang, G. Li, B. Ma, X. Xia, and Z. Jin, “Detecting code clones with graph neural network and flow-augmented abstract syntax tree,” in 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER) , pp. 261–271, IEEE, 2020

    2020

  71. [79]

    On the relationship between comment update practices and software bugs,

    W. M. Ibrahim, N. Bettenburg, B. Adams, and A. E. Hassan, “On the relationship between comment update practices and software bugs,” Journal of Systems and Software , vol. 85, no. 10, pp. 2293–2304, 2012

    2012

  72. [80]

    Understanding the rationale for updating a function’s comment,

    H. Malik, I. Chowdhury, H.-M. Tsou, Z. M. Jiang, and A. E. Hassan, “Understanding the rationale for updating a function’s comment,” in 2008 IEEE International Conference on Software Maintenance, pp. 167– 176, IEEE, 2008

    2008

  73. [81]

    Do code and comments co-evolve? on the relation between source code and comment changes,

    B. Fluri, M. Wursch, and H. C. Gall, “Do code and comments co-evolve? on the relation between source code and comment changes,” in 14th Working Conference on Reverse Engineering (WCRE 2007) , pp. 70–79, IEEE, 2007

    2007

  74. [82]

    On the impact of sample duplication in machine-learning-based android malware detection,

    Y . Zhao, L. Li, H. Wang, H. Cai, T. F. Bissyand´e, J. Klein, and J. Grundy, “On the impact of sample duplication in machine-learning-based android malware detection,” ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 30, no. 3, pp. 1–38, 2021

    2021

  75. [83]

    Sim- plifying deep-learning-based model for code search,

    C. Liu, X. Xia, D. Lo, Z. Liu, A. E. Hassan, and S. Li, “Sim- plifying deep-learning-based model for code search,” arXiv preprint arXiv:2005.14373, 2020

    2005 arXiv

  76. [84]

    Data quality matters: A case study of obsolete comment detection,

    S. Xu, Y . Yao, F. Xu, T. Gu, J. Xu, and X. Ma, “Data quality matters: A case study of obsolete comment detection,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pp. 781–793, IEEE, 2023. 12

    2023

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.