Pith. sign in

REVIEW 4 major objections 4 minor 77 references

CCISolver: End-to-End Detection and Repair of Method-Level Code-Comment Inconsistency

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A two-stage detector-plus-fixer pipeline detects and repairs Java code-comment mismatches, reaching F1 89.54% and a 65.33% human-judged fix success rate.

desk verdict Solid CCI paper with a real dataset and honest analysis, but the absolute numbers measure agreement with LLM labelers, not ground truth. read the letter →

arxiv 2506.20558 v1 pith:E7U3MLU3 submitted 2025-06-25 cs.SE

Renyi Zhong , Yintong Huo , Wenwei Gu , Jinxi Kuang , Zhihan Jiang , Guangba Yu , Yichen Li , David Lo
show 1 more author
Michael R. Lyu
This is my paper
classification cs.SE
keywords code-commentinconsistencyjust-in-timedetectioncommentrepairLLMvotingdatasetcleaningJavamethodcommentsLoRAfine-tuningdeeplearningdetector
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that code-comment inconsistency (CCI) work has been held back by dirty labels and single-model solutions. A manual audit of 600 positive cases from the widely used JITDATA dataset finds that 45.67% are false positives, mostly from comment additions/deletions, typo fixes, case changes, and lexical rewording. To replace that resource, the authors build CCIBench, a cleaned Java method-level dataset filtered by syntactic rules and by majority voting among three LLMs, and they use it to train and evaluate CCISolver, a two-stage pipeline with a lightweight deep-learning detector (CCIDetector) and a fine-tuned LLM fixer (CCIFixer). They report that CCISolver outperforms all compared baselines on detection (F1 89.54% on the full test set, 92.20% on the validated subset), on repair quality (GLEU 72.77, an 18.84% relative gain over C4RLLaMA), and on human-judged fix success (0.6533), while cutting end-to-end inference time by about 36% relative to the monolithic C4RLLaMA. The paper therefore claims that separating cheap detection from expensive repair is both more accurate and more practical than using one large model for both tasks.

What carries the argument

The central mechanism is the two-stage pipeline itself: CCIDetector scans all code-comment pairs cheaply and forwards only predicted inconsistencies to CCIFixer. The detector's input representation matters as much as the architecture: code diffs are rendered as add/del/rep/keep edit actions with replace spans, raw comment and new code are embedded with UniXcoder, and a classifier head scores consistency with a cosine-similarity term in the loss. The fixer is a LoRA fine-tuned code LLM (Qwen2.5-Coder-14B) put through KTO alignment so its rewrites stay close to the dataset's comment style. The dataset pipeline is equally load-bearing: syntactic filters remove trivial edits and a three-LLM voting rule with 4-shot in-context examples decides which semantic changes remain as genuine inconsistencies.

What would settle it

Take a random sample of, say, 300 CCIBench test cases plus 100 CCIFixer outputs and have two developers who did not build the dataset independently label whether the comment contradicts the code and whether the fix is correct; if their labels match CCIBench's labels and the paper's human success annotations only slightly better than chance, then the reported F1, GLEU, and success rates are artifacts of the labeling procedure.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that the CCI problem decomposes: a small, fast model can decide whether an old comment contradicts new code, and a larger generative model needs to run only on the cases flagged. The authors support this with CCIBench, whose construction is itself part of the contribution: after de-duplication, syntactic rules strip typo/case/stopword/lexical changes, and three LLM voters with dynamic 4-shot prompting decide semantic inconsistencies, keeping cases with at least two-thirds agreement and manually validating a 300-case test subset (Cohen's kappa 0.95). The detector encodes old comment, new code, and a structured code diff through UniXcoder, Bi-GRU, and multi-head attention, with a classifier that combines binary cross-entropy with a comment-code cosine similarity term, and is iteratively boosted on LLM-synthesized versions of its own mistakes. The fixer is Qwen2.5-Coder-14B fine-tuned with LoRA and then aligned with KTO on the cleaned inconsistent cases. With this design the paper reports F1 of 89.54% and 92.20% (full and validated test sets), GLEU of 72.77 and 74.90, human fix success of 0.6533, and per-instance inference time of 0.6164 s versus 0.9618 s for C4RLLaMA.

Load-bearing premise

Every reported performance number assumes the CCIBench labels are correct, but most labels come from majority votes among three large language models, so if those models systematically disagree with developer judgment, the F1, GLEU, and success-rate figures measure agreement with the labelers rather than real code-comment inconsistency.

Editorial extensions

If this is right

  • If the CCIBench labels are trusted, prior evaluations on JITDATA were inflated, since 45.67% of sampled positive labels were false positives; future CCI work should be measured on the cleaned benchmark.
  • A developer integrating CCISolver into a just-in-time commit workflow could scan a batch of changed Java methods, have the detector flag candidate inconsistencies, and send only those to the LLM fixer, obtaining better fixes than C4RLLaMA while spending about 36% less inference time per instance.
  • The fixer's human-evaluated success rate of 0.6533 implies that roughly two of every three generated comments are found acceptable by two independent annotators, setting a higher bar than DeepJIT (0.4867) and C4RLLaMA (0.5867).
  • The iterative enhancement results (F1 86.6 without augmentation versus 89.54 with 10 iterations) imply that synthesizing the detector's own hard cases is a usable substitute for collecting more labeled data.
  • Because only flagged cases reach the LLM, real-world savings should grow as the true inconsistency rate drops below the test set's rate, a point the authors make in their discussion.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same cheap-filter/expensive-repair split could be applied to inline comments, class comments, or other languages, since nothing in the architecture is Java-specific; the gains would need testing, but the bottleneck it removes is generic.
  • Beyond the paper, the reliance on LLM voters for ground truth means CCIBench's main risk is shared blind spots; publishing an independently human-labeled sample alongside the dataset would let users calibrate all downstream numbers.
  • Beyond the paper, because the paper itself notes that BLEU/GLEU scores do not reliably track fix correctness, an LLM-as-judge metric calibrated against the human annotations would be a natural, lower-cost way to scale fix evaluation.
  • Beyond the paper, conditioning the fixer on the detector's identified inconsistency type (return type, signature, or logic) could improve fixes, since the case study shows the fixer's errors differ across those categories.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper first performs a manual study of 600 JITDATA cases and reports that 45.67% of positive labels are false positives, attributing them to information add/delete, typo fixes, case changes, and lexical changes. It then constructs a new dataset, CCIBench, by de-duplicating JITDATA, applying syntactic cleaning rules, and filtering with majority voting among GPT-4o, Claude3.5-Sonnet, and LLaMA3.1-405b, with a 300-case validated test set built from unanimously flagged cases and manually verified. The proposed CCISolver combines a deep-learning detector (CCIDetector), which uses UnixCoder, Bi-GRU, multi-head attention, a similarity-based classifier, and iterative LLM-based data enhancement, with an LLM fixer (CCIFixer) based on Qwen2.5-Coder-14B fine-tuned with LoRA and KTO alignment. The paper reports state-of-the-art detection (F1 89.54% on the full CCIBench test set and 92.20% on the validated set), fixing quality (GLEU 72.77 on the full inconsistent test set, an 18.84% relative gain over C4RLLaMA), a human-evaluated fix success rate of 0.6533, and about 36% faster end-to-end inference than C4RLLaMA.

Significance. The potential contributions are real: the quantitative mislabeling study is a concrete empirical finding; CCIBench is a substantial new resource; the two-stage detector-fixer architecture is a sensible engineering response to the cost of LLM-only pipelines; and the evaluation is extensive, including five-run medians, annotation protocols with reported Cohen's kappa, ablation studies, a case study with failures, and a public data link. As currently presented, however, the headline numbers are not fully supported. The benchmark labels are produced by the same LLM voting process that filters the training data, so absolute F1/GLEU values measure agreement with the labelers rather than with an independent ground truth, and the validated set is not a random sample of the full test set. In addition, the default hyperparameters appear to be selected on the test sets used for the final numbers. If the authors add a random human-labeled evaluation sample and a proper validation protocol, the relative comparisons on CCIBench would remain a meaningful contribution even if the absolute real-world performance numbers change.

major comments (4)
  1. [§3.0.3, §3.0.4, Tables 5 and 7] The ground-truth labels of CCIBench are produced by majority voting among GPT-4o, Claude3.5-Sonnet, and LLaMA3.1-405b (§3.0.3), and the validated test set is composed of cases unanimously flagged as inconsistent and then manually verified (§3.0.4). Because the detector and fixer are trained on data filtered by the same voting process, the full-test-set F1 and GLEU numbers in Tables 5 and 7 quantify agreement with the labelers' judgment rather than with an independent ground truth. The 300-case validated subset cannot resolve this on its own: it is a unanimity slice, not a random sample of the full test set, so it does not measure performance on non-unanimous or negative cases. To support the absolute claims, the authors should human-annotate a random stratified sample of the full test set (including non-unanimous and negative cases) and report detection and fix metrics on that sample; the relative comparisons to baselines on the same benchmark remain informative regardless.
  2. [§5.3, §6.3 (Figure 7, Table 6)] The default hyperparameters (epoch = 10, loss weight λ = 1, iterative-enhancement iterations = 10) are reported as producing the highest F1 on both the full and validated test sets in Figure 7 and Table 6. This indicates that the hyperparameters were selected on the test sets whose numbers are later presented as final results, and no separate validation split is used. The reported 89.54/92.20 F1 scores are therefore optimistically biased, and the comparison against baselines, whose hyperparameters were not tuned on these same test sets, is not on equal footing. Please introduce a validation split for hyperparameter selection and early stopping, or otherwise demonstrate that the chosen configurations were not selected by test-set performance.
  3. [§6.6 RQ5, Table 8] The end-to-end fix metrics in Table 8 are not computed under the same evaluation protocol as the fixer metrics in Table 7. RQ3 deliberately removes consistent cases from the fix test set (Section 6.4.2) because unchanged comments in consistent cases make BLEU-4 and GLEU equal to 1 and inflate scores; RQ5, however, feeds the entire test set through the pipeline and reports fix metrics for the pipeline as a whole. As a result, the Table 8 GLEU values (CCISolver 76.31 vs. C4RLLaMA 63.24) are not comparable with the Table 7 values (72.77 vs. 61.23) and may be inflated by correct "no-change" outputs on consistent cases or by missed inconsistent cases whose old comments are scored as predictions. The paper should state exactly how fix metrics are computed in RQ5 and report them either on the flagged subset or under the RQ3 protocol so that the end-to-end comparison is apples-to-apples.
  4. [§3.0.3, §3.0.4, §6.4.2] The construction of CCIBench's negative instances and of the validated set is under-specified and appears internally inconsistent. The semantic filtering in §3.0.3 is described as keeping only cases in which at least two of three LLMs detect an inconsistency; if applied to all instances, this would leave no negative cases, yet the detection experiments in Table 5 require negative labels, and RQ3 (Section 6.4.2) reports that filtering out consistent cases leaves 150 cases in the validated test set, implying the validated set originally contained 150 consistent cases. The paper should state explicitly how negative labels are generated and verified, and how the 300 validated cases were selected. If the negative cases are inherited unchanged from JITDATA, they may contain false negatives under the very heuristic the paper argues is unreliable, which would bias the detection F1 scores.
minor comments (4)
  1. [§4.2.3, Eq. (2)] Equation (2) is mathematically equivalent to L = BCE + λ(1 − mean cosine similarity); rewriting it in this form would remove the confusing outer minus sign and the phrase "constraint ensuring the total loss remains positive." In addition, Section 6.3.2 refers to an "α" hyperparameter in the loss function, while Equation (2) and Figure 7 use λ; the notation should be unified.
  2. [Throughout] There are several typos that should be corrected in revision: "Eavaluation" in the Section 5.2 heading, "Preious" in Section 6.6.1, "deimensions" in Section 6.6.2, "contribusions" in Section 1, "maintainance" in the Index Terms, and "framkework" in Section 8.1.
  3. [§5.3, §6.2, §6.4] The paper reports medians over five runs but does not report standard deviations or confidence intervals for any metric; given the large performance differences claimed, at least for the smaller validated test sets it would be useful to know the spread across runs.
  4. [Abstract and §6.4] The phrase "significantly surpasses" is used for the human-evaluation success rate (0.6533 vs. 0.5867 and 0.4867), but no statistical significance test is reported; consider softening the wording or adding a test such as McNemar's test for paired outcomes.

Circularity Check

2 steps flagged · score 5.0 of 10

CCIBench's test labels come from the same three-LLM vote that labeled the training data, and GPT4o both votes and generates the detector's synthetic training data, so the headline F1 measures agreement with the labeler rather than independent ground truth; the 300-case validated set is a unanimous-only slice and cannot break this loop.

  1. self definitional [Section 3.0.3 and Section 6.2.3 (RQ1, Table 5)]
    "Specifically, if the two-thirds majority or all of the voters identify an inconsistency between a comment and the corresponding code, we choose to keep this case in our cleaned dataset. ... The CCID ETECTOR method outperforms all other techniques in CCIB ENCH, establishing itself as the most proficient strategy for just-in-time CCI detection."

    The ground-truth labels of CCIBench are produced by the three-LLM voting rule in Section 3.0.3. CCIDetector is trained on those LLM-voted labels and then evaluated on held-out labels generated by the same rule. The reported F1 (89.54 on the full test set, 92.20 on the validated set) therefore measures agreement with the labeler's judgment rather than with an independent definition of code-comment inconsistency. The full test set is never human-verified, and the 300-case validated set is restricted to cases where the same three voters were unanimous, so it cannot validate the full-set numbers. The central detection claim reduces to fitting the voting process that generated both training and test labels.

  2. fitted input called prediction [Section 4.2.4, Section 5.3, and Section 3.0.3]
    "We opted for GPT4o as the teacher LLM in CCID ETECTOR ... These selected cases, which highlight the model’s limitations, are then fed into the LLM. ... The synthesized data is subsequently merged with the existing training dataset to produce an updated training set Di+1. ... we employ a trio of popular and powerful LLMs to develop a voting mechanism: GPT4o [13], Claude3.5-Sonnet [14], and LLama3.1-405b [15]."

    GPT4o is one of the three voters whose majority defines the CCIBench labels, and Section 5.3 selects GPT4o as the teacher LLM that synthesizes new training examples for CCIDetector's iterative enhancement. The detector is therefore trained on data generated by a model whose judgment is a component of the evaluation label. Table 6 shows F1 rising from 86.6 at 0 iterations to 89.54 at 10 iterations; part of this gain is the detector learning to reproduce GPT4o's labeling preferences, which directly increases agreement with the GPT4o component of the test label. The 'prediction' is thus statistically forced toward the training-data generator rather than toward an external ground truth.

full rationale

CCISolver also has independent content: the fixer is fine-tuned on human-written developer-updated comments, the fix human evaluation is conducted by the authors with a reported Cohen's Kappa, and the relative ordering of methods on the same CCIBench is informative regardless of label source. No load-bearing self-citation chain was found; the paper's self-citations ([4], [24], [58]) are routine and do not carry the derivation. However, the headline detection F1 and the end-to-end F1/GLEU results in Tables 5, 7, and 8 are scored against labels generated by the same three-LLM voting process used to construct the training set, and GPT4o additionally acts as the teacher for synthetic training data. The paper's own threat-to-validity section states 'we utilized LLMs to autonomously assess inconsistencies, although this approach does not guarantee dataset quality.' Because the central detection claim reduces to agreement with the labeler, the score is elevated; because the fixer targets, relative rankings, and human evaluation provide partial independent grounding, it is not total circularity.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central empirical claims rest primarily on the validity of LLM-generated labels and the representativeness of the cleaned JITDATA-derived sample; no equations are derived in the paper, so no free parameters are hidden in derivations. The main free parameters are hyperparameters tuned on the test set.

free parameters (5)
  • loss weight lambda = 1
    Chosen via test-set sweep (Figure 7); lambda=1 gives the highest F1 on both full and validated test sets.
  • iterative enhancement iterations = 10
    Selected from {0,1,5,10,20} based on test F1 (Table 6); 10 yields the best score 89.54.
  • training epochs = 10
    Swept in Figure 7; default 10 outperforms 5 and 15 on test sets.
  • LLM voting threshold = 2/3 majority
    Design choice in Section 3.0.3; cases kept only if at least two of three LLMs flag inconsistency.
  • misclassification sampling rate = 0.1
    Default sampling rate for iterative enhancement in Algorithm 1 and Section 4.2.4; not ablated.
assumptions (6)
  • domain assumption JITDATA negative instances are correctly labeled; only positive instances are filtered.
    Section 3.0.1-3.0.3: the cleaning pipeline removes duplicates and filters positives but never audits negative labels, so negative-set noise remains.
  • domain assumption LLM majority voting provides correct ground truth for code-comment inconsistency.
    Section 3.0.3: labels in CCIBench are decided by two-thirds majority of GPT4o, Claude3.5-Sonnet, and LLaMA3.1-405b; no external verification except the 300-case validated subset.
  • domain assumption A comment update in a commit indicates the old comment should be inconsistent with new code, after cleaning.
    Section 2.1.3 and Figure 2: the JITDATA construction assumption, inherited by CCIBench.
  • domain assumption The four syntactic rules (typo, case, stopword, lexical) identify all non-CCI comment changes of those types.
    Section 3.0.2: rules were refined by manually reviewing 20% of filtered results, but no recall estimate is given.
  • domain assumption The 600-case sample is representative of JITDATA positives.
    Section 2.2.1: sampling 200 each from @return, @param, summary at 95% confidence, 5% interval; assumes within-stratum representativeness.
  • domain assumption Manually written new comment C' in JITDATA is the correct fix target (ground truth).
    Section 4.3 and RQ3: the fixer is trained and evaluated against the developer's updated comment, which may not be the only valid fix.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CCISolver: End-to-End Detection and Repair of Method-Level Code-Comment Inconsistency." pith.science (2026). https://pith.science/paper/E7U3MLU3

@misc{pith2026250620558,
  author       = {Pith},
  title        = {Pith review of: CCISolver: End-to-End Detection and Repair of Method-Level Code-Comment Inconsistency},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E7U3MLU3}},
  note         = {Machine review of arXiv:2506.20558}
}
read the original abstract

Comments within code serve as a crucial foundation for software documentation, facilitating developers to communicate and understand the code effectively. However, code-comment inconsistency (CCI) can negatively affect software development, testing, and maintenance. Recent efforts to mitigate this issue have emerged, but existing studies often suffer from inaccurate datasets and inadequate solutions, weakening their practical effectiveness. In this study, we first conduct a quantitative analysis of existing datasets, revealing a substantial portion of sampled data are mislabeled. To address these data limitations, we introduce CCIBench, a refined dataset comprising high-quality data, to support the training and evaluation of method-level CCI methods. Furthermore, we present an innovative end-to-end LLM-based framework, CCISolver, designed to improve code quality by identifying and rectifying CCIs. Comprehensive evaluations demonstrate CCISolver's superior performance. For detection, it establishes a new state-of-the-art with an F1-score of 89.54%. In fixing task, it achieves a remarkable 18.84% relative improvement in GLEU score over the strongest baseline. This superiority is confirmed by human evaluation, where CCISolver's fixing success rate of 0.6533 significantly surpasses existing methods. Critically, in a practical end-to-end setting, CCISolver's innovative architecture is approximately 36% faster for inference than the baseline model, underscoring its scalability and real-world applicability.

Figures

Figures reproduced from arXiv: 2506.20558 by the authors.

Figure 1
Figure 1. A CCI example in spring-data-mongodb [6]. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The presumptive scenario (left portion) and the actual [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Three inconsistent cases in JITDATA, where old comment is inconsistent with the new code. TABLE 1: Preliminary study result. @return @param summary Total Real Inconsistency 113 124 89 326 False Positive 87 76 111 274 – add/delete information 69 60 90 219 – fix typo 4 3 9 16 – change case 5 7 4 16 – change lexical 9 6 8 23 2.2.2 Two-stage examination process To ensure a high-quality manual analysis, we establish a mu… view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: CCIBENCH construction process. TABLE 2: Four false positive examples. Type Comment change add/delete information Removes the account with the primary key from the database. Removes the account from the database. fix typo Returns the result of interpretting the object a…
Figure 6
Figure 6. Figure 6: Code diff representation related to Figure 3c, removed [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 5
Figure 5. Figure 5: The framework of CCISOLVER. Conversely, if fewer than two-thirds register an inconsis￾tency, we discard the data. Appendix A provides a detailed prompt used for this semantic filtering. In addition to this mechanism, we apply a dynamic 4-shot in-context learning (ICL) …
Figure 7
Figure 7. Figure 7: Hyperparameter analysis. 6.3.3 Results [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Success rate of human evaluation for the fixing. [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: The fixing result of Qwen2.5-Coder-14B with different [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: The fixing result of LLaMA3.1-8B with different [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]
Figure 11
Figure 11. Figure 11: Return inconsistency case. Old code comment: @param now The current time in millis, as returned by System#currentTimeMillis() Related code change: - public CalciteSchema createSnapshot(long now) { + public CalciteSchema createSnapshot(SchemaVersion version) { Precondi…
Figure 12
Figure 12. Figure 12: Param inconsistency case. case. Hybrid Prediction shows significant shortcomings, of￾ten failing to adapt to code changes and relying on obsolete or incorrect references, which limits its utility. C4RLLaMA, while occasionally more descriptive, suffers from a lack of a…
Figure 14
Figure 14. Figure 14: Two failure cases in detection and repair stage. [PITH_FULL_IMAGE:figures/full_fig_p015_14.png]
Figure 15
Figure 15. Figure 15: The prompt template for semantic-based filtering. [PITH_FULL_IMAGE:figures/full_fig_p019_15.png]
Figure 16
Figure 16. Figure 16: The prompt template for iterative enhancement. [PITH_FULL_IMAGE:figures/full_fig_p019_16.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

77 extracted references · 71 canonical work pages

  1. [1]

    Classifying code comments in java software systems,

    L. Pascarella, M. Bruntink, and A. Bacchelli, “Classifying code comments in java software systems,” Empirical Software Engineer- ing (EMSE), vol. 24, no. 3, pp. 1499–1537, 2019

    work page 2019

  2. [2]

    Listening to program- mers—taxonomies and characteristics of comments in operating system code,

    Y. Padioleau, L. Tan, and Y. Zhou, “Listening to program- mers—taxonomies and characteristics of comments in operating system code,” in 2009 IEEE 31st International Conference on Software Engineering (ICSE). IEEE, 2009, pp. 331–341

    work page 2009

  3. [3]

    Learning a metric for code readabil- ity,

    R. P . Buse and W. R. Weimer, “Learning a metric for code readabil- ity,” IEEE Transactions on software engineering (TSE) , vol. 36, no. 4, pp. 546–558, 2009

    work page 2009

  4. [4]

    Towards Quality Assurance of Natural Language in Code ,

    R. Zhong, “ Towards Quality Assurance of Natural Language in Code ,” in 2025 IEEE/ACM 47th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). IEEE, May 2025, pp. 187–189

    work page 2025

  5. [5]

    Just-In-Time Obsolete Comment Detection and Update,

    Z. Liu, X. Xia, D. Lo, M. Yan, and S. Li, “Just-In-Time Obsolete Comment Detection and Update,” IEEE Transactions on Software Engineering (TSE), vol. 49, no. 1, pp. 1–23, 2023

    work page 2023

  6. [6]

    Code comment inconsistency cases in spring-data-mongodb,

    S. Projects, “Code comment inconsistency cases in spring-data-mongodb,” https://github. com/spring-projects/spring-data-mongodb/blob/ 74654cd7c788cdc53b05afe892607396a00f50a2/ spring-data-mongodb/src/main/java/org/springframework/ data/mongodb/core/mapreduce/MapReduceResults.java#L142, 2024, accessed: 2024-12-15

    work page 2024

  7. [7]

    Data are available at https://drive.google.com/drive/folders/1c4iTYKoe UnXnV9qUXKwp95Fs6eolyLiY?usp=sharing

    work page

  8. [8]

    Automatic Detection of Outdated Comments During Code Changes,

    Z. Liu, H. Chen, X. Chen, X. Luo, and F. Zhou, “Automatic Detection of Outdated Comments During Code Changes,” in 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC), vol. 01, 2018, pp. 154–163

    work page 2018

Show all 77 references
  1. [9]

    Deep Just- In-Time Inconsistency Detection Between Comments and Source Code,

    S. Panthaplackel, J. J. Li, M. Gligoric, and R. J. Mooney, “Deep Just- In-Time Inconsistency Detection Between Comments and Source Code,” Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), vol. 35, no. 1, pp. 427–435, 2021

    2021

  2. [10]

    Data Quality Matters: A Case Study of Obsolete Comment Detection,

    S. Xu, Y. Yao, F. Xu, T. Gu, J. Xu, and X. Ma, “Data Quality Matters: A Case Study of Obsolete Comment Detection,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). Melbourne, Australia: IEEE, 2023, pp. 781–793

    2023

  3. [11]

    Code comment inconsistency detection and rectification using a large language model,

    G. Rong, Y. Yu, S. Liu, X. Tan, T. Zhang, H. Shen, and J. Hu, “Code comment inconsistency detection and rectification using a large language model,” in 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 2025, pp. 432–443

    2025

  4. [12]

    Code Comment Inconsistency Detection Based on Confidence Learning,

    Z. Xu, S. Guo, Y. Wang, R. Chen, H. Li, X. Li, and H. Jiang, “Code Comment Inconsistency Detection Based on Confidence Learning,” IEEE Transactions on Software Engineering (TSE), vol. 50, no. 3, pp. 598–617, 2024

    2024

  5. [13]

    Automatic programming: Large language models and beyond,

    M. R. Lyu, B. Ray, A. Roychoudhury, S. H. Tan, and P . Thong- tanunam, “Automatic programming: Large language models and beyond,” ACM Trans. Softw. Eng. Methodol. , vol. 34, no. 5, May 2025

    2025

  6. [14]

    [Online]

    OpenAI, “Gpt-4o,” 2024, accessed: 2024-06-18. [Online]. Available: https://openai.com/index/hello-gpt-4o/

    2024

  7. [15]

    Claude 3.5: Sonnet,

    Anthropic, “Claude 3.5: Sonnet,” 2024, accessed: 2024- 07-10. [Online]. Available: https://www.anthropic.com/news/ claude-3-5-sonnet

    2024

  8. [16]

    Llama 3 model card,

    AI@Meta, “Llama 3 model card,” 2024. [Online]. Available: https://github.com/meta-llama/llama3/blob/main/ MODEL CARD.md

    2024

  9. [17]

    Asso- ciating natural language comment and source code entities,

    S. Panthaplackel, M. Gligoric, R. J. Mooney, and J. J. Li, “Asso- ciating natural language comment and source code entities,” in Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , vol. 34, no. 05, 2020, pp. 8592–8599

    2020

  10. [18]

    Boslaugh, Statistics in a nutshell: A desktop quick reference

    S. Boslaugh, Statistics in a nutshell: A desktop quick reference . O’Reilly Media, Inc., 2012

    2012

  11. [19]

    The kappa statistic in reliability studies: use, interpretation, and sample size requirements,

    J. Sim and C. C. Wright, “The kappa statistic in reliability studies: use, interpretation, and sample size requirements,” Physical ther- apy, vol. 85, no. 3, pp. 257–268, 2005

    2005

  12. [20]

    Doc- Checker: Bootstrapping Code Large Language Model for Detect- ing and Resolving Code-Comment Inconsistencies,

    A. T. V . Dau, J. L. C. Guo, and N. D. Q. Bui, “Doc- Checker: Bootstrapping Code Large Language Model for Detect- ing and Resolving Code-Comment Inconsistencies,” arXiv preprint arXiv:2306.06347, 2024

    2024 arXiv

  13. [21]

    Code comment inconsistency detection with bert and longformer,

    T. Steiner and R. Zhang, “Code comment inconsistency detection with bert and longformer,” arXiv preprint arXiv:2207.14444, 2022

    2022 arXiv

  14. [22]

    Judging llm-as-a-judge with mt- bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing et al., “Judging llm-as-a-judge with mt- bench and chatbot arena,”Advances in Neural Information Processing Systems (NeurIPS), vol. 36, pp. 46 595–46 623, 2023

    2023

  15. [23]

    Livecodebench leaderboard,

    LiveCodeBench, “Livecodebench leaderboard,” https: //livecodebench.github.io/leaderboard.html, 2024, accessed: 2024-12-27

    2024

  16. [24]

    Search-based llms for code optimization,

    S. Gao, C. Gao, W. Gu, and M. Lyu, “Search-based llms for code optimization,” in 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 2025, pp. 254–266

    2025

  17. [25]

    Larger is not always better: Exploring small open-source language models in logging statement generation,

    R. Zhong, Y. Li, G. Yu, W. Gu, J. Kuang, Y. Huo, and M. R. Lyu, “Larger is not always better: Exploring small open-source language models in logging statement generation,” arXiv preprint arXiv:2505.16590, 2025

    2025 arXiv

  18. [26]

    Ask me anything: A simple strategy for prompting language models,

    S. Arora, A. Narayan, M. F. Chen, L. Orr, N. Guha, K. Bhatia, I. Chami, and C. Re, “Ask me anything: A simple strategy for prompting language models,” in The Eleventh International Confer- ence on Learning Representations, 2023

    2023

  19. [27]

    What makes good in-context demonstrations for code in- telligence tasks with llms?

    S. Gao, X.-C. Wen, C. Gao, W. Wang, H. Zhang, and M. R. Lyu, “What makes good in-context demonstrations for code in- telligence tasks with llms?” in 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 2023, pp. 761–773

    2023

  20. [28]

    A neural model for generat- ing natural language summaries of program subroutines,

    A. LeClair, S. Jiang, and C. McMillan, “A neural model for generat- ing natural language summaries of program subroutines,” in 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 2019, pp. 795–806

    2019

  21. [29]

    P . S. Foundation, difflib, 2025, accessed: 2025-11-27. [Online]. Available: https://docs.python.org/3/library/difflib.html MANUSCRIPT 18

    2025

  22. [30]

    Learning to Update Natural Language Comments Based on Code Changes,

    S. Panthaplackel, P . Nie, M. Gligoric, J. J. Li, and R. Mooney, “Learning to Update Natural Language Comments Based on Code Changes,” in Proceedings of the 58th Annual Meeting of the Associa- tion for Computational Linguistics (ACL) . Online: Association for Computational Lin...

    2020

  23. [31]

    Unix- coder: Unified cross-modal pre-training for code representation,

    D. Guo, S. Lu, N. Duan, Y. Wang, M. Zhou, and J. Yin, “Unix- coder: Unified cross-modal pre-training for code representation,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2022, pp. 7212– 7225

    2022

  24. [32]

    Unified language model pre-training for natural language understanding and generation,

    L. Dong, N. Yang, W. Wang, F. Wei, X. Liu, Y. Wang, J. Gao, M. Zhou, and H.-W. Hon, “Unified language model pre-training for natural language understanding and generation,” Advances in neural information processing systems (NeurIPS), vol. 32, 2019

    2019

  25. [33]

    Automatic commit message generation: A critical review and di- rections for future work,

    Y. Zhang, Z. Qiu, K.-J. Stol, W. Zhu, J. Zhu, Y. Tian, and H. Liu, “Automatic commit message generation: A critical review and di- rections for future work,” IEEE Transactions on Software Engineering (TSE), 2024

    2024

  26. [34]

    Ex- ploring the potential of chatgpt in automated code refinement: An empirical study,

    Q. Guo, J. Cao, X. Xie, S. Liu, X. Li, B. Chen, and X. Peng, “Ex- ploring the potential of chatgpt in automated code refinement: An empirical study,” in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering (ICSE). IEEE, 2024, pp. 1–13

    2024

  27. [35]

    Eda: Easy data augmentation techniques for boosting performance on text classification tasks,

    J. Wei and K. Zou, “Eda: Easy data augmentation techniques for boosting performance on text classification tasks,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Process...

    2019

  28. [36]

    Text data augmen- tation for deep learning,

    C. Shorten, T. M. Khoshgoftaar, and B. Furht, “Text data augmen- tation for deep learning,” Journal of big Data , vol. 8, no. 1, p. 101, 2021

    2021

  29. [37]

    When to say what: Learning to find condition-message inconsistencies,

    I. Bouzenia and M. Pradel, “When to say what: Learning to find condition-message inconsistencies,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023, pp. 868–880

    2023

  30. [38]

    Auggpt: Leveraging chatgpt for text data augmentation,

    H. Dai, Z. Liu, W. Liao, X. Huang, Y. Cao, Z. Wu, L. Zhao, S. Xu, F. Zeng, W. Liu et al., “Auggpt: Leveraging chatgpt for text data augmentation,” IEEE Transactions on Big Data, 2025

    2025

  31. [39]

    Llm2llm: Boosting llms with novel iterative data enhancement,

    N. Lee, T. Wattanawong, S. Kim, K. Mangalam, S. Shen, G. Anu- manchipalli, M. Mahoney, K. Keutzer, and A. Gholami, “Llm2llm: Boosting llms with novel iterative data enhancement,” in Findings of the Association for Computational Linguistics ACL 2024 , 2024, pp. 6498–6526

    2024

  32. [40]

    Self-instruct: Aligning language models with self- generated instructions,

    Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi, “Self-instruct: Aligning language models with self- generated instructions,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2...

    2023

  33. [41]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, P . Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen et al., “Lora: Low-rank adaptation of large language models,” in International Conference on Learning Representations

  34. [42]

    Kto: Model alignment as prospect theoretic optimization,

    K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela, “Kto: Model alignment as prospect theoretic optimization,” arXiv preprint arXiv:2402.01306, 2024

    2024 arXiv

  35. [43]

    Direct preference optimization: Your language model is secretly a reward model,

    R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,” Advances in Neural Information Processing Systems (NeurIPS), vol. 36, 2024

    2024

  36. [44]

    Large language models for equivalent mutant detection: How far are we?

    Z. Tian, H. Shu, D. Wang, X. Cao, Y. Kamei, and J. Chen, “Large language models for equivalent mutant detection: How far are we?” in Proceedings of the 33rd ACM SIGSOFT International Sympo- sium on Software Testing and Analysis (ISSTA), 2024, pp. 1733–1745

    2024

  37. [45]

    Exploring the effectiveness of llms in automated log- ging statement generation: An empirical study,

    Y. Li, Y. Huo, Z. Jiang, R. Zhong, P . He, Y. Su, L. C. Briand, and M. R. Lyu, “Exploring the effectiveness of llms in automated log- ging statement generation: An empirical study,” IEEE Transactions on Software Engineering (TSE), vol. 50, no. 12, pp. 3188–3207, 2024

    2024

  38. [46]

    Large Language Models are Few-Shot Summarizers: Multi-Intent Comment Generation via In-Context Learning,

    M. Geng, S. Wang, D. Dong, H. Wang, G. Li, Z. Jin, X. Mao, and X. Liao, “Large Language Models are Few-Shot Summarizers: Multi-Intent Comment Generation via In-Context Learning,” in Proceedings of the IEEE/ACM 46th International Conference on Soft- ware Engineering (ICSE). Lis...

    2024

  39. [47]

    Codebert: A pre-trained model for programming and natural languages,

    Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang et al., “Codebert: A pre-trained model for programming and natural languages,” inFindings of the Association for Computational Linguistics: EMNLP 2020, 2020, pp. 1536–1547

    2020

  40. [48]

    Gpt-3.5,

    OpenAI., “Gpt-3.5,” Mar 2022. [Online]. Available: https: //platform.openai.com/docs/models/gpt-3-5

    2022

  41. [49]

    Deepseek-coder-v2: Breaking the bar- rier of closed-source models in code intelligence,

    Q. Zhu, D. Guo, Z. Shao, D. Yang, P . Wang, R. Xu, Y. Wu, Y. Li, H. Gao, S. Ma et al., “Deepseek-coder-v2: Breaking the bar- rier of closed-source models in code intelligence,” arXiv preprint arXiv:2406.11931, 2024

    2024 arXiv

  42. [50]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting of the Association for Computational Linguis- tics (ACL), 2002, pp. 311–318

    2002

  43. [51]

    Meteor: An automatic metric for mt evaluation with improved correlation with human judgments,

    S. Banerjee and A. Lavie, “Meteor: An automatic metric for mt evaluation with improved correlation with human judgments,” in Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization , 2005, pp. 65– 72

    2005

  44. [52]

    Optimizing statistical machine translation for text simplifica- tion,

    W. Xu, C. Napoles, E. Pavlick, Q. Chen, and C. Callison-Burch, “Optimizing statistical machine translation for text simplifica- tion,” Transactions of the Association for Computational Linguistics , vol. 4, pp. 401–415, 2016

    2016

  45. [53]

    Ground Truth for Grammatical Error Correction Metrics,

    C. Napoles, K. Sakaguchi, M. Post, and J. Tetreault, “Ground Truth for Grammatical Error Correction Metrics,” in Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Process- ing ...

    2015

  46. [54]

    The probabilistic relevance framework: Bm25 and beyond,

    S. Robertson, H. Zaragoza et al. , “The probabilistic relevance framework: Bm25 and beyond,” Foundations and Trends® in Infor- mation Retrieval, vol. 3, no. 4, pp. 333–389, 2009

    2009

  47. [55]

    Zeroshotdataaug: Gener- ating and augmenting training data with chatgpt,

    S. Ubani, S. O. Polat, and R. Nielsen, “Zeroshotdataaug: Gener- ating and augmenting training data with chatgpt,” arXiv preprint arXiv:2304.14334, 2023

    2023 arXiv

  48. [56]

    Want to reduce labeling cost? gpt-3 can help,

    S. Wang, Y. Liu, Y. Xu, C. Zhu, and M. Zeng, “Want to reduce labeling cost? gpt-3 can help,” in Findings of the Association for Computational Linguistics: EMNLP 2021, 2021, pp. 4195–4205

    2021

  49. [57]

    Magicoder: empowering code generation with oss-instruct,

    Y. Wei, Z. Wang, J. Liu, Y. Ding, and L. Zhang, “Magicoder: empowering code generation with oss-instruct,” in Proceedings of the 41st International Conference on Machine Learning (ICML) , 2024, pp. 52 632–52 657

    2024

  50. [58]

    Llm-assisted code cleaning for training accurate code generators,

    N. Jain, T. Zhang, W.-L. Chiang, J. E. Gonzalez, K. Sen, and I. Stoica, “Llm-assisted code cleaning for training accurate code generators,” in The Twelfth International Conference on Learning Representations (ICLR), 2023

    2023

  51. [59]

    Logupdater: Automated detection and repair of specific defects in logging statements,

    R. Zhong, Y. Li, J. Kuang, W. Gu, Y. Huo, and M. R. Lyu, “Logupdater: Automated detection and repair of specific defects in logging statements,” ACM Trans. Softw. Eng. Methodol. (TOSEM), Apr. 2025

    2025

  52. [60]

    Qwen2.5-coder,

    Q. Team, “Qwen2.5-coder,” 2025, accessed: 2025-12-13. [Online]. Available: https://qwenlm.github.io/blog/qwen2. 5-coder-family/

    2025

  53. [61]

    Detecting fragile comments,

    I. K. Ratol and M. P . Robillard, “Detecting fragile comments,” in 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE). Urbana, IL: IEEE, 2017, pp. 112–122

    2017

  54. [62]

    /* icomment: Bugs or bad comments?*,

    L. Tan, D. Yuan, G. Krishna, and Y. Zhou, “/* icomment: Bugs or bad comments?*,” in Proceedings of twenty-first ACM SIGOPS symposium on Operating systems principles (SOSP) , 2007, pp. 145– 158

    2007

  55. [63]

    Smartcoco: Checking comment-code inconsistency in smart contracts via constraint propagation and binding,

    S. Hao, Y. Nan, Z. Zheng, and X. Liu, “Smartcoco: Checking comment-code inconsistency in smart contracts via constraint propagation and binding,” in 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 2023, pp. 294–306

    2023

  56. [64]

    Au- tomating the removal of obsolete todo comments,

    Z. Gao, X. Xia, D. Lo, J. Grundy, and T. Zimmermann, “Au- tomating the removal of obsolete todo comments,” in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), 2021, pp. 218–229

    2021

  57. [65]

    Bert: Pre- training of deep bidirectional transformers for language under- standing,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre- training of deep bidirectional transformers for language under- standing,” in Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technol...

    2019

  58. [66]

    Longformer: The long- document transformer,

    I. Beltagy, M. E. Peters, and A. Cohan, “Longformer: The long- document transformer,” arXiv preprint arXiv:2004.05150, 2020

    2004 arXiv

  59. [67]

    An ensemble approach to detect code comment inconsistencies using topic modeling

    F. Rabbi, M. N. Haque, M. E. Kadir, M. S. Siddik, and A. Kabir, “An ensemble approach to detect code comment inconsistencies using topic modeling.” in SEKE, 2020, pp. 392–395

    2020

  60. [68]

    Towards detecting and understanding code-document violations in rust,

    W. Ouyang and B. Hua, “Towards detecting and understanding code-document violations in rust,” in 2021 IEEE International MANUSCRIPT 19 Symposium on Software Reliability Engineering Workshops (ISSREW) . IEEE, 2021, pp. 189–197

    2021

  61. [69]

    How do developers document database usages in source code?(n),

    M. Linares-V ´asquez, B. Li, C. Vendome, and D. Poshyvanyk, “How do developers document database usages in source code?(n),” in 2015 30th IEEE/ACM International Conference on Auto- mated Software Engineering (ASE). IEEE, 2015, pp. 36–41

    2015

  62. [70]

    On the relationship between comment update practices and software bugs,

    W. M. Ibrahim, N. Bettenburg, B. Adams, and A. E. Hassan, “On the relationship between comment update practices and software bugs,” Journal of Systems and Software (JSS) , vol. 85, no. 10, pp. 2293–2304, 2012

    2012

  63. [71]

    Taxonomy of inline code comment smells,

    E. Jabrayilzade, A. Yurto ˘glu, and E. T ¨uz ¨un, “Taxonomy of inline code comment smells,” Empirical Software Engineering (EMSE) , vol. 29, no. 3, p. 58, 2024

    2024

  64. [72]

    Suboptimal Comments in Java Projects: From Independent Comment Changes to Commenting Practices,

    C. Wang, H. He, U. Pal, D. Marinov, and M. Zhou, “Suboptimal Comments in Java Projects: From Independent Comment Changes to Commenting Practices,” ACM Transactions on Software Engineer- ing and Methodology (TOSEM), vol. 32, no. 2, pp. 1–33, 2023

    2023

  65. [73]

    CPC: Automatically classifying and propagating natural language comments via program analysis,

    J. Zhai, X. Xu, Y. Shi, G. Tao, M. Pan, S. Ma, L. Xu, W. Zhang, L. Tan, and X. Zhang, “CPC: Automatically classifying and propagating natural language comments via program analysis,” in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineer- ing (ICSE)....

    2020

  66. [74]

    A Large-Scale Empiri- cal Study on Code-Comment Inconsistencies,

    F. Wen, C. Nagy, G. Bavota, and M. Lanza, “A Large-Scale Empiri- cal Study on Code-Comment Inconsistencies,” in 2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC) . Montreal, QC, Canada: IEEE, 2019, pp. 53–64

    2019

  67. [75]

    Predictive comment updating with heuristics and ast-path-based neural learning: A two-phase approach,

    B. Lin, S. Wang, Z. Liu, X. Xia, and X. Mao, “Predictive comment updating with heuristics and ast-path-based neural learning: A two-phase approach,” IEEE Transactions on Software Engineering (TSE), vol. 49, no. 4, pp. 1640–1660, 2022

    2022

  68. [76]

    Why my code summa- rization model does not work: Code comment improvement with category prediction,

    Q. Chen, X. Xia, H. Hu, D. Lo, and S. Li, “Why my code summa- rization model does not work: Code comment improvement with category prediction,” ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 30, no. 2, pp. 1–29, 2021

    2021

  69. [77]

    Can llms replace human evaluators? an empirical study of llm-as-a-judge in software engineering,

    R. Wang, J. Guo, C. Gao, G. Fan, C. Y. Chong, and X. Xia, “Can llms replace human evaluators? an empirical study of llm-as-a-judge in software engineering,” arXiv preprint arXiv:2502.06193, 2025. APPENDIX A Prompt:Please help me judge whether the comment is inconsistent with t...

    2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.