Pith. sign in

REVIEW 4 major objections 4 minor 4 cited by

DocPrism claims decomposing doc-code inconsistency detection into local yes/no questions plus an external under-promise filter cuts LLM flag rate from 98% to 14%.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A zero-shot LLM prompting scheme (local categorization + external filtering) detects code-documentation incorrectness with low flag rates and about 0.6 precision across Python, TypeScript, C++, and Java.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection LCEF is a real and useful prompt-engineering idea, and the paper reads honestly, but the headline ablation is in-sample, so the 98%-to-14% flag-rate drop is not yet established. the 4 major comments →

arxiv 2511.00215 v2 pith:WJHXVTCL submitted 2025-10-31 cs.SE

DocPrism: Multi-lingual Detection of Incorrectness Inconsistencies between Code and Documentation

Xiaomeng Xu , Zahin Wahab , Reid Holmes , Caroline Lemieux This is my paper
classification cs.SE
keywords code-documentation inconsistencyLLM zero-shot detectionover-promisedirect mismatchunder-promise filteringlocal categorizationexternal filteringmulti-language software analysis
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DocPrism sets out to show that large language models can be made useful for spotting incorrectness inconsistencies between function-level documentation and code—specifically over-promises and direct mismatches—without fine-tuning. The paper argues that plain prompting makes LLMs flag nearly every function because they report benign under-promises, cases where the code does more than the documentation says. The proposed remedy, Local Categorization and External Filtering (LCEF), splits detection into per-category yes/no questions encoded in a JSON schema and then discards under-promise answers during post-processing. Across 1,615 real functions in Python, TypeScript, C++, and Java, DocPrism holds a 14-18% flag rate with roughly 0.6 precision; an ablation shows flag rate dropping from 98% to 14% and F1 rising from 0.22 to 0.77. If correct, developers can run practical post-hoc doc-code mismatch detection on existing codebases in many languages with a standard LLM and no training data.

Core claim

The paper's central claim is that the standard way of prompting LLMs for code-documentation consistency is structurally wrong: asking an LLM to 'find inconsistencies' exploits its tendency to agree and its difficulty ignoring benign abstraction gaps, so it flags 90% or more of functions. The fix is to make the model answer three narrow yes/no questions—one per inconsistency kind—embedded as JSON keys with self-contained follow-ups, and to drop the under-promise answers after generation. With this Local Categorization + External Filtering recipe, the paper reports flag rate falling from 98% to 14% and F1 rising from 0.22 to 0.77 on its 52-function construction set, and, on 1,615 real function

What carries the argument

Local Categorization, External Filtering (LCEF). The system prompt is a JSON schema whose entries are check-in keys (CK)—short yes/no questions, one per inconsistency category (over-promise, direct mismatch, under-promise)—paired with self-contained, verbose follow-up keys (FK) that are filled only when the check-in answer calls for it. This forces the LLM to act as a completion engine on each local field rather than performing open-ended reasoning, and the external filtering stage simply removes the under-promise JSON entries before producing the report. The mechanism does two jobs: it makes category assignment reliable enough that the filter can trust the labels, and it is the primary leve

Load-bearing premise

The entire evaluation rests on the authors' own hand-written labels of what counts as a reportable inconsistency—with only a single Cohen's kappa of 0.78 computed on 50 functions in one language—so if other developers draw the under-promise boundary differently, both the precision numbers and the design decision to suppress under-promises lose their foundation.

What would settle it

Have a fresh set of annotators who are blind to the authors' labels independently mark which of the 1,615 code-documentation pairs contain an over-promise or direct mismatch, then run DocPrism and compute precision against those labels; the central claim fails if precision falls below roughly 0.5 or if a majority of the fresh annotators judge a substantial share of the filtered-away under-promises as actionable. A simpler observational check: if a replication on a new language (e.g., Go or Rust) under independent labeling yields a flag rate above 25% or precision below 0.4, the claimed general

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • A standard LLM, prompted with LCEF and no fine-tuning, can flag over-promises and direct mismatches in Python, TypeScript, C++, and Java at a flag rate of 14-18% and precision around 0.6, tractable for manual developer review.
  • Because DocPrism needs no diff or edit history, it can be run post hoc on existing repositories to detect latent inconsistencies that just-in-time tools miss.
  • Across four popular real-world projects, at least 11% of function-documentation pairs contain an incorrectness inconsistency (over-promise or direct mismatch); this is a conservative lower bound since unflagged pairs were not exhaustively checked.
  • The ablation isolates the gain: pairing check-in questions with verbose follow-up keys reduces the under-promise rate from 50% to 0% relative to the closest instructing-and-filtering alternative, and the effect appears to transfer to a different LLM family in a supplementary run reported in the paper.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The check-in/follow-up schema is a generic antidote to LLM over-reporting in verification-style tasks; analogous 'is this specific property violated?' schemas could reduce false positives in security triage, test generation, or compliance review, where the same sycophantic bias exists.
  • The paper assumes under-promises are never worth showing; a tool that surfaces them as optional low-severity warnings, rather than dropping them outright, might capture the cases where an omitted detail is a real trap (the Tesseract example in the paper hints the LLM thought so) without losing the flag-rate benefit.
  • Because the ground-truth labels were produced and adjudicated by the authors themselves, with only one 50-function kappa check, an independent re-labeling study is the natural next step; if other developers disagree about which under-promises are harmless, the 0.52-0.70 precision numbers may not transfer to new settings.
  • The 11% lower bound likely understates the real rate: recall is not measured on the extension datasets, and the construction dataset's labeled recall of 0.64 suggests actual inconsistencies may be more frequent than the flagged ones.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents DocPrism, a zero-shot LLM-based tool for detecting incorrectness inconsistencies between function-level documentation and code. To avoid the high false-positive rates of plain LLM prompting, the authors introduce Local Categorization, External Filtering (LCEF): the LLM is prompted to answer per-category yes/no check-in questions encoded in a JSON schema, and under-promise findings are removed by a deterministic external filter after generation. The tool is evaluated on a manually labeled Python dataset (124 functions, 14 inconsistent) and on extension datasets in TypeScript, C++, and Java (1,491 functions), with manual inspection of flagged items. The paper reports low flag rates (11–18%), precision around 0.6, and a headline ablation result in which LCEF reduces the flag rate from 98% to 14% and increases F1 from 0.22 to 0.77 on a 52-benchmark construction dataset.

Significance. If the reported results hold, the paper makes a useful practical contribution: it demonstrates that a simple, fine-tuning-free prompt structure can make LLM-based inconsistency detection tractable for developers by keeping the flag rate low while preserving moderate precision, and it goes beyond binary classification by producing highlighted snippets and explanations. The LCEF idea—decomposing a categorization task into local check-in questions and delegating filtering to deterministic code—is simple, plausible, and potentially transferable to other LLM-based SE tasks. The paper is also transparent about its design trade-offs, releases an artifact, performs manual error analysis, and includes a limited cross-LLM check with GPT-4.1. The main weakness is that the central comparative claim for LCEF is supported only by an in-sample ablation, and some abstract-level claims are not backed by experiments in the body.

major comments (4)
  1. [§4.2.1, §4.7.2, Table 6] The headline ablation result (98%→14% flag rate, F1 0.22→0.77) is computed on the 52-benchmark construction dataset that was explicitly used while developing DocPrism and LCEF. Section 4.2.1 states that this subset was examined at tool-creation time. The paper does not report the same ablation on the 72 held-out Python benchmarks or on the full 124-benchmark labeled set; Table 1 reports only DocPrism's final performance on the full set, not the V1–V7 variants. Because the JSON schema, check-in wording, and external-filter design could have been selected on these 52 cases, the observed improvement may be substantially inflated by selection. This is load-bearing for the abstract's central claim. The authors should either run the ablation on the held-out or full labeled Python set, or explicitly restrict the claim to the construction dataset and temper the general statement.
  2. [Abstract (arXiv metadata)] The provided abstract states: 'DocPrism achieves precision comparable to the state-of-the-art on an established synthetic dataset, but substantially outperforms it on our real-world Java dataset in precision (DocPrism: 0.47-0.67 vs. SOTA: 0.05-0.14).' It also claims 'a conservative lower bound across four programming languages... 11% of code-documentation pairs.' Neither of these claims is supported by any experiment, table, or derivation in the manuscript body. Section 5 only mentions prior precision numbers (icomment 0.61, tcomment 0.48, clean-dataset precision 0.54–0.61) but does not compare them to DocPrism on the same data. If these comparisons exist, they must be added to the paper; if not, the abstract must be corrected. An unsupported state-of-the-art comparison is a serious accuracy issue in the submission's summary.
  3. [§4.5.1, Table 1] The fully labeled Python evaluation rests on only 14 inconsistent functions (9 TP + 5 FN). The reported precision, recall, and F1 of 0.64 are therefore estimates based on a small positive denominator, and no confidence intervals are provided. For example, a 95% Wilson interval for 9/14 is roughly 0.39–0.84, which spans values that would change the qualitative conclusion. The same issue affects the construction-set ablation in Table 6, where the positive count is likely even smaller. The paper should report confidence intervals or at least explicitly acknowledge the width of these estimates. This is load-bearing for the absolute metric claims, though it does not invalidate the design itself.
  4. [§4.2.2, §4.6, Table 4] For the extension datasets, recall, accuracy, and F1 cannot be computed because only flagged functions were manually labeled. The paper states this clearly, but the multi-language generalization claim in the abstract ('maintains a low flag rate... precision') is consequently limited to precision and flag rate. This is acceptable as a scoping statement, but the paper should be explicit that the LCEF benefit—particularly the recall/F1 improvement—is not validated across languages. The current wording in the introduction ('across this diverse dataset it achieves... a flag rate of 15%, and a precision of 0.62') reads as a broad performance claim without this caveat. Please add the caveat to the abstract and introduction.
minor comments (4)
  1. [Abstract vs. full text] The arXiv metadata abstract and the paper's own abstract disagree on several numbers: metadata says F1 0.22→0.77 and flag rate 17%, precision 0.63; the paper text says accuracy 14%→94% and flag rate 15%, precision 0.62. Both are based on the same tables, so one version should be aligned. Also, RQ3 heading in Section 4 misspells 'External' as 'Exernal'.
  2. [§3.1.2] The JSON key for the direct-mismatch check-in is printed as 'Does_the_code_correctly_implement_what_is_mentioned in_the_documentation' with a space before 'in_the_documentation'. This appears to be a formatting artifact but should be corrected for clarity, since the schema is central to the method.
  3. [§6 (Generalization to Other LLMs)] The GPT-4.1 robustness check reports only flag rates (87%→20%) and no precision, recall, or number of benchmarks. This is a useful sanity check but is too coarse to support the claim that 'LCEF remains effective' beyond flag-rate reduction. Consider reporting the same metrics as Table 6 if feasible.
  4. [§4.4] The definition of inconsistency-level precision is never explicitly written, though it is used in Tables 1 and 4. Please add the formula or a verbal definition for consistency with the function-level metrics.

Circularity Check

0 steps flagged

No definitional or self-citation circularity; the LCEF ablation's in-sample nature is a validity threat, not an equation-level reduction.

full rationale

I found no step where a claimed result reduces to its own inputs by construction. The paper contains no fitted equations and no load-bearing self-citations. The external filter does not definitionally guarantee LCEF's headline numbers: on the extension datasets, 3--8% of reported inconsistencies are still under-promises (Table 4, U.P. rate), showing the filter can be defeated by misclassification; the 0% under-promise rate in Table 6 is an empirical outcome, not a logical consequence of the design. The most significant concern is disclosed by the authors in §4.2.1: "While creating DocPrism and the LCEF methodology, we examined performance on a subset of our Python dataset—52 benchmarks in total. We call this subset the construction dataset. The remaining 72 benchmarks were not used at tool-creation time." The main ablation (Table 6) is reported on exactly this construction set, so prompt and schema choices may be tuned to those 52 cases, and no holdout ablation with recall/accuracy is reported. This is a real in-sample-selection threat to the 98%→14% flag-rate and 0.22→0.77 F1 claims, but it is a validity threat, not circularity: the measured reduction is not statistically forced by the method's definition, and the extension flag-rate results plus the GPT-4.1 generalization check provide partial independent evidence. The author-made labels with a single Cohen's kappa on 50 functions are also a validity concern, again not an equation-level reduction. No self-citation chain or uniqueness theorem is invoked, and the under-promise filtering decision is tied to an external prior taxonomy rather than to the authors' own prior work. Overall, no significant circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

No numeric parameters were fit to data in the usual sense, but the method's prompt structure, check-in wording, temperature, and base model are hand-chosen design decisions that drive the reported performance. The key assumptions are the domain judgment about under-promises and the reliability of author labeling.

free parameters (4)
  • LCEF JSON schema design (8 keys, verbose follow-up keys) = N/A (hand-designed)
    Prompt structure chosen during early exploratory experiments on construction data (§3.1.2); it is the main driver of the reported flag-rate reduction.
  • Check-in key wording for direct mismatch = N/A (hand-designed)
    The paper itself notes the key 'Does the code correctly implement...' is semantically vague and causes category overlap (§6); wording was not derived from theory.
  • Temperature = 0
    Set to encourage deterministic LLM output (§3.2); affects reproducibility of all reported numbers.
  • Base LLM = LLaMA 3.1-70B
    All main results use this model; GPT-4.1 is only checked for flag rate (§6), so precision claims are model-specific.
axioms (5)
  • domain assumption Under-promises (undocumented implementation details) are not worth reporting and should be filtered from developer-facing output.
    Justifies the central LCEF objective; stated in §2 ('we do not believe they should be shown to developers') and supported by a prior taxonomy, but it is a value judgment about developer utility.
  • domain assumption Author-generated manual labels are an unbiased ground truth for code-documentation inconsistency.
    All metrics depend on labels; only one inter-coder check (kappa 0.78 on 50 functions in one language, §4.2) is reported and no blinding is described.
  • ad hoc to paper LLM JSON parse failures can be safely treated as 'consistent'.
    §3.2 sets detection result to 'consistent' when output doesn't match JSON format; this inflates true negatives if failures correlate with actual inconsistencies.
  • ad hoc to paper Verbose follow-up keys improve schema adherence; this was found in early exploratory experiments and built into the method.
    §3.1.2 states non-verbose keys caused the LLM to list inconsistencies even when the check-in answer was 'No'; the final prompt encodes this hand-tuning.
  • domain assumption LLaMA 3.1-70B's behavior generalizes to other LLMs for the full precision claim.
    The GPT-4.1 check (§6) only reports flag-rate reduction (87% to 20%), not precision/recall, so the broad precision claim is supported for only one model.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of DocPrism: Multi-lingual Detection of Incorrectness Inconsistencies between Code and Documentation." pith.science (2026). https://pith.science/paper/WJHXVTCL

@misc{pith2026251100215,
  author       = {Pith},
  title        = {Pith review of: DocPrism: Multi-lingual Detection of Incorrectness Inconsistencies between Code and Documentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WJHXVTCL}},
  note         = {Machine review of arXiv:2511.00215}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Code-documentation inconsistencies are common and undesirable: they can lead to developer misunderstandings and software defects. This paper introduces DocPrism, a lightweight multi-language, code-documentation inconsistency detection tool. DocPrism uses a standard large language model (LLM) to analyze and explain inconsistencies, and focuses on outputting incorrectness inconsistencies. Plain use of LLMs for this task yields unacceptably high inconsistency flag rates-i.e., over 90% of functions are flagged as inconsistent with their documentation. One substantial reason is that LLMs identify natural gaps between high-level documentation and code as incompleteness inconsistencies. We introduce and apply the Local Categorization, External Filtering (LCEF) methodology: LCEF uses an LLM's local completion skills, rather than its long-term reasoning skills, to focus on reporting incorrectness inconsistencies. In our ablation study, LCEF reduces DocPrism's inconsistency flag rate from 98% to 14%, and increases F1 score from 0.22 to 0.77, compared to standard prompting techniques. On a broad evaluation across Python, TypeScript, C++, and Java, DocPrism maintains a low flag rate of 17%, and achieves a precision of 0.63 without performing any fine-tuning. We also establish a conservative lower bound across four programming languages, showing that inconsistency errors are present in 11% of code-documentation pairs. In addition, DocPrism achieves precision comparable to the state-of-the-art on an established synthetic dataset, but substantially outperforms it on our real-world Java dataset in precision (DocPrism: 0.47-0.67 vs. SOTA: 0.05-0.14).

Figures

Figures reproduced from arXiv: 2511.00215 by Caroline Lemieux, Reid Holmes, Xiaomeng Xu, Zahin Wahab.

Figure 1
Figure 1. Figure 1: Three broad kinds of inconsistencies between function￾level documentation and its source code implementation [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: A Python over-promise identified by DocPrism, along with its highlighting and rationale (Pandas-48). Over-Promise. Over-promises are cases where the method-level documentation over-advertises the intended behaviors of the func￾tion. I.e., some features described in the documentation are not implemented in the code [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: A TypeScript direct mismatch identified by DocPrism, along with its highlighting and rationale (Hoppscotch-37). the highest OrderIndex plus one. However, the code simply returns rootCollectionCount[0].orderIndex without adding one, re￾vealing a logical conflict between the identified documentation and code snippet. These mismatches are problematic, as they represent contradictions between what the implemen… view at source ↗
Figure 4
Figure 4. Figure 4: A C++ under-promise identified by DocPrism, along with its highlighting and rationale (Tesseract-89). Core Challenge: Under-Promises. Using LLMs to perform in￾consistency detection is challenging because LLMs tend to answer questions in a way that aligns positively with user expectations [17]. In particular, LLMs are eager to flag under-promise inconsistencies, as they are differences between code and docu… view at source ↗
Figure 5
Figure 5. Figure 5: Schematic workflow of DocPrism. Section 3 explains each of its components in detail. {"Documentation_Summary": "...", "Code_Summary": "...", ...} Local Categorization Subtasks. The remaining three pairs of JSON keys apply the approach of local categorization, which locally and implicitly guides the LLM to identify over-promises, direct mismatches, and under-promises. Each pair has a check-in key (CK) and a… view at source ↗
Figure 6
Figure 6. Figure 6: An example of a false positive reported by DocPrism due to lack of API knowledge (Requests-26). The 13 correctly identified inconsistencies span five categories: 5 are unimplemented features , 4 are parameter type mismatches, 2 are parameter mismatches, 1 is functionality mismatch, and 1 is return type mismatch. We detail these categories next. 4.5.2 Classification of Correctly Identified Inconsistencies. … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Measuring LLM Trust Allocation Across Conflicting Software Artifacts

    cs.SE 2026-04 unverdicted novelty 7.0

    TRACE reveals that LLMs detect documentation bugs and contradictions better than subtle implementation drift, with asymmetric sensitivity and poor confidence calibration across seven models on 22k traces.

  2. Measuring LLM Trust Allocation Across Conflicting Software Artifacts

    cs.SE 2026-04 conditional novelty 7.0

    Seven LLMs detect documentation faults far more reliably than implementation drift; when only the code changes and Javadoc stays intact, inconsistency detection falls by 21–43 percentage points.

  3. We Must Have Missed This Comment: Detecting and Repairing Stale Function References in Linux Kernel Comments

    cs.SE 2026-08 accept novelty 6.0

    ReCite detects 869 stale function references in Linux kernel v6.18-rc1 comments, with 89% of sampled repairs judged useful and 42.5% directly applicable.

  4. Beyond Coverage and Kill Scores: Empirically Measuring Test Suite Behavioural Gaps

    cs.SE 2026-06 unverdicted novelty 6.0

    An empirical study extracts 20,729 expected behaviors from ten Java libraries and finds 17.5% remain untested, independent of line coverage and mutation scores.

Reference graph

Works this paper leans on

39 extracted references · 2 linked inside Pith · cited by 3 Pith papers

  1. [1]

    Emad Aghajani, Csaba Nagy, Mario Linares-Vásquez, Laura Moreno, Gabriele Bavota, Michele Lanza, and David C Shepherd. 2020. Software documentation: the practitioners’ perspective. InProceedings of the ACM/IEEE 42nd International Conference on Software Engineering. 590–601

    2020

  2. [2]

    Emad Aghajani, Csaba Nagy, Olga Lucero Vega-Márquez, Mario Linares-Vásquez, Laura Moreno, Gabriele Bavota, and Michele Lanza. 2019. Software documenta- tion issues unveiled. In2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 1199–1210

    2019

  3. [3]

    Zhanqi Cui, Shifan Liu, Li Li, and Liwei Zheng. 2025. SEOCD: Detecting obsolete code comments by fusing semantic features and expert features.Expert Systems with Applications280 (2025), 127470

    2025

  4. [4]

    Guo, and Nghi Bui

    Anh Dau, Jin L.c. Guo, and Nghi Bui. 2024. DocChecker: Bootstrapping Code Large Language Model for Detecting and Resolving Code-Comment Inconsis- tencies. InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, Nikolaos Ale- tras and Orphee De Clercq (Eds.). Association for Comp...

    2024

  5. [5]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models.arXiv preprint arXiv:2407.21783(2024)

    Pith/arXiv arXiv 2024

  6. [6]

    Hoppscotch Contributors. 2025. Hoppscotch: An open-source ecosystem for API development and testing. https://github.com/hoppscotch/hoppscotch. Accessed: 2025-06-16

    2025

  7. [7]

    Yuan Huang, Yinan Chen, Xiangping Chen, and Xiaocong Zhou. 2025. Are your comments outdated? Toward automatically detecting code-comment consistency. Journal of Software: Evolution and Process37, 1 (2025), e2718

    2025

  8. [8]

    Yuan Huang, Shaohao Huang, Huanchao Chen, Xiangping Chen, Zibin Zheng, Xiapu Luo, Nan Jia, Xinyu Hu, and Xiaocong Zhou. 2020. Towards automati- cally generating block comments for code snippets.Information and Software Technology127 (2020), 106373

    2020

  9. [9]

    Walid M Ibrahim, Nicolas Bettenburg, Bram Adams, and Ahmed E Hassan

  10. [10]

    Michael Dubem Igbomezie, Phuong T Nguyen, and Davide Di Ruscio. 2024. When simplicity meets effectiveness: Detecting code comments coherence with word embeddings and LSTM. InProceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering. 411–416

    2024

  11. [11]

    Zhiyong Liu, Huanchao Chen, Xiangping Chen, Xiaonan Luo, and Fan Zhou

  12. [12]

    Zhongxin Liu, Xin Xia, David Lo, Meng Yan, and Shanping Li. 2021. Just-in- time obsolete comment detection and update.IEEE Transactions on Software Engineering49, 1 (2021), 1–23

    2021

  13. [13]

    Mary McHugh. 2012. Interrater reliability: The kappa statistic.Biochemia medica : časopis Hrvatskoga društva medicinskih biokemičara / HDMB22 (10 2012), 276–82. doi:10.11613/BM.2012.031

    doi:10.11613/bm.2012.031 2012

  14. [14]

    Marcus Nachtigall, Lisa Nguyen Quang Do, and Eric Bodden. 2019. Explaining Static Analysis - A Perspective. In2019 34th IEEE/ACM International Conference on Automated Software Engineering Workshop (ASEW). 29–32. doi:10.1109/AS EW.2019.00023

    arXiv 2019

  15. [15]

    Sheena Panthaplackel, Junyi Jessy Li, Milos Gligoric, and Raymond J Mooney

  16. [16]

    Fazle Rabbi and Md Saeed Siddik. 2020. Detecting code comment inconsis- tency using siamese recurrent network. InProceedings of the 28th international conference on program comprehension. 371–375

    2020

  17. [17]

    Leonardo Ranaldi and Giulia Pucci. 2025. When Large Language Mod- els contradict humans? Large Language Models’ Sycophantic Behaviour. arXiv:2311.09410 [cs.CL] https://arxiv.org/abs/2311.09410

    Pith/arXiv arXiv 2025

  18. [18]

    Pooja Rani. 2021. Speculative analysis for quality assessment of code com- ments. In2021 IEEE/ACM 43rd International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). IEEE, 299–303

    2021

  19. [19]

    Robillard, Deeksha M

    Martin P. Robillard, Deeksha M. Arya, Neil A. Ernst, Jin L. C. Guo, Maxime Lamothe, Mathieu Nassif, Nicole Novielli, Alexander Serebrenik, Igor Stein- macher, and Klaas-Jan Stol. 2024. Communicating Study Design Trade-offs in Software Engineering.ACM Trans. Softw. Eng. Methodol.33, 5, Article 112 (June 2024), 10 pages. doi:10.1145/3649598

    doi:10.1145/3649598 2024

  20. [20]

    Guoping Rong, Yongda Yu, Song Liu, Xin Tan, Tianyi Zhang, Haifeng Shen, and Jidong Hu. 2025. Code Comment Inconsistency Detection and Rectification Using a Large Language Model . In2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE Computer Society, Los Alamitos, CA, USA, 432–443. doi:10.1109/ICSE55347.2025.00035

    arXiv 2025

  21. [21]

    Lakmal Silva, Michael Unterkalmsteiner, and Krzysztof Wnuk. 2022. MultiDimEr: a multi-dimensional bug analyzEr. InProceedings of the International Conference on Technical Debt. 66–70

    2022

  22. [22]

    Janice Singer. 1998. Practices of software maintenance. InProceedings of the International Conference on Software Maintenance (ICSM). 139–145

    1998

  23. [23]

    Nataliia Stulova, Arianna Blasi, Alessandra Gorla, and Oscar Nierstrasz. 2020. Towards detecting inconsistent comments in java source code automatically. In2020 IEEE 20th international working conference on source code analysis and manipulation (SCAM). IEEE, 65–69

    2020

  24. [24]

    Zhi Rui Tam, Cheng-Kuang Wu, Yi-Lin Tsai, Chieh-Yen Lin, Hung-yi Lee, and Yun-Nung Chen. 2024. Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Large Language Model Performance.. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, Franck Dernoncourt, Daniel Preoţiuc-Pietro, and Anas...

    2024

  25. [25]

    Jiaxi Tan, Shikai Guo, Zijian Tao, Zhiguo Yang, and Hui Li. 2025. Just-In-Time Detection of Outdated Comments in Software Development by Jointly Reasoning. IEEE Transactions on Consumer Electronics(2025)

    2025

  26. [26]

    Lin Tan. 2015. Code comment analysis for improving software quality. InThe art and science of analyzing software data. Elsevier, 493–517

    2015

  27. [27]

    Lin Tan, Ding Yuan, Gopal Krishna, and Yuanyuan Zhou. 2007. /*icomment: bugs or bad comments?*/. InProceedings of Twenty-First ACM SIGOPS Sym- posium on Operating Systems Principles(Stevenson, Washington, USA)(SOSP ’07). Association for Computing Machinery, New York, NY, USA, 145–158. doi:10.1145/1294261.1294276

    arXiv 2007

  28. [28]

    Shin Hwei Tan, Darko Marinov, Lin Tan, and Gary T Leavens. 2012. @ tcomment: Testing javadoc comments to detect comment-code inconsistencies. In2012 IEEE Fifth International Conference on Software Testing, Verification and Validation. IEEE, 260–269

    2012

  29. [29]

    Tesseract OCR Developers. 2025. Tesseract: Open-source OCR engine. https: //github.com/tesseract-ocr/tesseract. Accessed: 2025-06-16

    2025

  30. [30]

    The pandas development team. 2025. pandas: Powerful Python data analysis toolkit. https://github.com/pandas-dev/pandas. Accessed: 2025-06-16

    2025

  31. [31]

    The Requests development team. 2025. Requests library. https://github.com/psf /requests. Accessed: 2025-06-16

    2025

  32. [32]

    Gias Uddin and Martin P Robillard. 2015. How API documentation fails.IEEE Software32, 4 (2015), 68–75

    2015

  33. [33]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reason- ing in large language models.Advances in neural information processing systems 35 (2022), 24824–24837

    2022

  34. [34]

    Xiufeng Xu, Fuman Xie, Chenguang Zhu, Guangdong Bai, Sarfraz Khurshid, and Yi Li. 2025. Identifying Multi-Parameter Constraint Errors in Python Data Science Library API Documentations. In34th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA)

    2025

  35. [35]

    Zhengkang Xu, Shikai Guo, Yumiao Wang, Rong Chen, Hui Li, Xiaochen Li, and He Jiang. 2024. Code Comment Inconsistency Detection Based on Confidence Learning.IEEE Transactions on Software Engineering(2024)

    2024

  36. [36]

    Yichi Zhang, Zixi Liu, Yang Feng, and Baowen Xu. 2024. Leveraging Large Language Model to Assist Detecting Rust Code Comment Inconsistency. In Proceedings of the 39th IEEE/ACM International Conference on Automated Soft- ware Engineering(Sacramento, CA, USA)(ASE ’24). Association for Computing Machinery, New York, NY, USA, 356–366. doi:10.1145/3691620.3695010

    arXiv 2024

  37. [2012]

    On the relationship between comment update practices and software bugs.Journal of Systems and Software85, 10 (2012), 2293–2304

    2012

  38. [2018]

    In2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC), Vol

    Automatic detection of outdated comments during code changes. In2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC), Vol. 1. IEEE, 154–163

  39. [2021]

    InProceedings of the AAAI Conference on Artificial Intelligence, Vol

    Deep just-in-time inconsistency detection between comments and source code. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 427–435

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.