Pith. sign in

REVIEW 4 major objections 6 minor 28 references

CIDRe: A Reference-Free Multi-Aspect Criterion for Code Comment Quality Measurement

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A reference-free score built from four comment signals ranks code comments better than existing metrics.

desk verdict A useful Russian annotation benchmark and a plausible four-aspect criterion, but the headline comparison is rigged by construction and the significance claim is unsupported; needs major revision. read the letter →

arxiv 2505.19757 v1 pith:5XIEM4PS submitted 2025-05-26 cs.SE cs.AIcs.CLcs.LG

Maria Dziuba , Valentin Malykh This is my paper · ORCID
classification cs.SEcs.AIcs.CLcs.LG
keywords codecommentqualityreference-freemetricdatasetfilteringdocstringgenerationStRuComcross-entropyevaluationLLM-as-a-judgestructuredcomments
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that CIDRe, a reference-free score built from four aspects of a structured code comment (completeness, informativeness, description length, and relevance), ranks code-comment quality more accurately than existing metrics and can curate training data for comment generation. The authors manually label 840 comments from the Russian StRuCom corpus as good or bad, train a support-vector machine on the four-feature representation, and report a cross-entropy of 1.35 on a test set, versus 5.32 for SIDE and 7.55 for MIDQ. Filtering StRuCom with the CIDRe probability threshold of 0.5 then improves the side-by-side quality judgments of Qwen2.5-Coder models from 0.5B to 7B parameters. If the paper is right, dataset builders can drop gold-reference requirements and use one continuous score to filter docstrings, with the same four-part recipe re-trainable for new languages.

What carries the argument

The load-bearing object is the four-dimensional feature vector $(C,I,D,R)$ consumed by an SVM. $C$ is the ratio of documented structural elements to available ones, generalizing MIDQ's Documented Items Ratio beyond Java; $I$ is the attention-weighted fraction of code identifiers whose terms reappear in the comment, using SFR-Embedding-Code-400M_R attention weights and ConceptNet Numberbatch cosine matching; $D$ is raw comment length in characters; $R$ is a semantic alignment score from CodeSage-small-v2 fine-tuned with triplet loss on hard negatives mined from StRuCom. The SVM translates that vector into a probability of belonging to the 'good' class, and the probability itself is the criterion.

What would settle it

Have two independent annotator teams re-label the same 840 StRuCom comments plus a fresh balanced sample and measure inter-annotator agreement; if agreement is near chance, or if CIDRe's ranking of held-out comments is explained by description length alone after controlling for the other features, the quality claim is falsified.

Watch

Extended reading notes

Core claim

CIDRe is the claim that comment quality decomposes into four measurable signals: structural completeness (which parameters, exceptions, and return values are documented), informativeness (which code terms, weighted by attention, appear in the comment), description length (character count as a proxy for detail sufficiency), and relevance (semantic alignment between code and comment learned by fine-tuning CodeSage-small-v2 with triplet loss). These four features are fed to a binary SVM trained on 840 manually annotated good/bad comments from StRuCom, producing a score in $[0,1]$. The paper reports that the SVM version outperforms SIDE and MIDQ in cross-entropy (1.35 vs 5.32 and 7.55), and that filtering the training set at a 0.5 threshold yields statistically significant quality gains across five programming languages and four model sizes.

Load-bearing premise

The 840 manual good/bad labels are unbiased, reliable ground truth; the paper does not report annotation guidelines, annotator background, sampling method, or inter-annotator agreement, so any bias in those labels (for example, favoring longer or more structured comments) is baked into the SVM and inherited by the filtering results.

Editorial extensions

If this is right

  • Filtering StRuCom at a CIDRe threshold of 0.5 raises win rates in GPT-4o-mini side-by-side comparisons against GitHub Copilot across Python, Java, Go, C#, and JavaScript for Qwen2.5-Coder models from 0.5B to 7B.
  • Because CIDRe needs no reference comment, it can rank and curate dataset entries where gold summaries are unavailable, unlike BLEU or ROUGE-L.
  • The ablation shows all four components are load-bearing: removing any single feature drops the SVM F1 from 0.994 to at most 0.950.
  • The score's continuity in $[0,1]$ lets curators choose a filtering threshold to trade data volume against expected quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's 'language-agnostic' framing is stronger than its evidence: the annotations, hard-negative mining, and relevance fine-tuning are Russian-specific, so transfer to English or other languages should be verified rather than assumed.
  • A control the paper does not report, comparing CIDRe-filtered data with a random subset of the same size, would separate quality selection from the generic benefit of discarding noisy training examples.
  • Because description length enters as raw character count, the score could be learning verbosity; a length-normalized ablation would show whether detail sufficiency is independent of sheer length.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes CIDRe, a reference-free criterion for structured code comment quality, defined as a combination of four components: completeness, informativeness, description length, and relevance. The authors manually annotate 840 comments from StRuCom, train a binary classifier (SVM, LightGBM, logistic regression) on the four component scores, and report cross-entropy comparisons against SIDE and MIDQ. They also use CIDRe to filter the StRuCom training set and evaluate fine-tuned Qwen2.5-Coder models with GPT-4o-mini side-by-side judgments. The central claims are that CIDRe outperforms existing metrics and that CIDRe-based filtering yields statistically significant quality gains.

Significance. If the claims were established, CIDRe would be a useful resource for dataset curation in code comment generation, particularly for Russian. The paper's decomposition into four aspects is reasonable, and the manual annotation effort is potentially valuable. However, the current validation does not support the headline claims: the main cross-entropy comparison is unfair as implemented, the ground-truth annotation process is undocumented, and the significance claim in the abstract is not backed by any statistical test. The paper would need substantial re-evaluation before the central claims can be accepted.

major comments (4)
  1. [§5, Table 2] The headline cross-entropy comparison does not establish CIDRe's superiority. CIDRe-SVM is a classifier trained on the 840 manual labels, whereas SIDE and MIDQ are fixed metrics with no described calibration step. Cross-entropy is a proper scoring rule that rewards calibrated probabilities; comparing raw SIDE/MIDQ scores with the output of a trained classifier is biased by construction. The authors must specify how SIDE/MIDQ scores were converted to probabilities, calibrate the baselines on a training split (e.g., Platt scaling), and state explicitly that the 100-comment test set is disjoint from the 840 training annotations. Without these steps, the values 1.35 vs. 5.32 vs. 7.55 cannot support the superiority claim.
  2. [§4, §5] The manual annotation ground truth is not documented to a standard that supports training a quality criterion. No annotation guidelines, annotator background, inter-annotator agreement, or sampling method are reported for the 840 comments. Since the final CIDRe score is an SVM trained on these labels, any bias in the labels (e.g., preference for longer comments or a particular docstring style) is baked into the criterion. The authors should report Cohen's kappa or equivalent, the class distribution, and the exact train/test split, and describe the annotation instructions.
  3. [§7, Table 4, Abstract] The abstract claims 'statistically significant quality gains', but no statistical test is reported. Table 4 gives raw side-by-side percentages only; there are no confidence intervals, no p-values, and no statement of how many pairwise judgments were collected. Even with the position-bias reversal described in Appendix E, the paper needs a proper paired significance test (e.g., Wilcoxon signed-rank or bootstrap) over the judged instances, or the significance claim must be removed.
  4. [§6, Table 3] The ablation study reports F1 = 0.994 for the full feature set, but it is unclear whether this is training or held-out performance, and no variance or cross-validation details are given. This is especially important because the component definitions contain hand-set parameters: the completeness weights in Appendix A, the cosine threshold 0.5 in Appendix D, and the triplet margin 0.3 in Appendix C. The authors should report the evaluation protocol for Table 3 (e.g., repeated stratified cross-validation) and provide a sensitivity analysis for the hand-set parameters.
minor comments (6)
  1. [Abstract vs. §9] The abstract says 'language-agnostic', but Section 9 states the metric is 'specifically optimized for Russian-language documentation'; either qualify the claim or add multilingual experiments.
  2. [Table 4] The Qwen2.5-Coder-0.5B-Instruct Python row contains eight slash-separated values instead of four, making the row unreadable and suggesting a formatting error.
  3. [§7] The evaluation against GitHub Copilot is not described with enough detail, including the Copilot version, the prompt template, and how the test subset was sampled.
  4. [§5] The 'independent test set of 100 code comments' is not described in terms of sampling strategy or class balance, which is needed to interpret the cross-entropy numbers.
  5. [General] The paper does not state whether code and data will be released; for a metric paper this is important for reproducibility.
  6. [Figure 1] Figure 1 and its caption contain formatting artifacts, including stray Russian text and line numbers, which should be cleaned before submission.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the low cross-entropy of CIDRe reflects a supervised model fitted to the same label distribution, but the paper describes an independent 100-comment test set, so the result is not forced by construction.

full rationale

CIDRe is defined as a binary classifier trained on 840 manual good/bad labels, and its four components are computed by explicit formulas (completeness ratio, term-overlap ratio, character length, and a fine-tuned relevance embedding) that do not themselves contain the target labels. Section 5 reports evaluation on an 'independent test set of 100 code comments' with cross-entropy, so on the paper's own description the Table 2 figures are held-out predictions rather than fits to the training labels. The comparison is methodologically questionable: SIDE and MIDQ are fixed, uncalibrated scores while CIDRe is a probability-calibrated classifier, and no calibration procedure or significance test is reported for the side-by-side results in Table 4. Those are evaluation-validity concerns, not a reduction of the prediction to its inputs by construction. The only self-citations are to the authors' StRuCom dataset and prior models; the dataset is an experimental resource, and no uniqueness theorem or ansatz is imported from those citations to force CIDRe's form. The limitation that the metric is optimized for Russian structured docstrings weakens the 'language-agnostic' claim, but that is a scope limitation rather than circular reasoning. No equation or fitted parameter was found that is equivalent by definition to the claimed outcome.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central score is a trained classifier on manual labels; the four component formulas contain hand-set weights and thresholds, and the annotation ground truth is assumed without reliability evidence. No invented entities are introduced.

free parameters (4)
  • Completeness formula weights = 2 (exceptions), 3 (parameters), 1 (return/description), varying by language
    Hand-chosen coefficients in Appendix A, Equations 1-4; no principled derivation, and they directly determine the completeness score.
  • Informativeness cosine threshold = 0.5
    Appendix D: a term is considered found if cosine similarity is greater than 0.5; threshold chosen by hand.
  • Relevance triplet margin = 0.3
    Appendix C: Triplet Loss margin for fine-tuning CodeSage-small-v2; no sensitivity analysis is provided.
  • SVM classifier (criterion fusion) = trained decision boundary on 840 labels
    The final CIDRe score is the output of an SVM trained on the four components against manual labels; the fitted weights are learned from the validation data, so the criterion is a fitted classifier, not a parameter-free formula.
assumptions (5)
  • domain assumption Competent programmers assign semantically meaningful names to code identifiers, so identifier terms carry functional information.
    Section 3.2 Informativeness, basis for term extraction and weighting.
  • domain assumption Longer comments with detailed explanations are higher quality.
    Section 3.3 Description Length; used as a positive quality component despite no causal evidence.
  • domain assumption Manual binary annotations of 840 comments are reliable and representative ground truth.
    No annotation guidelines, agreement metrics, or sampling details are reported; the entire validation rests on this premise.
  • domain assumption GPT-4o-mini is a valid judge for code comment quality relative to human judgment.
    Appendix E relies on the LLM-as-a-judge paradigm, citing Zheng et al. 2023, but does not measure agreement on this specific task.
  • domain assumption Attention weights from SFR-Embedding-Code-400M_R reflect term importance in code comments.
    Appendix B selects this model based on a single qualitative example; no quantitative evaluation is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CIDRe: A Reference-Free Multi-Aspect Criterion for Code Comment Quality Measurement." pith.science (2026). https://pith.science/paper/5XIEM4PS

@misc{pith2026250519757,
  author       = {Pith},
  title        = {Pith review of: CIDRe: A Reference-Free Multi-Aspect Criterion for Code Comment Quality Measurement},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5XIEM4PS}},
  note         = {Machine review of arXiv:2505.19757}
}
read the original abstract

Effective generation of structured code comments requires robust quality metrics for dataset curation, yet existing approaches (SIDE, MIDQ, STASIS) suffer from limited code-comment analysis. We propose CIDRe, a language-agnostic reference-free quality criterion combining four synergistic aspects: (1) relevance (code-comment semantic alignment), (2) informativeness (functional coverage), (3) completeness (presence of all structure sections), and (4) description length (detail sufficiency). We validate our criterion on a manually annotated dataset. Experiments demonstrate CIDRe's superiority over existing metrics, achieving improvement in cross-entropy evaluation. When applied to filter comments, the models finetuned on CIDRe-filtered data show statistically significant quality gains in GPT-4o-mini assessments.

Figures

Figures reproduced from arXiv: 2505.19757 by the authors.

Figure 1
Figure 1. Criteria pipeline. The proposed four quality [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. An example of completeness calculation SFR-Embedding-Code-400M_R was selected as the baseline architecture for informativeness computa￾tion. Key experimental findings include: 1. The Qwen2.5-Coder and deepseek-coder￾1.3b-base models exhibited excessive selec￾tivity, disregarding semantically significant terms (e.g., the term player in test cases re￾ceived insufficient attention weights). 2. Unixcoder-base-nine assig… view at source ↗
Figure 3
Figure 3. Visualization of the weighting of terms by importance using different models [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 12 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171--4186

    2019

  4. [4]

    Maria Dziuba and Valentin Malykh. 2025. https://arxiv.org/abs/2505.11026 Strucom: A novel dataset of structured code comments in russian . Preprint, arXiv:2505.11026

    work page Pith review arXiv 2025

  5. [5]

    Rudolf Flesch. 1979. How to write plain english. University of Canterbury. Available at http://www. mang. canterbury. ac. nz/writing\_guide/writing/flesch. shtml.[Retrieved 5 February 2016]

    1979

  6. [6]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3

    2022

  7. [7]

    Summarizing source code with transferred api knowledge.(2018)

    Xing HU, Ge LI, Xin XIA, David LO, Shuai LU, and Zhi JIN. Summarizing source code with transferred api knowledge.(2018). In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelli-gence (IJCAI 2018), Stockholm, Sweden, 2018 July 13, volume 19, pages 2269--2275

    work page 2018

  8. [8]

    Yuan Huang, Yinan Chen, Xiangping Chen, and Xiaocong Zhou. 2025. Are your comments outdated? toward automatically detecting code-comment consistency. Journal of Software: Evolution and Process, 37(1):e2718

    work page 2025

Show all 28 references
  1. [9]

    Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. https://doi.org/10.48550/arXiv.1909.09436 Codesearchnet challenge: Evaluating the state of semantic code search

    2019 doi arXiv

  2. [10]

    Denis Kocetkov, Raymond Li, Loubna Allal, Jia Li, Chenghao Mou, Carlos Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro Werra, and Harm Vries. 2022. https://doi.org/10.48550/arXiv.2211.15533 The stack: 3 tb of permissively licen...

    2022 doi arXiv

  3. [11]

    Alexander LeClair and Collin McMillan. 2019. Recommendations for datasets for source code summarization. arXiv preprint arXiv:1904.02660

    2019 arXiv

  4. [12]

    Yuhua Li, David McLean, Zuhair A Bandar, James D O'shea, and Keeley Crockett. 2006. Sentence similarity based on semantic nets and corpus statistics. IEEE transactions on knowledge and data engineering, 18(8):1138--1150

    2006

  5. [13]

    Ye Liu, Rui Meng, Shafiq Joty, Silvio Savarese, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. 2024. Codexembed: A generalist embedding model family for multiligual and multi-task code retrieval. arXiv preprint arXiv:2411.12644

    2024 arXiv

  6. [14]

    Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101

    2017 arXiv

  7. [15]

    Antonio Mastropaolo, Matteo Ciniselli, Massimiliano Di Penta, and Gabriele Bavota. 2024. Evaluating code summarization techniques: A new metric and an empirical characterization. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1--13

    2024

  8. [16]

    Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781

    2013 arXiv

  9. [17]

    George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39--41

    1995

  10. [18]

    Dung Nguyen, Le Nam, Anh Dau, Anh Nguyen, Khanh Nghiem, Jin Guo, and Nghi Bui. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.316 The vault: A comprehensive multilingual dataset for advancing code understanding and generation . In Findings of the Association for Computa...

    2023 doi

  11. [19]

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1--16. IEEE

    2020

  12. [20]

    Simone Scalabrino, Gabriele Bavota, Christopher Vendome, Mario Linares-V \'a squez, Denys Poshyvanyk, and Rocco Oliveto. 2017. Automatically assessing code understandability: How far are we? In 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE)...

    2017

  13. [21]

    Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815--823

    2015

  14. [22]

    Lin Shi, Fangwen Mu, Xiao Chen, Song Wang, Junjie Wang, Ye Yang, Ge Li, Xin Xia, and Qing Wang. 2022. Are we building on the rock? on the importance of data preprocessing for code summarization. In Proceedings of the 30th ACM Joint European Software Engineering Conference and ...

    2022

  15. [23]

    Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. Mpnet: Masked and permuted pre-training for language understanding. Advances in neural information processing systems, 33:16857--16867

    2020

  16. [24]

    Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. http://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14972 ConceptNet 5.5: An open multilingual graph of general knowledge . pages 4444--4451

    2017

  17. [25]

    Antonio Vitale, Antonio Mastropaolo, Rocco Oliveto, Massimiliano Di Penta, and Simone Scalabrino. 2025. Optimizing datasets for code summarization: Is code-comment coherence enough? arXiv preprint arXiv:2502.07611

    2025 arXiv

  18. [26]

    Xu, and Graham Neubig

    Zhiruo Wang, Grace Cuenca, Shuyan Zhou, Frank F. Xu, and Graham Neubig. 2023. https://doi.org/10.18653/v1/2023.findings-eacl.20 MC o N a L a: A benchmark for code generation from multiple natural languages . In Findings of the Association for Computational Linguistics: EACL 20...

    2023 doi

  19. [27]

    Dejiao Zhang, Wasi Uddin Ahmad, Ming Tan, Hantian Ding, Ramesh Nallapati, Dan Roth, Xiaofei Ma, and Bing Xiang. 2024. https://openreview.net/forum?id=vfzRRjumpX CODE REPRESENTATION LEARNING AT SCALE . In The Twelfth International Conference on Learning Representations

    2024

  20. [28]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595--46623

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.