{"id":"808091f7-ad93-4fae-bb1a-ed02faf9f582","arxiv_id":"2505.19757","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"CIDRe is a reference-free, four-part quality score for structured code comments, validated on 840 manually labeled Russian comments and reported to improve comment generation after dataset filtering.","lead":"This paper introduces CIDRe, a scoring system for code comments that needs no reference answer, combining four checks: relevance, informativeness, completeness, and length. It is tested on Russian code comments, and the authors say filtering data with it improves code-comment generation models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cross-entropy comparison is unfair: CIDRe is a trained classifier while SIDE/MIDQ are uncalibrated fixed metrics, so Table 2 does not establish superiority.","rationale":"The reader's weakest assumption focuses on the reliability of the 840 manual annotations as ground truth. That is a legitimate concern, and the paper indeed omits annotation guidelines, inter-annotator agreement, and sampling details. However, the more immediate and decisive problem is that even granting perfect labels, the central quantitative evidence in Table 2 does not support the claim. CIDRe is a trained probabilistic classifier, while SIDE and MIDQ are presented as fixed metrics; comparing them via cross-entropy without calibrating the baselines is structurally unfair. A trained model will naturally achieve lower CE on a label distribution it was fitted to, making the comparison circular in an operational sense. The downstream filtering results also lack any reported significance testing despite the abstract's claim of statistical significance. These issues are independent of annotation bias and directly undermine the paper's central claim. For these reasons, I agree with the REJECT verdict, though I locate the primary load-bearing weakness in the evaluation protocol rather than solely in the annotation data. A concrete calibration check on the baselines would settle whether the superiority claim is real or an artifact.","tokens_in":10054,"tokens_out":3158,"duration_ms":37608,"concrete_test":"Reconstruct the train/test partition used for the SVM (§5). On the training split, fit a monotone calibration map (e.g., isotonic regression or Platt scaling) to convert raw SIDE and MIDQ scores into probabilities. Recompute cross-entropy on the same 100-comment test set used in Table 2. If calibrated SIDE or MIDQ CE approaches or falls below CIDRe-SVM's 1.35, the claimed superiority is an artifact of uncalibrated baselines. As a calibration-agnostic secondary check, compute Spearman rank correlation with the 100 human labels for CIDRe, SIDE, and MIDQ; if CIDRe does not significantly outperform on correlation, the criterion's practical advantage is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing flaw is the evaluation protocol behind the headline CE numbers in Table 2, not only the annotation quality. CIDRe's final score is an SVM trained on the 840 manual labels (§4, §5), whereas SIDE and MIDQ are fixed, uncalibrated heuristics. Cross-entropy is a proper scoring rule that rewards calibrated probabilistic predictions; applying it to raw SIDE/MIDQ scores without any calibration step all but guarantees that a classifier fitted on the same label distribution will appear better. Thus the reported CE values (CIDRe-SVM 1.35 vs SIDE 5.32 vs MIDQ 7.55) show only that a supervised model can fit the label distribution better than an uncalibrated heuristic, not that CIDRe is a superior quality criterion. The paper does not describe how SIDE/MIDQ scores were converted to probabilities, whether the 100-comment test set was disjoint from the 840 training annotations, or whether a train/test split was respected. Additionally, the abstract claims 'statistically significant quality gains', but §7 reports only raw side-by-side percentages in Table 4, with no significance test, confidence interval, or number of judged instances. Even under perfect annotation labels, the central superiority claim collapses because the comparison metric is rigged by construction.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CIDRe, a reference-free criterion for structured code comment quality, defined as a combination of four components: completeness, informativeness, description length, and relevance. The authors manually annotate 840 comments from StRuCom, train a binary classifier (SVM, LightGBM, logistic regression) on the four component scores, and report cross-entropy comparisons against SIDE and MIDQ. They also use CIDRe to filter the StRuCom training set and evaluate fine-tuned Qwen2.5-Coder models with GPT-4o-mini side-by-side judgments. The central claims are that CIDRe outperforms existing metrics and that CIDRe-based filtering yields statistically significant quality gains.","tokens_in":10329,"tokens_out":6972,"duration_ms":71771,"significance":"If the claims were established, CIDRe would be a useful resource for dataset curation in code comment generation, particularly for Russian. The paper's decomposition into four aspects is reasonable, and the manual annotation effort is potentially valuable. However, the current validation does not support the headline claims: the main cross-entropy comparison is unfair as implemented, the ground-truth annotation process is undocumented, and the significance claim in the abstract is not backed by any statistical test. The paper would need substantial re-evaluation before the central claims can be accepted.","major_comments":[{"comment":"The headline cross-entropy comparison does not establish CIDRe's superiority. CIDRe-SVM is a classifier trained on the 840 manual labels, whereas SIDE and MIDQ are fixed metrics with no described calibration step. Cross-entropy is a proper scoring rule that rewards calibrated probabilities; comparing raw SIDE/MIDQ scores with the output of a trained classifier is biased by construction. The authors must specify how SIDE/MIDQ scores were converted to probabilities, calibrate the baselines on a training split (e.g., Platt scaling), and state explicitly that the 100-comment test set is disjoint from the 840 training annotations. Without these steps, the values 1.35 vs. 5.32 vs. 7.55 cannot support the superiority claim.","section":"§5, Table 2"},{"comment":"The manual annotation ground truth is not documented to a standard that supports training a quality criterion. No annotation guidelines, annotator background, inter-annotator agreement, or sampling method are reported for the 840 comments. Since the final CIDRe score is an SVM trained on these labels, any bias in the labels (e.g., preference for longer comments or a particular docstring style) is baked into the criterion. The authors should report Cohen's kappa or equivalent, the class distribution, and the exact train/test split, and describe the annotation instructions.","section":"§4, §5"},{"comment":"The abstract claims 'statistically significant quality gains', but no statistical test is reported. Table 4 gives raw side-by-side percentages only; there are no confidence intervals, no p-values, and no statement of how many pairwise judgments were collected. Even with the position-bias reversal described in Appendix E, the paper needs a proper paired significance test (e.g., Wilcoxon signed-rank or bootstrap) over the judged instances, or the significance claim must be removed.","section":"§7, Table 4, Abstract"},{"comment":"The ablation study reports F1 = 0.994 for the full feature set, but it is unclear whether this is training or held-out performance, and no variance or cross-validation details are given. This is especially important because the component definitions contain hand-set parameters: the completeness weights in Appendix A, the cosine threshold 0.5 in Appendix D, and the triplet margin 0.3 in Appendix C. The authors should report the evaluation protocol for Table 3 (e.g., repeated stratified cross-validation) and provide a sensitivity analysis for the hand-set parameters.","section":"§6, Table 3"}],"minor_comments":[{"comment":"The abstract says 'language-agnostic', but Section 9 states the metric is 'specifically optimized for Russian-language documentation'; either qualify the claim or add multilingual experiments.","section":"Abstract vs. §9"},{"comment":"The Qwen2.5-Coder-0.5B-Instruct Python row contains eight slash-separated values instead of four, making the row unreadable and suggesting a formatting error.","section":"Table 4"},{"comment":"The evaluation against GitHub Copilot is not described with enough detail, including the Copilot version, the prompt template, and how the test subset was sampled.","section":"§7"},{"comment":"The 'independent test set of 100 code comments' is not described in terms of sampling strategy or class balance, which is needed to interpret the cross-entropy numbers.","section":"§5"},{"comment":"The paper does not state whether code and data will be released; for a metric paper this is important for reproducibility.","section":"General"},{"comment":"Figure 1 and its caption contain formatting artifacts, including stray Russian text and line numbers, which should be cleaned before submission.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern in the reader's report is valid: Table 2 as it stands cannot support the paper's central claim. I nevertheless see a path to revision if the authors can re-run the comparison with calibrated baselines and a proper held-out test, and report annotation reliability and significance tests. If these data are not available or the experiments cannot be rerun, the claims should be scaled back substantially, and the central contribution would then be only the annotated dataset plus a set of features, not a validated quality criterion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is the 840-comment manually annotated Russian dataset and the four-aspect decomposition (relevance, informativeness, completeness, length) fused through a trained classifier. That benchmark is worth having, and the motivation—SIDE and MIDQ don't work well for Russian structured comments—is real. The paper is also honest enough to state in its limitations that the criterion is tuned for Russian docstrings, not a universal metric. I believe the authors are trying to solve a real curation problem, and the four-aspect framing is a reasonable starting point.\n\nThe problem is that the central superiority claim in Table 2 does not survive scrutiny. CIDRe-SVM gets CE 1.35 versus SIDE 5.32 and MIDQ 7.55, but that comparison is trained-versus-untrained. CIDRe is an SVM fitted on the 840 labels, while SIDE and MIDQ are fixed, uncalibrated heuristics. Cross-entropy is a proper scoring rule—it rewards calibrated probabilities. Without calibrating the baselines, the CE gap mostly shows that a supervised model can fit the label distribution better than raw heuristic scores, not that CIDRe is a better quality criterion. The paper also never says whether the 100-comment test set is disjoint from the 840 training annotations, and the baseline scores-to-probabilities conversion is not described. This is not a minor omission; it is the load-bearing comparison.\n\nThe downstream experiments have the same weakness in weaker form. The abstract claims statistically significant quality gains, but Table 4 is raw side-by-side percentages with no significance test, no confidence interval, and no stated number of judged cases. Some rows genuinely look better with filtration (e.g., Qwen2.5-Coder-7B Python), but others are mixed or worse, so without a test the claim is unsubstantiated. There is also no inter-annotator agreement, no annotation guideline description, and no artifact release—all of which matter because the final classifier is literally trained on those annotations.\n\nI want to be fair: the ablation study is internally consistent, the components are individually motivated, and the hand-set thresholds (cosine 0.5, triplet margin 0.3, completeness weights) are more of a tuning detail than a fatal flaw by themselves. But the evaluation protocol and the missing significance testing are fatal to the paper as currently written.\n\nWho should read it? Researchers working on code summarization dataset curation, especially for low-resource languages, will want to know this benchmark exists. But they should not cite the Table 2 numbers as evidence of superiority. If the authors recalibrate the baselines, report split details, add annotation reliability, and actually test the downstream differences, this could become a solid contribution. As is, it is a reject-and-revise, not a desk-reject-without-comment.\n\nRecommendation: send it to peer review only as a major-revision candidate, with the burden on the authors to fix the comparison protocol and provide real statistical support. The benchmark itself deserves referee time; the current claims do not.","headline":"A useful Russian annotation benchmark and a plausible four-aspect criterion, but the headline comparison is rigged by construction and the significance claim is unsupported; needs major revision.","tokens_in":10817,"tokens_out":1750,"would_cite":false,"duration_ms":22789,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A reference-free score built from four comment signals ranks code comments better than existing metrics.","keywords":["code comment quality","reference-free metric","dataset filtering","docstring generation","StRuCom","cross-entropy evaluation","LLM-as-a-judge","structured comments"],"falsifier":"Have two independent annotator teams re-label the same 840 StRuCom comments plus a fresh balanced sample and measure inter-annotator agreement; if agreement is near chance, or if CIDRe's ranking of held-out comments is explained by description length alone after controlling for the other features, the quality claim is falsified.","tokens_in":9875,"feed_emoji":"💬","tokens_out":7933,"duration_ms":74128,"temperature":0.7,"pith_summary":"This paper tries to establish that CIDRe, a reference-free score built from four aspects of a structured code comment (completeness, informativeness, description length, and relevance), ranks code-comment quality more accurately than existing metrics and can curate training data for comment generation. The authors manually label 840 comments from the Russian StRuCom corpus as good or bad, train a support-vector machine on the four-feature representation, and report a cross-entropy of 1.35 on a test set, versus 5.32 for SIDE and 7.55 for MIDQ. Filtering StRuCom with the CIDRe probability threshold of 0.5 then improves the side-by-side quality judgments of Qwen2.5-Coder models from 0.5B to 7B parameters. If the paper is right, dataset builders can drop gold-reference requirements and use one continuous score to filter docstrings, with the same four-part recipe re-trainable for new languages.","feed_headline":"Four-part score judges code comments without a reference","feed_subtitle":"CIDRe filters docstring datasets, beating SIDE and MIDQ and lifting fine-tuned model quality.","key_machinery":"The load-bearing object is the four-dimensional feature vector $(C,I,D,R)$ consumed by an SVM. $C$ is the ratio of documented structural elements to available ones, generalizing MIDQ's Documented Items Ratio beyond Java; $I$ is the attention-weighted fraction of code identifiers whose terms reappear in the comment, using SFR-Embedding-Code-400M_R attention weights and ConceptNet Numberbatch cosine matching; $D$ is raw comment length in characters; $R$ is a semantic alignment score from CodeSage-small-v2 fine-tuned with triplet loss on hard negatives mined from StRuCom. The SVM translates that vector into a probability of belonging to the 'good' class, and the probability itself is the criterion.","core_discovery":"CIDRe is the claim that comment quality decomposes into four measurable signals: structural completeness (which parameters, exceptions, and return values are documented), informativeness (which code terms, weighted by attention, appear in the comment), description length (character count as a proxy for detail sufficiency), and relevance (semantic alignment between code and comment learned by fine-tuning CodeSage-small-v2 with triplet loss). These four features are fed to a binary SVM trained on 840 manually annotated good/bad comments from StRuCom, producing a score in $[0,1]$. The paper reports that the SVM version outperforms SIDE and MIDQ in cross-entropy (1.35 vs 5.32 and 7.55), and that filtering the training set at a 0.5 threshold yields statistically significant quality gains across five programming languages and four model sizes.","pith_inferences":["The paper's 'language-agnostic' framing is stronger than its evidence: the annotations, hard-negative mining, and relevance fine-tuning are Russian-specific, so transfer to English or other languages should be verified rather than assumed.","A control the paper does not report, comparing CIDRe-filtered data with a random subset of the same size, would separate quality selection from the generic benefit of discarding noisy training examples.","Because description length enters as raw character count, the score could be learning verbosity; a length-normalized ablation would show whether detail sufficiency is independent of sheer length."],"forward_implications":["Filtering StRuCom at a CIDRe threshold of 0.5 raises win rates in GPT-4o-mini side-by-side comparisons against GitHub Copilot across Python, Java, Go, C#, and JavaScript for Qwen2.5-Coder models from 0.5B to 7B.","Because CIDRe needs no reference comment, it can rank and curate dataset entries where gold summaries are unavailable, unlike BLEU or ROUGE-L.","The ablation shows all four components are load-bearing: removing any single feature drops the SVM F1 from 0.994 to at most 0.950.","The score's continuity in $[0,1]$ lets curators choose a filtering threshold to trade data volume against expected quality."],"supporting_citations":[{"why":"Supplies the StRuCom corpus of 153K Russian code-comment pairs that CIDRe filters and that provides the comments for manual annotation.","marker":"(Dziuba and Malykh, 2025)"},{"why":"Defines the SIDE baseline and supplies the reference-free coherence idea that the relevance component adapts.","marker":"(Mastropaolo et al., 2024)"},{"why":"Defines the MIDQ baseline and the Documented Items Ratio that the completeness component generalizes beyond Java.","marker":"(Scalabrino et al., 2017)"},{"why":"Provides the CodeSage-small-v2 embedding model that is fine-tuned with triplet loss to compute relevance.","marker":"(Zhang et al., 2024)"},{"why":"Provides the CodeXEmbed/SFR-Embedding-Code-400M_R model whose attention weights are used to weight terms in the informativeness component.","marker":"(Liu et al., 2024)"},{"why":"Supplies ConceptNet Numberbatch multilingual embeddings used for cosine matching between code terms and comment words.","marker":"(Speer et al., 2017)"},{"why":"Establishes the LLM-as-a-judge protocol used for the GPT-4o-mini side-by-side evaluation.","marker":"(Zheng et al., 2023)"},{"why":"Defines STASIS, the term-coverage idea that informativeness extends by adding attention-based term weights.","marker":"(Li et al., 2006)"},{"why":"Provides the triplet-loss objective used to fine-tune the relevance embedding model.","marker":"(Schroff et al., 2015)"}],"fun_headline_variants":["CIDRe: four signals score comments without a reference","Reference-free comment scoring with four quality signals","Four-part CIDRe filter lifts docstring quality across languages","CIDRe beats SIDE and MIDQ in comment quality scoring"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 840 manual good/bad labels are unbiased, reliable ground truth; the paper does not report annotation guidelines, annotator background, sampling method, or inter-annotator agreement, so any bias in those labels (for example, favoring longer or more structured comments) is baked into the SVM and inherited by the filtering results.","fun_headline_variants_meta":{"raw":{"variants":["CIDRe: four signals score comments without a reference","Reference-free comment scoring with four quality signals","Four-part CIDRe filter lifts docstring quality across languages","CIDRe beats SIDE and MIDQ in comment quality scoring"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000188,"raw_usage":{"total_tokens":1280,"prompt_tokens":844,"completion_tokens":436,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":460,"completion_tokens_details":{"reasoning_tokens":372}},"tokens_in":460,"tokens_out":436,"duration_ms":5272,"temperature":1.0,"reasoning_tokens":372,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:06:40.160853+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have two independent annotator teams re-label the same 840 StRuCom comments plus a fresh balanced sample and measure inter-annotator agreement; if agreement is near chance, or if CIDRe's ranking of held-out comments is explained by description length alone after controlling for the other features, the quality claim is falsified.","supporting_citations":[{"cited_title":"StRuCom: A Novel Dataset of Structured Code Comments in Russian","cited_arxiv_id":"2505.11026","evidence_quote":"Supplies the StRuCom corpus of 153K Russian code-comment pairs that CIDRe filters and that provides the comments for manual annotation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the SIDE baseline and supplies the reference-free coherence idea that the relevance component adapts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the MIDQ baseline and the Documented Items Ratio that the completeness component generalizes beyond Java."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the CodeSage-small-v2 embedding model that is fine-tuned with triplet loss to compute relevance."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the triplet-loss objective used to fine-tune the relevance embedding model."}],"review_version":1}