{"id":"416ada03-f80f-46f6-a7a0-26d71290ac82","arxiv_id":"2505.01136","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CppSATD introduces a large C++ comment dataset in which 13,044 comments were confirmed as self-admitted technical debt by human annotators and the remainder were labeled NON-SATD largely by pattern-based inference.","lead":"Researchers built CppSATD, a dataset of 531,367 comments mined from five major C++ open-source projects, labeled with five types of self-admitted technical debt plus a NON-SATD class. It is advertised as the first C++ multiclass SATD dataset with surrounding code context, meant to support detection tools and cross-language studies.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The NON-SATD half of the dataset is inferred, not annotated; the paper's own sample found 25 SATD comments in that set and the released data still labels them NON-SATD, so the dataset ships known mislabeled negatives.","rationale":"The reader's weakest assumption is exactly the pattern-based NON-SATD inference, and the paper's own text supports this concern concretely: the sample found 25 SATD comments that were not added back, and the construct-validity paragraph admits the risk while offering a statistically imprecise reassurance. My independent check of the manuscript confirmed this is the most load-bearing issue. The paper has real strengths: the manual annotation of 18,973 candidate SATD comments produced 13,044 multiclass labels with a kappa of 0.86, the replication package is public and includes scripts, and the limitations section is candid. The concern does not rise to rejection because the estimated mislabeling rate in the negative class is under 0.25% at 99% confidence, which is small for training but still matters for evaluation and should be disclosed or corrected. The abstract's '531,000 annotated comments' overstates the manual annotation effort, but the Records and Storage section and the separate manual_annotations.csv partly mitigate this. Therefore the reader's CONDITIONAL verdict is appropriate, and my analysis does not move it.","tokens_in":10732,"tokens_out":6997,"duration_ms":69961,"concrete_test":"Download the Zenodo replication package, open manual_annotations.csv, and extract the 25 comments that the second author labeled as SATD within the 16,125-comment NON-SATD sample (the SATD-positive rows from Table 6's sample). Match them by comment id or text to cppsatd.csv and check the 'annotation' field for each. If any of these 25 are labeled NON-SATD in the released dataset, the known mislabeling is shipped; the descriptor should then either correct the labels, exclude those comments, or explicitly document the known errors. If all 25 are corrected or absent, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that CppSATD contains 531,367 comments 'annotated' as one of five SATD types or NON-SATD and can serve as training/evaluation data. The load-bearing condition is that the labels are trustworthy. That condition is weakest for the 512,394 comments that did not match any of the 21 ETF or 656 HTF patterns from Pentacet (Section C 'Data Preparation'). These are labeled NON-SATD without manual review, based on the assumption that the pattern set has near-complete recall in C++. The only evidence offered is a 16,125-comment random sample manually annotated by the second author, which found 25 SATD comments (0.155%, Table 6). The paper then states that 'we applied the NON-SATD label to all leftover comments after ETF and HTF pattern filtering' (Section D), meaning the 25 discovered SATD comments were left in the dataset as NON-SATD. This is a known, shipped mislabeling. The statistical argument that 'with a confidence level of 99% the remaining set has less than 1% chance of containing some SATD comments' is also a misreading: the 99% confidence interval for the proportion has an upper bound near 0.24%, not a 1% probability of containing any SATD. For binary SATD detection, a ~0.15% false-negative rate in the negative class is small, but for evaluation it will systematically depress precision, and the dataset descriptor does not disclose the known errors in the released file. The abstract overstates manual annotation when it calls all 531,367 comments 'annotated' while the replication package separates only 35,098 manually annotated comments. These issues do not invalidate the positive SATD annotations (13,044 manual labels, kappa 0.86), but they do undercut the claim that the full dataset is a reliable benchmark without caveats.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CppSATD, a dataset of 531,367 C++ comments extracted from five large open-source repositories (TensorFlow, React Native, Godot, Bitcoin, Swift), labeled into five SATD types (Design/Code, Requirement, Defect, Test, Documentation) or NON-SATD, and augmented with preceding and succeeding code contexts. The construction pipeline uses srcML for comment extraction, then filters comments against 21 ETF and 656 HTF patterns from Sridharan et al. [12], producing 18,973 candidate SATD comments that were manually annotated by the first author (13,044 SATD and 5,929 NON-SATD). The remaining 512,394 comments are treated as NON-SATD after a random sample of 16,125 was manually annotated by the second author; that sample found 25 SATD comments. Inter-rater agreement was assessed on 385-comment samples, yielding Cohen's kappa 0.86 for SATD annotations and 1.0 for NON-SATD annotations. The authors position CppSATD as the first C++ SATD dataset with multiclass annotations and code contexts, intended to support detection and cross-language SATD research.","tokens_in":10964,"tokens_out":7326,"duration_ms":74595,"significance":"If the label quality is trustworthy, CppSATD fills a real gap: it offers a large C++ corpus with code contexts, a rare multiclass SATD annotation, and a replication package containing scripts, patterns, and the annotation reference document. The manually annotated core (13,044 positives and the 18,973-comment candidate set) is a solid starting point, and the reported kappa of 0.86 for the SATD set is credible. However, the dataset's usefulness for training and, especially, for evaluation depends on the validity of the 512,394 NON-SATD labels, and this is precisely where the descriptor overstates the evidence and ships known label errors. The artifact is valuable in principle, but the descriptor needs substantive corrections to make the provenance and limitations of each label explicit.","major_comments":[{"comment":"The central claim that CppSATD contains 531,367 'annotated' comments is not supported by the described procedure. Section D and Records and Storage show that only 35,098 comments (18,973 candidate SATD plus 16,125 candidate NON-SATD) were manually annotated; the remaining 496,269 comments received the NON-SATD label by pattern-based inference. Moreover, Section D reports that the second author found 25 SATD comments in the 16,125-comment random sample, yet then states that 'we applied the NON-SATD label to all leftover comments after ETF and HTF pattern filtering.' The released dataset therefore contains known false negatives, and the abstract overstates the annotation effort. Please revise the wording to distinguish 'manually annotated' from 'inferred/labeled,' document the 25 known SATD instances explicitly, and either correct their labels in cppsatd.csv or provide a supplementary file listing known mislabeled comments.","section":"Abstract, Section D, Records and Storage"},{"comment":"The statistical statement 'we can conclude with a confidence level of 99% that the remaining set has less than 1% chance of containing some SATD comments' misstates what the confidence interval supports. With 25 SATD comments out of 16,125, the observed proportion is 0.155%, and a 99% confidence interval's upper bound is around 0.24%, not a 1% probability that the set contains any SATD. The correct claim is that the data are consistent at 99% confidence with a true prevalence below about 0.24% (and hence also below 1%). In addition, Equation (1) with z=2.58, p=0.5, and E=0.01 gives n=16,589, not 16,125; if a finite-population correction was used, it should be stated. Please report the interval, the correction, and the exact claim precisely.","section":"Section D, Eq. (1), Construct Validity"},{"comment":"The decision to label all 512,394 non-matching comments as NON-SATD assumes that the 21 ETF and 656 HTF patterns have near-complete recall for C++. The paper provides no precision/recall evaluation of these patterns on C++ comments, and the only empirical check, the 16,125-comment random sample, already falsifies the zero-hidden-SATD assumption by finding 25 SATD comments. Under simple random sampling, 25/16,125 extrapolates to roughly 794 mislabeled comments in the full 512,394 set. I request a direct estimate: report pattern precision and recall against the 18,973 manually annotated candidate set and the 16,125-comment sample, and use the resulting false-negative rate to state the expected number of mislabeled NON-SATD comments in the full dataset. This number should appear in the dataset descriptor so that downstream users can calibrate evaluation metrics.","section":"Section C and Construct Validity"}],"minor_comments":[{"comment":"Please replace 'annotated' with 'labeled' when referring to all 531,367 comments and reserve 'manually annotated' for the 35,098 comments in manual_annotations.csv; this would align the text with the actual annotation effort.","section":"Abstract and Records and Storage"},{"comment":"The phrase 'the first C++ SATD dataset with multiclass SATD annotations' should be substantiated with a brief literature search or softened to 'to our knowledge, the first'; otherwise the novelty claim is unverifiable.","section":"Background"},{"comment":"Please state whether the sample size of 16,125 was computed with a finite-population correction and which z-value and margin of error were used in the final calculation, since the formula and parameter values as printed yield a different number.","section":"Section C, Equation (1)"},{"comment":"The headers 'ETFSATD' and 'HTFSATD' should read 'ETF SATD' and 'HTF SATD' for consistency with the text.","section":"Section C, Table 3"},{"comment":"The dataset DOI appears as 10.5281/zenodo.15275192 in the article metadata and as 10.5281/zenodo.15562944 in reference [22]; please verify which DOI is current and use it consistently.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The NON-SATD inference relies on the pattern set of Sridharan et al. [12], which is co-authored by one of the present authors; I recommend requiring a clear statement of this dependency and an independent or at least transparent validation of pattern recall. The discrepancy between the two DOIs should also be resolved before publication, as it affects the replication package's accessibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Net: this is a solid dataset paper. It delivers what it claims in the abstract — a C++ SATD set with five types and code context — and it is packaged well enough to become the default C++ resource. But the abstract oversells the annotation, and the NON-SATD half has a small known error that should be fixed or disclosed before release.\n\nWhat is new: previous SATD resources are Java-centric (Maldonado-62k) or multi-language without a reported C++ multiclass subset (Pentacet). CppSATD provides 531,367 comments from five well-known C++ repos with surrounding code, 13,044 manually annotated SATD comments across five types, and a kappa of 0.86 on a stratified sample. The annotation reference document, code, and data are public. That is real value.\n\nWhere the soft spots are:\n\n- The abstract says 'over 531,000 annotated comments.' Actually 35,098 were manually labeled; the other 512,394 got NON-SATD by not matching the pattern set. That is a reasonable design, but 'annotated' overstates it.\n- Section D reveals 25 SATD comments found in the 16,125-comment NON-SATD sample. The paper then labels all leftover comments as NON-SATD anyway. So the released dataset knowingly contains at least these 25 positives as negatives. In absolute terms this is tiny (0.155%), but for anyone evaluating binary detectors it silently deflates precision. The fix is either to add the 25 back or to document them as a known limitation. Right now neither is done.\n- The statistical justification is misstated. The 16,125 sample gives a confidence interval for the proportion, not a 'less than 1% chance' that any SATD exists. The proper statement is that the true proportion is likely below roughly 0.25%. Minor, but it deserves a correction.\n- The paper reuses ETF/HTF patterns from Pentacet, by one of the authors. That is fine for a dataset, but a direct comparison of coverage or label distributions with Pentacet is still missing. Not a blocker.\n\nThe positive annotations themselves look trustworthy: two annotators with cross-review, kappa 0.86, stratification by type. The limitations section is candid but understates the NON-SATD issue.\n\nBottom line: for anyone building or evaluating C++ SATD detectors, this is the resource to start from. It deserves a serious referee; I would recommend acceptance with minor revision — fix the abstract, correct the statistics, and either repair or explicitly disclose the 25 mislabeled negatives.","headline":"Genuinely useful first C++ multiclass SATD dataset with code contexts, but the NON-SATD set is inferred, and 25 known SATD comments are shipped as negatives.","tokens_in":11702,"tokens_out":3099,"would_cite":true,"duration_ms":30083,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper presents CppSATD, a dataset of 531,367 C++ comments labeled into five self-admitted technical debt types or NON-SATD, with surrounding code context, claimed to be the first multiclass C++ SATD dataset.","keywords":["technical debt","self-admitted technical debt","SATD","C++","code comments","code context","multiclass annotation","mining software repositories"],"falsifier":"Take a fresh random sample of at least 1,000 comments from the 512,394 pattern-missed pool, have two independent annotators label them with the same reference document, and compare the SATD rate to the paper's claimed below-one-percent bound; also re-examine the 16,125-comment sample to see whether the 25 SATD comments were correctly handled. SATD rates meaningfully above one percent, or a re-annotation that finds many missed debt comments, would falsify the NON-SATD certification.","tokens_in":10415,"feed_emoji":"💬","tokens_out":11215,"duration_ms":106082,"temperature":0.7,"pith_summary":"This paper introduces CppSATD, a dataset of 531,367 comments mined from five large, active open-source C++ projects, each labeled as one of five self-admitted technical debt (SATD) types — Design/Code, Requirement, Defect, Test, Documentation — or as NON-SATD, and paired with the code immediately before and after it. The authors' central claim is that this is the first C++ SATD dataset with multiclass annotations and surrounding code contexts, filling a gap left by the Java-dominated SATD literature. If the claim holds, researchers gain a large labeled resource for training and evaluating C++ SATD detection, for testing whether Java-era findings generalize across languages, and for studying how debt is admitted in C++ code.","feed_headline":"Dataset labels 531K C++ comments by technical-debt type","feed_subtitle":"Five debt types plus code context turn half a million comments into training data for C++ tools.","key_machinery":"The mechanism that carries the argument is a two-stage labeling pipeline. Stage one uses pattern matching: 21 Easy-to-Find (ETF) patterns and 656 Hard-to-Find (HTF) patterns — short markers such as 'todo' and longer phrasings such as 'needs to be improved' — to split 531,367 comments into 18,973 candidate SATD comments and 512,394 candidate NON-SATD comments. Stage one also extracts, via the XML-marker tool srcML, the code context preceding and succeeding each comment. Stage two is human annotation: one annotator labels every candidate SATD comment, a second annotator labels a random sample of 16,125 from the candidate NON-SATD pool, and a kappa agreement coefficient on 385-comment subsets (0.86 and 1.0) is used to certify agreement. The statistical sampling step is what converts 'no pattern matched' into a defended NON-SATD label.","core_discovery":"On the paper's own terms, the contribution is an artifact with a validation argument. CppSATD is a dataset of 531,367 comments from five popular open-source C++ projects, each comment carrying one of five SATD type labels or NON-SATD, plus the preceding and succeeding code snippets. The authors report that they built it by pattern-filtering with 21 easy-to-find and 656 hard-to-find SATD patterns to isolate 18,973 candidate debt comments, manually annotating those plus a 16,125-comment random sample of the remaining pool, and validating with inter-rater agreement scores of 0.86 for SATD and 1.0 for NON-SATD. Their stated result is that 13,044 comments are positively labeled SATD, the most common types being Design/Code and Requirement debt, and that the rest can be treated as NON-SATD with claimed 99% confidence that hidden SATD in the unfiltered pool is below 1%.","pith_inferences":["Since the 25 SATD comments found inside the NON-SATD sample were left labeled NON-SATD rather than reclassified, the dataset's 'clean' pool contains known label noise; models trained on it will likely underestimate SATD prevalence and show optimistic precision.","A natural next experiment the paper does not run is a context-ablation study: train the same classifier on comment text with and without the code context fields to quantify the value of the surrounding snippets.","Cross-language comparisons with the standard Java SATD benchmark will require taxonomy reconciliation, because this dataset merges design and code debt into a single category; headline rate differences between languages should be read after that mapping.","The pipeline is repeatable, so the same five repositories could be re-extracted with an expanded pattern list to produce versioned dataset releases with cleaner NON-SATD pools."],"forward_implications":["C++ SATD detectors can now be trained and evaluated on five debt types rather than a single binary label, using both comment text and surrounding code context as input.","Java-trained SATD detectors can be benchmarked against C++ data, producing direct evidence on whether SATD patterns and type distributions transfer across languages.","The 13,044 positively labeled debt comments provide a large corpus for discovering recurring C++ SATD patterns that existing pattern lists miss.","The inclusion of code context enables experiments measuring how much preceding and succeeding code improves detection accuracy over comment text alone.","Because the dataset comes from five active projects in different domains, it supports studies of how SATD type distributions vary across project domains."],"supporting_citations":[{"why":"Supplies the SATD-type ontology and definitions that the five annotation categories adapt.","marker":"[4]"},{"why":"The widely used Java SATD dataset that motivates the C++ gap and provides the five-type annotation precedent.","marker":"[5]"},{"why":"Provides the contextualized vocabulary and indicators used to define each SATD type's indicators.","marker":"[10]"},{"why":"Supplies the 21 ETF and 656 HTF text patterns used to filter likely SATD comments from the full comment pool.","marker":"[12]"},{"why":"Provides the srcML tool used to transform source files to XML so comments and code contexts can be extracted.","marker":"[14]"},{"why":"Supplies the rule for assigning a single SATD type when a comment admits several types.","marker":"[15]"},{"why":"Introduces the kappa coefficient used to validate inter-rater agreement.","marker":"[16]"}],"fun_headline_variants":["C++ dataset labels 531K comments for technical debt types","Half a million C++ comments tagged for debt detection","New C++ corpus: 531K comments with SATD labels","CppSATD: 531K C++ comments labeled for technical debt","Largest C++ SATD dataset: 531K comments, five debt types"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The NON-SATD label on the roughly 512,000 comments that matched no debt pattern is certified by a single random sample of 16,125 comments annotated by one person; if that sample was not representative, or if the 25 debt comments it found were not the only ones, the 'clean' half of the dataset is silently contaminated.","fun_headline_variants_meta":{"raw":{"variants":["C++ dataset labels 531K comments for technical debt types","Half a million C++ comments tagged for debt detection","New C++ corpus: 531K comments with SATD labels","CppSATD: 531K C++ comments labeled for technical debt","Largest C++ SATD dataset: 531K comments, five debt types"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00042,"raw_usage":{"total_tokens":2144,"prompt_tokens":909,"completion_tokens":1235,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":1143}},"tokens_in":525,"tokens_out":1235,"duration_ms":9272,"temperature":1.0,"reasoning_tokens":1143,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:25:46.110512+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fresh random sample of at least 1,000 comments from the 512,394 pattern-missed pool, have two independent annotators label them with the same reference document, and compare the SATD rate to the paper's claimed below-one-percent bound; also re-examine the 16,125-comment sample to see whether the 25 SATD comments were correctly handled. SATD rates meaningfully above one percent, or a re-annotation that finds many missed debt comments, would falsify the NON-SATD certification.","supporting_citations":[{"cited_title":"Towards an ontology of terms on technical debt,","cited_arxiv_id":null,"evidence_quote":"Supplies the SATD-type ontology and definitions that the five annotation categories adapt."},{"cited_title":"Identifying self-admitted technical debt through code comment analysis with a contextualized vocabulary,","cited_arxiv_id":null,"evidence_quote":"Provides the contextualized vocabulary and indicators used to define each SATD type's indicators."},{"cited_title":"Pentacet data-23 million contextual code comments and 250,000 satd comments,","cited_arxiv_id":null,"evidence_quote":"Supplies the 21 ETF and 656 HTF text patterns used to filter likely SATD comments from the full comment pool."},{"cited_title":"srcml: An infrastructure for the exploration, analysis, and manipulation of source code: A tool demonstration,","cited_arxiv_id":null,"evidence_quote":"Provides the srcML tool used to transform source files to XML so comments and code contexts can be extracted."},{"cited_title":"Detecting and quantifying different types of self-admitted technical debt,","cited_arxiv_id":null,"evidence_quote":"Supplies the rule for assigning a single SATD type when a comment admits several types."}],"review_version":1}