{"id":"cf930fa5-8af1-4b9f-b3b8-1882703e0596","arxiv_id":"2412.14217","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A survey of seven SATD detection tools finds most are broken, inaccessible, or obsolete, with only three still workable.","lead":"This paper reviews seven tools for detecting self-admitted technical debt and finds only three are currently accessible and usable. It recommends open-source, FAIR, and industry collaboration practices to stop research tools from disappearing.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The tool census hinges on an unverifiable screening of the companion SLR; the 60+7=67 arithmetic gap and absent access protocol mean the 3-of-7 accessibility claim is not yet independently checkable.","rationale":"The reader's weakest assumption correctly identifies the load-bearing condition: the SLR-based census must both be complete and be screened accurately for tool-providing papers. I agree with that assessment and sharpen it with two concrete, checkable symptoms. First, the arithmetic in Section 3 does not reconcile: 60 non-tool papers plus the 7 tools in Table 1 yields 67, not 68. This does not by itself falsify the claim, but it signals that the screening procedure is not fully accounted for. Second, the accessibility classifications are presented without the evidence needed to reproduce them: no URLs, access dates, or execution environments. Since the headline claim is a state-of-practice statement about the whole ecosystem, both issues undermine the confidence one can place in the 3-of-7 figure. The paper's qualitative recommendations about FAIR principles and open-source practices are reasonable and supported by citations, but they are secondary to the empirical premise. If the authors supply the tool inventory and a reproducible access protocol, the report would support its conclusion; if independent screening reveals additional tools or misclassifications, the headline would need qualification. Therefore, the existing CONDITIONAL verdict remains appropriate, with the conditions being the release of the census data and protocol.","tokens_in":8146,"tokens_out":5124,"duration_ms":47026,"concrete_test":"Obtain from the authors the full list of the 68 SLR papers with per-paper screening outcomes (tool/no-tool), and have two independent raters re-classify every paper using the paper's own definition (proposes an approach and offers a prototype/ready-to-use tool). Also have the authors provide for each of the seven tools the exact URL/repository, access date, OS/dependency environment, and run instructions. Then rerun the accessibility check from a clean environment. The concern lands if the independent screening finds additional tool papers among the 68 (resolving the 60+7=67 gap) or if any of the three 'Accessible' classifications fail in the reproduced environment; it is neutralized if the independent census reproduces exactly seven tools and the three accessibility classifications.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative assertion—'Only a small number of tools are actively maintained' (Abstract; Section 3)—rests entirely on the claim that, of 68 SATD-detection papers in the companion SLR [7], exactly seven provided tools and that three of those are currently accessible. That census is load-bearing because the conclusion is a denominator claim: any missed tool-containing paper or any erroneous accessibility classification changes the headline. Section 3 reports 60 papers without tools and Table 1 lists 7 tools; 60+7=67, not 68, so the screening itself contains an unexplained count error. Moreover, the paper provides no protocol for the accessibility assessment: no URLs, access dates, execution environment, or per-tool evidence for the 'Accessible', 'Broken link', 'Inaccessible', or 'Obsolete' classifications. The classification of 'DebtViz', 'SATDBailiff', and 'DebtHunter' as 'Accessible' cannot be reproduced. Because the paper also self-cites the SLR [7] without summarizing its inclusion/exclusion criteria, the possibility that the 68-paper corpus missed tools outside the indexed literature is not addressed. This is not an accusation of misreporting; it is a statement that the most important empirical premise is currently underspecified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper is a state-of-practice report on tools for Self-Admitted Technical Debt (SATD) detection. Building on the authors' companion systematic literature review of 68 papers, it identifies seven SATD detection tools proposed between 2014 and 2024, classifies their current accessibility (three accessible, two broken links, one inaccessible, one obsolete), and derives anti-patterns and recommendations centered on FAIR principles and academia-industry collaboration. The central claim, stated in the abstract and in Section 3, is that only a small number of tools are actively maintained, hindering their widespread adoption.","tokens_in":8368,"tokens_out":4613,"duration_ms":41540,"significance":"If the tool census were reproducible, this would be a useful contribution: it shifts attention from algorithmic accuracy to the sustainability of research software, provides a compact inventory of seven tools, and makes concrete, actionable recommendations such as versioning, Zenodo/GitHub archiving, and FAIR-USE4OS guidelines. The authors also deserve credit for intentionally avoiding the term 'tool' in the search string, which reduces a common sampling bias, and for being transparent that the paper builds on their own prior SLR. However, the significance is tempered by the lack of an auditable protocol and by a count inconsistency, so the headline 3-of-7 availability rate is currently an assertion rather than an established empirical result.","major_comments":[{"comment":"Section 3 reports that 60 of the 68 papers in the companion SLR did not provide prototypes or tools, and Table 1 lists 7 tools; 60 + 7 = 67, not 68. If one paper contributes two tools, that needs to be stated explicitly; if one tool-containing paper was omitted from Table 1, the denominator changes. Because the conclusion that only a few tools are maintained depends entirely on this census, please reconcile the count and provide the paper-to-tool mapping.","section":"Section 3, Table 1"},{"comment":"The classification of each tool into 'Accessible and functional', 'Inaccessible or broken link', or 'Obsolete or non-functional' is not supported by any access protocol. The paper gives no URLs, access dates, execution environment, or per-tool evidence such as the last commit, attempted installation steps, or error messages. Without this information, a reader cannot verify that DebtViz, SATDBailiff, and DebtHunter currently run, or that SATD Detector is incompatible with modern platforms. Please add an appendix with the exact artifact identifiers, versions, access dates, and test outcomes for each tool.","section":"Section 3"},{"comment":"The abstract's claim that 'Only a small number of tools are actively maintained' is not directly measured by the methodology. Table 1's categories capture accessibility and functionality at a single point in time, not maintenance activity such as release frequency, commit history, or issue responsiveness. A tool can be accessible but unmaintained, and a tool can be temporarily inaccessible despite active maintenance. Please either reframe the central claim as current accessibility/functionality or add maintenance metrics to support the word 'maintained'.","section":"Abstract and Section 3"},{"comment":"The entire tool census is drawn from the authors' companion systematic literature review [7], but the paper does not summarize that SLR's search databases, time span, inclusion/exclusion criteria, or number of records screened. If the SLR excluded gray literature, industry tools, or tools described only in non-indexed venues, the reported availability rate could understate the true landscape. This is not a circularity objection; it is a completeness risk. Please describe the SLR protocol or state its search date and venue coverage, and discuss the potential for missed tools outside the indexed literature.","section":"Section 3, companion SLR [7]"}],"minor_comments":[{"comment":"The sentence 'The review employed search terms such as:' is a grammatical fragment; consider rewriting as 'The review employed the following search terms' and closing the quotation before the period.","section":"Section 3"},{"comment":"The row labeled 'A browser extension' should either give the tool's actual name if one is presented in reference [19], or explicitly state that the artifact is unnamed in that paper, in addition to providing the repository link.","section":"Table 1"},{"comment":"Reference [7] is cited as an arXiv preprint without a version identifier or an access date; since the paper's results depend on that SLR, please provide the arXiv version number and the retrieval date.","section":"References [7]"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the manuscript reads as a workshop-scale empirical report, and its main weakness is not novelty but auditability. The arithmetic mismatch in Section 3 and the absent access protocol are fixable with an artifact appendix and a short protocol description. I would not reject on self-citation grounds alone, because building on one's own SLR is a legitimate design choice, but the authors must make the tool census independently checkable before the paper's central claim can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe useful thing in this paper is the specific snapshot: of seven SATD detection tools found via the authors' own SLR, three are currently accessible (DebtViz, SATDBailiff, DebtHunter), two have broken links, one has no link, one is obsolete. That 3-of-7 bottom line is new and worth knowing for anyone who works on SATD replication.\n\nThe paper also does a decent job of connecting that finding to known anti-patterns in research software (obsolescence, broken links, missing docs) and to FAIR principles. The recommendations—host on GitHub/Zenodo, license properly, document, benchmark—are generic and well established in the research-software literature, so the novelty is in the census, not the advice.\n\nThe soft spots are real and fixable. First, the arithmetic doesn't add up: the text says 60 of the 68 papers lack tools, and Table 1 lists 7 tools; 60 + 7 = 67, not 68. One paper is unaccounted for. Second, the access protocol is not reported. We don't get the URLs, the dates of access, the test environment, or per-tool evidence for each category. \"Accessible\" is not reproducible from the paper. Third, the sample is entirely dependent on the authors' own companion SLR; if that SLR missed tools outside indexed venues or industry tools, the denominator is wrong. These are not accusations of fraud, just the load-bearing premise being underspecified.\n\nThe reader's conditional verdict is about right. The central claim is plausible—the SATD tool ecosystem is fragile—but the paper as written doesn't give the reader enough to verify the specifics. The stress-test note's concern about the 60+7 gap and the absent protocol is correct and lands.\n\nWho is this for? The SATD research community and people working on research-software sustainability. A reader in that niche will get a quick, useful state-of-play, even if they already know FAIR principles.\n\nRecommendation: worth engaging with as a workshop-level paper, but it needs a revision that supplies the tool list with links, access dates, environment details, and fixes the paper count. I'd send it to peer review with that expectation, not desk-reject it.","headline":"A useful but thinly documented census of seven SATD tools—worth a revision, not a desk reject.","tokens_in":8897,"tokens_out":2309,"would_cite":false,"duration_ms":20322,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Just 3 of 7 technical-debt detection tools remain usable","keywords":["self-admitted technical debt","SATD detection","tool maintenance","FAIR principles","open-source software","software reproducibility","technical debt management","systematic literature review"],"falsifier":"Install and run SATD Detector in a current Java/Eclipse environment; if it runs correctly, the 'obsolete' classification, and the paper's clearest obsolescence example, is wrong. Alternatively, expand the tool search beyond the 68 papers and recalculate the functional share; a share much higher than 3 of 7 would contradict the central claim.","tokens_in":7926,"feed_emoji":"🔧","tokens_out":6226,"duration_ms":51269,"temperature":0.7,"pith_summary":"This paper reports on the real-world availability of self-admitted technical debt (SATD) detection tools built in the last decade. It asks how many of the proposed tools are actually accessible, and it finds that only three of seven are currently usable, with the rest dead links, missing links, or obsolete software. The authors argue that this fragile tool ecosystem threatens the reliability and generalizability of empirical SATD research, and they propose anti-pattern remedies: FAIR principles, open-source maintenance, and closer academia-industry collaboration.","feed_headline":"Just 3 of 7 technical-debt detection tools remain usable","feed_subtitle":"A 68-paper review finds dead links, obsolete code, and a push for FAIR, open-source maintenance.","key_machinery":"The argument runs on a three-way accessibility classification: accessible and functional, inaccessible or broken link, and obsolete or non-functional, assigned by attempting the links and repositories given in each paper. The companion systematic literature review supplies the pool of 68 papers, and screening those papers for actual prototypes yields the seven tools that produce the 3-of-7 count. The classification both produces the main statistic and organizes the anti-pattern discussion that follows.","core_discovery":"The paper's central claim is that the SATD detection tool ecosystem is largely unusable: of seven tools identified from a systematic review of 68 papers, only DebtViz, SATDBailiff, and DebtHunter are accessible and functional. FixMe and eXcomment have broken links, a browser extension for rOpenSci packages has no working link, and SATD Detector is obsolete on current platforms. The paper then names five anti-patterns: threats to reliability of findings, bias in analysis, incomplete coverage of debt types, technological obsolescence, and poor sustainability and reproducibility, and it proposes countermeasures around FAIR principles, open-source practices, and academia-industry collaboration.","pith_inferences":["Re-checking the same seven tools at yearly intervals would show whether availability decay is ongoing, turning the 3-of-7 metric into a trend rather than a snapshot.","The paper's numbers add to 67, not 68: 60 papers without tools plus 7 tools, so one screened paper is unaccounted for and the denominator deserves a second look.","Repository activity, such as last commit date, open issues, and release tags, could serve as a low-cost predictor of whether a newly proposed SATD tool will remain usable, extending the anti-pattern narrative into a testable model."],"forward_implications":["Studies that rely only on the three accessible tools may miss the variety of SATD detection techniques and debt types, reducing generalizability.","SATD Detector and eXcomment cannot be rerun on modern platforms, so replicating earlier findings means rebuilding the tools from scratch.","Hosting tools on persistent, versioned repositories with DOIs would make tool availability measurable and more stable over time.","Combining multiple diverse detectors reduces false positives and negatives, making tool preservation a direct contributor to detection quality."],"supporting_citations":[{"why":"Supplies the 68-paper pool from which all seven tools are drawn; the availability claim depends entirely on this review's completeness.","marker":"[7]"},{"why":"One of the three tools rated accessible, representing the SATDBailiff approach to mining and tracking SATD.","marker":"[4]"},{"why":"One of the three tools rated accessible, DebtHunter is a central example of machine-learning-based SATD detection.","marker":"[5]"},{"why":"One of the three accessible tools, DebtViz is the paper's example of a multi-debt-type platform.","marker":"[14]"},{"why":"Rated obsolete, SATD Detector's incompatibility with modern environments is the paper's main obsolescence case.","marker":"[13]"},{"why":"Rated inaccessible, the browser extension's missing link supports the no-valid-link category.","marker":"[19]"},{"why":"Rated broken link, FixMe's dead repository supports the link-decay finding.","marker":"[20]"},{"why":"Rated broken link, eXcomment's dead links are the second link-decay case.","marker":"[21, 22]"}],"fun_headline_variants":["Only 3 of 7 SATD detection tools still work","SATD tool landscape: 3 usable, 4 broken or obsolete","Just 3 of 7 technical-debt detectors remain functional","4 of 7 SATD tools are dead links or obsolete code","Most SATD detection tools unusable; only 3 of 7 work"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole availability statistic rests on the companion systematic literature review having found every SATD detection tool that exists, so if that review missed tools, the 3-of-7 result may not describe the field.","fun_headline_variants_meta":{"raw":{"variants":["Only 3 of 7 SATD detection tools still work","SATD tool landscape: 3 usable, 4 broken or obsolete","Just 3 of 7 technical-debt detectors remain functional","4 of 7 SATD tools are dead links or obsolete code","Most SATD detection tools unusable; only 3 of 7 work"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000692,"raw_usage":{"total_tokens":3087,"prompt_tokens":854,"completion_tokens":2233,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":470,"completion_tokens_details":{"reasoning_tokens":2141}},"tokens_in":470,"tokens_out":2233,"duration_ms":16966,"temperature":1.0,"reasoning_tokens":2141,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:48:21.671555+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Install and run SATD Detector in a current Java/Eclipse environment; if it runs correctly, the 'obsolete' classification, and the paper's clearest obsolescence example, is wrong. Alternatively, expand the tool search beyond the 68 papers and recalculate the functional share; a share much higher than 3 of 7 would contradict the central claim.","supporting_citations":[{"cited_title":"Self-Admitted Technical Debt Detection Approaches: A Decade Systematic Review","cited_arxiv_id":"2312.15020","evidence_quote":"Supplies the 68-paper pool from which all seven tools are drawn; the availability claim depends entirely on this review's completeness."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"One of the three tools rated accessible, representing the SATDBailiff approach to mining and tracking SATD."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"One of the three tools rated accessible, DebtHunter is a central example of machine-learning-based SATD detection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"One of the three accessible tools, DebtViz is the paper's example of a multi-debt-type platform."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Rated obsolete, SATD Detector's incompatibility with modern environments is the paper's main obsolescence case."},{"cited_title":"Phaithoon, S","cited_arxiv_id":null,"evidence_quote":"Rated broken link, FixMe's dead repository supports the link-decay finding."}],"review_version":1}