Pith. sign in

REVIEW 4 major objections 3 minor 1 cited by

Development and Adoption of SATD Detection Tools: A State-of-practice Report

T0 review · 4 major / 3 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Just 3 of 7 technical-debt detection tools remain usable

desk verdict A useful but thinly documented census of seven SATD tools—worth a revision, not a desk reject. read the letter →

arxiv 2412.14217 v1 pith:XQCJZ4MT submitted 2024-12-18 cs.SE

Edi Sutoyo , Andrea Capiluppi This is my paper · ORCID
classification cs.SE
keywords self-admittedtechnicaldebtSATDdetectiontoolmaintenanceFAIRprinciplesopen-sourcesoftwarereproducibilitymanagementsystematicliteraturereview
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports on the real-world availability of self-admitted technical debt (SATD) detection tools built in the last decade. It asks how many of the proposed tools are actually accessible, and it finds that only three of seven are currently usable, with the rest dead links, missing links, or obsolete software. The authors argue that this fragile tool ecosystem threatens the reliability and generalizability of empirical SATD research, and they propose anti-pattern remedies: FAIR principles, open-source maintenance, and closer academia-industry collaboration.

What carries the argument

The argument runs on a three-way accessibility classification: accessible and functional, inaccessible or broken link, and obsolete or non-functional, assigned by attempting the links and repositories given in each paper. The companion systematic literature review supplies the pool of 68 papers, and screening those papers for actual prototypes yields the seven tools that produce the 3-of-7 count. The classification both produces the main statistic and organizes the anti-pattern discussion that follows.

What would settle it

Install and run SATD Detector in a current Java/Eclipse environment; if it runs correctly, the 'obsolete' classification, and the paper's clearest obsolescence example, is wrong. Alternatively, expand the tool search beyond the 68 papers and recalculate the functional share; a share much higher than 3 of 7 would contradict the central claim.

Watch

Extended reading notes

Core claim

The paper's central claim is that the SATD detection tool ecosystem is largely unusable: of seven tools identified from a systematic review of 68 papers, only DebtViz, SATDBailiff, and DebtHunter are accessible and functional. FixMe and eXcomment have broken links, a browser extension for rOpenSci packages has no working link, and SATD Detector is obsolete on current platforms. The paper then names five anti-patterns: threats to reliability of findings, bias in analysis, incomplete coverage of debt types, technological obsolescence, and poor sustainability and reproducibility, and it proposes countermeasures around FAIR principles, open-source practices, and academia-industry collaboration.

Load-bearing premise

The whole availability statistic rests on the companion systematic literature review having found every SATD detection tool that exists, so if that review missed tools, the 3-of-7 result may not describe the field.

Editorial extensions

If this is right

  • Studies that rely only on the three accessible tools may miss the variety of SATD detection techniques and debt types, reducing generalizability.
  • SATD Detector and eXcomment cannot be rerun on modern platforms, so replicating earlier findings means rebuilding the tools from scratch.
  • Hosting tools on persistent, versioned repositories with DOIs would make tool availability measurable and more stable over time.
  • Combining multiple diverse detectors reduces false positives and negatives, making tool preservation a direct contributor to detection quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Re-checking the same seven tools at yearly intervals would show whether availability decay is ongoing, turning the 3-of-7 metric into a trend rather than a snapshot.
  • The paper's numbers add to 67, not 68: 60 papers without tools plus 7 tools, so one screened paper is unaccounted for and the denominator deserves a second look.
  • Repository activity, such as last commit date, open issues, and release tags, could serve as a low-cost predictor of whether a newly proposed SATD tool will remain usable, extending the anti-pattern narrative into a testable model.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper is a state-of-practice report on tools for Self-Admitted Technical Debt (SATD) detection. Building on the authors' companion systematic literature review of 68 papers, it identifies seven SATD detection tools proposed between 2014 and 2024, classifies their current accessibility (three accessible, two broken links, one inaccessible, one obsolete), and derives anti-patterns and recommendations centered on FAIR principles and academia-industry collaboration. The central claim, stated in the abstract and in Section 3, is that only a small number of tools are actively maintained, hindering their widespread adoption.

Significance. If the tool census were reproducible, this would be a useful contribution: it shifts attention from algorithmic accuracy to the sustainability of research software, provides a compact inventory of seven tools, and makes concrete, actionable recommendations such as versioning, Zenodo/GitHub archiving, and FAIR-USE4OS guidelines. The authors also deserve credit for intentionally avoiding the term 'tool' in the search string, which reduces a common sampling bias, and for being transparent that the paper builds on their own prior SLR. However, the significance is tempered by the lack of an auditable protocol and by a count inconsistency, so the headline 3-of-7 availability rate is currently an assertion rather than an established empirical result.

major comments (4)
  1. [Section 3, Table 1] Section 3 reports that 60 of the 68 papers in the companion SLR did not provide prototypes or tools, and Table 1 lists 7 tools; 60 + 7 = 67, not 68. If one paper contributes two tools, that needs to be stated explicitly; if one tool-containing paper was omitted from Table 1, the denominator changes. Because the conclusion that only a few tools are maintained depends entirely on this census, please reconcile the count and provide the paper-to-tool mapping.
  2. [Section 3] The classification of each tool into 'Accessible and functional', 'Inaccessible or broken link', or 'Obsolete or non-functional' is not supported by any access protocol. The paper gives no URLs, access dates, execution environment, or per-tool evidence such as the last commit, attempted installation steps, or error messages. Without this information, a reader cannot verify that DebtViz, SATDBailiff, and DebtHunter currently run, or that SATD Detector is incompatible with modern platforms. Please add an appendix with the exact artifact identifiers, versions, access dates, and test outcomes for each tool.
  3. [Abstract and Section 3] The abstract's claim that 'Only a small number of tools are actively maintained' is not directly measured by the methodology. Table 1's categories capture accessibility and functionality at a single point in time, not maintenance activity such as release frequency, commit history, or issue responsiveness. A tool can be accessible but unmaintained, and a tool can be temporarily inaccessible despite active maintenance. Please either reframe the central claim as current accessibility/functionality or add maintenance metrics to support the word 'maintained'.
  4. [Section 3, companion SLR [7]] The entire tool census is drawn from the authors' companion systematic literature review [7], but the paper does not summarize that SLR's search databases, time span, inclusion/exclusion criteria, or number of records screened. If the SLR excluded gray literature, industry tools, or tools described only in non-indexed venues, the reported availability rate could understate the true landscape. This is not a circularity objection; it is a completeness risk. Please describe the SLR protocol or state its search date and venue coverage, and discuss the potential for missed tools outside the indexed literature.
minor comments (3)
  1. [Section 3] The sentence 'The review employed search terms such as:' is a grammatical fragment; consider rewriting as 'The review employed the following search terms' and closing the quotation before the period.
  2. [Table 1] The row labeled 'A browser extension' should either give the tool's actual name if one is presented in reference [19], or explicitly state that the artifact is unnamed in that paper, in addition to providing the repository link.
  3. [References [7]] Reference [7] is cited as an arXiv preprint without a version identifier or an access date; since the paper's results depend on that SLR, please provide the arXiv version number and the retrieval date.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation; the paper's own accessibility checks support the headline finding, though the tool sample comes from a companion self-cited SLR.

full rationale

This is a state-of-practice report rather than a derivation, so the prediction-equivalence circularity patterns do not apply. The central claim that only three of seven surveyed SATD tools are accessible is supported by the direct accessibility evaluation reported in Section 3 and Table 1, not by a fitted parameter or by an equation reused from the input. That evaluation is new, externally checkable, and falsifiable: one could revisit the links and repositories and dispute the 'Accessible', 'Broken link', 'Inaccessible', or 'Obsolete' labels. The companion systematic literature review [7] is self-cited and supplies the candidate pool of 68 papers and 7 tools, so the sample frame is not fully independent of the authors' prior work. This creates a transparency limitation, especially because the paper does not reproduce the SLR's inclusion/exclusion criteria and reports a count inconsistency (60 papers without tools plus the 7 listed tools sum to 67, not 68). However, those are validity and reproducibility concerns rather than circularity: the paper does not define its conclusion in terms of its inputs, and the accessibility result has independent empirical content beyond the self-cited review. Accordingly, no specific circular step is identified, and the score reflects only the minor self-citation dependency in the sampling frame.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper makes no mathematical claims and fits no parameters. Its empirical findings depend on two domain assumptions: the completeness of the self-cited companion SLR that supplies the 68-paper sample, and the validity of an undocumented accessibility check. No invented entities are introduced.

assumptions (3)
  • domain assumption The companion SLR (Sutoyo and Capiluppi, arXiv:2312.15020) provides a complete and valid set of 68 SATD detection papers.
    The entire tool sample is drawn from this self-cited SLR; if it is incomplete or biased, the accessibility findings do not generalize.
  • domain assumption Checking tool links and attempting to run the tools at a single point in time is a reliable measure of tool availability and maintenance.
    The classification into accessible, broken, and obsolete depends on an undocumented access procedure with no date, environment, or reproducibility criteria.
  • domain assumption Tools classified as 'accessible and functional' were actually run successfully, and 'obsolete' tools could not be run in any modern environment.
    The paper states these criteria but provides no logs, screenshots, or test details.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Development and Adoption of SATD Detection Tools: A State-of-practice Report." pith.science (2026). https://pith.science/paper/XQCJZ4MT

@misc{pith2026241214217,
  author       = {Pith},
  title        = {Pith review of: Development and Adoption of SATD Detection Tools: A State-of-practice Report},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XQCJZ4MT}},
  note         = {Machine review of arXiv:2412.14217}
}
read the original abstract

Self-Admitted Technical Debt (SATD) refers to instances where developers knowingly introduce suboptimal solutions into code and document them, often through textual artifacts. This paper provides a comprehensive state-of-practice report on the development and adoption of SATD detection tools. Through a systematic review of the available literature and tools, we examined their overall accessibility. Our findings reveal that, although SATD detection tools are crucial for maintaining software quality, many face challenges such as technological obsolescence, poor maintenance, and limited platform compatibility. Only a small number of tools are actively maintained, hindering their widespread adoption. This report discusses common anti-patterns in tool development, proposes corrections, and highlights the need for implementing Findable, Accessible, Interoperable, and Reusable (FAIR) principles and fostering greater collaboration between academia and industry to ensure the sustainability and efficacy of these tools. The insights presented here aim to drive more robust management of technical debt and enhance the reliability of SATD tools.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Tracing the Lifecycle of Architecture Technical Debt in Software Systems: A Dependency Approach

    cs.SE 2025-01 conditional novelty 6.0 of 10

    Repaying architectural technical debt in 16 usable Apache codebase items increased average class connectivity, but the effect is negligible and smaller than changes seen in non-ATD files.

Reference graph

Works this paper leans on

38 extracted references · 35 canonical work pages · cited by 1 Pith paper

  1. [7]

    Self-Admitted Technical Debt Detection Approaches: A Decade Systematic Review

    E. Sutoyo, A. Capiluppi, Self-admitted technical debt detection approaches: A decade systematic review, arXiv preprint arXiv:2312.15020 (2024)

    work page Pith review arXiv 2024

  2. [1]

    Cunningham, The wycash portfolio management system, ACM Sigplan Oops Messenger 4 (1992) 29–30

    W. Cunningham, The wycash portfolio management system, ACM Sigplan Oops Messenger 4 (1992) 29–30

    work page 1992

  3. [2]

    Potdar, E

    A. Potdar, E. Shihab, An exploratory study on self-admitted technical debt, in: 2014 IEEE International Conference on Software Maintenance and Evolution, IEEE, 2014, pp. 91–100

    2014

  4. [3]

    E. d. S. Maldonado, E. Shihab, Detecting and quantifying different types of self-admitted technical debt, in: 2015 IEEE 7Th international workshop on managing technical debt (MTD), IEEE, 2015, pp. 9–15

    work page 2015

  5. [4]

    E. A. AlOmar, B. Christians, M. Busho, A. H. AlKhalid, A. Ouni, C. Newman, M. W. Mkaouer, Satdbailiff-mining and tracking self-admitted technical debt, Science of Computer Programming 213 (2022) 102693

    work page 2022

  6. [5]

    I. Sala, A. Tommasel, F. Arcelli Fontana, Debthunter: A machine learning-based approach for detecting self-admitted technical debt, in: Proceedings of the 25th International Conference on Evaluation and Assessment in Software Engineering, 2021, pp. 278–283

    work page 2021

  7. [6]

    E. D. S. Maldonado, R. Abdalkareem, E. Shihab, A. Serebrenik, An empirical study on the removal of self-admitted technical debt, in: 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME), IEEE, 2017, pp. 238–248

    work page 2017

  8. [8]

    Fortunato, D

    Graham Lee, Sebastian Bacon, Ian Bush, L. Fortunato, D. Gavaghan, T. Lestang, Caroline Morton, M. Robinson, P. Rocca-Serra, Susanna-Assunta Sansone, Helena Webb, Barely sufficient practices in scientific computing, Patterns (2021)

    work page 2021

Show all 38 references
  1. [9]

    Torkar, Outliers and Replication in Software Engineering, Asia-Pacific Software Engineering Conference (2014)

    Henrik Larsson, Erik Lindqvist, R. Torkar, Outliers and Replication in Software Engineering, Asia-Pacific Software Engineering Conference (2014)

    2014

  2. [10]

    Brito, Jun Z Li, J

    J. Brito, Jun Z Li, J. H. Moore, C. Greene, Nicole A Nogoy, L. Garmire, S. Mangul, Recommendations to enhance rigor and reproducibility in biomedical research, GigaScience (2020)

    2020

  3. [11]

    Madeyski, B

    L. Madeyski, B. Kitchenham, Would wider adoption of reproducible research be beneficial for empirical software engineering research?, Journal of Intelligent & Fuzzy Systems (2017)

    2017

  4. [12]

    R. D. Peng, Reproducible Research in Computational Science, Science 334 (2011) 1226–1227

    2011

  5. [13]

    Zhongxin Liu, Qiao Huang, Xin Xia, Emad Shihab, D. Lo, Shanping Li, Satd Detector: A Text- Mining-Based Self-Admitted Technical Debt Detection Tool, 2018 IEEE/ACM 40th International Conference on Software Engineering: Companion (ICSE-Companion) (2018)

    2018

  6. [14]

    Y. Li, M. Soliman, P. Avgeriou, Automatic identification of self-admitted technical debt from four different sources, Empirical Software Engineering 28 (2023)

    2023

  7. [15]

    Huang, E

    Q. Huang, E. Shihab, X. Xia, D. Lo, S. Li, Identifying self-admitted technical debt in open source projects using text mining, Empirical Software Engineering 23 (2017) 418–451

    2017

  8. [16]

    Zampetti, C

    F. Zampetti, C. Noiseux, G. Antoniol, F. Khomh, M. Di Penta, Recommending when Design Technical Debt Should be Self-Admitted, in: 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME), volume 3, IEEE, 2017, pp. 216–226

    2017

  9. [17]

    Maipradit, C

    R. Maipradit, C. Treude, H. Hata, K. Matsumoto, Wait for it: identifying “On-Hold” self-admitted technical debt, Empirical Software Engineering 25 (2020) 3770–3798

    2020

  10. [18]

    Pavlič, T

    L. Pavlič, T. Hliš, M. Heričko, T. Beranič, The Gap between the Admitted and the Measured Technical Debt: An Empirical Study, Applied Sciences 12 (2022) 7482

    2022

  11. [19]

    J. Y. Khan, G. Uddin, Automatic Detection and Analysis of Technical Debts in Peer-Review Documentation of R Packages, in: Proceedings - 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2022, Institute of Electrical and Electronics Eng...

    2022

  12. [20]

    Phaithoon, S

    S. Phaithoon, S. Wongnil, P. Pussawong, M. Choetkiertikul, C. Ragkhitwetsagul, T. Sunetnanta, R. Maipradit, H. Hata, K. Matsumoto, Fixme: A github bot for detecting and monitoring on-hold self-admitted technical debt, in: 2021 36th IEEE/ACM International Conference on Automate...

    2021

  13. [21]

    M. A. D. F. Farias, M. G. D. M. Neto, A. B. D. Silva, R. O. Spinola, A Contextualized Vocabulary Model for identifying technical debt on code comments, in: 2015 IEEE 7th International Workshop on Managing Technical Debt, MTD 2015 - Proceedings, Institute of Electrical and Elec...

    2015

  14. [22]

    Farias, T

    M. Farias, T. S. Mendes, M. G. Mendonça, R. O. Spínola, On comment patterns that are good indicators of the presence of self-admitted technical debt and those that lead to false positive items., in: AMCIS, 2021

    2021

  15. [23]

    Costa, P

    J. Costa, P. Meirelles, C. Chavez, On the sustainability of academic software, in: Proceedings of the XXXII Brazilian Symposium on Software Engineering, volume 6, ACM, 2018, pp. 202–207

    2018

  16. [24]

    A. Deo, S. K. Dash, G. Suarez-Tangil, V. Vovk, L. Cavallaro, Prescience, in: Proceedings of the 2016 ACM Workshop on Artificial Intelligence and Security, ACM, 2016, pp. 71–82

    2016

  17. [25]

    Algaith, P

    A. Algaith, P. Nunes, F. Jose, I. Gashi, M. Vieira, Finding SQL Injection and Cross Site Scripting Vulnerabilities with Diverse Static Analysis Tools, in: 2018 14th European Dependable Computing Conference (EDCC), IEEE, 2018, pp. 57–64

    2018

  18. [26]

    Patel, S

    B. Patel, S. Soundarajan, H. Ménager, Z. Hu, Making Biomedical Research Software FAIR: Action- able Step-by-step Guidelines with a User-support Tool (2022)

    2022

  19. [27]

    Sonabend, H

    R. Sonabend, H. Gruson, L. Wolansky, A. Kiragga, D. S. Katz, Fair-USE4OS: Guidelines for creating impactful open-source software, PLOS Computational Biology 20 (2024) e1012045

    2024

  20. [28]

    Hasselbring, L

    W. Hasselbring, L. Carr, S. Hettrick, H. Packer, T. Tiropanis, From FAIR research data toward FAIR and open research software, it - Information Technology 62 (2020) 39–47

    2020

  21. [29]

    Audemard, L

    G. Audemard, L. Paulevé, L. Simon, SAT Heritage: A Community-Driven Effort for Archiving, Building and Running More Than Thousand SAT Solvers, Springer International Publishing, 2020, pp. 107–113

    2020

  22. [30]

    E. M. del Pico, J. L. Gelpi, S. Capella-Gutiérrez, Fairsoft-a practical implementation of fair principles for research software, bioRxiv (2022) 2022–05

    2022

  23. [31]

    Spinellis, Version control systems, IEEE software 22 (2005) 108–109

    D. Spinellis, Version control systems, IEEE software 22 (2005) 108–109

    2005

  24. [32]

    Stirbu, T

    V. Stirbu, T. Mikkonen, Introducing traceability in github for medical software development, in: Product-Focused Software Process Improvement: 22nd International Conference, PROFES 2021, Turin, Italy, November 26, 2021, Proceedings 22, Springer, 2021, pp. 152–164

    2021

  25. [33]

    Klein, L

    M. Klein, L. Balakireva, On the persistence of persistent identifiers of the scholarly web, in: Digital Libraries for Open Knowledge: 24th International Conference on Theory and Practice of Digital Libraries, TPDL 2020, Lyon, France, August 25–27, 2020, Proceedings 24, Springe...

    2020

  26. [34]

    R. C. Jiménez, M. Kuzak, M. Alhamdoosh, M. Barker, B. Batut, M. Borg, S. Capella-Gutierrez, N. Chue Hong, M. Cook, M. Corpas, M. Flannery, L. Garcia, J. L. Gelpí, S. Gladman, C. Goble, M. González Ferreiro, A. Gonzalez-Beltran, P. C. Griffin, B. Grüning, J. Hagberg, P. Holub, ...

    2017

  27. [35]

    Corpas, D

    M. Corpas, D. Mellor, Four simple recommendations to encourage best practices in research software (2019)

    2019

  28. [36]

    T. Gamblin, Picking an Open Source License at LLNL: Guidance and Recommendations from the Computing Directorate, Technical Report, Lawrence Livermore National Lab.(LLNL), Livermore, CA (United States), 2021

    2021

  29. [37]

    Bikard, K

    M. Bikard, K. Vakili, F. Teodoridis, When collaboration bridges institutions: The impact of university–industry collaboration on academic productivity, Organization Science 30 (2019) 426–445

    2019

  30. [38]

    P. M. Duvall, S. Matyas, A. Glover, Continuous integration: improving software quality and reducing risk, Pearson Education, 2007

    2007

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.