pith. sign in

arxiv: 2604.10631 · v1 · submitted 2026-04-12 · 💻 cs.SE

Investigating CI/CD-based Technical Debt Management in Open-source Projects

Jo\~ao Paulo Biazotto , Daniel Feitosa , Paris Avgeriou , Elisa Yumi Nakagawa This is my paper

Pith reviewed 2026-05-10 15:50 UTC · model grok-4.3

classification 💻 cs.SE
keywords technical debt managementCI/CD pipelinesTravis CIconfiguration anti-patternsopen-source projectsmining software repositoriescontinuous integration
0
0 comments X

The pith

TDM tools in open-source CI/CD pipelines are mostly invoked via external scripts, and absent feedback is the most common anti-pattern.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper examines how technical debt management tools get embedded in continuous integration and delivery pipelines by analyzing a large collection of Travis CI configuration files from GitHub. Researchers collected hundreds of thousands of these files and supporting scripts, then isolated thousands of pipelines that use at least one TDM tool. Their results indicate that direct embedding in the main configuration is rare, with most tools instead called through separate scripts, and that pipelines frequently omit any feedback mechanism about debt levels. This setup matters because it shows how automation intended to reduce long-term maintenance costs can itself introduce configuration habits that limit visibility and consistency. The findings supply concrete data that developers and tool builders can apply when designing or adopting CI/CD workflows for debt management.

Core claim

A mining study of roughly 600,000 Travis CI files and 50,000 scripts identified 3,684 pipelines containing TDM tools; within those pipelines, execution and integration occur predominantly through external scripts, and the Absent Feedback configuration anti-pattern appears most often.

What carries the argument

Large-scale mining of Travis CI configuration files and scripts to detect presence of TDM tools and to classify configuration anti-patterns.

If this is right

  • Practitioners integrating TDM tools should expect to rely on external scripts for most current usage patterns.
  • Many pipelines fail to close the loop by providing debt-related feedback to developers.
  • Guidelines for CI/CD-based TDM can be refined around script-centric integration and feedback mechanisms.
  • Tool developers gain evidence on how their products are actually invoked in pipelines today.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same script-heavy pattern may hold on other CI platforms, pointing to a general integration challenge across ecosystems.
  • Native support inside popular CI configuration formats could reduce the need for separate scripts and lower maintenance overhead.
  • Making feedback a default or required step in TDM pipelines might change how teams track and act on debt over time.

Load-bearing premise

Mining GitHub Travis CI files and scripts accurately reflects typical TDM tool usage and correctly flags anti-patterns without major bias from data selection or parsing limits.

What would settle it

A larger or differently sampled study that finds direct native integration without external scripts to be the majority approach, or that ranks a different anti-pattern above absent feedback, would undermine the reported prevalence results.

Figures

Figures reproduced from arXiv: 2604.10631 by Daniel Feitosa, Elisa Yumi Nakagawa, Jo\~ao Paulo Biazotto, Paris Avgeriou.

Figure 1
Figure 1. Figure 1: Tools that co-occur in at least 20 different pipelines [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: reports the distribution of types for script-based and direct executions. In direct calls, mixed job pipelines account for a little more than a half of the cases (1,963/3,283), while script-based setups raise this to about two-thirds (2,119/2,993). Direct calls show a larger portion of dedicated job within stage (870/3,283), whereas dedicated stage (450/3,283) is a minority. In script-based TDM, dedicated … view at source ↗
read the original abstract

Managing technical debt (TD) is critical to ensure the sustainability of long-term software projects. However, the time and cost involved in technical debt management (TDM) often discourage practitioners from performing this activity consistently. Continuous Integration and Continuous Delivery (CI/CD) pipelines offer an opportunity to support TDM by embedding automated practices directly into the development workflow. Despite this potential, it remains unclear how TDM tools could be integrated into CI/CD pipelines, and we still lack established best practices for this process. To address this problem, the objective of this study is to understand how TDM tools have been used in CI/CD pipelines and also identify potential configuration anti-patterns. To this end, we conducted a large-scale mining software repository (MSR) study on GitHub. In total, we collected around 600,000 Travis CI configuration files and 50,000 supporting scripts, and identified 3,684 pipelines that contain at least one TDM tool. We applied descriptive statistics to analyze the prevalence of tools and anti-patterns, and our findings show that most tools are executed and integrated using an external script; in addition, \textit{Absent Feedback} is the most common configuration anti-pattern. We believe that researchers and practitioners can use the evidence of this study to further investigate how to improve both the tools that are integrated in CI/CD and the integration practices.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper reports a large-scale mining software repository study of GitHub projects using Travis CI. From ~600,000 .travis.yml files and 50,000 supporting scripts the authors identify 3,684 pipelines containing at least one technical debt management (TDM) tool. Descriptive statistics are applied to characterize integration methods and configuration anti-patterns, with the headline results that most TDM tools are invoked via external scripts and that Absent Feedback is the most common anti-pattern.

Significance. If the classification pipeline is shown to be reliable, the work supplies useful empirical prevalence data on how TDM tools are actually embedded in open-source CI/CD workflows. The scale of the corpus (hundreds of thousands of configurations) is a clear strength for descriptive claims and can inform both tool developers and practitioners seeking integration guidelines.

major comments (2)
  1. [Methodology] Methodology section (tool identification and classification pipeline): the criteria, heuristics, and validation steps used to detect TDM tool invocations inside .travis.yml files and the 50k scripts are not described in sufficient detail. It is therefore impossible to assess precision/recall for the distinction between direct integration and external-script usage or to evaluate handling of includes, variables, or false positives. This classification step is load-bearing for both headline claims.
  2. [Results] Results on anti-patterns (Absent Feedback): the operational definition and automated detection rule for Absent Feedback (and the other anti-patterns) are not specified, nor is any manual validation or inter-rater agreement reported. Because the paper asserts this is the most prevalent anti-pattern, any systematic under- or over-detection directly undermines the prevalence ranking.
minor comments (1)
  1. [Abstract] Abstract and §3: the exact filtering steps that reduce the initial 600k files to the final 3,684 pipelines should be stated explicitly (including any exclusion criteria) to support reproducibility.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. The comments correctly identify areas where the manuscript would benefit from greater methodological transparency. We address each point below and will revise the paper to incorporate additional details on the classification pipeline and anti-pattern definitions.

read point-by-point responses

  1. Referee: [Methodology] Methodology section (tool identification and classification pipeline): the criteria, heuristics, and validation steps used to detect TDM tool invocations inside .travis.yml files and the 50k scripts are not described in sufficient detail. It is therefore impossible to assess precision/recall for the distinction between direct integration and external-script usage or to evaluate handling of includes, variables, or false positives. This classification step is load-bearing for both headline claims.

    Authors: We agree that the methodology section requires expansion for reproducibility. In the revised manuscript we will add a dedicated subsection detailing the exact criteria and heuristics applied to identify TDM tool invocations within .travis.yml files and the supporting scripts. This will include how direct invocations were distinguished from external-script usage, the handling of includes and variable substitution, and any steps taken to mitigate false positives. We will also report any validation performed and, where feasible, provide estimates of precision and recall for the classification. revision: yes

  2. Referee: [Results] Results on anti-patterns (Absent Feedback): the operational definition and automated detection rule for Absent Feedback (and the other anti-patterns) are not specified, nor is any manual validation or inter-rater agreement reported. Because the paper asserts this is the most prevalent anti-pattern, any systematic under- or over-detection directly undermines the prevalence ranking.

    Authors: We accept that the operational definitions and automated detection rules for the configuration anti-patterns were not presented with sufficient explicitness. The revised version will include clear definitions for each anti-pattern (with Absent Feedback treated in detail) together with the precise rules used for automated detection. No manual validation or inter-rater agreement study was conducted in the original work; we will therefore state this limitation explicitly and discuss its potential effect on the reported prevalence ordering rather than claiming validation that does not exist. revision: partial

Circularity Check

0 steps flagged

No significant circularity in empirical MSR study

full rationale

The paper performs a direct mining software repository analysis: it collects ~600k Travis CI files and 50k scripts, identifies 3,684 pipelines containing TDM tools via parsing, and applies descriptive statistics to tabulate integration methods and anti-pattern frequencies. No equations, derivations, fitted parameters, or predictions exist that could reduce to inputs by construction. Claims rest on observed counts and categorizations from the mined data rather than any self-referential loop, self-citation chain, or renamed ansatz. The analysis is therefore self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The claims depend on assumptions about the representativeness of GitHub Travis CI data and the accuracy of heuristic-based detection of TDM tools and anti-patterns; no free parameters or invented entities are evident from the abstract.

axioms (2)
  • domain assumption GitHub repositories using Travis CI provide a representative sample of open-source CI/CD practices involving TDM tools
    The study uses this data source as the basis for prevalence statistics without external validation.
  • domain assumption TDM tools and configuration anti-patterns can be reliably identified from CI configuration files and scripts
    Descriptive analysis rests on this classification step.

pith-pipeline@v0.9.0 · 5556 in / 1230 out tokens · 42790 ms · 2026-05-10T15:50:55.932814+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 34 canonical work pages

  1. [1]

    A systematic mapping study on technical debt and its management,

    Z. Li, P. Avgeriou, and P. Liang, “A systematic mapping study on technical debt and its management,”Journal of Systems and Software, vol. 101, pp. 193–220, Mar. 2015

    work page 2015

  2. [2]

    A tertiary study on technical debt: Types, management strategies, research trends, and base information for practitioners,

    N. Rios, M. G. de Mendonc ¸a Neto, and R. O. Sp ´ınola, “A tertiary study on technical debt: Types, management strategies, research trends, and base information for practitioners,”Information and Software Technol- ogy, vol. 102, pp. 117–145, 2018

    work page 2018

  3. [3]

    Technical debt cripples software developer productivity: A longitudinal study on developers’ daily soft- ware development work,

    T. Besker, A. Martini, and J. Bosch, “Technical debt cripples software developer productivity: A longitudinal study on developers’ daily soft- ware development work,” inInternational Conference on Technical Debt (TechDebt), 2018, p. 105–114

    work page 2018

  4. [4]

    Exploring technical debt tools: A systematic mapping study,

    J. D. S. Silva, J. G. Neto, U. Kulesza, G. Freitas, R. Rebouc ¸as, and R. Coelho, “Exploring technical debt tools: A systematic mapping study,” inInternational Conference on Enterprise Information Systems (ICEIS), 2022, pp. 280–303

    work page 2022

  5. [5]

    Un- derstanding practitioners’ reasoning and requirements for efficient tool support in technical debt management,

    J. a. P. Biazotto, D. Feitosa, P. Avgeriou, and E. Y . Nakagawa, “Un- derstanding practitioners’ reasoning and requirements for efficient tool support in technical debt management,”Empirical Software Engineering, vol. 30, no. 5, Jul. 2025

    work page 2025

  6. [6]

    Technical debt management automation: State of the art and future perspectives,

    ——, “Technical debt management automation: State of the art and future perspectives,”Information and Software Technology, p. 107375, Dec. 2023

    work page 2023

  7. [7]

    N. Rios, R. O. Sp ´ınola, M. Mendonc ¸a, and C. Seaman, “The practi- tioners’ point of view on the concept of technical debt and its causes and consequences: a design for a global family of industrial surveys and its first results from Brazil,”Empirical Software Engineering, vol. 25, no. 5, pp. 3216–3287, Jun. 2020

    work page 2020

  8. [8]

    An architectural technical debt index based on machine learning and architectural smells,

    D. Sas and P. Avgeriou, “An architectural technical debt index based on machine learning and architectural smells,”IEEE Transactions on Software Engineering, vol. 49, no. 8, pp. 4169–4195, 2023

    work page 2023

  9. [9]

    Breakbot: analyzing the impact of breaking changes to assist library evolution,

    L. Ochoa, T. Degueule, and J.-R. Falleri, “Breakbot: analyzing the impact of breaking changes to assist library evolution,” inProceedings of the ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results, ser. ICSE ’22. ACM, May 2022

    work page 2022

  10. [10]

    Technical debt tracking: Current state of practice: A survey and multiple case study in 15 large organiza- tions,

    A. Martini, T. Besker, and J. Bosch, “Technical debt tracking: Current state of practice: A survey and multiple case study in 15 large organiza- tions,”Science of Computer Programming, vol. 163, pp. 42–61, 2018

    work page 2018

  11. [11]

    Manifesto from dagstuhl perspectives workshop 24452 – reframing technical debt,

    P. Avgeriou, I. Ozkaya, H. Koziolek, Z. Codabux, and N. Ernst, “Manifesto from dagstuhl perspectives workshop 24452 – reframing technical debt,” 2025

    work page 2025

  12. [12]

    Technical debt in continuous software engineering: An overview of the state of the art and future trends,

    L. d. O. Carvalho, J. P. Biazotto, D. Feitosa, and E. Y . Nakagawa, “Technical debt in continuous software engineering: An overview of the state of the art and future trends,” inAnais do XXVII Congresso Ibero-Americano em Engenharia de Software (CIbSE 2024), ser. CIbSE

    work page 2024

  13. [13]

    Sociedade Brasileira de Computac ¸ ˜ao, May 2024, p. 313–326. [Online]. Available: http://dx.doi.org/10.5753/cibse.2024.28456

    work page doi:10.5753/cibse.2024.28456 2024

  14. [14]

    Continuous software engineering: A roadmap and agenda,

    B. Fitzgerald and K.-J. Stol, “Continuous software engineering: A roadmap and agenda,”Journal of Systems and Software, vol. 123, pp. 176–189, 2017

    work page 2017

  15. [15]

    Use and misuse of continuous integration features: An empirical study of projects that (mis)use travis ci,

    K. Gallaba and S. McIntosh, “Use and misuse of continuous integration features: An empirical study of projects that (mis)use travis ci,”IEEE Transactions on Software Engineering, vol. 46, no. 1, p. 33–50, Jan

    work page

  16. [16]

    Available: http://dx.doi.org/10.1109/TSE.2018.2838131

    [Online]. Available: http://dx.doi.org/10.1109/TSE.2018.2838131

    work page doi:10.1109/tse.2018.2838131 2018

  17. [17]

    InInt’l Conf

    C. Vassallo, S. Proksch, H. C. Gall, and M. Di Penta, “Automated reporting of anti-patterns and decay in continuous integration,” in2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, May 2019. [Online]. Available: http: //dx.doi.org/10.1109/ICSE.2019.00028

    work page doi:10.1109/icse.2019.00028 2019

  18. [18]

    In continuous software development, tools are the message for documentation,

    T. Theunissen, S. Hoppenbrouwers, and S. Overbeek, “In continuous software development, tools are the message for documentation,” inPro- ceedings of the 23rd International Conference on Enterprise Information Systems. SCITEPRESS - Science and Technology Publications, 2021

    work page 2021

  19. [19]

    The most common causes and effects of technical debt: First results from a global family of industrial surveys,

    N. Rios, R. O. Sp ´ınola, M. Mendonc ¸a, and C. Seaman, “The most common causes and effects of technical debt: First results from a global family of industrial surveys,” in12th ACM/IEEE International Sympo- sium on Empirical Software Engineering and Measurement (ESEM), 2018, pp. 1–10

    work page 2018

  20. [20]

    An overview and comparison of technical debt measurement tools,

    P. Avgeriou, D. Taibi, A. Ampatzoglou, F. A. Fontana, T. Besker, A. Chatzigeorgiou, V . Lenarduzzi, A. Martini, A. Moschou, I. Pigazzini, N. Saarimaki, D. D. Sas, S. S. de Toledo, and A. A. Tsintzira, “An overview and comparison of technical debt measurement tools,”IEEE Software, vol. 38, no. 3, pp. 61–71, May 2021

    work page 2021

  21. [21]

    The need for holistic technical debt management across the value stream: Lessons learnt and open chal- lenges,

    S. Malakuti and J. Heuschkel, “The need for holistic technical debt management across the value stream: Lessons learnt and open chal- lenges,” in2021 IEEE/ACM International Conference on Technical Debt (TechDebt), 2021, pp. 109–113

    work page 2021

  22. [22]

    Automat- ing technical debt management: Insights from practitioner discussions in stack exchange,

    J. a. Biazotto, D. Feitosa, P. Avgeriou, and E. Nakagawa, “Automat- ing technical debt management: Insights from practitioner discussions in stack exchange,” in2025 IEEE/ACM International Conference on Technical Debt (TechDebt), 2025

    work page 2025

  23. [23]

    Consolidating a common perspective on technical debt and its management through a tertiary study,

    H. Jeronimo Junior and G. H. Travassos, “Consolidating a common perspective on technical debt and its management through a tertiary study,”Information and Software Technology, vol. 149, p. 106964, 2022

    work page 2022

  24. [24]

    Auto- mated measurement of technical debt: A systematic literature review,

    I. Khomyakov, Z. Makhmutov, R. Mirgalimova, and A. Sillitti, “Auto- mated measurement of technical debt: A systematic literature review,” in21st International Conference on Enterprise Information Systems (ICEIS), 2019, pp. 95–106

    work page 2019

  25. [25]

    Code red: the business impact of code quality - a quantitative study of 39 proprietary production codebases,

    A. Tornhill and M. Borg, “Code red: the business impact of code quality - a quantitative study of 39 proprietary production codebases,” inProceedings of the International Conference on Technical Debt, ser. TechDebt ’22. New York, NY , USA: Association for Computing Machinery, 2022, p. 11–20

    work page 2022

  26. [26]

    Verdecchia, K

    R. Verdecchia, K. Maggi, L. Scommegna, and E. Vicario,Technical Debt in Microservices: A Mixed-Method Case Study. Springer Nature Switzerland, 2024, p. 217–236

    work page 2024

  27. [27]

    Goal question metric (GQM) approach,

    R. van Solingen, V . Basili, G. Caldiera, and H. D. Rombach, “Goal question metric (GQM) approach,” Jan. 2002

    work page 2002

  28. [28]

    An empirical study on self-admitted technical debt in modern code review,

    Y . Kashiwa, R. Nishikawa, Y . Kamei, M. Kondo, E. Shihab, R. Sato, and N. Ubayashi, “An empirical study on self-admitted technical debt in modern code review,”Information and Software Technology, vol. 146, p. 106855, 2022

    work page 2022

  29. [29]

    The lifecycle of technical debt that manifests in both source code and issue trackers,

    J. Tan, D. Feitosa, and P. Avgeriou, “The lifecycle of technical debt that manifests in both source code and issue trackers,”Information and Software Technology, vol. 159, p. 107216, 2023

    work page 2023

  30. [30]

    Identifying emerging smells in soft- ware designs based on predicting package dependencies,

    A. Tommasel and J. A. Diaz-Pace, “Identifying emerging smells in soft- ware designs based on predicting package dependencies,”Engineering Applications of Artificial Intelligence, vol. 115, p. 105209, 2022

    work page 2022

  31. [31]

    Characterizing the usage of ci tools in ml projects,

    D. E. Rzig, F. Hassan, C. Bansal, and N. Nagappan, “Characterizing the usage of ci tools in ml projects,” inProceedings of the 16th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement, ser. ESEM ’22. New York, NY , USA: Association for Computing Machinery, 2022, p. 69–79. [Online]. Available: https://doi.org/10.1145/354490...

    work page doi:10.1145/3544902.3546237 2022

  32. [32]

    Analyzing the impact of ci sub-practices on continuous code quality in open-source projects: An empirical study,

    G. Freitas, J. a. H. Bernardo, G. Siz ´ıLio, D. A. D. Costa, and U. Kulesza, “Analyzing the impact of ci sub-practices on continuous code quality in open-source projects: An empirical study,” inProceedings of the XXXVII Brazilian Symposium on Software Engineering, ser. SBES ’23. New York, NY , USA: Association for Computing Machinery, 2023, p. 1–10. [Onli...

    work page doi:10.1145/3613372.3613403 2023

  33. [33]

    2025.To- wards the Versioning of LLM-Agent-Based Software

    L. Chomkatek, J. Papuga, P. Nowak, and A. Poniszewska-Mara ´nda, “Decoding ci/cd practices in open-source projects with llm insights,” inProceedings of the 33rd ACM International Conference on the Foundations of Software Engineering, ser. FSE Companion ’25. New York, NY , USA: Association for Computing Machinery, 2025, p. 1638–1644. [Online]. Available: h...

    work page doi:10.1145/3696630 2025

  34. [34]

    Monitoring technical debt in an industrial setting,

    E.-M. Arvanitou, A. Ampatzoglou, S. Bibi, A. Chatzigeorgiou, and I. Stamelos, “Monitoring technical debt in an industrial setting,” in Proceedings of the 23rd International Conference on Evaluation and Assessment in Software Engineering, ser. EASE ’19. New York, NY , USA: Association for Computing Machinery, 2019, p. 123–132. [Online]. Available: https://...

    work page doi:10.1145/3319008.3319019 2019