Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

This paper claims that a BERT classifier guided by the Breaking Ties active-learning strategy needs only 51% of expert-labeled Jira issues to outperform a model trained on all labels, cutting annotation effort by 49%.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 19:13 UTC pith:XRD5V2B5

load-bearing objection A useful ATD dataset and a plausible active-learning pipeline, but the central 'Breaking Ties is best' claim rests on single runs and missing same-budget random baselines. the 4 major comments →

arxiv 2603.02944 v2 pith:XRD5V2B5 submitted 2026-03-03 cs.SE

Reducing Labeling Effort in Architecture Technical Debt Detection through Active Learning and Explainable AI

Edi Sutoyo , Paris Avgeriou , Andrea Capiluppi This is my paper
classification cs.SE
keywords architecture technical debtactive learningBreaking TiesBERTkeyword filteringexplainable AILIMESHAP
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to show that detecting architecture technical debt (ATD) in issue trackers does not require labeling tens of thousands of issues by hand. It builds a pipeline that first uses expert-validated keywords to filter a large pool of candidate issues, then active learning with the Breaking Ties query strategy selects the most informative issues for annotation. The central result is that a BERT classifier trained on only 51% of the labeled data reaches F1 0.72, beating the same model trained on the full dataset (F1 0.66) and beating all compared baselines. This matters because expert annotation of ATD is scarce and expensive, so nearly halving the required labels could make large-scale ATD detection practical. The paper also argues that LIME and SHAP explanations are useful to experts, with LIME preferred for clarity.

Core claim

The central discovery is that uncertainty-based active learning, specifically Breaking Ties, lets a BERT classifier match or exceed full-supervision performance while using roughly half the labels. On a dataset built from ten open-source Java projects' Jira issues, Breaking Ties reached F1 0.72 at 1,700 training instances (51% of the pool), compared with F1 0.66 for BERT trained on the entire labeled set. The paper also reports that keyword filtering alone catches only 21–33% of true ATD issues but filters out 83–85% of non-ATD issues, so it is useful as a pre-filter rather than a standalone detector. For explainability, both LIME and SHAP provide reasonable local explanations, with experts

What carries the argument

The mechanism carrying the argument is a hybrid three-stage pipeline. First, keyword extraction (TF-IDF, KeyBERT, CS KeyBERT) with n-gram chunking and BERT-based cosine similarity flags candidate ATD issues. Second, pool-based active learning with six query strategies—prediction entropy, least confidence, Breaking Ties, embedding k-means, contrastive active learning, and random—iteratively selects the most informative issues for expert labeling; Breaking Ties, which chooses samples where the top-two class probabilities are closest, is the strategy that yields the reported gains. Third, LIME and SHAP generate token-level explanations of the BERT classifier's decisions. The load-bearing compar

Load-bearing premise

The dataset used to train and evaluate the model is a keyword-filtered sample of Jira issues, and the paper assumes this pool represents the population of architecture technical debt; if ATD issues that do not resemble the extracted keywords are common, both the F1 0.72 and the 49% effort reduction may not hold for unfiltered issue streams.

What would settle it

Take a random sample of all resolved Jira issues from the same ten projects without keyword filtering, have experts label them, and evaluate the Breaking Ties model on that sample. If its F1 drops well below 0.72, or if the 49% annotation saving disappears because the initial pool excludes many true ATD cases, the central claim is falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • A new ATD detection dataset can be built in about half the expert-labeling time, because the Breaking Ties strategy reaches F1 0.72 with only 51% of the labels.
  • The same BERT model trained on the full dataset ends up at F1 0.66, so the choice of which issues to label can matter more than labeling everything.
  • Keyword-based filtering is useful mainly as a pre-filter: it removes most non-ATD issues but misses 67–79% of true ATD issues, so supervised learning on top is necessary.
  • Model explanations from LIME and SHAP are considered useful by technical debt experts, and LIME's clarity makes it the preferred default for stakeholder-facing interpretation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Going beyond the paper: if the reported 49% reduction holds, active learning could be applied to other scarce-label software engineering tasks, such as detecting design debt or requirement debt in issue comments, where expert annotation is similarly expensive.
  • A testable extension would be to run the same Breaking Ties pipeline on GitHub-issue-based projects or non-Java languages; the current dataset is Java/Jira-only, so the effort reduction may not transfer to ecosystems with different issue-writing styles.
  • The keyword filter's 15–17% false-negative rate suggests that the ATD pool itself may be biased toward issues whose language resembles the seed keywords; a model trained on this pool may under-detect ATD expressed in unusual or project-specific terms.
  • Expert preference for LIME implies that explanation design should be human-centered and evaluated with domain users, hinting that future XAI tools for technical debt should prioritize accessible visualizations over formal completeness.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper addresses the cost of manually labeling Jira issues for Architecture Technical Debt (ATD) detection. The authors refine 116 previously labeled ATD issues into 57 expert-validated items, extract keywords via TF-IDF, KeyBERT, and CS KeyBERT, and use semantic-similarity matching to build a candidate pool of over 103,000 issues from ten Apache/Jira projects. They manually validate samples from this pool, then train a BERT classifier under six active-learning query strategies, compare it with supervised baselines, and assess LIME and SHAP explanations through an expert study. The central claim is that the Breaking Ties query strategy achieves an F1-score of 0.72 while using only 51% of the labeled data, corresponding to a 49% reduction in annotation effort relative to training BERT on all labels.

Significance. If the central claim is statistically supported, the paper makes a useful empirical contribution: it provides a new ATD-labeled dataset from Jira issues, demonstrates that active learning can reduce annotation cost for a specialized software-engineering classification task, and offers a rare expert evaluation of XAI outputs in the SATD/ATD domain. The pipeline is clearly described, the replication package is mentioned, and the RQ1 analysis of keyword-filter reliability is a useful negative result that motivates the hybrid design. However, the headline claims rest on single-run comparisons without uncertainty quantification, and the evaluation pool is constructed through keyword/semantic filtering. Both issues must be addressed before the paper's main conclusions can be considered established.

major comments (4)
  1. [Section 4.2, Tables 8 and 9, Figure 4] The comparison among query strategies is based on one F1 value per strategy and model, with no standard deviation, confidence interval, or significance test. Active learning is stochastic here: the initial 100-example seed set is randomly drawn (Section 4.2) and BERT is re-initialized and fine-tuned each iteration. The differences among strategies are small—Breaking Ties 0.72 vs. Prediction Entropy/Contrastive 0.70 vs. Random 0.68 in the combined configuration—and the 49% effort-reduction claim compares a single 0.72 at 1,700 labels with a single 0.66 for the fully supervised BERT. Repeated seed runs and appropriate statistical tests are needed to know whether Breaking Ties genuinely outperforms the alternatives, and whether the effort reduction is real rather than noise.
  2. [Section 3.2 and Section 4.1, Table 6] There is an internal inconsistency in the reported sample sizes. The text says 2,161 ATD-labeled instances were manually reviewed, but the sample sizes in Table 6 sum to 1,959 (379+377+373+279+224+275+12+13+27). Similarly, Section 4.1 states that the evaluation 'expanded the initial ATD dataset of 57 expert-validated items to 1,100 ATD instances,' which is neither 1,959 nor the total number of keyword-flagged items. The relationship among these figures—sampled, validated, deduplicated, and final training/evaluation sets—must be reconciled and stated precisely, since it directly affects reproducibility and the interpretation of all downstream results.
  3. [Section 3.2 and Section 4.1, false-negative analysis] The training and evaluation pool is defined by keyword extraction and semantic-similarity filtering, and the authors report that 15–17% of Non-ATD samples are misclassified false negatives. This means ATD issues whose wording does not resemble the extracted keywords are systematically underrepresented in the pool. The reported F1 of 0.72 and the 49% effort reduction are therefore measured on a filtered distribution, not on a random sample of Jira issues. The paper should either evaluate the final classifier on a sample drawn from the unfiltered issue population or explicitly delimit the claim to the keyword-filtered pool. As written, the external-validity caveat in Section 5.4 does not fully address this load-bearing limitation.
  4. [Section 3.2 and Section 4.2, True-ATD vs. Weak-ATD] The headline result uses the merged 'True- and Weak-ATD' class, where Weak-ATD is defined as cases on which only one annotator identified ATD and the other did not. Merging these borderline cases with True-ATD changes the classification task and may make the positive class easier to detect while also obscuring annotation disagreement. The paper already reports True-ATD-only results in Table 8; the discussion should justify why the merged class is the appropriate basis for the main claim, and should report separate error-analysis results for True-ATD and Weak-ATD rather than only aggregate F1 scores.
minor comments (5)
  1. [Throughout] There are frequent spacing/typo issues in rendered text, e.g., 'W eak-A TD' and 'Na ¨ ıve Bayes'; the manuscript should be copyedited for these artifacts.
  2. [Table 1] Apache Solr is described as a 'Load balancer'; the project is a search platform. This appears to be a factual error that should be corrected.
  3. [Section 4.1.1] The claimed annotation workload is based on 2,161 items, but Table 6 sums to 1,959. The per-item time estimates and total hours should be recomputed once the sample count is corrected.
  4. [Section 4.2] The class-imbalance discussion is qualitative; no class ratio or label distribution is reported for the training/evaluation pool. Please report the number of ATD and Non-ATD instances in the final dataset, and the split used for the supervised comparisons.
  5. [Section 4.3.1] The expert evaluation uses ten experts and reports mean scores without variability or significance tests. This is acceptable as a descriptive study, but the conclusions such as 'LIME preferred overall' should be phrased as descriptive findings, not as evidence of a general preference.

Circularity Check

0 steps flagged

No significant circularity: the F1 and 49% effort-reduction results are empirical outputs from a BERT model trained on manually validated labels, not quantities forced by construction.

full rationale

The paper's derivation chain is not circular. ATD keywords are extracted from a small expert-validated seed set (Section 3.2, unsupervised learning step) and used only to filter candidate Jira issues; the resulting candidate pool is then manually labeled by the authors with independent annotation and majority voting (Section 4.1.2). The active-learning classifier in RQ2 is trained and evaluated on these manual labels, not on the keyword scores: Table 8 and Figure 4 report F1 values from training BERT with each query strategy, and Table 9 compares the resulting model to baselines. The headline 'F1-score of 0.72 and a 49% reduction in annotation effort' is thus an empirical measurement, not a quantity derived by definition from the same inputs. The only self-citation in the pipeline is the initial 116-item dataset from Li et al. [31]; the authors explicitly relabel and validate it to 57 agreed items, so the published result does not rest on an unverified self-cited claim. The paper's own Threats to Validity sections acknowledge limitations such as single-run active-learning seeds, project-specific terminology, and keyword-filter representativeness; these are validity and robustness concerns, not circularity. No equation-level reduction, fitted-input-as-prediction, or load-bearing self-citation chain is present.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 1 invented entities

The central experimental results rest on several domain assumptions about what ATD looks like in Jira text, plus hand-set parameters for the keyword filter and AL initialization. None of these is 'fitted' in a formal sense, but the 0.9 cosine threshold, the chosen seed keywords, and the Weak-ATD merge directly shape the dataset and the reported F1.

free parameters (4)
  • semantic similarity threshold = 0.9
    Issues with an n-gram cosine similarity above 0.9 to an ATD keyword are automatically labeled ATD; chosen by hand, with no sensitivity analysis reported.
  • seed keywords for CS KeyBERT = move, refactor, remove, dependency, couple, update
    Selected by the authors based on frequency in the 57 ATD items and discussion; these seeds guide keyword extraction and affect all downstream filtering.
  • initial AL seed set size = 100
    Each active-learning query strategy starts with 100 randomly chosen labeled texts; this choice shapes early-iteration behavior and the final comparison.
  • n-gram sizes = unigrams, bigrams, trigrams
    Keyword matching uses uni-/bi-/tri-grams; no justification is given that these sizes capture ATD expressions better than other lengths.
axioms (4)
  • domain assumption Resolved Jira issues from ten Apache Java projects are representative of ATD discussions in issue trackers.
    Stage 1 restricts data to resolved issues and Apache projects, shaping both the 103k pool and the final dataset; stated in Section 3.2.
  • domain assumption Combining each issue's summary and description into one text is sufficient input for ATD detection.
    Section 3.2 follows Diamantopoulos et al. and ignores code/commit context; if ATD signals live elsewhere, the dataset and model miss them.
  • domain assumption True-ATD and Weak-ATD can be merged into a single positive class without changing the construct being learned.
    Section 4.2 pools True- and Weak-ATD labels; this raises F1 and is acknowledged as a construct-validity threat in Section 5.4.
  • domain assumption Manual labels assigned by the authors (with majority voting) are valid ground truth for both training and evaluation.
    Two authors label 3,313 samples; Cohen's kappa is reported only for the 1,152 Non-ATD set (0.81), not for the 2,161 ATD set, so ATD-label reliability is untested.
invented entities (1)
  • Weak-ATD annotation class no independent evidence
    purpose: Retains issues one annotator labeled 'Maybe-ATD' and merges them with True-ATD to enlarge the positive training class; the F1=0.72 result is reported on this merged class.
    The class is defined by the authors, has no external validation, and changes the target construct; the paper acknowledges this as a construct-validity threat (Section 5.4).

pith-pipeline@v1.3.0-alltime-deepseek · 26489 in / 17856 out tokens · 154030 ms · 2026-08-02T19:13:31.137103+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Reducing Labeling Effort in Architecture Technical Debt Detection through Active Learning and Explainable AI." pith.science (2026). https://pith.science/paper/XRD5V2B5

@misc{pith2026260302944,
  author       = {Pith},
  title        = {Pith review of: Reducing Labeling Effort in Architecture Technical Debt Detection through Active Learning and Explainable AI},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XRD5V2B5}},
  note         = {Machine review of arXiv:2603.02944}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Self-Admitted Technical Debt (SATD) refers to technical compromises explicitly admitted by developers in natural language artifacts, such as code comments, commit messages, and issue trackers. Among its types, Architecture Technical Debt (ATD) is particularly difficult to detect due to its abstract and context-dependent nature. Manual annotation of ATD is costly, time-consuming, and challenging to scale. To reduce labeling effort, this study combines keyword-based filtering, active learning, and explainable AI for ATD detection. We refined an existing dataset of ATD-related Jira issues to obtain an expert-validated seed set used to extract representative keywords. These keywords were then applied to identify more than 103k candidate issues across 10 open-source projects. To assess the reliability of keyword-based filtering, we qualitatively evaluated a statistically representative sample of labeled issues. Building on the resulting dataset, we applied active learning with multiple query strategies to prioritize informative samples for annotation. The results show that Breaking Ties achieved the best performance, with an F1-score of 0.72 and a 49% reduction in annotation effort. To improve transparency, we used SHAP and LIME to explain ATD classification results. Expert evaluation showed that both methods provided useful explanations, with LIME generally preferred for its clarity and ease of use.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. The Dangers of Non-Self-Fixed Architecture Technical Debt and Its Impact on Time-to-Fix

    cs.SE 2026-05 conditional novelty 5.0

    Non-self-fixed architectural technical debt persists longer than self-fixed debt in Apache projects, with repayment speed linked to the spread of changes across developers.

Reference graph

Works this paper leans on

72 extracted references · 9 linked inside Pith · cited by 1 Pith paper

  1. [1]

    IEEE access6, 52138–52160 (2018)

    Adadi, A., Berrada, M.: Peeking inside the black-box: a survey on explainable artificial intelligence (xai). IEEE access6, 52138–52160 (2018)

    2018

  2. [2]

    Human Behavior and Emerging Technologies2024(1), 4628855 (2024)

    Al-Ansari, N., Al-Thani, D., Al-Mansoori, R.S.: User-centered evaluation of explain- able artificial intelligence (xai): A systematic literature review. Human Behavior and Emerging Technologies2024(1), 4628855 (2024)

    2024

  3. [3]

    Information fusion58, 82–115 (2020)

    Arrieta, A.B., D ´ ıaz-Rodr ´ ıguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garc ´ ıa, S., Gil-L´ opez, S., Molina, D., Benjamins, R., et al.: Explainable artificial intelli- gence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information fusion58, 82–115 (2020)

    2020

  4. [4]

    arXiv preprint arXiv:2505.13009 (2025) 6 https://github.com/edisutoyo/ATD-ISSUES Title Suppressed Due to Excessive Length 39

    Avgeriou, P., Ozkaya, I., Koziolek, H., Codabux, Z., Ernst, N.: Manifesto from dagstuhl perspectives workshop 24452–reframing technical debt. arXiv preprint arXiv:2505.13009 (2025) 6 https://github.com/edisutoyo/ATD-ISSUES Title Suppressed Due to Excessive Length 39

    Pith/arXiv arXiv 2025

  5. [5]

    In: Proceedings of the 12th European Conference on Software Architecture: Companion Proceedings, pp

    Carrillo, C., Capilla, R.: Ripple effect to evaluate the impact of changes in architec- tural design decisions. In: Proceedings of the 12th European Conference on Software Architecture: Companion Proceedings, pp. 1–8 (2018)

    2018

  6. [6]

    In: 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR), pp

    Codabux, Z., Vidoni, M., Fard, F.H.: Technical debt in the peer-review documentation of r packages: A ropensci case study. In: 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR), pp. 195–206. IEEE (2021)

    2021

  7. [7]

    ACM Sigplan Oops Mes- senger4(2), 29–30 (1992)

    Cunningham, W.: The wycash portfolio management system. ACM Sigplan Oops Mes- senger4(2), 29–30 (1992)

    1992

  8. [8]

    IEEE software 30(1), 37–43 (2012)

    Dabbish, L., Stuart, C., Tsay, J., Herbsleb, J.: Leveraging transparency. IEEE software 30(1), 37–43 (2012)

    2012

  9. [9]

    In: QuASoQ@ APSEC, pp

    Dai, K., Kruchten, P.: Detecting technical debt through issue trackers. In: QuASoQ@ APSEC, pp. 59–65 (2017)

    2017

  10. [10]

    ¨O., Chouseinoglou, O., Tarhan, A.K.: Factors affecting architectural decision- making process and challenges in software projects: An industrial survey

    Demir, M. ¨O., Chouseinoglou, O., Tarhan, A.K.: Factors affecting architectural decision- making process and challenges in software projects: An industrial survey. Journal of Software: Evolution and Process36(10), e2703 (2024)

    2024

  11. [11]

    In: 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR), pp

    Diamantopoulos, T., Nastos, D.N., Symeonidis, A.: Semantically-enriched jira issue tracking data. In: 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR), pp. 218–222. IEEE (2023)

    2023

  12. [12]

    Information Sciences325, 98–117 (2015)

    D ´ ıez-Pastor, J.F., Rodr ´ ıguez, J.J., Garc ´ ıa-Osorio, C.I., Kuncheva, L.I.: Diversity tech- niques improve the performance of the best imbalance learning ensembles. Information Sciences325, 98–117 (2015)

    2015

  13. [13]

    In: Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pp

    Dor, L.E., Halfon, A., Gera, A., Shnarch, E., Dankin, L., Choshen, L., Danilevsky, M., Aharonov, R., Katz, Y., Slonim, N.: Active learning for bert: an empirical study. In: Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pp. 7949–7962 (2020)

    2020

  14. [14]

    ACM Computing Surveys55(9), 1–33 (2023)

    Dwivedi, R., Dave, D., Naik, H., Singhal, S., Omer, R., Patel, P., Qian, B., Wen, Z., Shah, T., Morgan, G., et al.: Explainable ai (xai): Core ideas, techniques, and solutions. ACM Computing Surveys55(9), 1–33 (2023)

    2023

  15. [15]

    Automated Software Engineering27(3), 369–392 (2020)

    Esteves, G., Figueiredo, E., Veloso, A., Viggiato, M., Ziviani, N.: Understanding machine learning software defect predictions. Automated Software Engineering27(3), 369–392 (2020)

    2020

  16. [16]

    Journal of Software: Evolution and Process37(4), e70018 (2025)

    Gezici Ge¸ cer, B., Kolukısa Tarhan, A.: Explainable ai framework for software defect prediction. Journal of Software: Evolution and Process37(4), e70018 (2025)

    2025

  17. [17]

    Grootendorst, M.: Keybert: Minimal keyword extraction with bert. (2020)

    2020

  18. [18]

    In: 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp

    Holub, A., Perona, P., Burl, M.C.: Entropy-based active learning for object recogni- tion. In: 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp. 1–8. IEEE (2008)

    2008

  19. [19]

    Honnibal, M., Montani, I., Van Landeghem, S., Boyd, A., et al.: spacy: Industrial- strength natural language processing in python (2020)

    2020

  20. [20]

    arXiv preprint arXiv:1802.07427 (2018)

    Hu, P., Lipton, Z.C., Anandkumar, A., Ramanan, D.: Active learning with partial feed- back. arXiv preprint arXiv:1802.07427 (2018)

    Pith/arXiv arXiv 2018

  21. [21]

    In: Benelux Conference on Artificial Intelligence, pp

    Jacobs, P.F., Maillette de Buy Wenniger, G., Wiering, M., Schomaker, L.: Active learn- ing for reducing labeling effort in text classification tasks. In: Benelux Conference on Artificial Intelligence, pp. 3–29. Springer (2021)

    2021

  22. [22]

    Jan´ ak, J.: Issue tracking systems. Ph.D. thesis, Masarykova univerzita, Fakulta infor- matiky (2009)

    2009

  23. [23]

    In: 2021 IEEE/ACM 18th Int

    Jiarpakdee, J., Tantithamthavorn, C.K., Grundy, J.: Practitioners’ perceptions of the goals and visual explanations of defect prediction models. In: 2021 IEEE/ACM 18th Int. Conf. on Mining Software Repositories (MSR), pp. 432–443. IEEE (2021)

    2021

  24. [24]

    In: International conference on machine learning, pp

    Kumar, I.E., Venkatasubramanian, S., Scheidegger, C., Friedler, S.: Problems with shapley-value-based explanations as feature importance measures. In: International conference on machine learning, pp. 5491–5500. PMLR (2020)

    2020

  25. [25]

    Journal of Computer Science and Technology35, 913–945 (2020)

    Kumar, P., Gupta, A.: Active learning query strategies for classification, regression, and clustering: A survey. Journal of Computer Science and Technology35, 913–945 (2020)

    2020

  26. [26]

    biometrics pp

    Landis, J.R., Koch, G.G.: The measurement of observer agreement for categorical data. biometrics pp. 159–174 (1977)

    1977

  27. [27]

    a systematic literature review

    Lenarduzzi, V., Besker, T., Taibi, D., Martini, A., Fontana, F.A.: Technical debt prioriti- zation: State of the art. a systematic literature review. arXiv preprint arXiv:1904.12538 (2019) 40 E. Sutoyo et al

    Pith/arXiv arXiv 1904

  28. [28]

    IEEE transactions on pattern analysis and machine intelligence28(8), 1251–1261 (2006)

    Li, M., Sethi, I.K.: Confidence-based active learning. IEEE transactions on pattern analysis and machine intelligence28(8), 1251–1261 (2006)

    2006

  29. [29]

    In: 2020 46th Euromicro conference on software engineering and advanced applications (SEAA), pp

    Li, Y., Soliman, M., Avgeriou, P.: Identification and remediation of self-admitted tech- nical debt in issue trackers. In: 2020 46th Euromicro conference on software engineering and advanced applications (SEAA), pp. 495–503. IEEE (2020)

    2020

  30. [30]

    Empirical Software Engineering27(6), 131 (2022)

    Li, Y., Soliman, M., Avgeriou, P.: Identifying self-admitted technical debt in issue track- ing systems using machine learning. Empirical Software Engineering27(6), 131 (2022)

    2022

  31. [31]

    Empirical Software Engineering28(3), 65 (2023)

    Li, Y., Soliman, M., Avgeriou, P.: Automatic identification of self-admitted technical debt from four different sources. Empirical Software Engineering28(3), 65 (2023)

    2023

  32. [32]

    In: 2015 12th Working IEEE/IFIP Conference on Software Architecture, pp

    Li, Z., Liang, P., Avgeriou, P.: Architectural technical debt identification based on archi- tecture decisions and change scenarios. In: 2015 12th Working IEEE/IFIP Conference on Software Architecture, pp. 65–74. IEEE (2015)

    2015

  33. [33]

    IEEE software29(6), 22–27 (2012)

    Lim, E., Taksande, N., Seaman, C.: A balancing act: What software practitioners have to say about technical debt. IEEE software29(6), 22–27 (2012)

    2012

  34. [34]

    Ad- vances in neural information processing systems30(2017)

    Lundberg, S.M., Lee, S.I.: A unified approach to interpreting model predictions. Ad- vances in neural information processing systems30(2017)

    2017

  35. [35]

    Journal of Machine Learning Research6(4) (2005)

    Luo, T., Kramer, K., Goldgof, D.B., Hall, L.O., Samson, S., Remsen, A., Hopkins, T., Cohn, D.: Active learning to recognize multiple types of plankton. Journal of Machine Learning Research6(4) (2005)

    2005

  36. [36]

    Margatina, K., Vernikos, G., Barrault, L., Aletras, N.: Active learning by acquiring contrastive examples. In: M.F. Moens, X. Huang, L. Specia, S.W.t. Yih (eds.) Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 650–663. Association for Computational Linguistics, Online and Punta Cana, Domini- can Republic (2021)

    2021

  37. [37]

    In: 2015 12th Working IEEE/IFIP Conference on Software Architecture, pp

    Martini, A., Bosch, J.: The danger of architectural technical debt: Contagious debt and vicious circles. In: 2015 12th Working IEEE/IFIP Conference on Software Architecture, pp. 1–10. IEEE (2015)

    2015

  38. [38]

    In: 2014 40th EUROMICRO Conference on Software Engi- neering and Advanced Applications, pp

    Martini, A., Bosch, J., Chaudron, M.: Architecture technical debt: Understanding causes and a qualitative model. In: 2014 40th EUROMICRO Conference on Software Engi- neering and Advanced Applications, pp. 85–92. IEEE (2014)

    2014

  39. [39]

    arXiv preprint arXiv:2407.14085 (2024)

    Meisenbacher, S., Schopf, T., Yan, W., Holl, P., Matthes, F.: An improved method for class-specific keyword extraction: A case study in the german business registry. arXiv preprint arXiv:2407.14085 (2024)

    Pith/arXiv arXiv 2024

  40. [40]

    Political Analysis28(4), 532–551 (2020)

    Miller, B., Linder, F., Mebane Jr, W.R.: Active learning approaches for labeling text: Review and assessment of the performance of active learning approaches. Political Analysis28(4), 532–551 (2020)

    2020

  41. [41]

    In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Prac- tice (ICSE-SEIP), pp

    Nayebi, M., Cai, Y., Kazman, R., Ruhe, G., Feng, Q., Carlson, C., Chew, F.: A longi- tudinal study of identifying and paying down architecture debt. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Prac- tice (ICSE-SEIP), pp. 171–180. IEEE (2019)

    2019

  42. [42]

    In: 2014 IEEE Int

    Potdar, A., Shihab, E.: An exploratory study on self-admitted technical debt. In: 2014 IEEE Int. Conf. on Software Maintenance and Evolution, pp. 91–100. IEEE (2014)

    2014

  43. [43]

    IEEE Transactions on Software Engineering48(12), 5050–5067 (2021)

    Prenner, J.A., Robbes, R.: Making the most of small software engineering datasets with modern machine learning. IEEE Transactions on Software Engineering48(12), 5050–5067 (2021)

    2021

  44. [44]

    In: Proceedings of the first instructional conference on machine learning, vol

    Ramos, J., et al.: Using tf-idf to determine word relevance in document queries. In: Proceedings of the first instructional conference on machine learning, vol. 242, pp. 29–

  45. [45]

    In: 2020 46th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp

    Rantala, L., M¨ antyl¨ a, M., Lo, D.: Prevalence, contents and automatic detection of kl- satd. In: 2020 46th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 385–388. IEEE (2020)

    2020

  46. [46]

    arXiv preprint arXiv:1908.10084 (2019)

    Reimers, N.: Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084 (2019)

    Pith/arXiv arXiv 1908

  47. [47]

    ACM computing surveys54(9), 1–40 (2021)

    Ren, P., Xiao, Y., Chang, X., Huang, P.Y., Li, Z., Gupta, B.B., Chen, X., Wang, X.: A survey of deep active learning. ACM computing surveys54(9), 1–40 (2021)

    2021

  48. [48]

    In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp

    Ribeiro, M.T., Singh, S., Guestrin, C.: ” why should i trust you?” explaining the pre- dictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135–1144 (2016) Title Suppressed Due to Excessive Length 41

    2016

  49. [49]

    arXiv preprint arXiv:1606.05386 (2016)

    Ribeiro, M.T., Singh, S., Guestrin, C.: Model-agnostic interpretability of machine learn- ing. arXiv preprint arXiv:1606.05386 (2016)

    Pith/arXiv arXiv 2016

  50. [50]

    Advanced Intelligent Systems7(1), 2400304 (2025)

    Salih, A.M., Raisi-Estabragh, Z., Galazzo, I.B., Radeva, P., Petersen, S.E., Lekadir, K., Menegaz, G.: A perspective on explainable artificial intelligence methods: Shap and lime. Advanced Intelligent Systems7(1), 2400304 (2025)

    2025

  51. [51]

    arXiv preprint arXiv:2107.10314 (2021)

    Schr¨ oder, C., M¨ uller, L., Niekler, A., Potthast, M.: Small-text: Active learning for text classification in python. arXiv preprint arXiv:2107.10314 (2021)

    Pith/arXiv arXiv 2021

  52. [52]

    Empirical Software Engineering 29(4), 93 (2024)

    Schulte, L., Ledel, B., Herbold, S.: Studying the explanations for the automated pre- diction of bug and non-bug issues using lime and shap. Empirical Software Engineering 29(4), 93 (2024)

    2024

  53. [53]

    Settles, B.: Active learning literature survey (2009)

    2009

  54. [54]

    Shapley, L.S., et al.: A value for n-person games (1953)

    1953

  55. [55]

    Automated Software Engineering29(2), 53 (2022)

    Sharma, R., Shahbazi, R., Fard, F.H., Codabux, Z., Vidoni, M.: Self-admitted technical debt in r: detection and causes. Automated Software Engineering29(2), 53 (2022)

    2022

  56. [56]

    Empirical Software Engineering29(6), 159 (2024)

    Sheikhaei, M.S., Tian, Y., Wang, S., Xu, B.: An empirical study on the effectiveness of large language models for satd identification and classification. Empirical Software Engineering29(6), 159 (2024)

    2024

  57. [57]

    Journal of Systems and Software152, 70–82 (2019)

    Sierra, G., Shihab, E., Kamei, Y.: A survey of self-admitted technical debt. Journal of Systems and Software152, 70–82 (2019)

    2019

  58. [58]

    IEEE Transactions on Software Engineering43(11), 1044–1062 (2017)

    da Silva Maldonado, E., Shihab, E., Tsantalis, N.: Using natural language processing to automatically detect self-admitted technical debt. IEEE Transactions on Software Engineering43(11), 1044–1062 (2017)

    2017

  59. [59]

    In: 2023 ACM/IEEE Int

    Skryseth, D., Shivashankar, K., Pil´ an, I., Martini, A.: Technical debt classification in issue trackers using natural language processing based on transformers. In: 2023 ACM/IEEE Int. Conf. on Technical Debt (TechDebt), pp. 92–101. IEEE (2023)

    2023

  60. [60]

    ACM Transactions on Software Engineering and Methodology (TOSEM)27(3), 1–51 (2018)

    Stol, K.J., Fitzgerald, B.: The abc of software engineering research. ACM Transactions on Software Engineering and Methodology (TOSEM)27(3), 1–51 (2018)

    2018

  61. [61]

    arXiv preprint arXiv:2312.15020 (2023)

    Sutoyo, E., Capiluppi, A.: Self-admitted technical debt detection approaches: A decade systematic review. arXiv preprint arXiv:2312.15020 (2023)

    Pith/arXiv arXiv 2023

  62. [62]

    In: 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp

    Tantithamthavorn, C.K., Jiarpakdee, J.: Explainable ai for software engineering. In: 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 1–2. IEEE (2021)

    2021

  63. [63]

    Journal of Systems and Software86(6), 1498–1516 (2013)

    Tom, E., Aurum, A., Vidgen, R.: An exploration of technical debt. Journal of Systems and Software86(6), 1498–1516 (2013)

    2013

  64. [64]

    IEEE Transactions on Software Engineering (2024)

    Tsoukalas, D., Mittas, N., Arvanitou, E.M., Ampatzoglou, A., Chatzigeorgiou, A., Kechagias, D.: Local and global explainability for technical debt identification. IEEE Transactions on Software Engineering (2024)

    2024

  65. [65]

    Encyclopedia of software engineering (2002)

    Van Solingen, R., Basili, V., Caldiera, G., Rombach, H.D.: Goal question metric (gqm) approach. Encyclopedia of software engineering (2002)

    2002

  66. [66]

    Machine Learning with Applications16, 100557 (2024)

    Vujinovi´ c, A., Luburi´ c, N., Slivka, J., Kovaˇ cevi´ c, A.: Using chatgpt to annotate a dataset: A case study in intelligent tutoring systems. Machine Learning with Applications16, 100557 (2024)

    2024

  67. [67]

    Journal of Systems and Software160, 110456 (2020)

    Wu, X., Zheng, W., Chen, X., Wang, F., Mu, D.: Cve-assisted large-scale security bug report dataset construction method. Journal of Systems and Software160, 110456 (2020)

    2020

  68. [68]

    In: Proceedings of the 38th international conference on software engineering, pp

    Xiao, L., Cai, Y., Kazman, R., Mo, R., Feng, Q.: Identifying and quantifying architec- tural debt. In: Proceedings of the 38th international conference on software engineering, pp. 488–498 (2016)

    2016

  69. [69]

    Pattern Recognition83, 401–415 (2018)

    Yang, Y., Loog, M.: A benchmark and comparison of active learning for logistic regres- sion. Pattern Recognition83, 401–415 (2018)

    2018

  70. [70]

    Journal of Systems and Software120, 195–218 (2016)

    Yli-Huumo, J., Maglyas, A., Smolander, K.: How do software development teams man- age technical debt?–an empirical study. Journal of Systems and Software120, 195–218 (2016)

    2016

  71. [71]

    arXiv preprint arXiv:2010.09535 (2020)

    Yuan, M., Lin, H.T., Boyd-Graber, J.: Cold-start active learning through self-supervised language modeling. arXiv preprint arXiv:2010.09535 (2020)

    Pith/arXiv arXiv 2010

  72. [72]

    In: IJCAI, pp

    Zhan, X., Liu, H., Li, Q., Chan, A.B.: A comparative survey: Benchmarking for pool- based active learning. In: IJCAI, pp. 4679–4686 (2021)

    2021