Pith. sign in

REVIEW 3 major objections 6 minor 47 references

Rethinking Issue Resolution for AI/ML Systems

T0 review · 3 major / 6 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Issue resolution in AI/ML systems loops through experimentation, monitoring, and artifact changes, which traditional code-centric frameworks fail to capture.

desk verdict A useful, honestly hedged reconnaissance study that gives SE4ML a common vocabulary for issue resolution, but the cross-stage claim needs per-issue data before it can carry the vision. read the letter →

arxiv 2607.14657 v1 pith:TYZMBLCE submitted 2026-07-16 cs.SE

Ahmed Adnan , Mushfiqur Rahman , Antu Saha , Oscar Chaparro This is my paper
classification cs.SE
keywords issueresolutionAI/MLmaintenancequalitativestudynondeterminismreproducibilityheterogeneousartifactsMLOpsreportanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that fixing issues in AI/ML systems follows a different pattern from traditional software debugging. Through a qualitative analysis of 100 closed issue reports and pull requests across four widely used open-source AI/ML projects, the authors find that resolution involves recurring AI/ML-specific activities—model performance monitoring, parameter tuning and retraining, data modification, and model functionality analysis—that reappear across multiple stages instead of forming a single sequential workflow. Developers also face nondeterminism and reproducibility problems, and they respond with repeated experiments, statistical comparisons, and environment-specific validation rather than simple pass/fail tests. The paper reports that in nearly half of the AI/ML issues studied, the fix changed artifacts beyond source code, such as datasets, prompts, hyperparameters, and configurations. The authors conclude that existing issue resolution frameworks, built for deterministic and code-centric maintenance, need to be rethought for AI/ML.

What carries the argument

The argument's load-bearing machinery is the qualitative open-coding study itself: 988 snippets from 100 closed issue threads and their associated pull requests were annotated with fine-grained activity, challenge, and mitigation codes, building on an existing issue-resolution taxonomy. The key analytical move is the observation that the AI/ML-specific activities are cross-cutting—they appear in multiple traditional stages and co-occur with the challenges that the paper catalogues. This recurrence, not any single activity, is what motivates the proposed extension of issue resolution frameworks.

What would settle it

Directly observing AI/ML developers at work (e.g., IDE logs or screen recordings during issue resolution) and finding that they typically implement a fix once, validate it with deterministic unit tests, and merge without repeated experimentation, monitoring, or artifact coordination, would contradict the paper's central claim. Alternatively, a larger replication on systems beyond the four studied that finds these activities in few issues would weaken the generality.

Watch

Extended reading notes

Core claim

The central discovery is a set of four recurring AI/ML-specific resolution activities—model performance monitoring (in 27% of the sampled issues), parameter tuning and training (11%), data modification (7%), and model functionality analysis (5%)—that occur across traditional resolution stages (issue analysis, solution design, reproduction, implementation, verification) rather than in a single phase. These activities come with distinctive challenges: verifying fixes under nondeterministic behavior, reproducing issues across environments, understanding complex model architectures, and scaling fixes to larger or distributed settings. Developers mitigate them with repeated runs, statistical test

Load-bearing premise

The central claim rests on the assumption that what developers write in public issue threads and pull requests faithfully reflects the complete resolution process; if offline experimentation and private-branch changes go unmentioned, the recurring AI/ML activities could be documentation artifacts rather than real workflows.

Editorial extensions

If this is right

  • Issue resolution frameworks and issue triage should treat datasets, prompts, model configurations, and environments as first-class artifacts with provenance, not as attachments to code.
  • Verification of fixes in AI/ML systems should include repeated execution, statistical comparison, and environment-specific testing instead of deterministic pass/fail only.
  • AI-assisted maintenance tools and coding agents need to reason about stochastic behavior and heterogeneous artifacts to support AI/ML issue resolution.
  • Reproducibility support—experiment replay, environment isolation, and seed management—belongs inside the resolution workflow, not as an optional add-on.
  • Newcomer onboarding and documentation for AI/ML projects should reflect iterative, cross-stage resolution workflows rather than a linear fix pipeline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the findings scale to larger and more diverse systems, continuous integration for AI/ML may need statistical gates (e.g., confidence intervals on performance metrics) rather than binary test outcomes.
  • The paper's design suggests a natural extension: instrumenting developer workflows to capture offline experimentation, since public threads may under-represent trial-and-error and private branch work.
  • The cross-stage recurrence hints that AI/ML maintenance could be modeled as an experimentation loop akin to the scientific method, where each fix is a hypothesis validated across varying conditions.
  • A practical testable extension is to cluster issues by artifact type and measure whether resolutions grouped that way require different triage and review procedures.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper argues that traditional issue-resolution frameworks, which assume deterministic, code-centric, sequential workflows, are insufficient for AI/ML systems and advocates for tailored AI/ML issue-resolution frameworks. To motivate this, it reports a preliminary qualitative study of 100 closed issues and associated pull requests from TensorFlow, scikit-learn, MLflow, and AutoGPT. Using iterative open coding based on the authors' prior issue-resolution codebook, the study identifies four AI/ML-specific activities (model performance monitoring, parameter tuning and training, data modification, and model functionality analysis), several challenges (nondeterminism, reproduction difficulty, scaling, understanding model architectures), and adaptive validation and mitigation strategies. The paper uses these findings to propose five capabilities for future AI/ML issue-resolution frameworks and outlines research challenges.

Significance. If the empirical claims are supported, the paper addresses a real gap: existing SE4ML work focuses on ML bug taxonomies, fault localization, and MLOps challenges, but not on issue-resolution workflows as such. The study provides concrete examples, a replication package, and a systematic coding procedure grounded in a prior traditional-issue-resolution taxonomy, which is a strength. The proposed vision and research agenda are useful for framing future work. However, the current evidence base is narrow (25 issues per project, four projects) and one of the load-bearing empirical claims—cross-stage recurrence of AI/ML activities—is not established by the aggregate data reported. The paper is explicitly preliminary, but several of its central conclusions and the first proposed capability rest on this under-supported claim.

major comments (3)
  1. [III-B2, Table I; IV-A1] Finding 2 claims that AI/ML activities 'often ... span multiple resolution stages' and 'recur across multiple traditional issue resolution stages.' Table I reports only aggregate per-stage counts for each activity. For example, MODEL PERFORMANCE MONITORING is associated with 22 issues and stage counts totaling 28 (6+11+5+4+2), which implies that some issues were coded in more than one stage, but it does not report how many distinct issues were multi-stage or which stage combinations occurred. The pattern could be produced by different issues each contributing a single stage occurrence. Because Section IV-A1 (Iterative and Cross-Stage Resolution Workflows) is built directly on this finding, the cross-stage conclusion is not currently supported by the evidence as presented. Please report per-issue stage distributions (e.g., number of issues with two or more stages per activity, or a breakd
  2. [III-A3] The coding procedure explicitly states: 'Instead of calculating inter-coder agreement metrics, reliability was established throughout the coding process through critical, consensus-driven discussions and iterative refinement.' This is a notable limitation for a study whose findings are used to support a general vision. The assignment of snippets to AI/ML-specific activities and traditional stages is a central analytic step, and without any quantitative reliability measure, readers cannot calibrate the reproducibility of the coding. While consensus-based open coding is acceptable in some qualitative traditions, for an empirical software engineering audience I request at least a sample-based inter-coder reliability statistic (e.g., Cohen's kappa) on the final codebook, or a more detailed justification of why consensus coding alone is sufficient to support the weight placed on the findings.
  3. [III-A2, III-C] The study treats closed issue threads and associated pull requests as end-to-end evidence of the whole resolution process. However, issue threads are self-selected documentation: iterative experimentation, dataset or prompt tweaks, and parameter search may happen offline or in private branches before contributors report back. The Threats to Validity section acknowledges subjective coding and limited sample size, but does not address this documentation bias. Since several findings (e.g., Finding 2 and the characterization of experimentation-driven workflows) concern activities that are likely under-recorded in public threads, the authors should either triangulate with commit history or other repository artifacts, or explicitly bound the claims to 'publicly documented resolution activities' rather than all resolution activity.
minor comments (6)
  1. [III-A4, III-A5] The text reports '41 AI/ML-specific unique activities' in the coding results but later synthesizes '4 AI/ML-specific activities.' The relationship between these numbers should be clarified: are the 41 raw fine-grained codes that were later grouped into four high-level categories? Please state this explicitly.
  2. [References [3] and [23]] References [3] and [23] both cite Zeller's 'Why Programs Fail' and appear to be duplicates. Please consolidate or distinguish the editions.
  3. [Replication package (Ref. [25])] The replication package URL contains a space ('AI-ML Replication Package') and is likely not directly accessible. Provide a clean, valid URL.
  4. [Table I] The project names in Table I are inconsistent ('Tensorflow' vs. 'TensorFlow' elsewhere). Also, consider adding the total number of issues per project to the table so readers can interpret the per-project counts in context.
  5. [Figure 1] The diagram is dense and the labels are small; in a camera-ready version, ensure the activities, stages, and challenges are legible and that the mapping between them is clearly described in the caption or text.
  6. [III-B5] The statement that '83% of the non-AI/ML issues and 77% of the hybrid issues required only production code changes' would be easier to verify if the absolute numbers were given alongside the percentages, especially since the total number of non-AI/ML and hybrid issues is small (18 each).

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the study is an externally grounded qualitative analysis with no fitted predictions or derivation-by-definition.

full rationale

The paper makes no mathematical derivation and fits no parameters; its central claims are empirical generalizations from 100 externally sampled GitHub issues and pull requests across four AI/ML projects. The only self-citation is to the authors' prior ICSE'25 study of Firefox issue resolution [1], used as a baseline taxonomy of traditional resolution stages (Section III-A3: 'Starting from our prior study's codebook [1]'). This is not load-bearing circularity: that prior work analyzed non-AI/ML issues in a different project, so it provides an external, independent baseline rather than encoding the target result. The AI/ML-specific activities were identified by iteratively expanding the code catalog to capture activities that did not align with existing stages (Section III-A5), and the findings are supported by concrete issue examples (e.g., TensorFlow #46168, MLflow #12790, AutoGPT #2711). There is no uniqueness theorem imported from the authors, no ansatz smuggled via citation, and no fitted value renamed as a prediction. The skeptic's concern that Table I's aggregate per-stage counts do not fully demonstrate cross-stage recurrence within individual issues is a threat to evidence sufficiency, not a circularity; it does not show that any claimed result reduces by construction to its inputs. Accordingly, the circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claims rest on methodological assumptions about what public issue threads reveal, how the traditional codebook is extended, how coding reliability is established, and how representative the sample is. There are no mathematical free parameters or invented physical entities.

assumptions (4)
  • domain assumption Closed issue reports and associated PRs provide reliable end-to-end evidence of the issue resolution process.
    Section III-A2 justifies sampling closed issues because they offer end-to-end evidence, but written threads may omit offline or private experimentation.
  • domain assumption Coding can start from a traditional issue-resolution codebook and still reliably identify AI/ML-specific activities through iterative open coding.
    Section III-A3 starts from the authors' prior codebook [1]; anchoring on traditional stages is mitigated by iterative expansion but not quantitatively assessed.
  • domain assumption Consensus-driven reconciliation without inter-coder agreement metrics establishes sufficient coding reliability for the reported frequencies.
    Section III-A3 explicitly forgoes inter-coder agreement metrics; this is defensible in qualitative research but limits auditability of the counts.
  • domain assumption A sample of 100 issues from four projects is enough to identify recurring cross-project patterns.
    Section III-C acknowledges the sample may not generalize; the recurring language depends on this assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rethinking Issue Resolution for AI/ML Systems." pith.science (2026). https://pith.science/paper/TYZMBLCE

@misc{pith2026260714657,
  author       = {Pith},
  title        = {Pith review of: Rethinking Issue Resolution for AI/ML Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TYZMBLCE}},
  note         = {Machine review of arXiv:2607.14657}
}
read the original abstract

We advocate for AI/ML issue resolution frameworks tailored to maintenance workflows and the nature of modern AI/ML systems. Existing issue resolution frameworks largely emerged for traditional software maintenance practices and do not explicitly account for characteristics common in AI/ML systems, such as stochastic behavior, experimentation-driven workflows, and heterogeneous artifacts beyond source code. To identify the unique characteristics of issue resolution in AI/ML systems and motivate the need for tailored frameworks, we conducted a qualitative study of issue resolution workflows documented in 100 issue reports and pull requests across four widely used AI/ML systems: TensorFlow, scikit-learn, MLflow, and AutoGPT. Our findings suggest that issue resolution in AI/ML systems involves: recurring AI/ML-related activities that span multiple resolution stages; iterative experimentation and adaptive verification; and coordinated changes across artifacts such as datasets, prompts, and model configurations. We also observed challenges related to reproducibility, nondeterministic behavior, and artifact coordination. Building on these findings, we present a vision for AI/ML issue resolution frameworks and discuss research directions and tooling support needed to realize this vision.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

47 extracted references · 1 linked inside Pith

  1. [1]

    Decoding the issue resolution process in practice via issue report analysis: A case study of firefox,

    A. Saha and O. Chaparro, “Decoding the issue resolution process in practice via issue report analysis: A case study of firefox,” inICSE’25, 2025, pp. 2316–2328

    2025

  2. [2]

    A literature review of research in bug resolution: Tasks, challenges and future directions,

    T. Zhang, H. Jiang, X. Luo, and A. T. Chan, “A literature review of research in bug resolution: Tasks, challenges and future directions,”The Computer Journal, vol. 59, no. 5, pp. 741–773, 2016

    2016

  3. [3]

    Zeller,Why Programs Fail: A Guide to Systematic Debugging, 2nd ed

    A. Zeller,Why Programs Fail: A Guide to Systematic Debugging, 2nd ed. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 2009

    2009

  4. [4]

    Software engineering for machine learning: A case study,

    S. Amershi, A. Begel, C. Bird, R. DeLine, H. Gall, E. Kamar, N. Nagappan, B. Nushi, and T. Zimmermann, “Software engineering for machine learning: A case study,” inICSE-SEIP’19. IEEE, 2019, pp. 291–300

    2019

  5. [5]

    Copiloting the future: How generative ai transforms software engineering,

    L. Banh, F. Holldack, and G. Strobel, “Copiloting the future: How generative ai transforms software engineering,”Information and Software Technology, vol. 183, p. 107751, 2025

    2025

  6. [6]

    A comprehensive study on deep learning bug characteristics,

    M. J. Islam, G. Nguyen, R. Pan, and H. Rajan, “A comprehensive study on deep learning bug characteristics,” inESEC’19, 2019, pp. 510–520

    2019

  7. [7]

    Comparative analysis of real issues in open-source machine learning projects,

    T. D. Lai, A. Simmons, S. Barnett, J.-G. Schneider, and R. Vasa, “Comparative analysis of real issues in open-source machine learning projects,”EMSE, vol. 29, no. 3, p. 60, 2024

    2024

  8. [8]

    A systematic survey on debugging techniques for machine learning systems,

    T.-D. Nguyen, H. Tian, B. Le, P. Thongtanunam, and S. McIntosh, “A systematic survey on debugging techniques for machine learning systems,” arXiv e-prints, pp. arXiv–2503, 2025

    2025

Show all 47 references
  1. [9]

    Taxonomy of real faults in deep learning systems,

    N. Humbatova, G. Jahangirova, G. Bavota, V . Riccio, A. Stocco, and P. Tonella, “Taxonomy of real faults in deep learning systems,” in ICSE’20, 2020, pp. 1110–1121

    2020

  2. [10]

    Tensorflow: An open source machine learning framework,

    TensorFlow, “Tensorflow: An open source machine learning framework,” https://github.com/tensorflow/tensorflow, 2025, accessed: 2025-09-25

    2025

  3. [11]

    scikit-learn: Machine learning in python,

    Scikit-learn, “scikit-learn: Machine learning in python,” https://github. com/scikit-learn/scikit-learn, 2025, accessed: 2025-09-25

    2025

  4. [12]

    MLflow, “Mlflow,” https://github.com/mlflow/mlflow, 2025, accessed: 2025-09-25

    2025

  5. [13]

    Autogpt: An autonomous gpt-4 experiment,

    Significant-Gravitas, “Autogpt: An autonomous gpt-4 experiment,” https: //github.com/Significant-Gravitas/AutoGPT, 2025, accessed: 2025-09-25

    2025

  6. [14]

    Spencer,Card sorting: Designing usable categories

    D. Spencer,Card sorting: Designing usable categories. Rosenfeld Media, 2009

    2009

  7. [15]

    How does machine learning change software development practices?

    Z. Wan, X. Xia, D. Lo, and G. C. Murphy, “How does machine learning change software development practices?”TSE, vol. 47, no. 9, pp. 1857– 1871, 2019

    2019

  8. [16]

    A multivocal review of mlops practices, challenges and open issues,

    B. Eken, S. Pallewatta, N. Tran, A. Tosun, and M. A. Babar, “A multivocal review of mlops practices, challenges and open issues,”ACM Computing Surveys, vol. 58, no. 2, pp. 1–35, 2025

    2025

  9. [17]

    Maintainability challenges in ml: A systematic literature review,

    K. Shivashankar and A. Martini, “Maintainability challenges in ml: A systematic literature review,” in2022 48th Euromicro Conference on Software Engineering and Advanced Applications (SEAA). IEEE, 2022, pp. 60–67

    2022

  10. [18]

    Scalability and maintainability challenges and solutions in machine learning: Systematic literature review,

    K. Shivashankar, G. S. A. Hajj, and A. Martini, “Scalability and maintainability challenges and solutions in machine learning: Systematic literature review,”arXiv preprint arXiv:2504.11079, 2025

    2025 arXiv

  11. [19]

    Quality issues in machine learning software systems,

    P.-O. C ˆot´e, A. Nikanjam, R. Bouchoucha, I. Basta, M. Abidi, and F. Khomh, “Quality issues in machine learning software systems,” Empirical Software Engineering, vol. 29, no. 6, p. 149, 2024

    2024

  12. [20]

    An empirical study of pre-trained model reuse in the hugging face deep learning model registry,

    W. Jiang, N. Synovic, M. Hyatt, T. R. Schorlemmer, R. Sethi, Y .- H. Lu, G. K. Thiruvathukal, and J. C. Davis, “An empirical study of pre-trained model reuse in the hugging face deep learning model registry,” inProceedings of the 45th International Conference on Software Engin...

    2023

  13. [21]

    An empirical study on real bugs for machine learning programs,

    X. Sun, T. Zhou, G. Li, J. Hu, H. Yang, and B. Li, “An empirical study on real bugs for machine learning programs,” inAPSEC’17, ’17

  14. [22]

    Rajlich,Software engineering: The current practice

    V . Rajlich,Software engineering: The current practice. Crc Press, 2011

    2011

  15. [23]

    Zeller,Why programs fail: a guide to systematic debugging

    A. Zeller,Why programs fail: a guide to systematic debugging. Elsevier, 2009

    2009

  16. [24]

    Understanding the triaging and fixing processes of long lived bugs,

    R. K. Saha, S. Khurshid, and D. E. Perry, “Understanding the triaging and fixing processes of long lived bugs,”Information and software technology, vol. 65, pp. 114–128, 2015

    2015

  17. [25]

    Replication package,

    “Replication package,” 2026, accessed: 2026-5-12. [Online]. Available: https://github.com/mushfiqurgalib/AI-ML Replication Package/

    2026

  18. [26]

    Github api,

    Git, “Github api,” https://docs.github.com/en/rest, 2025, accessed: 2025- 09-25

    2025

  19. [27]

    Strauss and J

    A. Strauss and J. M. Corbin,Basics of qualitative research: Grounded theory procedures and techniques. Sage Publications, Inc, 1990

    1990

  20. [28]

    Hypothesis: Web annotation tool,

    Hypothesis, “Hypothesis: Web annotation tool,” 2023. [Online]. Available: https://web.hypothes.is/

    2023

  21. [29]

    Issue #46168,

    TensorFlow, “Issue #46168,” https://github.com/tensorflow/tensorflow/ issues/46168, 2021, gitHub issue. Accessed: 2026-02-27

    2021

  22. [30]

    [bug] mlflow.evaluate function crash on binary classi- fication evaluation,

    ehcalabres, “[bug] mlflow.evaluate function crash on binary classi- fication evaluation,” https://github.com/mlflow/mlflow/issues/12790, 2024, gitHub issue, opened Jul 25, 2024. Accessed: 2026-02-27

    2024

  23. [31]

    [bug] loading more runs in the experiment ui becomes very slow with a large number of rows,

    adamreeve, “[bug] loading more runs in the experiment ui becomes very slow with a large number of rows,” https://github.com/mlflow/mlflow/ issues/5653, 2022, gitHub issue, opened Apr 11, 2022. Accessed: 2026- 02-27

    2022

  24. [32]

    Multiworkermirroredstrategy keras example hangs #35878,

    TensorFlow contributors, “Multiworkermirroredstrategy keras example hangs #35878,” GitHub Issue, 2020. [Online]. Available: https: //github.com/tensorflow/tensorflow/issues/35878

    2020

  25. [33]

    scikit issue #29229,

    scikit, “scikit issue #29229,” https://github.com/scikit-learn/scikit-learn/ issues/29229

  26. [34]

    Issue #8414: (discussed in stackoverflow context) proposed application of the pipeline subset of steps,

    scikit-learn contributors, “Issue #8414: (discussed in stackoverflow context) proposed application of the pipeline subset of steps,” GitHub Issue, 2018. [Online]. Available: https://github.com/scikit-learn/ scikit-learn/issues/8414

    2018

  27. [35]

    Non-determinism from ‘tf.data.dataset.map‘ with random ops #13932,

    TensorFlow contributors, “Non-determinism from ‘tf.data.dataset.map‘ with random ops #13932,” GitHub Issue, 2017. [Online]. Available: https://github.com/tensorflow/tensorflow/issues/13932

    2017

  28. [36]

    Issue #75945,

    TensorFlow, “Issue #75945,” https://github.com/tensorflow/tensorflow/ issues/75945, 2026, gitHub issue. Accessed: 2026-03-03

    2026

  29. [37]

    Groupkfold inconsistent under ties in group sizes. #29495,

    scikit-learn contributors, “Groupkfold inconsistent under ties in group sizes. #29495,” GitHub Issue, 2024. [Online]. Available: https://github.com/scikit-learn/scikit-learn/issues/29495

    2024

  30. [38]

    AutoGPT Issue #2711,

    Significant-Gravitas, “AutoGPT Issue #2711,” https://github.com/ Significant-Gravitas/AutoGPT/issues/2711, 2026, gitHub issue, accessed 2026-05-12

    2026

  31. [39]

    “Dvc,”URL: https://dvc.org/, 2024

    2024

  32. [40]

    Hugging Face: The AI community building the future,

    Hugging Face, “Hugging Face: The AI community building the future,” https://huggingface.co, accessed: 2026-05-13

    2026

  33. [41]

    On automated and explainable provenance of ai-generated code,

    A. Velasco, N. Wintersgill, T. Stalnaker, O. Chaparro, and D. Poshyvanyk, “On automated and explainable provenance of ai-generated code,” in ICSME’26, 2025, p. (to appear)

    2025

  34. [42]

    Ai agentic programming: A survey of techniques, challenges, and opportunities,

    H. Wang, J. Gong, H. Zhang, J. Xu, and Z. Wang, “Ai agentic programming: A survey of techniques, challenges, and opportunities,” arXiv preprint arXiv:2508.11126, 2025

    2025

  35. [43]

    Lon- gitudinal studies,

    E. J. Caruana, M. Roman, J. Hern ´andez-S´anchez, and P. Solli, “Lon- gitudinal studies,”Journal of thoracic disease, vol. 7, no. 11, p. E537, 2015

    2015

  36. [44]

    Human-in-the-loop artificial intelligence: A systematic review of concepts, methods, and applications,

    K. Lazaros, A. G. Vrahatis, and S. Kotsiantis, “Human-in-the-loop artificial intelligence: A systematic review of concepts, methods, and applications,”Entropy, vol. 28, no. 4, p. 377, 2026

    2026

  37. [45]

    Cybergym: Evaluating ai agents’ real-world cybersecurity capabilities at scale,

    Z. Wang, T. Shi, J. He, M. Cai, J. Zhang, and D. Song, “Cybergym: Evaluating ai agents’ real-world cybersecurity capabilities at scale,”arXiv preprint arXiv:2506.02548, 2025

    2025

  38. [46]

    Stochastic debugging based reliability growth models for open source software project,

    S. Singhal, P. Kapur, V . Kumar, and S. Panwar, “Stochastic debugging based reliability growth models for open source software project,”Annals of Operations Research, vol. 340, no. 1, pp. 531–569, 2024

    2024

  39. [47]

    Probabilistic models for monitoring and fault diagnosis,

    V . Verma, J. Fernandez, R. Simmons, and R. Chatila, “Probabilistic models for monitoring and fault diagnosis,” inThe Second IARP and IEEE/RAS Joint Workshop on Technical Challenges for Dependable Robots in Human Environments. Ed. Raja Chatila, 2002

    2002

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.