Explainable Artificial Intelligence Techniques for Software Development Lifecycle: A Phase-specific Survey

Aman Raj; Ankit Shetgaonkar; Dipen Pradhan; Lakshit Arora; Sanjay Surendranath Girija; Shashank Kapoor

REVIEW 4 major objections 5 minor 41 references

Explainable Artificial Intelligence Techniques for Software Development Lifecycle: A Phase-specific Survey

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This survey claims to be the first comprehensive, phase-specific mapping of explainable AI techniques onto every stage of the software development lifecycle, from requirements elicitation through maintenance.

desk verdict A clearly written but over-claimed survey of XAI-for-SDLC phases; treat its phase-specific 'most effective' rankings as hypotheses, not findings. read the letter →

arxiv 2505.07058 v1 pith:DNXKP5SF submitted 2025-05-11 cs.SE cs.AIcs.LG

Lakshit Arora , Sanjay Surendranath Girija , Shashank Kapoor , Aman Raj , Dipen Pradhan , Ankit Shetgaonkar This is my paper · ORCID

classification cs.SEcs.AIcs.LG

keywords ExplainableAISoftwareDevelopmentLifecycleSDLCphasesLIMESHAPCounterfactualexplanationsRuleextractionengineeringsurvey

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that explainability for AI-assisted software development is not a one-size-fits-all concern: each phase of the software development lifecycle (requirements, design, development, testing, deployment, maintenance) faces distinct explainability challenges, and different XAI techniques address them differently. It offers a phase-by-phase map that pairs techniques such as LIME, SHAP, rule extraction, counterfactuals, attention mechanisms, concept-based explanations, and example-based explanations with each phase's AI applications and explainability challenges. The paper also claims, citing prior systematic reviews, that XAI research in software engineering is heavily skewed toward maintenance (68%) and sparse in requirements and management (8% each), which motivates its emphasis on earlier phases. On top of the mapping, the paper claims to be the first comprehensive survey of XAI techniques for every SDLC phase. A sympathetic reader would take this as a practical guide and gap map intended to help practitioners choose XAI methods by phase and to steer research toward under-covered phases.

What carries the argument

The organizing device is a taxonomy-to-phase matrix. XAI techniques are classified by stage (ante-hoc vs post-hoc), scope (local vs global), and input/output format, then each SDLC phase is broken into AI applications, explainability challenges, and nominally most effective XAI techniques. The pairing does the argument's work: for example, counterfactual explanations answer 'what would need to differ in the input to change the outcome?' in design, testing, deployment, and maintenance, while LIME/SHAP attribute feature importance in requirements, development, testing, deployment, and maintenance. The mechanism is this repeatable technique-to-challenge pairing, underpinned by the distribution claim that most XAI-in-software-engineering research sits in maintenance.

What would settle it

A reader could re-run the search protocol the paper describes (peer-reviewed articles from the last six years across the major computing and scientific databases, using the stated keywords) and count how many XAI-in-software-engineering papers address each SDLC phase. If the resulting distribution differs materially from 68/16/8/8, or if design and testing turn out to contain substantial XAI research, the paper's gap-driven phase recommendations lose their evidential basis.

Watch

Extended reading notes

Core claim

The central claim is that XAI techniques should be selected per SDLC phase rather than applied uniformly, and that every phase has workable XAI options. For requirements elicitation, LIME/SHAP can surface latent requirements and detect bias, while counterfactuals clarify trade-offs; for design, counterfactuals, rule extraction, and concept-based explanations justify architecture choices; for development, LIME/SHAP, example-based explanations, and counterfactuals explain code generation and debug suggestions; for testing, LIME/SHAP and counterfactuals diagnose test failures; for deployment and monitoring, the same attribution and counterfactual tools explain anomaly flags and scaling decisions; and for maintenance, LIME/SHAP, counterfactuals, and attention mechanisms explain bug prediction, fixes, and summaries. The paper asserts that this phase-specific coverage makes it the first comprehensive survey of XAI across the whole software development lifecycle.

Load-bearing premise

The survey's recommendations rest on the assumption that the cited phase-distribution statistics—68% of XAI-in-software-engineering research in maintenance, 16% in development, and 8% each in management and requirements—are accurate and representative, so the claimed gaps in requirements and design are real gaps rather than artifacts of how the prior review searched.

Editorial extensions

If this is right

If the mapping is correct, practitioners can choose XAI techniques per phase instead of relying on a single universal method.
Requirements and design phases become concrete targets for XAI adoption and research, countering the maintenance-heavy distribution.
Per-phase XAI can expose latent requirements and bias early, through SHAP-style feature attribution on user interactions and existing systems.
Counterfactual explanations provide a uniform way to expose trade-offs across design, testing, deployment, and maintenance, because the same 'what would change the outcome?' question applies everywhere.
Standardized evaluation metrics and benchmarking structures are needed to compare XAI methods across phases, which the paper identifies as necessary future work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's per-phase 'most effective technique' rankings are narrative assignments rather than measured comparisons; a natural extension is a benchmark that runs the same attribution and counterfactual methods on requirements-elicitation and maintenance tasks to test whether the phase-specific recommendations survive.
If the 68/16/8/8 distribution is accurate, the highest-leverage move for the field is not a new XAI algorithm but shifting evaluation effort toward requirements and design, where the paper's own evidence says research is thinnest.
The taxonomy-to-phase matrix could be operationalized as a decision procedure: given an AI-assisted task, pick the technique by explanation scope (local vs global) and stage (ante-hoc vs post-hoc) rather than by technique familiarity.
A competing systematic map that counts papers per phase with a different protocol would directly test the survey's novelty and gap claims, since the distribution numbers are the load-bearing evidence for where XAI is missing.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

This is a survey paper, not a research result, and it should be read as one. What it does well: the descriptions of LIME, SHAP, counterfactuals, rule extraction, attention, and concept-based methods are accurate and accessible. The mapping of these techniques to SDLC phases is a useful organizing scaffold, especially for practitioners who want a quick overview of where XAI might apply. The writing is clean and the structure is sensible.

What is not new: the claim of being the first comprehensive phase-aware survey does not hold up against the paper's own cited references. Mohammadkhani et al. [7] already aligns XAI with software engineering phases and provides the 68/16/8/8 distribution. Clement et al. [10] explicitly aligns XAI with the software development process. This paper mostly rebuilds that map with slightly different phase boundaries and adds example use cases. The technique summaries are textbook material. That is fine for a tutorial, but it is not a novel contribution.

The bigger problem is the evidence gap behind the central claims. The paper states in Section II.C, citing [7], that design and testing have not been researched. Then Sections III.B and III.D assert that "the most effective XAI techniques" for those phases are LIME/SHAP, counterfactuals, and rule extraction. If there are no primary studies in those phases, no literature review could establish effectiveness rankings. Those claims are extrapolations or hypotheses, but they are presented as findings. The stress-test note is right about this tension. Also, the paper says it conducted an SLR but provides no protocol, screening counts, study list, or corpus table. The distribution numbers and effectiveness rankings are therefore unverifiable, and the "first comprehensive" claim rests on that unverifiable base.

Is it a bad paper? No. It is a useful secondary source for someone new to the area, and the phase-specific examples are often sensible. But the authors overreach. A revision that (a) removes or softens the novelty claim, (b) labels per-phase recommendations as hypotheses grounded in reasoning rather than systematic evidence, and (c) either reports the SLR details or explicitly depends on [7]'s corpus would make this an honest survey. As it stands, the load-bearing claims about effectiveness are not auditable.

My take: send it to peer review, but expect major revision. The core descriptions are sound and the phase map has practical value, but the evidence for the rankings and the first-comprehensive claim need fixing before publication. For my own work, I would cite the primary surveys, not this one.

Recommendation: engage with it as a candidate survey after major revision; do not desk reject, but do not let the current claims through as-is.

Best, [You]

Referee Report

4 major / 5 minor

Summary. This paper surveys Explainable AI (XAI) techniques organized by Software Development Life Cycle (SDLC) phase. It claims to be the first comprehensive survey of XAI techniques covering every SDLC phase, and it maps techniques such as LIME, SHAP, counterfactual explanations, rule extraction, attention mechanisms, concept-based explanations, and example-based explanations to requirements elicitation, design, development, testing, deployment/monitoring, and maintenance/evolution. The paper also reports, from a prior systematic literature review [7], that 68% of XAI-in-SE research focuses on maintenance, 16% on development, and 8% each on management and requirements. It draws conclusions about which XAI techniques are 'most effective' in each phase and recommends phase-specific XAI adoption.

Significance. If the phase-specific mapping were backed by a transparent and auditable literature review, the paper would be a useful practitioner-oriented reference: it gives generally accurate descriptions of LIME, SHAP, counterfactuals, rule extraction, and attention mechanisms, and it organizes these methods by SDLC phase in a way that could help practitioners choose candidates for explainability. The expository descriptions of individual XAI methods (Sections II.B.1–II.B.4) are broadly correct, and the paper makes a reasonable high-level case that different SDLC phases pose different explainability challenges. However, the central claims—first comprehensiveness, the 68/16/8/8 distribution, and the per-phase 'most effective' rankings—rest on an unverifiable evidentiary base, since the paper does not provide a corpus table, screening counts, or a reproducible search protocol. The significance of the paper as a survey is therefore currently limited by its failure to substantiate its empirical claims.

major comments (4)

[Section II.C vs. Section III.B/III.D] There is a direct internal contradiction between the paper's stated evidence and its phase-specific effectiveness claims. Section II.C, summarizing reference [7], states that 'Software design and testing have not been researched.' Yet Section III.B (Design, item 3) and Section III.D (Testing, item 3) both assert that 'during the literature review, the most effective XAI techniques in addressing XAI challenges were found to be' LIME/SHAP, counterfactuals, rule extraction, etc. If no primary studies exist for these phases, no literature review could have 'found' which techniques are most effective; these are proposals or expert judgments presented as empirical findings. The authors must either reclassify these as recommended/plausible techniques rather than literature-derived findings, or provide the primary sources that support effectiveness for design and testing.
[Abstract, Section I, Section II.C] The 68/16/8/8 distribution of XAI-in-SE research across SDLC phases is presented three times (abstract, Introduction, and Section II.C) and motivates the entire paper, but the manuscript provides no way to verify it. The Introduction describes search venues and inclusion criteria at a high level, but the paper does not report the number of studies screened, the number included, exclusion criteria, or a study list/corpus table. The statistics are imported from [7], a single arXiv preprint, and the paper does not assess the quality or representativeness of that source. Without these details, the motivating statistics and the implied research gap are not auditable.
[Abstract, Section I, Section V] The claim 'this is the first comprehensive survey of XAI techniques for every phase of the SDLC' is not supported by the evidence presented. The survey's own literature summary (Section II.C) states that design and testing have not been researched, so the content for those phases consists of proposed mappings rather than surveyed existing work. Additionally, the paper does not compare against the closely related prior work [10] (XAIR, a systematic metareview aligned to the software development process) beyond a citation, so the novelty claim is not established. Either provide a detailed comparison that demonstrates no prior phase-by-phase mapping exists, or soften the claim to 'first phase-specific proposal'.
[Section III (all phase subsections)] The phrase 'the most effective XAI techniques' is used repeatedly (e.g., Sections III.A.3, III.B.3, III.C.3, III.D.3, III.E.3, III.F.3) without any stated criterion for effectiveness. No evaluation framework, user study, fidelity measure, or comparative analysis is provided. If the paper intends to make a descriptive claim about the literature, it needs to show that the cited works actually evaluated these techniques and compared alternatives; if the claim is prescriptive, it should be worded as a recommendation. As written, the label 'most effective' is a comparative empirical claim with no supporting evidence.

minor comments (5)

[Abstract and Section III] The SDLC phase naming is inconsistent: the abstract says 'requirements elicitation, design and development, testing and deployment, and evolution' while Section III separates design, development, testing, and deployment/monitoring into distinct subsections. Align the phase list across the abstract, introduction, and body.
[Section II.C] The bullet 'Other tasks: Software design and testing have not been researched' uses the word 'tasks,' but design and testing are SDLC phases in this paper's own taxonomy; rephrase to avoid confusion.
[Figure 1] Figure 1 is referenced in Section II.B as a 'summarization and classification of standard XAI techniques,' but the figure content is not visible in the manuscript text and no figure caption is provided. Please include the figure or remove the reference.
[Reference [10]] The paper cites [10] (XAIR) as related work but does not explicitly state how its phase-level contribution differs from [10]'s alignment of XAI techniques to the software development process. A brief differentiation would strengthen the novelty discussion.
[References] Several references have incomplete author lists (e.g., [2] 'Cheligeer C, et. al' and [5] 'L. Chazette, et. al') and inconsistent rendering of 'et al.'; please correct to the journal's reference style.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the survey synthesizes external prior work without reducing any claim to its own input.

full rationale

This is an expository survey with no fitted parameters, no quantitative derivations, and no original equations to reduce. The phase-specific XAI mapping is a narrative synthesis of prior external literature, primarily the SLR of Mohammadkhani et al. [7], the XAI taxonomy of Vilone and Longo [30], and other cited surveys. Importing distribution statistics (68% maintenance, 16% development, 8% management/requirements) from [7] and then organizing XAI techniques by SDLC phase is normal scholarly dependency rather than circular reasoning, because the paper's output is a new organizational mapping, not a restatement of those statistics as a derived prediction. The 'first comprehensive survey' claim is a novelty assertion, not a derivation chain. The recurring phrase 'most effective XAI techniques' is an evidentiary concern: the paper's own Section II.C states that design and testing 'have not been researched,' while Sections III.B and III.D assert that literature review found effective techniques for those phases. That is an internal-evidence/validity inconsistency, not a circularity, because no step is defined in terms of its own conclusion and no cited result is invoked to forbid alternatives. There are no self-citations that carry load, no fitted inputs renamed as predictions, and no ansatz smuggled in via citation. Accordingly, no specific circular step can be quoted, and the score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The survey's conclusions rest on imported statistics, an accepted XAI taxonomy, and unvalidated feasibility assumptions; no free parameters or invented entities are introduced.

assumptions (3)

domain assumption The distribution of XAI research across SDLC phases (68% maintenance, 16% development, 8% management/requirements) from [7] is accurate and representative of the field.
The paper's motivation and gap analysis rest on these numbers, but it does not report the SLR protocol or corpus needed to check them; see Section I and II.C.
domain assumption Standard XAI taxonomies (ante-hoc/post-hoc, local/global, LIME/SHAP/rule extraction/counterfactuals) from [30] and related surveys are accepted as the complete relevant method space.
Section II.B presents these as the technique set; no method is added or validated for software engineering contexts.
domain assumption The illustrative XAI applications in each SDLC phase are feasible and would improve explainability without empirical validation.
Section III gives recommendations ('most effective') based on narrative reasoning and hypothetical examples, not on controlled studies.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Explainable Artificial Intelligence Techniques for Software Development Lifecycle: A Phase-specific Survey." pith.science (2026). https://pith.science/paper/DNXKP5SF

@misc{pith2026250507058,
  author       = {Pith},
  title        = {Pith review of: Explainable Artificial Intelligence Techniques for Software Development Lifecycle: A Phase-specific Survey},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DNXKP5SF}},
  note         = {Machine review of arXiv:2505.07058}
}

read the original abstract

Artificial Intelligence (AI) is rapidly expanding and integrating more into daily life to automate tasks, guide decision making, and enhance efficiency. However, complex AI models, which make decisions without providing clear explanations (known as the "black-box problem"), currently restrict trust and widespread adoption of AI. Explainable Artificial Intelligence (XAI) has emerged to address the black-box problem of making AI systems more interpretable and transparent so stakeholders can trust, verify, and act upon AI-based outcomes. Researchers have developed various techniques to foster XAI in the Software Development Lifecycle. However, there are gaps in applying XAI techniques in the Software Engineering phases. Literature review shows that 68% of XAI in Software Engineering research is focused on maintenance as opposed to 8% on software management and requirements. In this paper, we present a comprehensive survey of the applications of XAI methods such as concept-based explanations, Local Interpretable Model-agnostic Explanations (LIME), SHapley Additive exPlanations (SHAP), rule extraction, attention mechanisms, counterfactual explanations, and example-based explanations to the different phases of the Software Development Life Cycle (SDLC), including requirements elicitation, design and development, testing and deployment, and evolution. To the best of our knowledge, this paper presents the first comprehensive survey of XAI techniques for every phase of the Software Development Life Cycle (SDLC). This survey aims to promote explainable AI in Software Engineering and facilitate the practical application of complex AI models in AI-driven software development.

Figures

Figures reproduced from arXiv: 2505.07058 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 19 canonical work pages

[7]

Most common AI Applications: A literature review from 2023 conducted by Cheligeer C et. al. demonstrated how AI offers significant potential for automating and enhancing requirements elicitation based on the analysis of existing documents to infer key information and even create an initial draft of requirements [2]. Natural Language Processing (NLP) and L...

work page 2023
[10]

Software Engineering for AI-Based Systems: A Survey,

S. Martínez-Fernández et al., “Software Engineering for AI-Based Systems: A Survey,” ACM Transactions on Software Engineering and Methodology, vol. 31, no. 2, pp. 1–59, Apr. 2022, doi: 10.1145/3487043

doi:10.1145/3487043 2022
[1]

black-box problem

Explainable Artificial Intelligence Techniques for Software Development Lifecycle: A Phase-specific Survey Lakshit Arora lakshit@google.com Sanjay Surendranath Girija sanjaysg@google.com Aman Raj amanraj@google.com Dipen Pradhan dipenp@google.com Google Ankit Shetgaonkar ankiit@google.com Shashank Kapoor shashankkapoor@google.com Abstract— Artificial Inte...

work page 2024
[2]

by analogy

Instance based (or example based) techniques: These methods show examples (either real or synthetic) to illustrate the model's behavior. They explain "by analogy" or by showing "what-if" scenarios. Common techniques include: • Counterfactual Explanations [19]: Counterfactual explanations are a post-hoc, generally model-agnostic method for local explanatio...

work page 2020
[3]

They tell which features matter most

Feature Attribution based techniques: These techniques center around explaining predictions by assigning importance or relevance scores to the input features. They tell which features matter most. The most common feature attribution based techniques are: • LIME (Local Interpretable Model-agnostic Explanations) [12],[31]: LIME is a post-hoc, model-agnostic...

work page 2016
[4]

Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations

Mothilal, Ramaravind K., et al. “Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations.” Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, ACM, 2020, pp. 607–17. DOI.org (Crossref), https://doi.org/10.1145/3351095.3372850

arXiv 2020
[5]

striped” in an image dataset considerably influences the classification of images into “zebra

Concept based techniques: These techniques go beyond simple feature importance or examples and tries to explain in terms of abstract concepts that the model has (implicitly or explicitly) learned. • Concept-Based Explanations [12]: These are typically model-specific and post-hoc explanations and provide a global explanation of the model. They try to disco...

work page 2020
[6]

that has been applied to ensembles of decision trees and neural networks. C. XAI in Software Engineering XAI in Software Engineering involves applying XAI techniques to different Software Engineering phases and tasks. A Comprehensive literature study conducted by A. H. Mohammadkhani et. al. (2023) on XAI in Software Engineering presented the following fin...

work page 2023

Show all 41 references

[8]

Program Code Generation with Generative AIs,

I. Baskhad and S. Tim, “Program Code Generation with Generative AIs,” Algorithms, vol. 17, no. 2, 2024, doi: 10.3390/a17020062

2024 doi
[9]

al, Machine learning in requirements elicitation: a literature review

Cheligeer C, et. al, Machine learning in requirements elicitation: a literature review. Artificial Intelligence for Engineering Design, Analysis and Manufacturing. 2022;36:e32. doi:10.1017/S0890060422000166

2022 doi
[11]

The Use of AI in Software Engineering: A Synthetic Knowledge Synthesis of the Recent Research Literature,

K. Peter, “The Use of AI in Software Engineering: A Synthetic Knowledge Synthesis of the Recent Research Literature,” Information, vol. 15, no. 6, 2024, doi: 10.3390/info15060354

2024 doi
[12]

Explainable software systems: from requirements analysis to system evaluation,

L. Chazette, et. al, “Explainable software systems: from requirements analysis to system evaluation,” Requirements Engineering, 2022, doi: 10.1007/s00766-022-00393-5

2022 doi
[13]

A Rule Extraction Technique Applied to Ensembles of Neural Networks, Random Forests, and Gradient-Boosted Trees

Bologna, Guido. “A Rule Extraction Technique Applied to Ensembles of Neural Networks, Random Forests, and Gradient-Boosted Trees.” Algorithms, vol. 14, no. 12, Nov. 2021, p

2021
[14]

Four Principles of Explainable Artificial Intelligence,

P. J. Phillips et al., “Four Principles of Explainable Artificial Intelligence,” null, 2021, doi: 10.6028/nist.ir.8312

2021 doi
[15]

A Systematic Literature Review of Explainable AI for Software Engineering,

A. H. Mohammadkhani et. al, “A Systematic Literature Review of Explainable AI for Software Engineering,” arXiv.org, Feb. 2023, doi: 10.48550/arxiv.2302.06065

2023 doi arXiv
[16]

Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure

Hutchinson, Ben, et al. “Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure.” Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, ACM, 2021, pp. 560–75. DOI.org (Crossref), https://doi....

2021
[17]

A Practical tutorial on Explainable AI Techniques,

A. Bennetot et al., “A Practical tutorial on Explainable AI Techniques,” ACM Computing Surveys, 2024, doi: 10.1145/3670685

2024 doi
[18]

XAIR: A Systematic Metareview of Explainable AI (XAI) Aligned to the Software Development Process,

T. Clement, et al, “XAIR: A Systematic Metareview of Explainable AI (XAI) Aligned to the Software Development Process,” Machine Learning and Knowledge Extraction, vol. 5, no. 1, pp. 78–108, Jan. 2023, doi: 10.3390/make5010006

2023 doi
[19]

Explainable Artificial Intelligence: A Survey of Needs, Techniques, Applications, and Future Direction

Mersha, Melkamu, et al. “Explainable Artificial Intelligence: A Survey of Needs, Techniques, Applications, and Future Direction.” Neurocomputing, vol. 599, Sept. 2024, p. 128111. DOI.org (Crossref), https://doi.org/10.1016/j.neucom.2024.128111

2024
[20]

Explainable Artificial Intelligence (XAI): What we know and what is left to attain Trustworthy Artificial Intelligence,

S. Ali et al., “Explainable Artificial Intelligence (XAI): What we know and what is left to attain Trustworthy Artificial Intelligence,” Information Fusion, 2023, doi: 10.1016/j.inffus.2023.101805

2023
[21]

On completeness-aware concept-based explanations in deep neural networks

Yeh, Chih-Kuan, et al. "On completeness-aware concept-based explanations in deep neural networks." Advances in neural information processing systems 33 (2020): 20554-20565

2020
[22]

black-box

conducted a study on Software Engineering for Machine Learning (ML) systems in which they outlined the misconception that sometimes arises between Software Engineering for ML which refers to Software Engineering approaches to developing ML or AI systems versus ML for Software ...

2021
[23]

Large Language Models for Software Engineering: A Systematic Literature Review

Hou, Xinyi, et al. “Large Language Models for Software Engineering: A Systematic Literature Review.” ACM Transactions on Software Engineering and Methodology, vol. 33, no. 8, Nov. 2024, pp. 1–79. DOI.org (Crossref), https://doi.org/10.1145/3695988

2024 doi
[24]

Evaluating the Visualization of What a Deep Neural Network Has Learned

Samek, Wojciech, et al. “Evaluating the Visualization of What a Deep Neural Network Has Learned.” IEEE Transactions on Neural Networks and Learning Systems, vol. 28, no. 11, Nov. 2017, pp. 2660–73. IEEE Xplore, https://doi.org/10.1109/TNNLS.2016.2599820

2017
[25]

Model-Agnostic Interpretability with Shapley Values

Messalas, Andreas, et al. “Model-Agnostic Interpretability with Shapley Values.” 2019 10th International Conference on Information, Intelligence, Systems and Applications (IISA), IEEE, 2019, pp. 1–7. DOI.org (Crossref), https://doi.org/10.1109/IISA.2019.8900669

2019
[26]

Explainable artificial intelligence: a comprehensive review,

D. Minh, H. X. Wang, Y. F. Li, and T. N. Nguyen, “Explainable artificial intelligence: a comprehensive review,” Artificial Intelligence Review, pp. 1–66, Nov. 2021, doi: 10.1007/s10462-021-10088-y

2021 doi
[27]

Transparency and the Black Box Problem: Why We Do Not Trust AI,

W. J. von Eschenbach and J. R. Warren, “Transparency and the Black Box Problem: Why We Do Not Trust AI,” Philosophy & Technology, pp. 1–16, Sep. 2021, doi: 10.1007/s13347-021-00477-0

2021 doi
[28]

Exploring the Potential of Generative AI: Use Cases in Software Startups

Simaremare, Mario, et al. “Exploring the Potential of Generative AI: Use Cases in Software Startups.” Agile Processes in Software Engineering and Extreme Programming – Workshops, edited by Lodovica Marchesi et al., vol. 524, Springer Nature Switzerland, 2025, pp. 3–11. DOI.org...

2025 doi
[29]

A case study on the transformative potential of AI in software engineering on LeetCode and ChatGPT,

M. Merkel and J. Dorpinghaus, “A case study on the transformative potential of AI in software engineering on LeetCode and ChatGPT,” 2025, doi: https://doi.org/10.48550/arXiv.2501.03639

2025 doi arXiv
[30]

Classification of Explainable Artificial Intelligence Methods through Their Output Formats,

G. Vilone and L. Longo, “Classification of Explainable Artificial Intelligence Methods through Their Output Formats,” Machine Learning and Knowledge Extraction, vol. 3, no. 3, pp. 615–661, Aug. 2021, doi: 10.3390/make3030032

2021 doi
[31]

Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence,

V. Hassija et al., “Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence,” Cognitive Computation, vol. 16, pp. 45–74, Aug. 2023, doi: 10.1007/s12559-023-10179-8

2023 doi
[32]

Generative Artificial Intelligence for Software Engineering -- A Research Agenda,

A. Nguyen-Duc et al., “Generative Artificial Intelligence for Software Engineering -- A Research Agenda,” arXiv.org, Oct. 2023, doi: 10.48550/arxiv.2310.18648

2023 doi arXiv
[33]

The application of artificial intelligence in software engineering: a review challenging conventional wisdom,

F. A. Batarseh, R. Mohod, A. Kumar, and J. C. Bui, “The application of artificial intelligence in software engineering: a review challenging conventional wisdom,” Data Democracy, pp. 179–232, Jan. 2020, doi: 10.1016/b978-0-12-818366-3.00010-1

2020 doi
[34]

Navigating the Complexity of Generative AI Adoption in Software Engineering,

D. Russo, “Navigating the Complexity of Generative AI Adoption in Software Engineering,” ACM Transactions on Software Engineering and Methodology, 2023, doi: 10.1145/3652154

2023 doi
[36]

The Current Challenges of Software Engineering in the Era of Large Language Models,

C. Gao, X. Hu, S. Gao, X. Xia, and Z. Jin, “The Current Challenges of Software Engineering in the Era of Large Language Models,” ACM Transactions on Software Engineering and Methodology, 2025, doi: 10.1145/3712005

2025 doi
[37]

Explainable and secure artificial intelligence: taxonomy, cases of study, learned lessons, challenges and future directions,

K. A. Eldrandaly, M. Abdel-Basset, M. Ibrahim, and N. M. Abdel-Aziz, “Explainable and secure artificial intelligence: taxonomy, cases of study, learned lessons, challenges and future directions,” Enterprise Information Systems, Jul. 2022, doi: 10.1080/17517575.2022.2098537

2022
[39]

Survey on Explainable AI: From Approaches, Limitations and Applications Aspects,

W. Yang et al., “Survey on Explainable AI: From Approaches, Limitations and Applications Aspects,” Human-Centric Intelligent Systems, vol. 3, no. 3, pp. 161–188, Aug. 2023, doi: 10.1007/s44230-023-00038-y

2023 doi
[40]

Software Engineering Methods for Responsible Artificial Intelligence,

Z. U. Islam, “Software Engineering Methods for Responsible Artificial Intelligence,” Adaptive Agents and Multi-Agent Systems, 2021, doi: 10.5555/3463952.3464248

2021
[41]

Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI),

A. Adadi and M. Berrada, “Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI),” IEEE Access, vol. 6, pp. 52138–52160, Sep. 2018, doi: 10.1109/access.2018.2870052

2018
[339]

DOI.org (Crossref), https://doi.org/10.3390/a14120339

doi
[2021]

A software engineering perspective on engineering machine learning systems: State of the art and challenges. J. Syst. Softw. 180, C (Oct 2021). https://doi.org/10.1016/j.jss.2021.111031

2021

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [7]

Most common AI Applications: A literature review from 2023 conducted by Cheligeer C et. al. demonstrated how AI offers significant potential for automating and enhancing requirements elicitation based on the analysis of existing documents to infer key information and even create an initial draft of requirements [2]. Natural Language Processing (NLP) and L...

work page 2023

[2] [10]

Software Engineering for AI-Based Systems: A Survey,

S. Martínez-Fernández et al., “Software Engineering for AI-Based Systems: A Survey,” ACM Transactions on Software Engineering and Methodology, vol. 31, no. 2, pp. 1–59, Apr. 2022, doi: 10.1145/3487043

doi:10.1145/3487043 2022

[3] [1]

black-box problem

Explainable Artificial Intelligence Techniques for Software Development Lifecycle: A Phase-specific Survey Lakshit Arora lakshit@google.com Sanjay Surendranath Girija sanjaysg@google.com Aman Raj amanraj@google.com Dipen Pradhan dipenp@google.com Google Ankit Shetgaonkar ankiit@google.com Shashank Kapoor shashankkapoor@google.com Abstract— Artificial Inte...

work page 2024

[4] [2]

by analogy

Instance based (or example based) techniques: These methods show examples (either real or synthetic) to illustrate the model's behavior. They explain "by analogy" or by showing "what-if" scenarios. Common techniques include: • Counterfactual Explanations [19]: Counterfactual explanations are a post-hoc, generally model-agnostic method for local explanatio...

work page 2020

[5] [3]

They tell which features matter most

Feature Attribution based techniques: These techniques center around explaining predictions by assigning importance or relevance scores to the input features. They tell which features matter most. The most common feature attribution based techniques are: • LIME (Local Interpretable Model-agnostic Explanations) [12],[31]: LIME is a post-hoc, model-agnostic...

work page 2016

[6] [4]

Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations

Mothilal, Ramaravind K., et al. “Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations.” Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, ACM, 2020, pp. 607–17. DOI.org (Crossref), https://doi.org/10.1145/3351095.3372850

arXiv 2020

[7] [5]

striped” in an image dataset considerably influences the classification of images into “zebra

Concept based techniques: These techniques go beyond simple feature importance or examples and tries to explain in terms of abstract concepts that the model has (implicitly or explicitly) learned. • Concept-Based Explanations [12]: These are typically model-specific and post-hoc explanations and provide a global explanation of the model. They try to disco...

work page 2020

[8] [6]

that has been applied to ensembles of decision trees and neural networks. C. XAI in Software Engineering XAI in Software Engineering involves applying XAI techniques to different Software Engineering phases and tasks. A Comprehensive literature study conducted by A. H. Mohammadkhani et. al. (2023) on XAI in Software Engineering presented the following fin...

work page 2023