Pith. sign in

REVIEW 3 major objections 4 minor 15 references

Explainable Artificial Intelligence Credit Risk Assessment using Machine Learning

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read This paper claims that LightGBM, explained by SHAP and LIME, gives the best credit-risk accuracy and approval-default trade-off among three ensemble models.

desk verdict LightGBM's 'most business-optimal' claim is contradicted by the paper's own Table 3—lowest recall, 85.7% false-negative rate—so the main conclusion does not follow from the reported metrics. read the letter →

arxiv 2506.19383 v1 pith:GYZAXAQP submitted 2025-06-24 cs.LG

classification cs.LG
keywords creditriskassessmentloandefaultpredictionLightGBMXGBoostRandomForestSHAPLIMEexplainableAI
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a machine-learning pipeline can make credit-risk decisions both accurate and transparent, and that among three candidate ensemble models, LightGBM is the most business-optimal choice. Training XGBoost, LightGBM, and Random Forest on the same preprocessed and feature-engineered data, the authors report LightGBM at 90.07% accuracy and 0.7203 ROC-AUC, with the best trade-off between approving loans and avoiding defaults. They argue that adding SHAP and LIME explanations, together with per-applicant risk reports, makes the system usable by lenders and borrowers, not just data scientists. A sympathetic reader would care because it promises automated credit decisions that are both profitable and auditable.

What carries the argument

The load-bearing mechanism is the comparative evaluation pipeline rather than a single new identity: three ensemble classifiers trained on the same engineered financial features, with class imbalance corrected by SMOTE and hyperparameters selected by grid-search cross-validation, are scored by accuracy, precision, recall, and ROC-AUC and then translated into business metrics (approval rate, default rate, false-positive rate, false-negative rate). SHAP and LIME are the explainability layer, producing global feature rankings and local per-applicant reasons, while a rule-based risk engine converts model probabilities into Low, Moderate, and High risk bands mapped to Approve, Review, or Reject decisions.

What would settle it

Re-running the pipeline on a time-based or applicant-ID-disjoint split and recomputing ROC-AUC, approval rate, and default rate for all three models would settle the claim: if LightGBM no longer holds the highest ROC-AUC or the best default-versus-approval trade-off, the central conclusion fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that, after identical preprocessing, SMOTE balancing, grid-search hyperparameter tuning, and feature engineering, LightGBM dominates the other two ensemble models on the measured criteria: it reaches the highest accuracy (90.07%), the highest ROC-AUC (0.7203), and the best business metrics, sanctioning more loans while keeping the default share among approved loans lower than XGBoost or Random Forest. XGBoost comes close on accuracy (88.74%) but its approved loans carry a higher default risk, and Random Forest, despite the highest recall (0.3040), has the lowest accuracy (82.03%) and the worst business outcome. The authors further assert that SHAP feature rankings show all three models depend mainly on external credit-report variables, and that LIME explanations make each applicant's risk factors inspectable by lenders and borrowers.

Load-bearing premise

The reported ranking assumes the test set was genuinely unseen, with applicant-level data merged by ID strictly before the train/test split; the paper does not specify the split procedure, and any leakage across that boundary would make the accuracy and ROC-AUC figures optimistic.

Editorial extensions

If this is right

  • If LightGBM is the best model, lenders using this pipeline should expect more approved applicants than with XGBoost or Random Forest and fewer defaults among those approved loans.
  • The SHAP rankings imply that external credit-source data carries most of the predictive signal, so data quality on those bureau variables is more valuable than fine-tuning demographic features.
  • Because XGBoost's accuracy is close but its default rate among approved loans is higher, accuracy alone is not a safe model-selection criterion for credit decisions.
  • Per-applicant LIME and SHAP reports supply a concrete artifact for regulatory review and customer dispute, making automated decisions auditable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not explore how the Approve, Review, and Reject thresholds affect the LightGBM ranking; the reported comparison is at one operating point, and shifting thresholds could reorder the models.
  • All three models show low precision at the chosen threshold, meaning most applicants flagged as high-risk are actually non-defaulters; a lender-aware threshold tuned to the relative cost of false rejections versus false approvals could change the business comparison.
  • A direct follow-up test would be to audit the train/test split for applicant-ID leakage; if the split is clean and LightGBM still wins, the claim generalizes, and if not, the reported metrics are optimistic.
  • The dominance of external credit-source features suggests richer bureau data may help all three models; testing each model with and without those features would show how much of the advantage is data-driven rather than algorithm-driven.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper describes a credit-risk-assessment pipeline that preprocesses loan-application data, trains XGBoost, LightGBM, and Random Forest models with SMOTE and GridSearchCV, and uses SHAP and LIME to generate explanatory reports. The central claim, stated in the abstract and reiterated in Section 4D, is that LightGBM is the most business-optimal model, achieving the highest accuracy and the best trade-off between approval and default rates. The empirical section reports Table 3: LightGBM accuracy 90.07%, precision 0.2757, recall 0.1434, ROC AUC 0.7203; XGBoost accuracy 88.74%, precision 0.2199, recall 0.1566, ROC AUC 0.7033; Random Forest accuracy 82.03%, precision 0.1652, recall 0.3040, ROC AUC 0.6808.

Significance. A valid, reproducible comparison of explainable ensemble models for credit risk would be of practical interest, and the paper's use of a multi-source feature set with SHAP and LIME reports is a sensible design direction. However, the manuscript as written does not provide enough experimental detail to support its conclusions: the data are unnamed and the train/test split is unspecified, no uncertainty quantification is reported, and the headline conclusion is contradicted by the paper's own Table 3. The contribution is therefore currently at the level of a descriptive case study rather than a verified scientific claim.

major comments (3)
  1. [Section 4C/4D and Table 3] The central claim that LightGBM is the most business-optimal model is contradicted by the paper's own reported metrics. Table 3 gives LightGBM a recall of 0.1434, implying a false-negative rate of 85.7% on defaulters, the highest among the three models; Section 4C asserts that LightGBM is effective at minimizing false negatives, and Section 4D concludes it has the best default/approval trade-off. These statements cannot both hold at the same decision threshold, so the headline conclusion is not derivable from the reported results.
  2. [Section 3E and Section 5] The claimed approval/default trade-off is never operationalized. Section 3E states that thresholds are set 'as per user requirements', but no numerical thresholds, no cost matrix, no approval rates, and no default rates among approved loans are provided. Section 5 states that LightGBM achieves the 'highest approval rate of 95%', but no corresponding default rate is reported, making it impossible to verify the claimed business optimality even under the authors' best-case evaluation assumptions.
  3. [Proposed Methodology (Data and Evaluation)] The paper does not name the dataset, does not describe the train/test split, and does not report error bars or confidence intervals for any metric. The merging procedure described in Section 3A (aggregating multiple tables by applicant ID) could in principle introduce leakage if training and test rows are drawn from overlapping applicants, but the manuscript gives no information to rule this out. Consequently, the reported accuracy, precision, recall, and ROC AUC cannot be assessed for reliability or generalizability.
minor comments (4)
  1. [Various] There are numerous typographical errors, including 'LighGBM' in the caption of Figure 2, 'customed' in Section 3A, and 'pf' instead of 'of' in Section 4D; the paper would benefit from a careful proofreading pass.
  2. [Section 4D] The text says Random Forest 'comes with lots pf negative factors' and 'underperforms hence is unreliable', but Table 3 shows Random Forest has the highest recall (0.3040), which the paper itself acknowledges in Section 4B; the discussion should reconcile this rather than dismiss Random Forest without qualification.
  3. [Section 2] The related-work references are incomplete in places (for example, reference [4] has 'Kousarziya' as an author and an incomplete name) and the mapping between the numbered citations and the Table 1 summary is not always clear.
  4. [Section 5] The claim that report generation is 'the most remarkable novelty of the system' is overstated, as automated XAI report generation is already common in credit-scoring tools; this framing should be toned down.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical model comparison whose claims are not derived from fitted parameters, self-citations, or definitional equivalences.

full rationale

This paper is an empirical machine-learning comparison, not a derivation chain, so the circularity patterns enumerated do not apply. The central claim that LightGBM is 'most business-optimal' is asserted from reported evaluation metrics and business-impact language, but no equation or fitted parameter is reused as a prediction. Hyperparameter tuning with GridSearchCV on ROC-AUC and then reporting ROC-AUC is standard model selection, not a fitted input renamed as a prediction. The paper cites existing literature and tool documentation (SHAP, DNV), but none of these citations is load-bearing in a way that reduces the paper's own result to the cited work; there is no self-citation chain, no imported uniqueness theorem, and no ansatz smuggled in via citation. The internal inconsistency between Table 3 (LightGBM recall 0.1434, i.e., FNR of 85.7%) and the Section 4C claim that LightGBM minimizes false negatives is a correctness and evidentiary problem, not a circularity problem: the conclusion is unsupported by the reported numbers, but it is not equivalent to the inputs by construction. Similarly, the absence of approval-rate and default-rate tables means the 'best trade-off' claim is unverifiable, but that is a missing-evidence issue rather than circular reasoning. Under the stated hard rules, no circular step can be exhibited with a quotation showing Eq. X = Eq. Y by construction or a fitted parameter renamed as prediction, so the honest finding is no significant circularity (score 0).

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The paper's empirical comparison depends on several undocumented choices: the dataset itself, the train/test split, the SMOTE ratio, the outlier clipping bound, and the risk thresholds. None of these are reported, so the central claim rests on assumptions the reader cannot verify.

free parameters (4)
  • SMOTE oversampling ratio = not specified
    Balances classes before training; directly changes the precision/recall trade-off reported in Table 3.
  • Outlier clipping threshold = 3 standard deviations
    Chosen by hand; removes extreme values from all numerical features and shapes the input distribution.
  • Risk category thresholds = not specified
    Maps predicted default probability to Approve/Review/Reject; directly determines the approval and default rate business metrics.
  • Hyperparameters (XGBoost, LightGBM, RF) = not specified
    Tuned with GridSearchCV but not reported; all model metrics depend on these choices.
assumptions (3)
  • domain assumption The train/test split is leakage-free and the test set is representative.
    The paper mentions separate datasets but never describes the split or confirms no ID-based leakage, so the reported metrics rely on this unverified premise.
  • domain assumption The metrics (accuracy, ROC AUC) are computed on a held-out test set.
    The evaluation protocol is not described; the paper only states 'there are separate datasets for training and testing' without details.
  • ad hoc to paper Clipping numerical values at 3 standard deviations retains the signal relevant to default prediction.
    The choice is arbitrary and not justified against the data distribution; it affects all downstream model inputs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Explainable Artificial Intelligence Credit Risk Assessment using Machine Learning." pith.science (2026). https://pith.science/paper/GYZAXAQP

@misc{pith2026250619383,
  author       = {Pith},
  title        = {Pith review of: Explainable Artificial Intelligence Credit Risk Assessment using Machine Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GYZAXAQP}},
  note         = {Machine review of arXiv:2506.19383}
}
read the original abstract

This paper presents an intelligent and transparent AI-driven system for Credit Risk Assessment using three state-of-the-art ensemble machine learning models combined with Explainable AI (XAI) techniques. The system leverages XGBoost, LightGBM, and Random Forest algorithms for predictive analysis of loan default risks, addressing the challenges of model interpretability using SHAP and LIME. Preprocessing steps include custom imputation, one-hot encoding, and standardization. Class imbalance is managed using SMOTE, and hyperparameter tuning is performed with GridSearchCV. The model is evaluated on multiple performance metrics including ROC-AUC, precision, recall, and F1-score. LightGBM emerges as the most business-optimal model with the highest accuracy and best trade off between approval and default rates. Furthermore, the system generates applicant-specific XAI visual reports and business impact summaries to ensure transparent decision-making.

Figures

Figures reproduced from arXiv: 2506.19383 by the authors.

Figure 1
Figure 1. Risk Assessment Flow F. Explainability using SHAP and LIME Explainable AI (XAI) refers to techniques that allows human users to comprehend and trust the results predicted by algorithms in machine learning. XAI techniques provide insights into how AI models arrive at their predictions or actions. Models like XGBoost, LightGBM and Random Forest are “black-box” models that are highly accurate but do not provide much in… view at source ↗
Figure 2
Figure 2. SHAP Summary of LighGBM (Top Performer) LIME Feature Importance gives insights into particular instances (specific applications) providing users transparency about the model’s decision for each applicant. In the risk report generated for the applicants, these insights help understand model’s interpretability of the risk decision. This provides decision-makers, as well as applicants, ability to review their position … view at source ↗
Figure 3
Figure 3. LIME Feature Importance of High-Risk Applicant by LightGBM [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Evaluation metrics by model [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: ROC AUC for LightGBM 11 [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: ROC AUC for XGBoost [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: ROC AUC for RF C. Business Metrics Overall, LightGBM is most likely to sanction loan requests and is associated with lowest proportion of defaults among all approved loans. LightGBM is also effective in minimizing the false positives as well as false negatives. This is…
Figure 8
Figure 8. Figure 8: Business metrics by model D. Top Classifier LightGBM shows top performance in terms of accuracy (90.07 %), business impact, approval rates and ROC-AUC. XGBoost performs near to LightGBM in terms of accuracy (88.74%) and approval rate but gives fair performance when con…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 15 canonical work pages

  1. [1]

    The impact of artificial intelligence on credit risk assessment and busi- ness model transformation in the financial sector,

    D. A. Dura and A. M. Cazacu, “The impact of artificial intelligence on credit risk assessment and busi- ness model transformation in the financial sector,”Journal of Law and Economic Development , vol. 6, no. 1, pp. 199–203, 2024, [Online]. Available: https://ideas.repec.org/a/cbu/jrnlec/y2024v6ip199-203. html

  2. [2]

    Credit risk prediction using machine learning and deep learning: A study on credit card customers,

    V . Chang, S. Sivakulasingam, H. Wang, S. T. Wong, M. A. Ganatra, and J. Luo, “Credit risk prediction using machine learning and deep learning: A study on credit card customers,”Risks, vol. 12, no. 11, p. 174, Nov. 2024, [Online]. Available: https://www.mdpi.com/2227-9091/12/11/174

  3. [3]

    KACDP: A Highly Interpretable Credit Default Prediction Model

    K. Liu and J. Zhao, “Kacdp: A highly interpretable credit default prediction model,” arXiv preprint arXiv:2411.17783, Nov. 2024, [Online]. Available: https://arxiv.org/abs/2411.17783

  4. [4]

    Enhancing credit scoring mod- els with artificial intelligence: A comparative study of traditional methods and ai-powered techniques,

    A. Hussain, M. A. Khan, A. A. K. S, and K. Kousarziya, “Enhancing credit scoring mod- els with artificial intelligence: A comparative study of traditional methods and ai-powered techniques,” in A Modern Approach to AI – Integrating Machine Learning with Agile Prac- tices, Oct. 2024, pp. 99–107, [Online]. Available: https://www.researchgate.net/publication...

  5. [5]

    Predictive analytics in credit risk management for banks: A comprehensive re- view,

    W. A. Addy, C. E. Ugochukwu, A. T. Oyewole, O. C. Ofodile, O. B. Adeoye, and C. C. Okoye, “Predictive analytics in credit risk management for banks: A comprehensive re- view,” GSC Advanced Research and Reviews , vol. 18, no. 2, pp. 434–449, Nov. 2024, [On- line]. Available: https://www.researchgate.net/publication/378672514 Predictive analytics in credit ...

  6. [6]

    Credit risk assessment and financial decision support using explainable artificial intelligence,

    M. K. Nallakaruppan, H. Chaturvedi, V . Grover, B. Balusamy, P. Jaraut, J. Bahadur, V . P. Meena, and I. A. Hameed, “Credit risk assessment and financial decision support using explainable artificial intelligence,” Risks, vol. 12, no. 10, p. 164, Oct. 2024, [Online]. Available: https://www.mdpi.com/ 2227-9091/12/10/164. 14

  7. [7]

    Credit risk prediction with and without weights of evidence using conventional pre- dictive statistical models,

    A. S. Al-Afeef, “Credit risk prediction with and without weights of evidence using conventional pre- dictive statistical models,” Cogent Economics & Finance , vol. 12, no. 1, pp. 1–15, 2024, [Online]. Available: https://doi.org/10.1080/23322039.2024.2338971

  8. [8]

    Artificial intelligence in credit risk: A literature review,

    P. Widagdo, R. A. Pratiwi, H. Nurlinda, N. Nurbaeti, R. Ismiwati, F. R. Kurniawan, and S. Yusri- ani, “Artificial intelligence in credit risk: A literature review,” in Proc. 6th Int. Seminar on Busi- ness, Economics, Social Science, and Technology (ISBEST) , vol. 3, 2023, [Online]. Available: https: //conference.ut.ac.id/index.php/isbest/article/view/1472

Show all 15 references
  1. [9]

    Estimating value-at-risk in the eurusd currency cross from implied volatilities using machine learning methods and quantile regression,

    H. M. Blom, P. E. de Lange, and M. Risstad, “Estimating value-at-risk in the eurusd currency cross from implied volatilities using machine learning methods and quantile regression,”Journal of Risk and Financial Management, vol. 16, no. 7, p. 312, Jul. 2023, [Online]. Available...

  2. [10]

    The value of open banking data for application credit scoring: Case study of a norwegian bank,

    L. O. Hjelkrem, P. E. de Lange, and E. Nesset, “The value of open banking data for application credit scoring: Case study of a norwegian bank,”Journal of Risk and Financial Management, vol. 15, no. 12, p. 597, 2022, [Online]. Available: https://www.mdpi.com/1911-8074/15/12/597

  3. [11]

    Credit risk modeling using predictive analytics,

    M. Yusuff, “Credit risk modeling using predictive analytics,” Apr. 2021, [Online]. Available: https: //www.researchgate.net/publication/386887850 Credit Risk Modeling Using Predictive Analytics

  4. [12]

    Credit risk analysis using machine and deep learning mod- els,

    P. M. Addo, D. Guegan, and B. Hassani, “Credit risk analysis using machine and deep learning mod- els,” Risks, vol. 6, no. 2, p. 38, 2018, [Online]. Available: https://www.mdpi.com/2227-9091/6/2/38

  5. [13]

    Application of artificial intelligence (artificial neural network) to assess credit risk: A predictive model for credit card scoring,

    M. S. Islam, L. Zhou, and F. Li, “Application of artificial intelligence (artificial neural network) to assess credit risk: A predictive model for credit card scoring,” Master’s thesis, Blekinge Insti- tute of Technology, 2009, [Online]. Available: https://www.diva-portal.org/...

  6. [14]

    Optimizing human-ai collaboration,

    DNV , “Optimizing human-ai collaboration,” [Online]. Available: https://www.dnv.com/research/ future-of-digital-assurance/optimizing-human-ai-collaboration/

  7. [15]

    Shap: Beeswarm plot,

    S. M. Lundberg and H. Chen, “Shap: Beeswarm plot,” [Online]. Available: https://shap.readthedocs. io/en/latest/example notebooks/api examples/plots/beeswarm.html. 15

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.