{"id":"728b98d5-9d62-4377-a282-618803a9eb71","arxiv_id":"2506.19383","paper_version":1,"verdict":"REJECT","confidence":"LOW","novelty_score":1.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LightGBM achieves the highest accuracy (90.07%) and ROC AUC (0.7203) for credit default prediction in a comparison of three ensembles with SHAP/LIME explainability on a public lending dataset.","lead":"This paper compares three common machine learning models for predicting loan defaults, adding SHAP and LIME explanations, and reports LightGBM as the most accurate model. The work is a standard applied machine learning study on a lending dataset, of interest mainly as a template for transparent credit scoring.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central 'business-optimal' claim contradicts the paper's own Table 3: LightGBM has FNR=85.7%, yet §4C says it minimizes false negatives, so the headline conclusion is not derivable from reported metrics.","rationale":"The reader's REJECT verdict is appropriate, but the single most load-bearing weakness is not (or not only) the assumed absence of leakage. The paper's own quantitative results contradict its headline conclusion: LightGBM's recall of 0.1434 means it misses about 86% of actual defaulters, yet §4C claims it minimizes false negatives. Since credit losses are driven primarily by false negatives, a model with the highest false-negative rate cannot be declared business-optimal without an explicit cost model and threshold policy. The paper provides neither. Section 3E describes risk bands and decision rules only qualitatively, and Section 5 introduces an approval rate of 95% with no accompanying default rate. Thus the central claim fails on internal consistency, not merely on external verification. The data-leakage concern raised by the reader is real and important, but it is secondary: even a perfectly executed train/test split would not make the reported metrics imply the claimed business optimality. For this reason I partially agree with the reader's weakest-assumption identification and recommend keeping the REJECT outcome, while shifting the emphasis to the internal contradiction in the reported business metrics.","tokens_in":8324,"tokens_out":4082,"duration_ms":48633,"concrete_test":"Ask the authors to release the exact train/test split, the decision threshold or cost specification used for the business metrics, and the raw test-set confusion matrices. Then, for each model, reconstruct the counts in Table 3 and compute the approval rate, the default rate among approved applicants, and the expected loss under the stated business rule. If LightGBM's false-negative rate is indeed 85.7%, demonstrate how it can simultaneously 'minimize false negatives' and have the lowest default rate among approved loans. If no threshold/cost definition can be provided, or if the recomputed business metrics do not rank LightGBM first, the central claim should be withdrawn.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim (Abstract; §4D) that LightGBM is 'the most business-optimal model' is not supported by the paper's own reported metrics. Table 3 gives LightGBM recall 0.1434, i.e. a false-negative rate of 85.7%—the highest of the three models—while §4C asserts that LightGBM 'is also effective in minimizing the false positives as well as false negatives.' Those statements cannot both hold at the same decision threshold. Business optimality in lending depends on the relative costs of approving a defaulter versus rejecting a good applicant, and on the numerical thresholds used to map predicted probabilities to approve/review/reject decisions. Section 3E says thresholds are set 'as per user requirements' but gives no numeric values, no cost matrix, and no table of approval rates or default rates. Section 5 introduces a 'highest approval rate of 95%' for LightGBM without reporting the corresponding default rate among approved loans. Consequently, even if the test data are perfectly clean and leakage-free, the paper provides no way to verify the claimed approval/default trade-off; the headline ranking is an unsupported assertion. This internal inconsistency is more immediate than the separate data-leakage concern, because it invalidates the central conclusion even under the authors' best-case evaluation setup.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes a credit-risk-assessment pipeline that preprocesses loan-application data, trains XGBoost, LightGBM, and Random Forest models with SMOTE and GridSearchCV, and uses SHAP and LIME to generate explanatory reports. The central claim, stated in the abstract and reiterated in Section 4D, is that LightGBM is the most business-optimal model, achieving the highest accuracy and the best trade-off between approval and default rates. The empirical section reports Table 3: LightGBM accuracy 90.07%, precision 0.2757, recall 0.1434, ROC AUC 0.7203; XGBoost accuracy 88.74%, precision 0.2199, recall 0.1566, ROC AUC 0.7033; Random Forest accuracy 82.03%, precision 0.1652, recall 0.3040, ROC AUC 0.6808.","tokens_in":8636,"tokens_out":2272,"duration_ms":25885,"significance":"A valid, reproducible comparison of explainable ensemble models for credit risk would be of practical interest, and the paper's use of a multi-source feature set with SHAP and LIME reports is a sensible design direction. However, the manuscript as written does not provide enough experimental detail to support its conclusions: the data are unnamed and the train/test split is unspecified, no uncertainty quantification is reported, and the headline conclusion is contradicted by the paper's own Table 3. The contribution is therefore currently at the level of a descriptive case study rather than a verified scientific claim.","major_comments":[{"comment":"The central claim that LightGBM is the most business-optimal model is contradicted by the paper's own reported metrics. Table 3 gives LightGBM a recall of 0.1434, implying a false-negative rate of 85.7% on defaulters, the highest among the three models; Section 4C asserts that LightGBM is effective at minimizing false negatives, and Section 4D concludes it has the best default/approval trade-off. These statements cannot both hold at the same decision threshold, so the headline conclusion is not derivable from the reported results.","section":"Section 4C/4D and Table 3"},{"comment":"The claimed approval/default trade-off is never operationalized. Section 3E states that thresholds are set 'as per user requirements', but no numerical thresholds, no cost matrix, no approval rates, and no default rates among approved loans are provided. Section 5 states that LightGBM achieves the 'highest approval rate of 95%', but no corresponding default rate is reported, making it impossible to verify the claimed business optimality even under the authors' best-case evaluation assumptions.","section":"Section 3E and Section 5"},{"comment":"The paper does not name the dataset, does not describe the train/test split, and does not report error bars or confidence intervals for any metric. The merging procedure described in Section 3A (aggregating multiple tables by applicant ID) could in principle introduce leakage if training and test rows are drawn from overlapping applicants, but the manuscript gives no information to rule this out. Consequently, the reported accuracy, precision, recall, and ROC AUC cannot be assessed for reliability or generalizability.","section":"Proposed Methodology (Data and Evaluation)"}],"minor_comments":[{"comment":"There are numerous typographical errors, including 'LighGBM' in the caption of Figure 2, 'customed' in Section 3A, and 'pf' instead of 'of' in Section 4D; the paper would benefit from a careful proofreading pass.","section":"Various"},{"comment":"The text says Random Forest 'comes with lots pf negative factors' and 'underperforms hence is unreliable', but Table 3 shows Random Forest has the highest recall (0.3040), which the paper itself acknowledges in Section 4B; the discussion should reconcile this rather than dismiss Random Forest without qualification.","section":"Section 4D"},{"comment":"The related-work references are incomplete in places (for example, reference [4] has 'Kousarziya' as an author and an incomplete name) and the mapping between the numbered citations and the Table 1 summary is not always clear.","section":"Section 2"},{"comment":"The claim that report generation is 'the most remarkable novelty of the system' is overstated, as automated XAI report generation is already common in credit-scoring tools; this framing should be toned down.","section":"Section 5"}],"recommendation":"reject","confidential_remarks":"The manuscript reads as a course project report rather than a finished research paper. The main empirical conclusion is internally inconsistent with the reported table, and the absence of dataset identification and split details means the results cannot be checked. I do not see how these issues can be fixed without substantially redoing the experiments and rewriting the conclusions, so my recommendation is reject rather than major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper's headline claim—that LightGBM is the most business-optimal model—is contradicted by the paper's own Table 3. LightGBM has recall 0.1434, meaning it misses 85.7% of actual defaulters, the worst of the three models. Section 4C says LightGBM 'is also effective in minimizing the false positives as well as false negatives,' which cannot be true at the same decision threshold. Section 5 introduces a 95% approval rate for LightGBM without reporting the corresponding default rate. So even under best-case assumptions about data leakage, the central conclusion is not derivable from the reported metrics.\n\nWhat the paper does well: it is a clean, readable walkthrough of a standard pipeline—XGBoost/LightGBM/RF, SMOTE, GridSearchCV, SHAP, LIME—on what appears to be the Home Credit dataset. The feature importance findings (external credit scores dominate) are plausible. The automated HTML/JSON report generator is a nice engineering deliverable for practitioners who want to operationalize XAI in a lending workflow.\n\nSoft spots beyond the central contradiction: the dataset is never named; the train/test split is not described; no error bars or confidence intervals; no code or data released. Business optimality would require a cost matrix and explicit thresholds for approve/review/reject, but the thresholds are only described as 'as per user requirements.' The reader's strongest claim is correct. The stress-test note about the recall/FNR contradiction is also correct. This isn't a case of a missing reference or a minor gap; it's a load-bearing inconsistency in the main result.\n\nWho is this for? Students or applied teams looking for a template for explainable credit scoring. As a research contribution, it replicates previously published comparisons and adds an engineering feature. I would not cite it. A serious referee would reject it because the central claim fails on the paper's own numbers. Recommendation: desk reject, though the report-generation component might be worth a blog post or industry write-up.","headline":"LightGBM's 'most business-optimal' claim is contradicted by the paper's own Table 3—lowest recall, 85.7% false-negative rate—so the main conclusion does not follow from the reported metrics.","tokens_in":9121,"tokens_out":2139,"would_cite":false,"duration_ms":20386,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that LightGBM, explained by SHAP and LIME, gives the best credit-risk accuracy and approval-default trade-off among three ensemble models.","keywords":["credit risk assessment","loan default prediction","LightGBM","XGBoost","Random Forest","SHAP","LIME","explainable AI"],"falsifier":"Re-running the pipeline on a time-based or applicant-ID-disjoint split and recomputing ROC-AUC, approval rate, and default rate for all three models would settle the claim: if LightGBM no longer holds the highest ROC-AUC or the best default-versus-approval trade-off, the central conclusion fails.","tokens_in":8131,"feed_emoji":"🏦","tokens_out":9069,"duration_ms":83976,"temperature":0.7,"pith_summary":"This paper tries to establish that a machine-learning pipeline can make credit-risk decisions both accurate and transparent, and that among three candidate ensemble models, LightGBM is the most business-optimal choice. Training XGBoost, LightGBM, and Random Forest on the same preprocessed and feature-engineered data, the authors report LightGBM at 90.07% accuracy and 0.7203 ROC-AUC, with the best trade-off between approving loans and avoiding defaults. They argue that adding SHAP and LIME explanations, together with per-applicant risk reports, makes the system usable by lenders and borrowers, not just data scientists. A sympathetic reader would care because it promises automated credit decisions that are both profitable and auditable.","feed_headline":"LightGBM wins credit-risk model comparison with 90.07% accuracy","feed_subtitle":"A transparent loan-screening pipeline shows LightGBM balances approvals and defaults better than the alternatives.","key_machinery":"The load-bearing mechanism is the comparative evaluation pipeline rather than a single new identity: three ensemble classifiers trained on the same engineered financial features, with class imbalance corrected by SMOTE and hyperparameters selected by grid-search cross-validation, are scored by accuracy, precision, recall, and ROC-AUC and then translated into business metrics (approval rate, default rate, false-positive rate, false-negative rate). SHAP and LIME are the explainability layer, producing global feature rankings and local per-applicant reasons, while a rule-based risk engine converts model probabilities into Low, Moderate, and High risk bands mapped to Approve, Review, or Reject decisions.","core_discovery":"The paper's central claim is that, after identical preprocessing, SMOTE balancing, grid-search hyperparameter tuning, and feature engineering, LightGBM dominates the other two ensemble models on the measured criteria: it reaches the highest accuracy (90.07%), the highest ROC-AUC (0.7203), and the best business metrics, sanctioning more loans while keeping the default share among approved loans lower than XGBoost or Random Forest. XGBoost comes close on accuracy (88.74%) but its approved loans carry a higher default risk, and Random Forest, despite the highest recall (0.3040), has the lowest accuracy (82.03%) and the worst business outcome. The authors further assert that SHAP feature rankings show all three models depend mainly on external credit-report variables, and that LIME explanations make each applicant's risk factors inspectable by lenders and borrowers.","pith_inferences":["The paper does not explore how the Approve, Review, and Reject thresholds affect the LightGBM ranking; the reported comparison is at one operating point, and shifting thresholds could reorder the models.","All three models show low precision at the chosen threshold, meaning most applicants flagged as high-risk are actually non-defaulters; a lender-aware threshold tuned to the relative cost of false rejections versus false approvals could change the business comparison.","A direct follow-up test would be to audit the train/test split for applicant-ID leakage; if the split is clean and LightGBM still wins, the claim generalizes, and if not, the reported metrics are optimistic.","The dominance of external credit-source features suggests richer bureau data may help all three models; testing each model with and without those features would show how much of the advantage is data-driven rather than algorithm-driven."],"forward_implications":["If LightGBM is the best model, lenders using this pipeline should expect more approved applicants than with XGBoost or Random Forest and fewer defaults among those approved loans.","The SHAP rankings imply that external credit-source data carries most of the predictive signal, so data quality on those bureau variables is more valuable than fine-tuning demographic features.","Because XGBoost's accuracy is close but its default rate among approved loans is higher, accuracy alone is not a safe model-selection criterion for credit decisions.","Per-applicant LIME and SHAP reports supply a concrete artifact for regulatory review and customer dispute, making automated decisions auditable."],"supporting_citations":[{"why":"Documents XGBoost reaching 99.4% accuracy on credit-card default data, setting the strong XGBoost baseline this paper compares against.","marker":"[2]"},{"why":"Shows Random Forest achieving high accuracy with SHAP and LIME on peer-to-peer lending data, grounding the RF-plus-XAI comparison.","marker":"[6]"},{"why":"Reports LightGBM and CatBoost as top estimators for financial value-at-risk, supporting the expectation that LightGBM excels on financial data.","marker":"[9]"},{"why":"Demonstrates LightGBM outperforming an existing logistic-regression application credit scorer, the direct precedent for choosing LightGBM.","marker":"[10]"},{"why":"Establishes gradient boosting as the top-AUC model on a large financial dataset, the benchmark family that LightGBM extends.","marker":"[12]"},{"why":"Motivates the paper's interpretability requirement by linking explainability to human-AI trust in sensitive decisions.","marker":"[14]"},{"why":"Supplies the SHAP beeswarm summary plot method used to produce the global feature-importance rankings.","marker":"[15]"}],"fun_headline_variants":["LightGBM tops credit-risk models with 90% accuracy","LightGBM beats XGBoost, Random Forest in loan default prediction","Transparent AI credit scoring: LightGBM wins on accuracy and balance","Best credit risk model found: LightGBM with SHAP and LIME explanations","LightGBM approves more loans with fewer defaults than rivals"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported ranking assumes the test set was genuinely unseen, with applicant-level data merged by ID strictly before the train/test split; the paper does not specify the split procedure, and any leakage across that boundary would make the accuracy and ROC-AUC figures optimistic.","fun_headline_variants_meta":{"raw":{"variants":["LightGBM tops credit-risk models with 90% accuracy","LightGBM beats XGBoost, Random Forest in loan default prediction","Transparent AI credit scoring: LightGBM wins on accuracy and balance","Best credit risk model found: LightGBM with SHAP and LIME explanations","LightGBM approves more loans with fewer defaults than rivals"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000234,"raw_usage":{"total_tokens":1456,"prompt_tokens":863,"completion_tokens":593,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":479,"completion_tokens_details":{"reasoning_tokens":499}},"tokens_in":479,"tokens_out":593,"duration_ms":5049,"temperature":1.0,"reasoning_tokens":499,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:06:29.550837+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-running the pipeline on a time-based or applicant-ID-disjoint split and recomputing ROC-AUC, approval rate, and default rate for all three models would settle the claim: if LightGBM no longer holds the highest ROC-AUC or the best default-versus-approval trade-off, the central conclusion fails.","supporting_citations":[{"cited_title":"Credit risk prediction using machine learning and deep learning: A study on credit card customers,","cited_arxiv_id":null,"evidence_quote":"Documents XGBoost reaching 99.4% accuracy on credit-card default data, setting the strong XGBoost baseline this paper compares against."},{"cited_title":"Credit risk assessment and financial decision support using explainable artificial intelligence,","cited_arxiv_id":null,"evidence_quote":"Shows Random Forest achieving high accuracy with SHAP and LIME on peer-to-peer lending data, grounding the RF-plus-XAI comparison."},{"cited_title":"Estimating value-at-risk in the eurusd currency cross from implied volatilities using machine learning methods and quantile regression,","cited_arxiv_id":null,"evidence_quote":"Reports LightGBM and CatBoost as top estimators for financial value-at-risk, supporting the expectation that LightGBM excels on financial data."},{"cited_title":"The value of open banking data for application credit scoring: Case study of a norwegian bank,","cited_arxiv_id":null,"evidence_quote":"Demonstrates LightGBM outperforming an existing logistic-regression application credit scorer, the direct precedent for choosing LightGBM."},{"cited_title":"Credit risk analysis using machine and deep learning mod- els,","cited_arxiv_id":null,"evidence_quote":"Establishes gradient boosting as the top-AUC model on a large financial dataset, the benchmark family that LightGBM extends."},{"cited_title":"Optimizing human-ai collaboration,","cited_arxiv_id":null,"evidence_quote":"Motivates the paper's interpretability requirement by linking explainability to human-AI trust in sensitive decisions."},{"cited_title":"Shap: Beeswarm plot,","cited_arxiv_id":null,"evidence_quote":"Supplies the SHAP beeswarm summary plot method used to produce the global feature-importance rankings."}],"review_version":1}