REVIEW 3 major objections 4 minor 25 references
Impact of Sampling Techniques and Data Leakage on XGBoost Performance in Credit Card Fraud Detection
T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read This paper argues that pre-split sampling in credit card fraud detection inflates XGBoost's reported performance through data leakage, and that post-split sampling yields lower but trustworthy metrics without sacrificing the model's…
desk verdict The abstract promises a three-scenario comparison, but only the post-split arm is actually measured; the pre-split numbers are imported from other papers, so the central claim about inflated metrics rests on an uncontrolled cross-study comparison. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism carrying the argument is the ordering of resampling relative to the train-test split. When sampling precedes the split, synthetic or duplicated minority-class examples are generated from the whole dataset, so test-set information leaks into training and inflates evaluation metrics; when sampling is applied only to the training portion after the split, the test set remains a true holdout. The paper uses this contrast as its analytical lens, pulling pre-split results from other studies for comparison against its own post-split XGBoost runs.
What would settle it
Run the same XGBoost configuration on the same Kaggle credit-card data twice, once applying SMOTE before the train-test split and once applying it only to the training set, then compare metrics on an identical held-out test set. If the pre-split model's test scores are not substantially higher than the post-split model's, or if the post-split model's scores match the pre-split ones, the leakage-inflation claim would be falsified.
Extended reading notes
Core claim
The central claim is that applying sampling techniques before the train-test split creates data leakage and produces artificially inflated performance metrics for XGBoost in credit card fraud detection. The paper contrasts external pre-split results, such as an XGBoost model with SMOTE scoring 99.969% accuracy, 100% recall, and 99.969% AUC, with its own experiments where sampling is applied only to the training set after the split; those honest models achieve lower but credible scores, with the best configuration reaching 95.00% F1-score using CGAN augmentation and cost-sensitive learning. The conclusion is that XGBoost retains high detection performance without the inflation, so pre-split sampling is both unnecessary and a threat to evaluation validity.
Load-bearing premise
The comparison only works if the external pre-split results and the author's own post-split runs are directly comparable, even though they use different models, hyperparameters, feature engineering, and evaluation protocols.
Editorial extensions
If this is right
- Published fraud-detection benchmarks that balanced data before splitting likely overstate how well the models will perform in real deployment.
- Practitioners should treat reported metrics as trustworthy only when sampling was performed after the split, ideally inside each cross-validation fold.
- XGBoost maintains strong performance under honest evaluation, with F1-scores around 94 to 95 percent and accuracy above 99.9 percent on the tested dataset.
- Evaluation protocols for any imbalanced classification problem should mandate resampling after the split to avoid leakage-driven inflation.
- Hybrid approaches that combine synthetic data generation with cost-sensitive learning appear to be a viable path that improves sensitivity without compromising validity.
Reading between the lines
- A controlled head-to-head experiment with identical data and hyperparameters, changing only the sampling order, would convert the paper's cross-study comparison into a direct test of the leakage effect; the paper itself does not run its own pre-split condition.
- The size of the gap between pre-split and post-split scores is likely an upper bound on pure leakage effects, because the compared studies also differ in model configuration, preprocessing, and evaluation protocols.
- The same split-order caution generalizes to other heavily imbalanced domains such as medical diagnostics or network intrusion detection, where pre-split resampling carries the same inflation risk.
- One could isolate the leakage mechanism by training a model on a dataset where synthetic minority samples are generated from the full data but evaluated on a clean held-out set; if that reproduces the inflated scores, leakage is confirmed as the cause.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper claims to compare XGBoost performance for credit card fraud detection under three scenarios: no imbalance handling, sampling applied after the train-test split, and sampling applied before the train-test split. It reports its own post-split experiments on the Kaggle credit card dataset, imports pre-split XGBoost results from Qasim et al. [2], and imports additional XGBoost and Random Forest results from Mohbey et al. [20] and Chogugudza [10]. The abstract concludes that pre-split sampling artificially inflates metrics and that post-split sampling yields lower but evaluation-integrity-preserving results, with XGBoost remaining superior. The paper includes code appendices for SMOTE, CTGAN, and preprocessing.
Significance. If the central claim were established, the paper would provide a useful, concrete caution about a common methodological pitfall in fraud-detection benchmarks. The paper does articulate a relevant problem and provides code-level detail for several pipelines, which is a useful start. However, the load-bearing comparison is not actually run: the pre-split condition is imported from another study, the comparability of external results is unverified, and the preprocessing code in Appendix C itself contains a potential leakage path. As a result, the significance of the empirical findings is currently low: the paper does not demonstrate the inflation effect it claims to characterize, and it does not provide a controlled comparison supporting its "XGBoost remains superior" conclusion.
major comments (3)
- [Section 9.1, Table 1] The pre-split sampling condition, which is central to the abstract's three-scenario comparison, is not implemented in this paper. The text states "Table 1[2]" and the table's near-perfect values are taken from Qasim et al. [2], not from the authors' experiments. Section 9.2 and Table 3 contain only post-split experiments. Therefore the reported gap between Table 1's 99.969% F1-score and Table 3's 92.56% F1-score cannot be attributed to sampling timing: it may reflect differences in hyperparameters, data splits, evaluation thresholds, dataset preprocessing, or the specific dataset version. To support the abstract's causal claim, the authors must run their own pre-split sampling condition under the same protocol, or explicitly reframe the conclusion as a cross-study observation.
- [Appendix C] The preprocessing code shown in Appendix C calls StandardScaler().fit_transform on the Amount and Time columns of the full data object (lines 1-6) before any train-test split is shown. If this code reflects the actual experimental protocol, the scaler statistics incorporate test-set information, which is itself a form of data leakage independent of sampling. This undermines the claim in Section 9.2 that the post-split results "preserve the integrity of the evaluation process." The authors must either demonstrate that the split occurs before any fit_transform is applied, or revise the code and rerun the experiments.
- [Section 9.2, Tables 5 and 6] The comparison with external results is uncontrolled. Table 5 reproduces an XGBoost result from Mohbey et al. [20] and Table 6 reproduces a Random Forest result from Chogugudza [10], with no verification that the models, hyperparameters, data splits, evaluation metrics, or dataset preprocessing are commensurable with the authors' own runs. The concluding sentence "From these results it is now evident that the xgboost model is still superior even without the risk of data leakage" is not supported by the presented evidence. Additionally, Table 3 lacks essential experimental detail: the train-test split ratio, the number of independent runs, and any measure of variance are not reported, and Table 4 lists only a subset of hyperparameters, omitting the sampling_strategy values for SMOTE and RUS used in the main experiments (those appear only in one pipeline in Appendix A). Without this information, the metric differences in Table 3 could be within run-to-run noise.
minor comments (4)
- [References] There are duplicate references: [7] and [8] are the same SMOTE paper, [14] and [15] are identical Hashemi et al. entries, and [19] and [20] are the same Mohbey et al. entry. These should be consolidated.
- [Section 9.2] The text reads "quite descent results" and should read "quite decent results." Also, the name "Chougugudza" in the text does not match the spelling "Chogugudza" in reference [10].
- [Section 9.1, Table 1] The citation "Table 1[2]" interleaves a citation with a table reference; the original source should be credited in the caption and its experimental settings (dataset version, hyperparameters, split procedure) should be described in the text so that readers can judge comparability.
- [Appendix A, Listing 1] The comment "also consider u -> o" is unclear, and the pipeline order is defined as over-sampling then under-sampling. The comment suggests the reverse order should be considered; please clarify the intended design and justify the chosen order.
Circularity Check
No significant circularity: the paper's central comparison is empirical and does not reduce to its own inputs, though it rests on an uncontrolled cross-study comparison.
full rationale
The paper's argument is entirely empirical: it compares XGBoost performance under pre-split sampling, post-split sampling, and no sampling. The load-bearing pre-split results are not derived or fitted by the author; they are taken from another study, as the text states: "As shown in Table 1[2], the XGBoost model achieves near-perfect scores in all metrics." The post-split results come from the author's own experiments in Table 3. There is no equation that defines one result in terms of another, no parameter fitted to a target and then renamed as a prediction, and no self-citation chain supporting a uniqueness claim. The weakness that a reader might identify is that the pre-split and post-split numbers come from different papers, models, and protocols, so attributing the gap to sampling timing is not controlled. That is a validity or correctness concern, not circular reasoning. Even Appendix C's scaling of Amount and Time before an explicit split could constitute leakage, but it does not create a definitional loop between the paper's claims and its evidence. Accordingly, no circular step is present and the score is 0.
Assumptions & free parameters
free parameters (6)
- learning_rate =
0.4
- n_estimators =
1000
- SMOTE sampling_strategy =
0.8
- RandomUnderSampler sampling_strategy =
0.9
- scale_pos_weight =
577.27/10
- CTGAN epochs =
300
assumptions (4)
- domain assumption Pre-split sampling causes test-set information to enter training and inflates performance metrics.
- ad hoc to paper External results in Tables 1, 5, and 6 are comparable to the author's own experiments.
- domain assumption The Kaggle credit card dataset's labels and PCA features are reliable ground truth.
- ad hoc to paper The fixed XGBoost hyperparameters in Table 4 represent a fair baseline for all compared techniques.
Cite this review
Pith. "Pith review of Impact of Sampling Techniques and Data Leakage on XGBoost Performance in Credit Card Fraud Detection." pith.science (2026). https://pith.science/paper/GDLMFF4R
@misc{pith2026241207437,
author = {Pith},
title = {Pith review of: Impact of Sampling Techniques and Data Leakage on XGBoost Performance in Credit Card Fraud Detection},
year = {2026},
howpublished = {\url{https://pith.science/paper/GDLMFF4R}},
note = {Machine review of arXiv:2412.07437}
}
read the original abstract
Credit card fraud detection remains a critical challenge in financial security, with machine learning models like XGBoost(eXtreme gradient boosting) emerging as powerful tools for identifying fraudulent transactions. However, the inherent class imbalance in credit card transaction datasets poses significant challenges for model performance. Although sampling techniques are commonly used to address this imbalance, their implementation sometimes precedes the train-test split, potentially introducing data leakage. This study presents a comparative analysis of XGBoost's performance in credit card fraud detection under three scenarios: Firstly without any imbalance handling techniques, secondly with sampling techniques applied only to the training set after the train-test split, and third with sampling techniques applied before the train-test split. We utilized a dataset from Kaggle of 284,807 credit card transactions, containing 0.172\% fraudulent cases, to evaluate these approaches. Our findings show that although sampling strategies enhance model performance, the reliability of results is greatly impacted by when they are applied. Due to a data leakage issue that frequently occurs in machine learning models during the sampling phase, XGBoost models trained on data where sampling was applied prior to the train-test split may have displayed artificially inflated performance metrics. Surprisingly, models trained with sampling techniques applied solely to the training set demonstrated significantly lower results than those with pre-split sampling, all the while preserving the integrity of the evaluation process.
Figures
Reference graph
Works this paper leans on
-
[2]
Credit Card Fraud Detection Using XGBoost Algorithm
Ahmed Qasim Abdulghani, Osman Nuri UCAN, and Khattab M. Ali Alheeti. “Credit Card Fraud Detection Using XGBoost Algorithm”. In: 2021 14th Interna- tional Conference on Developments in eSystems Engineering (DeSE). 2021, pp. 487–
work page 2021
-
[20]
Credit-Card-Fraud-Prediction- Using-XGBoost -An-Ensemble-Learning-Approach
K. Mohbey, Mohammad Khan, and Ajay Indian. “Credit-Card-Fraud-Prediction- Using-XGBoost -An-Ensemble-Learning-Approach”. In: International Journal of Information Retrieval Research 12 (July 2022). doi: 10.4018/IJIRR.299940
-
[10]
Mcdonald Chogugudza. “The classification performance of ensemble decision tree classifiers: a case study of detecting fraud in credit card transactions”. Identifier: vital:69317. Master’s thesis. University of Fort Hare, Nov. 2022. url: https : / / vital.seals.ac.za/vital/access/manager/Repository/vital:69317?site_ name=GlobalView
work page 2022
-
[1]
Fraud detection sys- tem: A survey
Aisha Abdallah, Mohd Aizaini Maarof, and Anazida Zainal. “Fraud detection sys- tem: A survey”. In: Journal of Network and Computer Applications 68 (2016), pp. 90–113. issn: 1084-8045. doi: https : / / doi . org / 10 . 1016 / j . jnca . 2016 . 04 . 007. url: https : / / www . sciencedirect . com / science / article / pii / S1084804516300571
work page 2016
-
[3]
Survey of Credit Card Anomaly and Fraud Detection Using Sampling Techniques
M. Alamri and M. Ykhlef. “Survey of Credit Card Anomaly and Fraud Detection Using Sampling Techniques”. In: Electronics 11 (2022), p. 4003
work page 2022
-
[4]
Data mining for credit card fraud: A comparative study
S. Bhattacharyya et al. “Data mining for credit card fraud: A comparative study”. In: Decision Support Systems 50.3 (2011), pp. 602–613. doi: 10.1016/j.dss.2010. 08.008. 14
-
[5]
Leo Breiman. “Random forests”. In: Machine learning 45.1 (2001), pp. 5–32. doi: 10.1023/A:1010933404324
-
[6]
Lost, stolen or skimmed: Overcoming credit card fraud in South Africa
Trevor Budhram. “Lost, stolen or skimmed: Overcoming credit card fraud in South Africa”. In: South African Crime Quarterly 40 (2016), pp. 31–37. doi: 10.17159/ 2413-3108/2012/V0I40A843
work page 2016
Show all 25 references
-
[7]
SMOTE: Synthetic Minority Over-sampling Technique
Nitesh V Chawla et al. “SMOTE: Synthetic Minority Over-sampling Technique”. In: Journal of artificial intelligence research 16 (2002), pp. 321–357. doi: 10.1613/ jair.953
2002
-
[8]
SMOTE: Synthetic Minority Over-sampling Technique
Nitesh V. Chawla et al. “SMOTE: Synthetic Minority Over-sampling Technique”. In: Journal of Artificial Intelligence Research 16 (2002), pp. 321–357. doi: 10.1613/ jair.953
2002
-
[9]
Xgboost: A scalable tree boosting system
Tianqi Chen and Carlos Guestrin. “Xgboost: A scalable tree boosting system”. In: Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining . ACM. 2016, pp. 785–794. doi: 10.1145/2939672.2939785
2016
-
[11]
Generating multi-label discrete patient records using generative adversarial networks
E. Choi et al. “Generating multi-label discrete patient records using generative adversarial networks”. In: Proceedings of the Machine Learning for Healthcare Con- ference. 2017, pp. 286–305
2017
-
[12]
Effective data generation for imbalanced learning using conditional generative adversarial networks
Georgios Douzas and Fernando Bacao. “Effective data generation for imbalanced learning using conditional generative adversarial networks”. In: Expert Systems with Applications 91 (2018), pp. 464–471. issn: 0957-4174. doi: https://doi.org/10. 1016/j.eswa.2017.09.030 . url: http...
2018
-
[13]
A novel method for detecting credit card fraud problems
H. Du et al. “A novel method for detecting credit card fraud problems”. In: PLoS ONE 19.3 (2024), e0294537. doi: 10.1371/journal.pone.0294537 . url: https: //doi.org/10.1371/journal.pone.0294537
2024 doi
-
[15]
Fraud De- tection in Banking Data by Machine Learning Techniques
Seyedeh Khadijeh Hashemi, Seyedeh Leili Mirtaheri, and Sergio Greco. “Fraud De- tection in Banking Data by Machine Learning Techniques”. In: IEEE Access 11 (2023), pp. 3034–3043. doi: 10.1109/ACCESS.2022.3232287
2023
-
[16]
Fraud detection: Intention- ality and deception in cognition
Paul E. Johnson, Stefano Grazioli, and Karim Jamal. “Fraud detection: Intention- ality and deception in cognition”. In: Accounting, Organizations and Society 18.5 (1993), pp. 467–488. issn: 0361-3682. doi: https : / / doi . org / 10 . 1016 / 0361 - 3682(93)90042- 5. url: https...
1993
-
[17]
Keep it simple: random oversampling for imbalanced data
Firuz Kamalov, Ho-Hon Leung, and Aswani Kumar Cherukuri. “Keep it simple: random oversampling for imbalanced data”. In: 2023 Advances in Science and En- gineering Technology International Conferences (ASET) . 2023, pp. 1–4. doi: 10. 1109/ASET56582.2023.10180891. 15
2023
-
[18]
GAN-based imbalanced data intrusion detection system
J. Lee and K. Park. “GAN-based imbalanced data intrusion detection system”. In: Personal and Ubiquitous Computing 25 (2021), pp. 121–128. doi: 10.1007/s00779- 019-01332-y
2021 doi
-
[21]
A Comparison Study of Credit Card Fraud Detection: Supervised versus Unsupervised
Xuetong Niu, Li Wang, and Xulei Yang. A Comparison Study of Credit Card Fraud Detection: Supervised versus Unsupervised. 2019. arXiv: 1904.10604 [cs.LG]. url: https://arxiv.org/abs/1904.10604
2019 arXiv
-
[22]
Credit Card Fraud Detection
Andrea Dal Pozzolo et al. Credit Card Fraud Detection. https://www.kaggle.com/ datasets/mlg- ulb/creditcardfraud. Accessed: 2024-11-11. 2016. url: https: //www.kaggle.com/datasets/mlg-ulb/creditcardfraud
2024
-
[23]
Data leakage inflates prediction performance in connectome- based machine learning models
Matthew Rosenblatt et al. “Data leakage inflates prediction performance in connectome- based machine learning models”. In: Nature Communications 15.1 (2024), p. 1829. doi: 10.1038/s41467-024-46150-w
2024 doi
-
[24]
Hazards of data leakage in machine learning: a study on classification of breast cancer using deep neural networks
Ravi K Samala et al. “Hazards of data leakage in machine learning: a study on classification of breast cancer using deep neural networks”. In: Medical Imaging 2020: Computer-Aided Diagnosis . Vol. 11314. International Society for Optics and Photonics. 2020, p. 1131416. doi: 10...
2020 doi
-
[25]
Semi-supervised anomaly detection for EEG waveforms using deep belief nets
D. Wulsin et al. “Semi-supervised anomaly detection for EEG waveforms using deep belief nets”. In: Proceedings of the 2010 Ninth international conference on machine learning and applications . 2010, pp. 436–441
2010
-
[26]
Impact of random oversampling and random undersampling on the performance of prediction models developed using observational health data
C. Yang, E. A. Fridgeirsson, J. A. Kors, et al. “Impact of random oversampling and random undersampling on the performance of prediction models developed using observational health data”. In: Journal of Big Data 11 (2024), p. 7. doi: 10.1186/s40537-023-00857-7 . 16 Appendix A:...
2024 doi
-
[492]
doi: 10.1109/DeSE54285.2021.9719580
2021
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.