{"id":"3efea4a8-9f87-4b9a-88dc-c8c00a6fe93f","arxiv_id":"2501.10451","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"An XGBoost model with cost-sensitive learning reports Cohen's kappa of 0.81 against a bank committee's credit card limit decisions and is proposed to automate the process.","lead":"A machine learning model using XGBoost was trained to automate credit card limit increase decisions for a Venezuelan bank, matching the bank's human committee with a Cohen's kappa of 0.81. The work illustrates how cost-sensitive ML can replace manual committee reviews, though small samples and evaluation gaps limit how strongly the results can be trusted.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Effectiveness claim rests on agreement with the committee used as training labels; no realized-outcome validation, so kappa=0.81 may only show imitation.","rationale":"The reader's weakest assumption—that committee decisions are a valid gold standard—is also the most load-bearing issue here. The paper's strongest evidence for the XGBoost model is the kappa value, but that only shows agreement with the process that generated the training labels. The age-denial anecdote strengthens this concern: the model's disagreement with a biased committee decision is counted as a false positive, revealing that the evaluation is anchored to the committee's judgment rather than to any external measure of good outcomes. The model may well be a faithful and less biased version of the committee, but that is not established. The paper has real independent value: it uses a cost-sensitive objective, excludes demographic attributes, and evaluates on a temporally separated October sample (assuming those 153 CLAD postdate the September 2019–September 2024 training data). The 10-fold CV accuracy and cost figures, however, are in-sample model-selection metrics and do not resolve the gold-standard problem. The requested realized-outcome check would settle whether the central 'effective solution' claim holds. Because this concern matches the reader's stated weakest assumption and the reader's CONDITIONAL verdict already conditions on addressing it, no verdict change is needed.","tokens_in":5590,"tokens_out":5606,"duration_ms":55501,"concrete_test":"For the 153 October CLAD (and, if feasible, a random sample of training-period cases), collect realized outcomes over the following 12 months: whether each approved increase defaulted, and the incremental interest/fee revenue from the higher limit. Compute the realized cost using the Section 3 cost definitions (C_FP = defaulted adjusted limit; C_FN = lost minimum payment plus admin cost) for (a) the committee's actual decisions, (b) the model's predictions, and (c) a trivial baseline such as 'approve all.' If the model's realized cost is not lower than the committee's, or if kappa with the committee does not track realized cost, the effectiveness claim fails because agreement with the committee is not a valid proxy for quality.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that the XGBoost model is an 'effective solution' for credit limit adjustments—is supported mainly by Cohen's kappa = 0.81 against committee decisions (Section 4). But those same committee decisions are the labels used to train the model (Sections 2–3), so the evaluation measures imitation fidelity, not decision quality. The paper defines disagreement with the committee as FN/FP, making the committee the ground truth; this is circular for a system motivated by the need to avoid 'biased decisions.' The paper even reports a committee denial based on age, and the model's disagreement with that denial is scored as an error. No realized outcomes (defaults, revenue, profit) are tracked for the 153 October CLAD, and the cost comparison (809,660.81 vs 834,988.96 BS) is computed from the training set using undisclosed cost parameters. The kappa formula also has typos (Eq. 3 should be (P0-Pe)/(1-Pe); Eq. 4 should use TP+TN), although the reported 0.81 is plausibly consistent with the corrected formula. The evidence is therefore compatible with the model being a high-fidelity clone of a possibly flawed human process.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a cost-sensitive XGBoost model to automate credit card limit adjustment decisions (CLAD) at Venezolano de Crédito. Using 10,000 historical committee decisions as training labels, the authors compare XGBoost against a neural network via grid search with 10-fold cross-validation, selecting XGBoost for its lower cost and greater interpretability. The model is then compared with 153 committee decisions from the first half of October, and Cohen's kappa is reported as 0.81, which the authors interpret as almost perfect agreement.","tokens_in":5857,"tokens_out":4069,"duration_ms":42166,"significance":"If the claims are supported, the paper would be a practical demonstration of machine learning applied to a real credit decision process in a resource-constrained setting, with attention to misclassification costs and interpretability. The strengths include the use of real-world data, the deliberate exclusion of demographic attributes, a cost-sensitive objective, and a temporal evaluation on a separate set of October decisions. However, the current evidence supports only high-fidelity imitation of the committee's historical behavior, not an improvement in decision quality; the paper's central claim of 'effectiveness' therefore needs substantial reframing or additional validation.","major_comments":[{"comment":"The kappa equations as printed are mathematically incorrect: Eq. (3) should be κ=(P0−Pe)/(1−Pe), Eq. (4) should be P0=(TP+TN)/N, and Eqs. (5)-(6) need the correct chance-agreement terms (the sum of the products of marginal probabilities divided by N^2). The current text would produce values outside the valid kappa range. Please correct the equations and verify that the reported 0.81 is computed from the corrected formula.","section":"Section 4, Eqs. (3)-(6)"},{"comment":"The model is trained on historical CLAD decisions made by the same committee that is later used as the gold standard for the kappa evaluation. Because the paper explicitly defines false positives and false negatives relative to the committee's decisions, κ=0.81 quantifies imitation fidelity, not decision quality. The age-based denial example is scored as a model error even though the manuscript elsewhere motivates the model as a way to avoid biased decisions. To support the claim that the model is effective, the authors must either reframe the conclusion as automating the committee's policy or validate against realized outcomes such as subsequent defaults and revenue.","section":"Section 4, kappa evaluation"},{"comment":"The cost matrix in Figure 1 depends on parameters α, m_r, and an administrative cost, but their numerical values are not provided anywhere in the manuscript. Without these values, the reported total costs (834,988.96 BS and 809,660.81 BS) and the claimed savings of 25,328.15 BS are not reproducible, and the cost comparison cannot be independently verified. Please disclose the parameter values or provide a sensitivity analysis.","section":"Section 3, cost-sensitive learning"},{"comment":"All performance claims—accuracy, cost, and kappa—are reported as point estimates with no measures of uncertainty. The October sample has only 153 instances, so the kappa estimate of 0.81 has a wide standard error. The accuracy and cost figures come from cross-validation folds, but no standard deviation across folds is given. Report confidence intervals or standard errors for the key metrics.","section":"Section 4, results"},{"comment":"The hyperparameters are selected using grid search with 10-fold cross-validation, and the same cross-validation results are then used to report final accuracy and cost. This selection-on-test-data procedure can lead to optimistically biased estimates. The October holdout is used only for the kappa computation; a separate independent holdout (or nested cross-validation) should be used to report accuracy and cost for the final model.","section":"Sections 3-4, model selection"}],"minor_comments":[{"comment":"The notation alternates between 'k' and 'kappa'; use 'κ' consistently throughout.","section":"Section 4, Eq. (3)"},{"comment":"The confusion matrices show only true positives and true negatives in the text; please display all four cells (TP, FP, FN, TN) so the reader can verify the arithmetic.","section":"Figures 2 and 4"},{"comment":"Calling the agreement counts 'true positives' and 'true negatives' is misleading when there is no objective ground truth; consider 'positive agreement' and 'negative agreement'.","section":"Section 4, kappa section"},{"comment":"The NN is described as 'state-of-the-art' based on one comparison to XGBoost; this overstates the evidence, as only two model families are compared.","section":"Section 3, NN description"},{"comment":"The phrase 'an agency's credit score' is vague; clarify which credit agency and whether the score is a rating or a numeric score.","section":"Section 2, data description"}],"recommendation":"major_revision","confidential_remarks":"The paper is more of an application case study than a methodological contribution. The circularity issue (validating against the same committee used for labels) is the core concern; if the authors reframe the claims as automation fidelity and add uncertainty quantification, the paper could be acceptable. Given the venue's standards, I would insist on at least one additional validation with realized outcomes or a clear statement that such outcomes are unavailable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Take a look at this if you want a concrete example of cost-sensitive learning in credit risk at a Venezuelan bank. The authors train XGBoost and a small NN on 10,000 real CLAD decisions, pick the XGBoost model for cost and interpretability, and report Cohen's kappa of 0.81 against the bank committee's later decisions. It's a modest application paper, not a methods advance, and it doesn't open any new scientific questions. That's fine—the contribution is a worked deployment case.\n\nWhat's good: the cost-sensitive setup is appropriate. They use an instance-dependent cost matrix from prior work, and they deliberately exclude demographic attributes to avoid bias. The age-denial anecdote is candid: a committee member admitted denying a younger client, and the model, which doesn't see age, flagged the client as eligible. The model comparison is cost-driven rather than accuracy-driven, and they choose the interpretable model because regulators require explanations. These are the right instincts.\n\nThe soft spots are real but mostly fixable. First, the kappa equations are printed wrong. Eq. 3 should be (P0-Pe)/(1-Pe), and Eq. 4 should be (TP+TN)/N. As written they make no mathematical sense; the reported 0.81 is plausible, but the formulas must be corrected. Second, the evaluation uses the same committee as both the training-label source and the gold standard for agreement. The kappa measures imitation fidelity, not decision quality. The age case is exactly the problem: the model disagrees with a biased committee decision, and that disagreement is scored as an error. The paper doesn't acknowledge this circularity; it treats kappa as evidence of effectiveness. Third, there is no held-out test set. Model selection and performance reporting both come from the same 10-fold CV, with no error bars, so the cost figures (809,660.81 vs 834,988.96 BS) are not independently verifiable, especially with the cost parameters (alpha, minimum payment percentage, admin cost) undisclosed.\n\nNone of these are fatal for an extended abstract. The pipeline is standard and the authors are honest about data limitations. But the central claim that the model is an 'effective solution' is stronger than the evidence supports. For a serious referee, I'd ask for a corrected kappa, a proper holdout or at least error bars, and a discussion of the gold-standard problem, ideally with a small realized-outcome check (defaults, revenue) on the October cohort. If those are added, the paper is a solid applied contribution. I'd send it to review, conditional on revision.","headline":"A real-world cost-sensitive ML application with a genuine deployment story, but the evaluation leans heavily on agreement with the same committee that produced the labels.","tokens_in":6372,"tokens_out":5377,"would_cite":false,"duration_ms":46653,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper replaces a bank's manual credit card limit committee with a cost-sensitive XGBoost classifier and reports almost perfect agreement (kappa = 0.81) with the committee's decisions.","keywords":["credit card limit adjustment","cost-sensitive learning","XGBoost","neural networks","Cohen's kappa","banking automation","tabular data","grid search cross-validation"],"falsifier":"Compare the model's approvals to long-run repayment outcomes: if clients approved only by the model default at the same or higher rate than clients approved only by the committee, then the near-perfect kappa would show replication of the committee's judgment rather than correctness of the automated decision.","tokens_in":5389,"feed_emoji":"💳","tokens_out":11805,"duration_ms":105346,"temperature":0.7,"pith_summary":"The paper sets out to show that a binary classifier trained with cost-sensitive learning can take over a task banks in Venezuela have done by hand: deciding whether a credit card holder gets a fixed-percentage limit increase. It trains neural network and XGBoost models on 10,000 historical decisions from the risk committee of a Venezuelan bank and selects XGBoost because it is only 0.68 percentage points less accurate than the best neural network, produces fewer costly false positives, costs less overall, and can be explained to regulators. The paper reports 94.91 percent accuracy on test data and Cohen's kappa of 0.81 against 153 committee decisions from the first half of October, a level usually read as almost perfect agreement. The reason this matters is that manual committee decisions are becoming unfeasible as credit card volume grows, and they can inject bias, as in the disclosed case where a client was denied partly because of age.","feed_headline":"XGBoost model matches a bank's credit card limit committee","feed_subtitle":"Trained on 10,000 decisions, the tree model agrees with the committee (kappa 0.81) and costs less.","key_machinery":"The central object is the cost-sensitive loss function $C = \\sum_i y_i(1-c_i)C_{FN,i} + c_i(1-y_i)C_{FP,i}$, which replaces plain accuracy as the training objective. Each client has an individual cost matrix: a false positive costs $C_{FP,i}$, taken as the client's entire credit card limit after adjustment $C_i^a = C_i^b(1+\\alpha)$, while a false negative costs $C_{FN,i}$, the minimum payment of the first month plus an administrative cost. The mechanism is instance-dependent because limits vary among clients. Hyperparameters for neural networks and XGBoost are chosen by grid search with 10-fold cross-validation, and the final comparison with human judgment uses Cohen's kappa, a chance-corrected measure of agreement between two raters.","core_discovery":"The paper's central claim is that an XGBoost model with an instance-dependent cost matrix is a workable automated substitute for a bank's credit card limit adjustment committee. The task is binary classification: predict whether a client should receive an alpha-percentage increase. Misclassification costs are asymmetric and instance-dependent because a false positive costs the client's post-adjustment limit if the client defaults, while a false negative only costs the first minimum payment plus an administrative charge. On a test set, the chosen XGBoost model predicts 94.91 percent of cases correctly and returns a total cost of 809,660.81 bolívares, compared with 95.59 percent accuracy and 834,988.96 bolívares for the neural network; the paper prefers XGBoost for its lower cost and interpretability. Against 153 committee decisions from the first half of October, the model and committee agree with kappa = 0.81.","pith_inferences":["Editorial inference: the kappa = 0.81 result measures how faithfully the model reproduces this particular committee, not whether the decisions are good; a full assessment of decision quality would require comparing repayment outcomes for clients approved only by the model versus only by the committee.","Editorial inference: excluding age and gender removes the specific demographic bias disclosed in one disagreement, but if other features correlate with protected attributes the model could still inherit indirect bias; testing that would require inspecting feature attributions against protected groups.","Editorial inference: a natural next experiment is to run the same model against a second independent committee or against a later cohort with known defaults; if kappa drops materially, the almost perfect agreement is specific to this committee and time period."],"forward_implications":["The bank's risk committee can run the XGBoost model during its meetings and let the model prepare or make credit card limit adjustment decisions, reducing manual review as the number of card holders grows.","Because the model is a tree ensemble, each decision can be attributed through the Tree Explainer algorithm, which is the explainability property the paper says banking regulators require.","Larger future datasets will require alternative hyperparameter tuning methods because grid search with 10-fold cross-validation becomes less practical as the data grows.","Other Venezuelan banks facing a similar post-crisis increase in credit card demand could reuse the same cost-sensitive modeling approach, since the model is built from performance-based attributes rather than demographic data."],"supporting_citations":[{"why":"Supplies the XGBoost gradient-boosted tree algorithm that becomes the proposed model.","marker":"[4]"},{"why":"Supplies the instance-dependent cost matrix used to weight false positives and false negatives differently for each client.","marker":"[5]"},{"why":"Justifies the grid search with 10-fold cross-validation used to tune both model families.","marker":"[9]"},{"why":"Supplies Cohen's kappa as the inter-rater reliability measure for agreement between model and committee.","marker":"[3]"},{"why":"Provides the interpretation that kappa values from 0.8 to 0.9 count as almost perfect agreement.","marker":"[15]"},{"why":"Supplies the Tree Explainer algorithm the paper relies on for explaining XGBoost decisions to regulators.","marker":"[12]"},{"why":"Supports the choice of neural networks and XGBoost as strong classifiers for bank credit assessments.","marker":"[13]"},{"why":"Supports the expectation that neural networks and XGBoost are effective for predicting credit card repayment behavior.","marker":"[7]"}],"fun_headline_variants":["XGBoost matches bank's credit limit committee (kappa 0.81)","Cost-sensitive XGBoost automates credit limit increases","XGBoost beats neural net on cost, matches bank committee","Automating credit card limits with cost-sensitive XGBoost","XGBoost model does credit limit committee's job with kappa 0.81"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the historical committee decisions used as training labels are a valid gold standard: the model is built to imitate those decisions and is then judged by how well it imitates them, so any bias, inconsistency, or suboptimality in the committee is learned and reproduced.","fun_headline_variants_meta":{"raw":{"variants":["XGBoost matches bank's credit limit committee (kappa 0.81)","Cost-sensitive XGBoost automates credit limit increases","XGBoost beats neural net on cost, matches bank committee","Automating credit card limits with cost-sensitive XGBoost","XGBoost model does credit limit committee's job with kappa 0.81"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001121,"raw_usage":{"total_tokens":4628,"prompt_tokens":873,"completion_tokens":3755,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":489,"completion_tokens_details":{"reasoning_tokens":3659}},"tokens_in":489,"tokens_out":3755,"duration_ms":29003,"temperature":1.0,"reasoning_tokens":3659,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:29:25.915301+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the model's approvals to long-run repayment outcomes: if clients approved only by the model default at the same or higher rate than clients approved only by the committee, then the near-perfect kappa would show replication of the committee's judgment rather than correctness of the automated decision.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the XGBoost gradient-boosted tree algorithm that becomes the proposed model."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the instance-dependent cost matrix used to weight false positives and false negatives differently for each client."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Justifies the grid search with 10-fold cross-validation used to tune both model families."},{"cited_title":"Assessing agreement on classification tasks: the kappa statistic","cited_arxiv_id":"cmp-lg/9602004","evidence_quote":"Supplies Cohen's kappa as the inter-rater reliability measure for agreement between model and committee."},{"cited_title":"Uebersax","cited_arxiv_id":null,"evidence_quote":"Provides the interpretation that kappa values from 0.8 to 0.9 count as almost perfect agreement."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the choice of neural networks and XGBoost as strong classifiers for bank credit assessments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the expectation that neural networks and XGBoost are effective for predicting credit card repayment behavior."}],"review_version":1}