{"id":"74d5918e-1a11-44e6-b6d6-f0041742ae51","arxiv_id":"1908.01672","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"The authors built Imbalance-XGBoost, a Python package adding weighted and focal losses to XGBoost, with derived custom-objective derivatives and a single-dataset empirical evaluation.","lead":"This paper presents an open-source Python package that lets XGBoost use weighted and focal loss functions for imbalanced binary classification. It derives the gradient and Hessian formulas for both losses and reports F1 gains over one baseline on a Parkinson's disease dataset.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Weighted cross-entropy gradients in Eqs. (5)-(6) vanish for y_i=0; the true gradient is ŷ_i, so the implemented objective does not minimize Eq. (4) and reported F1 gains may reflect ignoring negatives.","rationale":"The reader's weakest assumption identifies the same load-bearing flaw: Eqs. (5)-(6) give zero gradient and Hessian for every negative example, whereas the true gradient of Eq. (4) for y_i=0 is ŷ_i and the true Hessian is ŷ_i(1-ŷ_i). This is not a stylistic or bibliographic issue; it changes what the training procedure actually minimizes. The XGBoost custom-objective framework uses exactly the supplied first- and second-order terms, so zero values on the negative class remove all learning signal from that class. Consequently, the paper's first algorithmic contribution, the weighted-loss derivative derivation, is incorrect as written, and the empirical F1 gains are plausibly explained in part by the model's ignoring negative samples rather than by cost-sensitive weighting of Eq. (4). The focal-loss derivation checks out, so the concern is specific to one of the two loss implementations. The empirical comparison is also weak on its own, but the algebraic error is the more decisive issue because it undermines the claimed mechanism. The available open-source code could resolve whether the code itself is correct, but the paper's equations remain wrong; therefore the reader's REJECT verdict stands without further adjustment.","tokens_in":12247,"tokens_out":6291,"duration_ms":60354,"concrete_test":"Recompute the derivative of Eq. (4) at y_i=0 symbolically or numerically at ŷ_i=0.5, then rerun the Table 2 weighted-XGBoost experiments with the corrected objective g_i=(1-y_i)ŷ_i+α y_i(ŷ_i-1), h_i=(1-y_i+α y_i)ŷ_i(1-ŷ_i) on the same PD split and grid. If the F1 scores and learned trees coincide with the paper's, Eqs. (5)-(6) are at most a typo with no empirical consequence; if they differ, the weighted-loss implementation does not minimize Eq. (4) and the contribution must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2's claimed weighted cross-entropy derivatives are wrong for the negative class. Differentiating Eq. (4), for y_i=0 the loss term is -log(1-ŷ_i), whose derivative with respect to z_i is ŷ_i, not -α y_i(y_i-ŷ_i)=0. The second derivative is ŷ_i(1-ŷ_i), not 0. Since XGBoost's custom-objective interface consumes only the supplied g_i and h_i, all negative examples are silently dropped from the training objective. The implemented loss is therefore not the weighted cross-entropy of Eq. (4); it is at best a positive-only loss. This matters directly for the abstract's 'algebraic derivation' contribution and for the empirical results: on the PD data (188 patients vs 64 healthy), a trivial all-positive classifier has F1 = 2*188/(2*188+64) ≈ 0.854, close to several 0.85 entries in Tables 2-3. The focal-loss derivatives (Eqs. 8-11) appear correct, but the weighted-loss half of the package's central claim rests on incorrect algebra.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Imbalance-XGBoost, an open-source Python package that implements weighted cross-entropy and focal losses as custom objectives for XGBoost in binary label-imbalanced classification. The authors describe the package design, provide first- and second-order derivative derivations for the two losses, and empirically evaluate the implementation on a Parkinson's disease classification dataset, reporting higher F1 scores than the baseline study and claiming multiple state-of-the-art performances.","tokens_in":12462,"tokens_out":9242,"duration_ms":93359,"significance":"The intended contribution is practically useful if it holds: an XGBoost-compatible implementation of weighted and focal losses would give practitioners two standard imbalance-robust objectives without leaving the XGBoost framework. The focal-loss derivative derivation appears algebraically self-contained and correct, and the package's scikit-learn integration plus public release on GitHub and PyPI are concrete engineering contributions. However, the weighted-loss part of the derivation is wrong for the negative class, and the evaluation protocol leaks information from the test data into model selection. As a result, the central claims about the weighted-loss implementation and the reported state-of-the-art performances are not currently supported.","major_comments":[{"comment":"The stated gradient and Hessian of the weighted cross-entropy loss are incorrect for y_i = 0. For y_i = 0, Eq. (4) reduces to L = -log(1 - ŷ_i), whose derivative with respect to the raw prediction z_i is ŷ_i, and whose second derivative is ŷ_i(1 - ŷ_i). Eq. (5) gives 0 and Eq. (6) gives 0. Because XGBoost's custom-objective interface consumes only the supplied g_i and h_i, all negative-class instances contribute nothing to the tree-fitting objective. The implemented 'weighted cross-entropy' therefore does not minimize the loss in Eq. (4); it is at best a positive-only loss. This is a load-bearing error for the paper's algebraic-derivation contribution and for the package's claim to implement weighted cross-entropy.","section":"§3.2, Eqs. (5)-(6)"},{"comment":"The experimental protocol leaks information from the evaluation data into model selection. In Listing 2, GridsearchCV is applied to the full dataset to select α and γ, and the same full dataset is then used for the leave-one-object-out evaluation in §5.1-5.2. Instantiating a new booster with the selected parameters does not remove this leakage because the parameter choice itself was informed by the full dataset. A nested cross-validation or a separate validation set is required before the reported F1 values can be interpreted as honest estimates, and the 'state-of-the-art performances' claim is not supported without such a protocol.","section":"Listing 2 and §5.1-5.2"},{"comment":"Several reported F1 values (for example 0.85 in Table 2 and 0.88-0.89 in Table 3) are close to the F1 of a trivial all-positive classifier on this dataset: with 188 patients and 64 healthy controls, per-record all-positive classification gives F1 = 2·188/(2·188 + 64) ≈ 0.854. The paper provides no vanilla-XGBoost baseline or class-weighted baseline, and no statistical comparison. Given that Eqs. (5)-(6) drop all negative examples, the empirical section cannot distinguish a genuine improvement from the effect of ignoring the negative class. The abstract's 'multiple state-of-the-art performances' claim is therefore not established.","section":"§5.2, Tables 2-3"}],"minor_comments":[{"comment":"The notation ∂L²_w/∂²z_i is nonstandard; it should be written as ∂²L_w/∂z_i².","section":"§3.2, Eq. (6)"},{"comment":"Eq. (4) has an unmatched closing parenthesis and should be rewritten for clarity.","section":"§3.2, Eq. (4)"},{"comment":"Variable names are inconsistent and misspelled (for example 'xgboster_focal' versus 'xgboost_focal_opt', and 'cross-validatoin'); these should be corrected.","section":"Listing 2 and §2.2"},{"comment":"There are typos such as 'Parkison's disease' and 'confusion metric' where 'confusion matrix' is intended.","section":"§5.1"},{"comment":"The explanation after Eq. (5) reads 'a αyi term is added to control the present of the parameter'; this should be clarified, since 'presence' is presumably intended.","section":"§3.2"},{"comment":"The conclusion states that the package 'successfully combines' both losses, but this claim is not supported for the weighted loss until Eqs. (5)-(6) are corrected and the experiments are re-run.","section":"§6"}],"recommendation":"reject","confidential_remarks":"The two main problems compound each other: the incorrect weighted-loss derivatives may explain the reported F1 gains, and the evaluation protocol cannot rule out leakage from the parameter search. A corrected implementation with a nested cross-validation protocol and re-run experiments would constitute a substantially new submission rather than a repair of the present claims. The focal-loss part might be salvageable in a shorter, narrowly scoped paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: the weighted cross-entropy half of this paper is mathematically wrong. Differentiating Eq. 4 gives g_i = -α y_i(1-ŷ_i) + (1-y_i)ŷ_i and h_i = (α y_i + 1 - y_i)ŷ_i(1-ŷ_i). The paper's Eqs. 5-6 give zero for both when y_i=0. Since XGBoost's custom-objective interface consumes only the supplied g and h, all negative-class examples vanish from the training objective. So the implemented 'weighted' loss is not minimizing Eq. 4; it is at best a positive-only loss. That is load-bearing, not cosmetic.\n\nThe focal-loss half is in better shape. The first and second derivatives in Eqs. 8-11 reduce correctly, and setting γ=0 recovers ordinary cross-entropy, which is a good check. The package itself is real, installable, and integrates with scikit-learn; making focal loss available as an XGBoost custom objective is a small but legitimate software artifact. Credit is due: the code is public, the paper is honest about prior work ([15] already had weighted XGBoost), and the citation pattern is not padded.\n\nWhere it falls apart: the experiments. One dataset, one baseline from [39], no error bars, no significance tests, and hyperparameters tuned by grid search on the full data before the same data is used for LOO evaluation. Worse, an all-positive classifier on this 188:64 (patient:healthy) split has F1 = 2·188/(2·188+64) ≈ 0.854, which is essentially the 0.85 reported for many feature sets in Tables 2-3. The 'state-of-the-art' claim does not survive contact with a trivial baseline. The paper also never compares against XGBoost's built-in scale_pos_weight, which is the obvious existing alternative and should have been the first benchmark.\n\nWho is this for? A practitioner who specifically wants focal loss in XGBoost might get value from the focal objective; the weighted loss should not be used until corrected. The paper deserves serious refereeing because the package is in the wild and the correctness of its central objective should be on the record. I would send it out with the expectation of major revision: fix the weighted derivatives or remove that contribution, redo the evaluation with proper nested CV and a trivial-classifier baseline, and add a scale_pos_weight comparison. As it stands, it should not be accepted.","headline":"The package is real and the focal-loss algebra is right, but the weighted-loss gradients are wrong and the empirical F1 gains sit close to a trivial all-positive baseline.","tokens_in":13035,"tokens_out":5898,"would_cite":false,"duration_ms":58076,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Imbalance-XGBoost lets XGBoost learn binary classification with skewed labels by plugging in weighted cross-entropy and focal losses, and reports F1 up to 0.89 on a Parkinson's dataset.","keywords":["imbalanced classification","XGBoost","Python package","weighted cross-entropy","focal loss","gradient boosting","binary classification","Parkinson's disease"],"falsifier":"Train the package's weighted objective on a small synthetic binary dataset with both classes present, then numerically compute the derivative of the loss in equation (4) at the fitted model; if that derivative is not near zero even though the boosting run has converged, the supplied gradient and Hessian do not match the loss the paper claims to minimize.","tokens_in":11999,"feed_emoji":"⚖️","tokens_out":10655,"duration_ms":103260,"temperature":0.7,"pith_summary":"The paper claims that XGBoost can be turned into a general-purpose learner for binary label-imbalanced classification by swapping its default objective for either a weighted cross-entropy loss or a focal loss. Its contribution is a small Python package, Imbalance-XGBoost, that implements both losses through XGBoost's custom-objective mechanism, together with an algebraic derivation of the first- and second-order derivatives the training loop requires. On a Parkinson's disease dataset with a patient-to-healthy ratio of roughly 3:1, the package reports F1 scores of 0.85–0.89 across feature groups, exceeding the comparison figures from the original dataset study, while accuracy falls slightly in each case. The result matters because XGBoost is one of the most widely used tools for skewed, large-scale classification tasks, and the package plugs the loss functions directly into that workflow.","feed_headline":"Imbalance-XGBoost adds weighted and focal losses to XGBoost","feed_subtitle":"Reports F1 up to 0.89 on a Parkinson's dataset, above the 0.84 prior best, for a small accuracy trade-off.","key_machinery":"The machinery is XGBoost's second-order approximation of the boosting objective, in which each split is scored from per-example first derivatives $g_i$ and second derivatives $h_i$ of the loss with respect to the raw prediction. Since XGBoost does not differentiate losses itself, Imbalance-XGBoost supplies these two numbers for each loss through custom objective classes: for weighted cross-entropy the paper derives merged-form expressions using $\\hat{y}_i=\\sigma(z_i)$ and the sigmoid identity $\\partial \\hat{y}_i/\\partial z_i=\\hat{y}_i(1-\\hat{y}_i)$; for focal loss it derives a lengthier expression (Eqs. 10 and 11) with shorthand variables $\\eta_1,\\dots,\\eta_5$ to keep the implementation vectorizable. These derivatives are what let the two losses run inside XGBoost without changing the boosting engine.","core_discovery":"The central discovery the paper argues for is that two imbalanced-aware loss functions, weighted cross-entropy and focal loss, can be implemented inside XGBoost purely through the software's custom objective framework by supplying hand-derived gradient and Hessian expressions. Weighted cross-entropy multiplies the loss on the positive class by a parameter α, while focal loss adds a $(1-\\hat{y})^{\\gamma}$ down-weighting factor so the classifier concentrates on hard examples. On the Parkinson's disease data, both variants raise F1 over the best previously reported results on the same feature sets while lowering accuracy, and the focal variant is the stronger of the two, reaching F1 0.89 on the top-50 features selected by a minimum-redundancy maximum-relevance criterion, versus 0.84 for the best model in the original study. The paper also claims that this is the first integrated implementation of the two losses in XGBoost, making the derivative derivation a substantive part of the contribution.","pith_inferences":["The merged-form derivative expressions are written without reference to a specific number of classes, so they can be carried over to one-vs-all multi-class ensembles or to other boosting libraries that accept custom objectives; that porting is mechanical once the package's formulas are transcribed.","If the weighted-loss gradient is corrected for negative examples, the accuracy-versus-F1 trade-off the paper reports for weighted-XGBoost may shift; a synthetic benchmark with a known optimal decision boundary would separate that correction from the focal-loss behavior.","The consistent pattern of lower accuracy with higher F1 across every feature group points to majority-class overprediction as the dominant failure mode of the prior classifiers, suggesting that F1 or a related metric should be the default reporting choice on skewed medical data."],"forward_implications":["A user can select the imbalanced objective at construction time with a keyword such as `special_objective='weighted'` or `'focal'`, then tune the corresponding parameter through grid search and cross-validation in the usual estimator workflow.","Models fit with the custom losses can be saved as plain XGBoost boosters, so a model trained with the package can be deployed later on machines that do not have the package installed.","On the Parkinson's dataset, the paper reports focal-XGBoost reaching F1 0.89 on the top-50 features, exceeding the 0.84 best previous comparison, with accuracy 0.83 instead of 0.86.","Because the losses are handled at the objective level, they can be combined with existing data-level resampling steps in an estimator pipeline, giving users both algorithm-level and data-level imbalance remedies.","The same hand-derived derivatives can be reused outside the package in any XGBoost-style workflow that accepts custom objectives, including one-vs-all multi-class setups."],"supporting_citations":[{"why":"Defines XGBoost's second-order approximation and custom objective mechanism, which the package plugs into through its supplied gradients and Hessians.","marker":"[1]"},{"why":"Introduces focal loss; the paper implements its binary form by deriving its first- and second-order derivatives.","marker":"[12]"},{"why":"Provides the Parkinson's disease dataset, the leave-one-object-out evaluation protocol, and the best prior results that serve as baselines.","marker":"[39]"},{"why":"Supplies the cost-sensitive classification context that motivates weighted cross-entropy as an algorithm-level remedy.","marker":"[10]"},{"why":"Describes an earlier use of weighted XGBoost for radar classification, which this paper extends with a packaged implementation and explicit derivative derivations.","marker":"[15]"},{"why":"Provides the estimator base classes and model-selection utilities that the package builds on for parameter tuning and evaluation.","marker":"[17]"}],"fun_headline_variants":["First XGBoost package for weighted and focal losses","Imbalance-XGBoost: tackle label imbalance inside XGBoost","New package adds focal loss to XGBoost, beats prior F1","Imbalance-XGBoost: custom objectives lift F1 on Parkinson's","Weighted and focal losses now native to XGBoost"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that equations (5) and (6) are the true gradient and Hessian of the weighted cross-entropy loss for every training example, including the negative class; when this algebraic identity fails, the weighted model is not minimizing the loss the paper says it minimizes.","fun_headline_variants_meta":{"raw":{"variants":["First XGBoost package for weighted and focal losses","Imbalance-XGBoost: tackle label imbalance inside XGBoost","New package adds focal loss to XGBoost, beats prior F1","Imbalance-XGBoost: custom objectives lift F1 on Parkinson's","Weighted and focal losses now native to XGBoost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000225,"raw_usage":{"total_tokens":1481,"prompt_tokens":975,"completion_tokens":506,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":412}},"tokens_in":591,"tokens_out":506,"duration_ms":4240,"temperature":1.0,"reasoning_tokens":412,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:06:43.646449+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the package's weighted objective on a small synthetic binary dataset with both classes present, then numerically compute the derivative of the loss in equation (4) at the fitted model; if that derivative is not near zero even though the boosting run has converged, the supplied gradient and Hessian do not match the loss the paper claims to minimize.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Parkinson's disease dataset, the leave-one-object-out evaluation protocol, and the best prior results that serve as baselines."},{"cited_title":"Classification of imbalanced data: A review","cited_arxiv_id":null,"evidence_quote":"Supplies the cost-sensitive classification context that motivates weighted cross-entropy as an algorithm-level remedy."},{"cited_title":"Radar emitter classification for large data set based on weighted-xgboost","cited_arxiv_id":null,"evidence_quote":"Describes an earlier use of weighted XGBoost for radar classification, which this paper extends with a packaged implementation and explicit derivative derivations."}],"review_version":1}