{"id":"4481b259-fcf8-4579-b27f-4c36ce3d25c3","arxiv_id":"2506.14306","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A three-parameter grid search finds resampling ratios that trade off disparate impact and MCC, improving fairness on doubly imbalanced fraud data.","lead":"This paper introduces a three-parameter resampling method that searches for the best balance of sensitive groups and class labels to train fairer classifiers on fraud data imbalanced in both ways. Standard debiasing tools fail on such data, and this method aims to improve fairness while keeping accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Grid-search parameters are selected on the same 5,000-row test partitions that produce Table 8; the only held-out larger test (Table 9) shows most DI ratios outside the acceptable band, so the claimed fairness gains do not transfer.","rationale":"The reader's weakest assumption is that parameters tuned on a 5,000-row sample do not transfer to the original-distribution test set; the paper's own Table 9 provides direct evidence of this failure. This is the single most load-bearing concern because the abstract's central contribution is exactly the fairness improvement, and the only truly held-out evaluation shows most models' DI ratios outside the acceptable range. The exploratory analysis of LFR on doubly imbalanced data is a useful descriptive contribution, and the sampling parametrization is clearly specified, but the proposed method's claimed benefit is not yet supported by the experimental protocol. The correct remedy is a nested validation-based selection with a disjoint test set, plus variance estimates over repeated sampling. This confirms the reader's CONDITIONAL verdict; no further verdict change is needed.","tokens_in":21874,"tokens_out":4435,"duration_ms":45652,"concrete_test":"Re-run the full two-level grid search with a nested split where a validation partition disjoint from the 10% larger test set is used to select (alpha, beta, gamma) and the threshold, and report DI ratio and MCC only on the larger original-distribution test set. If the validation-selected parameters reproduce Table 8's DI values on this disjoint test, the claim stands; if they resemble Table 9 (e.g., LR DI below 0.6), the Table 8 fairness gains are an artifact of tuning on the evaluation partition.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the proposed (alpha, beta, gamma) resampling search yields 'significant improvements in model fairness (DI Ratio)' on doubly imbalanced data. The empirical support for this claim is weakened by the evaluation protocol. In Section 6.3, the 5,000-instance sample is split into training, 'level 0 test', and 'level 1 test' partitions, and Algorithm 1 returns losses computed on the D' test partition. The two-level grid search selects the parameters and the classification threshold using these test partitions, so Table 8's near-ideal DI ratios are selected in-sample rather than independently validated. The only evaluation on a separately held-out larger collection, Table 9, shows that the selected parameters do not transfer: LR Level 0 DI drops from 1.001 to 0.231, RF from 0.995 to 0.535, SVM from 0.990 to 0.544, and LFR from 0.960 to 0.769. Only NB Level 0 remains inside the acceptable [0.8, 1.2] band. Since the independent test contradicts the headline fairness improvement for most models, the central claim is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses fairness in doubly imbalanced datasets, where both the label distribution and the sensitive-attribute group distribution are skewed. It proposes a sampling scheme parameterized by (alpha, beta, gamma) that controls the privilege-group balance, the label balance, and the favourable-rate ratio between groups, followed by a two-level grid search to minimize a combined loss of DI-ratio and MCC. Experiments are carried out on three fraud datasets with five classifiers (LR, NB, RF, SVM, and LFR). The manuscript reports near-ideal DI ratios and improved MCC on the tuning partitions, and also includes results on larger held-out test collections.","tokens_in":22154,"tokens_out":5392,"duration_ms":58676,"significance":"The problem is well motivated: most fairness mitigation methods fail under double imbalance, and a model-agnostic, interpretable sampling schedule would be practically useful. The exploratory analysis in Section 4 is a clear contribution, and the (alpha, beta, gamma) parameterization is transparent. The paper is less convincing as an empirical validation, because the headline numbers in Table 8 are selected on the same partitions used for evaluation, and the independent larger test in Table 9 shows substantial deterioration for most models. The authors do not provide code or data, which limits reproducibility. The central idea is defensible and the manuscript can be strengthened, but the current evidence is not sufficient to support the claim of fine-tuned, transferable optimal balancing.","major_comments":[{"comment":"The near-ideal DI ratios in Table 8 are fitted, not predictive. In Section 6.3, the two-level grid search selects (alpha, beta, gamma) and the classification threshold (the latter per Section 6.2) by minimizing the combined loss in Eq. (11) on the level-0 and level-1 test partitions, and Table 8 reports results on those same partitions. The only independent check, Table 9, shows that these configurations do not transfer to the larger test collection: for BAF, LR level-0 DI drops from 1.001 to 0.231, RF from 0.995 to 0.535, SVM from 0.990 to 0.544, and LFR from 0.960 to 0.769. The paper should use a separate validation partition for model selection and report final metrics on an untouched test set; without this, the optimality claim is not established.","section":"§6.2–6.3, Tables 7–8"},{"comment":"The statement that metric values 'remain similar except that DI Ratio and Combined Loss tend to be somewhat lower and higher respectively' is not supported by the table. NB level-1 degrades from DI 1.007, Recall 0.85, MCC 0.049 to DI 0.545, Recall 0.01, MCC 0.012; LR level-1 DI drops from 0.986 to 0.514. These are not minor or 'expected and acceptable' changes unless a tolerance is explicitly defined. The authors should quantify the acceptable deterioration, report repeated-run variability, and avoid claiming near-consistent performance on the larger test.","section":"§6.3, Table 9"},{"comment":"The transfer problem is not specific to the BAF dataset. On CCF level-1, the SVM DI ratio is 0.998 on the tuning partition but 2.681 on the larger test collection; on VIF level-1, the LFR DI ratio is 1.011 on the tuning partition but 0.572 on the larger test. These results are consistent with the grid search overfitting the tuning partition rather than finding a robust balance structure. The authors should either adopt a validation-based selection protocol in all three datasets or substantially soften the claim that the selected parameters are optimal for unseen data.","section":"§6.5, Tables 13 and 17"}],"minor_comments":[{"comment":"The word 'Parento' is misspelled and should be 'Pareto'.","section":"Figure captions 1–3"},{"comment":"The formula for MCC is typeset incorrectly: the denominator should be the square root of the product of the four terms, and the current rendering appears as a single fraction without the square root.","section":"Equation (2)"},{"comment":"The example and surrounding text use theta in place of gamma (e.g., 'parameters alpha, beta, theta = (0.5, 0.8, 0.4)' and the iteration over '(alpha, beta, theta)'), but only alpha, beta, and gamma are defined; this should be made consistent.","section":"§5.1, Equations (5)–(8)"},{"comment":"No code or data repository is provided, making the experiments difficult to reproduce; the authors should include an artifact link or explicit instructions for reproducing the grid search.","section":"General"},{"comment":"The paper would be stronger with a comparison to simple fixed-ratio resampling baselines (e.g., random undersampling or oversampling to a 50/50 label balance, with and without privilege balancing) to isolate the contribution of the grid-search component.","section":"§6.3"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper has a clean, interpretable resampling scheme for doubly imbalanced fairness data, but the main empirical claim is not established. The headline Table 8 results come from grid search that selects alpha, beta, gamma and the classification threshold on the same level-0/level-1 test partitions reported there, so those near-ideal DI ratios are fitted values. The only truly held-out BAF test (Table 9) shows DI ratios collapsing from about 1.0 to 0.231 (LR level 0), 0.535 (RF), 0.544 (SVM); only NB stays inside the 0.8–1.2 band. The paper calls this an accepted and expected deterioration, but it is the center of the claim not transferring. \n\nThe genuinely new piece is the three-parameter (alpha, beta, gamma) interpolation with two-level grid search and Pareto-front selection, applied to the doubly imbalanced setting. The sampling formulation is interpretable, with restrictions that prevent the privileged/unprivileged or favorable/unfavorable roles from flipping. The exploratory evidence that LFR breaks under double imbalance is a useful motivating observation. Credit is also due for including a larger held-out collection at all, which is more than many resampling papers do. But the interpretation of those results is too generous. \n\nSoft spots, in proportion: (1) selection on the evaluation partition is the load-bearing flaw, as above; (2) there are no baselines against SMOTE, random oversampling, reweighting, or standard fairness mitigation methods, so there is no evidence the grid search beats cheaper existing options; (3) no variance estimates for the main grid-search results, even though the exploratory section repeats LFR ten times; (4) the larger-test failures on VIF (LFR) and CCF (SVM) are mentioned without the weight they deserve. The sampling math is sound, the writing is clear, and the citations are adequate. \n\nWho this is for: practitioners wanting a model-agnostic knob to tune fairness versus accuracy in imbalanced tabular data. The idea deserves a serious referee because the fix is feasible: hold out parameter selection, add baselines, and report repeated runs. But as submitted, the central fairness claim does not hold, so the request should be for major revision, not acceptance.","headline":"The three-parameter resampling idea is clean and interpretable, but the headline fairness gains are fitted to the tuning partitions and largely vanish on the one truly held-out test, so the central claim is not yet established.","tokens_in":22691,"tokens_out":4132,"would_cite":false,"duration_ms":40849,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes a three-parameter resampling search that finds data balances making fraud classifiers fairer without sacrificing accuracy.","keywords":["algorithmic fairness","doubly imbalanced datasets","fraud detection","resampling","grid search","Pareto front","disparate impact ratio","Matthews correlation coefficient"],"falsifier":"Take the BAF dataset, hold out a test partition that is never touched by the grid search, and re-run the full two-level search on several independent 5,000-row training samples; if the selected $(\\alpha, \\beta, \\gamma)$ values vary widely across samples, or if the DI ratio on the untouched test set falls below 0.8 while the tuning-partition DI ratio stays near 1, then the reported fairness gain is an artifact of tuning on the evaluation sample.","tokens_in":1611,"feed_emoji":"⚖️","tokens_out":2206,"duration_ms":85050,"temperature":0.7,"pith_summary":"The paper addresses datasets that are imbalanced twice over: the fraud label is rare and one sensitive-attribute group is much smaller than the other. It argues that off-the-shelf debiasing methods fail on such doubly imbalanced data, and proposes a model-agnostic resampling solution that searches for the best balance between the two privilege groups and the two labels. The search is controlled by three interpretable parameters, and the outcome is a Pareto front showing the trade-off between fairness, measured by disparate impact ratio, and accuracy, measured by Matthews correlation coefficient. The authors demonstrate on three fraud datasets that the resulting balance brings the DI ratio close to 1 for several classifiers, including when combined with an existing debiasing method, without sacrificing and sometimes improving MCC.","feed_headline":"Three data-balance dials make fraud models fair and accurate","feed_subtitle":"Grid search over label and group balance pushes DI ratio near 1 while keeping or raising MCC.","key_machinery":"The load-bearing object is the three-parameter balance structure $(\\alpha, \\beta, \\gamma)$. Given the original four cell counts, privileged-favourable, privileged-unfavourable, unprivileged-favourable, and unprivileged-unfavourable, these parameters determine the four sampling ratios through a closed-form interpolation, with $\\alpha$ moving the privilege split toward 50/50, $\\beta$ moving the label split toward 50/50, and $\\gamma$ moving the favourable-label rates of the two groups toward equality. A two-level grid search, first with coarse steps of 0.1 and then with refinement around the top five points at steps of 0.01, evaluates the combined loss for every parameter triple and for each classifier's probability threshold. The Pareto front of the two loss components is reported so that a user can pick the operating point that best fits their fairness and accuracy requirements.","core_discovery":"On its own terms, the paper's discovery is that the optimal training distribution for a fair classifier on a doubly imbalanced dataset is not a fully balanced sample but a dataset-specific intermediate point that can be found by search. The proposed method defines the training sample by three parameters: $\\alpha$ controls the proportion of the unprivileged group, $\\beta$ controls the proportion of the unfavourable label, and $\\gamma$ controls the ratio of favourable-label rates between privileged and unprivileged groups. A two-level grid search evaluates the combined loss $c_1 \\cdot |1 - \\mathrm{MCC}| + c_2 \\cdot |1 - \\mathrm{DI}|$ for each parameter combination, and the resulting Pareto-optimal points let a user choose a fairness-accuracy trade-off. The experiments report that on the BAF, VIF, and CCF fraud datasets, the chosen balance yields DI ratios near 1 for logistic regression, random forest, SVM, naive Bayes, and Learning Fair Representations, with MCC values that improve on the unmodified baseline.","pith_inferences":["Because the optimal parameters differ across classifiers, the search likely compensates for each model's inductive bias; a testable extension is whether a single balance exists that serves an ensemble of models.","The same three-parameter search could be applied with other fairness metrics, such as equalized odds, by replacing the DI-ratio loss in the combined objective.","The paper evaluates fraud data only, but the method's promise extends to any doubly imbalanced binary classification task, such as credit approval or recidivism prediction, where both the sensitive attribute and the outcome are unevenly distributed."],"forward_implications":["A practitioner with a doubly imbalanced fraud dataset can improve group fairness by resampling the training data only, without changing the classifier or adding a separate debiasing algorithm.","The same search can be applied together with methods like Learning Fair Representations, so it acts as a pre-processing layer rather than a replacement for existing debiasing.","The Pareto front gives operators a principled way to choose a fairness-accuracy trade-off that satisfies a legal or ethical threshold such as a DI ratio of at least 0.8.","Because the three parameters are interpretable, the chosen balance can be audited and reported in plain terms, such as the unprivileged group share and the fraud label share in the training sample.","The method also covers singly imbalanced datasets as a boundary case where at least one parameter is fixed at 0, which is the common scenario in many data collections."],"supporting_citations":[{"why":"Supplies the BAF bank-fraud dataset suite used for the main experiments and the exploratory analysis.","marker":"[30]"},{"why":"Provides the Learning Fair Representations debiasing method used as the fairness baseline and as the method combined with the proposed sampling.","marker":"[39]"},{"why":"Defines the Disparate Impact ratio and the acceptable 0.8 to 1.2 range that the paper optimizes toward.","marker":"[1]"},{"why":"Establishes the Matthews Correlation Coefficient as the classification metric used in the combined loss and evaluations.","marker":"[37]"},{"why":"Justifies the choice of old versus young as the privileged and unprivileged groups in the BAF dataset.","marker":"[34]"}],"fun_headline_variants":["Three balance dials tune fairness and accuracy on skewed data","Grid search finds the fair spot in doubly imbalanced data","Searching data balance achieves fair models on doubly imbalanced sets","Fairness via data balance: a three-parameter search works","Dial in fairness and accuracy with three data balance knobs"],"cache_read_input_tokens":24832,"weakest_assumption_plain":"The load-bearing premise is that the balance parameters selected on a 5,000-row tuning sample (60% train) remain near-optimal for the model on a test set with the original distribution; the paper's own Table 9 shows the transfer is imperfect, with logistic regression's DI ratio dropping from 1.001 on the tuning partition to 0.231 on the larger test collection.","fun_headline_variants_meta":{"raw":{"variants":["Three balance dials tune fairness and accuracy on skewed data","Grid search finds the fair spot in doubly imbalanced data","Searching data balance achieves fair models on doubly imbalanced sets","Fairness via data balance: a three-parameter search works","Dial in fairness and accuracy with three data balance knobs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000173,"raw_usage":{"total_tokens":1237,"prompt_tokens":863,"completion_tokens":374,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":479,"completion_tokens_details":{"reasoning_tokens":291}},"tokens_in":479,"tokens_out":374,"duration_ms":4277,"temperature":1.0,"reasoning_tokens":291,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:17:47.229619+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the BAF dataset, hold out a test partition that is never touched by the grid search, and re-run the full two-level search on several independent 5,000-row training samples; if the selected $(\\alpha, \\beta, \\gamma)$ values vary widely across samples, or if the DI ratio on the untouched test set falls below 0.8 while the tuning-partition DI ratio stays near 1, then the reported fairness gain is an artifact of tuning on the evaluation sample.","supporting_citations":[{"cited_title":"Advances in Neural Information Processing Systems (2022)","cited_arxiv_id":null,"evidence_quote":"Supplies the BAF bank-fraud dataset suite used for the main experiments and the exploratory analysis."},{"cited_title":"In: Proceedings of the 30th International Conference on Machine Learning, pp","cited_arxiv_id":null,"evidence_quote":"Provides the Learning Fair Representations debiasing method used as the fairness baseline and as the method combined with the proposed sampling."},{"cited_title":"arXiv (2015)","cited_arxiv_id":null,"evidence_quote":"Defines the Disparate Impact ratio and the acceptable 0.8 to 1.2 range that the paper optimizes toward."},{"cited_title":"BMC genomics 21, 1–13 (2020)","cited_arxiv_id":null,"evidence_quote":"Establishes the Matthews Correlation Coefficient as the classification metric used in the combined loss and evaluations."}],"review_version":1}