Pith. sign in

REVIEW 4 major objections 5 minor 78 references

Investigating the Impact of Balancing, Filtering, and Complexity on Predictive Multiplicity: A Data-Centric Perspective

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read On 21 imbalanced datasets, balancing methods such as ANSMOTE enlarge the set of equally accurate but disagreeing models, while significance-test filtering shrinks it, with data complexity moderating both effects.

desk verdict A substantial benchmark with a filtering claim that the paper's own tests contradict; worth a careful revision rather than a rejection. read the letter →

arxiv 2412.09712 v1 pith:WPKNVFN3 submitted 2024-12-12 stat.ML cs.LG

Mustafa Cavus , Przemyslaw Biecek This is my paper · ORCID
classification stat.MLcs.LG
keywords predictivemultiplicityRashomoneffectimbalancedclassificationdata-centricAIbalancingmethodsfilteringdatacomplexitySMOTEvariants
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is an empirical study of how data preprocessing changes predictive multiplicity, the situation where many models fit a dataset about equally well but disagree on individual predictions. Working with 21 real-world imbalanced datasets, the authors build empirical Rashomon sets with a tree-based AutoML tool and measure disagreement using discrepancy and obscurity. Their central finding is that balancing methods, especially ANSMOTE, inflate both disagreement metrics relative to the original data, while filtering methods, especially the significance test, reduce them. Data complexity moderates both effects: methods like Near Miss and ADASYN become worse on high-complexity datasets, and complexity metrics from the linearity and neighborhood families correlate strongly with predictive multiplicity. The practical claim is that preprocessing choices should be matched to data complexity rather than applied uniformly.

What carries the argument

The empirical Rashomon set $\hat{R}_{L,\epsilon}(\hat{f}_R)$ is the collection of models in the searched hypothesis space whose empirical loss is within $\epsilon = 0.05$ of the best model found; the paper builds it with the tree-based AutoML tool forester using 5 Bayesian iterations and 10 random evaluations. The two outcome metrics are discrepancy, $\delta_\epsilon(\hat{f}_R) = \max_{f \in \hat{R}} \frac{1}{n}\sum_i \mathbb{1}[f(x_i) \neq \hat{f}_R(x_i)]$, and obscurity, which replaces the max with an average over models and observations. These metrics turn 'how many equally good models disagree' into numbers that can be compared across preprocessing pipelines, and the filtering methods are defined by variable selection with correlation and Wilcoxon rank-sum tests, both with Benjamini-Hochberg correction.

What would settle it

Rerun the same 21 dataset and preprocessing combinations with a much larger model search, such as hundreds of random evaluations or exhaustive enumeration on the smallest datasets, and compare the resulting discrepancy and obscurity medians as well as the ranking of balancing methods. If the medians or rankings shift materially, the paper's comparative claims are artifacts of the small Rashomon sample rather than properties of the datasets.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes a comparative result: among ten balancing methods, almost all enlarge the Rashomon set's disagreement, with ANSMOTE the worst (average discrepancy rising from about 0.4 to 0.6 and obscurity from 0.1 to 0.2), and filtering can partially reverse this, with the significance test shrinking distributions most consistently. For example, DBSMOTE combined with the significance test lowers discrepancy from about 0.5 to 0.3 and obscurity from 0.15 to 0.1. These effects are not uniform: data complexity changes which balancing method is safest, with Near Miss pushing discrepancy to 0.7 and obscurity to 0.2 on high-complexity datasets while RSLSMOTE and BLSMOTE stay near 0.3 and 0.05 on simpler ones. The paper also reports that linearity metrics (L1, L2, L3) and the neighborhood metric N4 correlate significantly with both disagreement metrics, and that balancing methods improve AUC while filtering alone does not significantly change it.

Load-bearing premise

The empirical Rashomon set produced by the AutoML search (epsilon 0.05, five Bayesian iterations, ten random evaluations) is treated as representative of all equally accurate models, and every discrepancy and obscurity comparison is computed on that small sample.

Editorial extensions

If this is right

  • Balancing a dataset before training can buy accuracy at the price of wider prediction disagreement, so accuracy-based model selection on balanced data carries hidden arbitrariness.
  • Significance-test filtering, particularly alongside DBSMOTE or SLSMOTE, is a workable lever for shrinking predictive multiplicity without sacrificing the performance gains of balancing.
  • The safest balancing method depends on data complexity: RSLSMOTE and BLSMOTE are stable across complexity levels, while Near Miss and ADASYN are risky on high-complexity data.
  • Complexity metrics from the linearity family (L1, L2, L3) and the neighborhood metric N4 are positively associated with both discrepancy and obscurity, giving practitioners a way to anticipate multiplicity before training many models.
  • Filtering alone does not significantly change AUC in the experiments, so its demonstrated value is in stabilizing predictions rather than boosting accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Going beyond the paper's comparisons, the strong correlations of L1, L2, L3, and N4 with multiplicity suggest a cheap screening rule: compute these complexity metrics before choosing a balancing method, and prefer safe-level SMOTE variants when linearity complexity is high.
  • The Rashomon sets here come from a tree-based AutoML search; an untested but plausible extension is whether the same ranking of balancing methods holds for linear or neural models, whose Rashomon sets have different shapes and sizes.
  • One testable extension is to check whether the benefit of significance filtering persists under different Rashomon parameters (for example, larger epsilon or many more searched models), since the reported effects are measured on small empirical sets.
  • If the complexity-multiplicity correlations reflect a causal relationship, then datasets with high L1/L2/L3 may need stronger filtering before any balancing is applied; this is an inference from the observed correlations, not a result the paper proves.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper studies how data preprocessing steps—ten balancing methods and two filtering methods (correlation and significance tests)—affect predictive multiplicity, measured by discrepancy and obscurity over empirically constructed Rashomon sets. Experiments are carried out on 21 imbalanced benchmark datasets with the forester AutoML tool, and datasets are grouped into three complexity clusters derived from complexity measures. The paper reports that balancing methods, especially ANSMOTE, inflate predictive multiplicity, that filtering methods, especially the Significance Test, reduce predictive multiplicity, and that data complexity moderates both effects.

Significance. If the reported findings were valid, the paper would offer practically useful guidance for data-centric AI: it would identify preprocessing choices that reduce the variance of predictions among equally accurate models. The study covers a broad set of balancing methods and 21 real-world datasets, uses established predictive-multiplicity metrics, and points to a reproducibility repository. However, the central filtering claim is contradicted by the paper's own statistical tests, and the empirical Rashomon sets are not validated as representative; these issues undermine the main contribution as it stands.

major comments (4)
  1. [Section 5 (RQ2) and Section 6 (Conclusion)] The conclusion states that 'Filtering methods, particularly the Significance Test, proved instrumental in reducing predictive multiplicity,' but this claim is not supported by the paper's own inferential results. For RQ2, the Kruskal-Wallis test across the three filtering conditions is non-significant for both metrics (discrepancy chi-squared(2)=0.26, p=0.879; obscurity chi-squared(2)=0.50, p=0.779). The subsequent Friedman test is significant only for obscurity (chi-squared_F(2)=9.25, p=0.010), and the text immediately after reports that 'the combined effects of filtering methods and balancing methods significantly increase the obscurity values'—the opposite of a reduction. In RQ4, the analogous Friedman test is non-significant for both metrics (p=0.7165 and p=0.7166). Thus no reported test establishes a reduction in predictive multiplicity from filtering; if anything, the only significant effect indicates an increase in obscurity. This internal inconsistency is load-bearing and requires either a reanalysis of the data or a substantial reframing of the paper's claims.
  2. [Section 4 (Experiments)] The empirical Rashomon set is built with forester using fixed parameters: bayes iter = 5, random evals = 10, and epsilon = 0.05. The paper does not provide any evidence that the ten or fewer models retrieved for each dataset are representative of the true Rashomon set for that dataset. All discrepancy and obscurity values, and hence every comparison across RQ1–RQ6, depend on this sample. If the retrieved models do not span the set of equally accurate models, the reported multiplicity levels and rankings are biased in an unknown direction. The authors should validate coverage, for example by increasing the search budget and checking stability of the metrics, or by using established Rashomon-set exploration methods.
  3. [Section 4 (Experiments) and Section 3.3 (Filtering Methods)] The manuscript does not state whether the variable filtering is applied before or after the train/test split. If the correlation and significance tests are applied to the full dataset before splitting, the test set informs feature selection, which is a form of target leakage and would bias the filtering comparisons. The text in Section 4 says 'During the balancing phase, adjustments are made to the training set, leaving the test set unchanged,' but no analogous statement appears for filtering. The authors should clarify the exact pipeline order and, if filtering is currently done on the full data, re-run the experiments with filtering inside the training fold only.
  4. [Section 5 (RQ3) and Section 6] The paper interprets the significant Kruskal-Wallis results in RQ3 as evidence that filtering reduces predictive multiplicity, but a significant difference among complexity groups only shows that the metric varies across clusters; it does not establish that filtering lowers the metrics relative to the unfiltered condition. The claims about the moderating role of complexity should be tied to explicit pairwise comparisons or interaction tests, rather than to a single omnibus test.
minor comments (5)
  1. [Introduction] The roadmap sentence states that 'Sect. 2 covers the essentials of the Rashomon effect and predictive multiplicity, Sect. 3 outlines the experiments; Sect. 4 interprets the findings,' but the actual sections are Methods (3), Experiments (4), Results (5), and Conclusion (6).
  2. [Introduction] Reference [5] appears as a placeholder '[4, ?]' in the first paragraph; the citation should be completed.
  3. [Section 5 (RQ2)] The sentence 'Consequently, it is observed that the combined effects of filtering methods and balancing methods significantly increase the obscurity values' is contradictory to the paper's overall conclusion and should either be reconciled or removed.
  4. [Appendix] In Table 10, MagicTelescope and mammography have NA entries for neighborhood measures. This is presumably due to dataset size, but the reason and the implications for the complexity clustering should be stated in the main text.
  5. [Supplemental Materials] The repository link is given as 'github.com/mcavs/data centric Rashomon paper'; it should be a proper URL with the exact repository name to be accessible.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports empirical measurements rather than deriving predictions from inputs, the self-citations supply reusable benchmark and metric artifacts rather than load-bearing justifications, and the filtering conclusion's conflict with the paper's own p-values is a statistical-support problem, not a circular reduction.

full rationale

The paper's central claims are empirical measurements: it constructs empirical Rashomon sets (Eq. 4), computes discrepancy (Eq. 5) and obscurity (Eq. 6) on 21 benchmark datasets under balancing and filtering interventions, and compares the resulting distributions using Kruskal-Wallis, Friedman, and Dunn tests. No parameter is fitted to the disagreement metrics and then renamed as a prediction; the only fitted objects are the forester models and the k-means complexity clusters, and neither is defined in terms of the multiplicity metrics. The self-citations to the benchmark dataset collection [12] and the obscurity metric [17] are reusable artifacts from prior work by the same group, but the present paper recomputes the effects from new model sets and does not rely on those citations to justify its conclusions. The complexity clusters are formed from complexity measures, not from the multiplicity outcomes, so stratifying by them is not circular. The most serious weakness is an internal inconsistency: the Kruskal-Wallis tests for filtering are non-significant for both discrepancy (p=0.879) and obscurity (p=0.779), and the Friedman test in RQ2 indicates that filtering combined with balancing significantly increases obscurity, contradicting the conclusion that the Significance Test 'proved instrumental in reducing predictive multiplicity.' That is a correctness and evidence-weighting problem, not a circularity problem, because the reported statistics are not equivalent to the conclusion by construction. No circular step could be identified by quoting an equation that reduces to its own input or a fitted parameter that is relabeled as a prediction.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central empirical claims depend on several user-chosen parameters (epsilon, AutoML rounds, target balance ratio, number of complexity clusters) and on unstated assumptions about the AutoML search's coverage of the Rashomon set, the mapping of k-means clusters to complexity labels, and the absence of test-set leakage in filtering. None of these are justified or validated.

free parameters (6)
  • Rashomon parameter epsilon = 0.05
    Defines which models are considered equally good; set by hand in Section 4 without sensitivity analysis.
  • bayes iter (forester) = 5
    Number of Bayesian optimization rounds, fixed in Section 4; controls the diversity of the Rashomon set.
  • random evals (forester) = 10
    Number of models trained, fixed in Section 4; caps the Rashomon set size.
  • Resampling ratio = 1
    Balancing target set to perfect balance in Section 4; other ratios are not explored.
  • Number of complexity clusters = 3
    k-means result on 21 datasets; k is chosen without stated justification and cluster labels are never defined.
  • Significance level alpha for filters = not stated (assumed 0.05)
    Used in the BH-adjusted tests in Section 3.3; the value is not reported.
assumptions (5)
  • standard math Empirical loss approximates expected loss (Eq. 4).
    The Rashomon set is defined on empirical loss because the true expected loss is not accessible; this is a standard approximation.
  • ad hoc to paper The forester AutoML run yields a representative sample of the Rashomon set.
    Section 4 fixes bayes iter=5 and random evals=10 and treats the resulting models as the empirical hypothesis space; no coverage check is provided.
  • ad hoc to paper The three k-means clusters map to low, medium, and high data complexity.
    Section 4.1 clusters datasets but never states which cluster number corresponds to which complexity label used in RQ3 and RQ4.
  • domain assumption Missing neighborhood complexity metrics for MagicTelescope and mammography do not affect clustering or correlations.
    Table 10 lists NA for six metrics for these datasets; the paper does not describe imputation or exclusion.
  • domain assumption Variable filtering does not leak test information into training.
    Section 4 does not specify whether correlation and significance filters are applied before the train/test split or inside cross-validation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Investigating the Impact of Balancing, Filtering, and Complexity on Predictive Multiplicity: A Data-Centric Perspective." pith.science (2026). https://pith.science/paper/WPKNVFN3

@misc{pith2026241209712,
  author       = {Pith},
  title        = {Pith review of: Investigating the Impact of Balancing, Filtering, and Complexity on Predictive Multiplicity: A Data-Centric Perspective},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WPKNVFN3}},
  note         = {Machine review of arXiv:2412.09712}
}
read the original abstract

The Rashomon effect presents a significant challenge in model selection. It occurs when multiple models achieve similar performance on a dataset but produce different predictions, resulting in predictive multiplicity. This is especially problematic in high-stakes environments, where arbitrary model outcomes can have serious consequences. Traditional model selection methods prioritize accuracy and fail to address this issue. Factors such as class imbalance and irrelevant variables further complicate the situation, making it harder for models to provide trustworthy predictions. Data-centric AI approaches can mitigate these problems by prioritizing data optimization, particularly through preprocessing techniques. However, recent studies suggest preprocessing methods may inadvertently inflate predictive multiplicity. This paper investigates how data preprocessing techniques like balancing and filtering methods impact predictive multiplicity and model stability, considering the complexity of the data. We conduct the experiments on 21 real-world datasets, applying various balancing and filtering techniques, and assess the level of predictive multiplicity introduced by these methods by leveraging the Rashomon effect. Additionally, we examine how filtering techniques reduce redundancy and enhance model generalization. The findings provide insights into the relationship between balancing methods, data complexity, and predictive multiplicity, demonstrating how data-centric AI strategies can improve model performance.

Figures

Figures reproduced from arXiv: 2412.09712 by the authors.

Figure 1
Figure 1. Illustration of a Rashomon cube with size 5 means that comprises five [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. The similarity analysis of datasets from our benchmark. Similarity is [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗
Figure 3
Figure 3. The distribution of the disagreement metrics obscurity and discrepancy [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The 2d density plot of the disagreement metrics obscurity and discrepancy [PITH_FULL_IMAGE:figures/full_fig_p017_4.png]
Figure 5
Figure 5. Figure 5: The 2d density plot of the disagreement metrics obscurity and discrepancy [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]
Figure 6
Figure 6. Figure 6: The 2d density plot of the disagreement metrics obscurity and discrepancy [PITH_FULL_IMAGE:figures/full_fig_p021_6.png]
Figure 7
Figure 7. Figure 7: The performance gain plots of obscurity and discrepancy for the balancing [PITH_FULL_IMAGE:figures/full_fig_p023_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

78 extracted references · 71 canonical work pages

  1. [1]

    Breiman, L. (2001). Statistical modeling: The two cultures. Statistical Science, 16(3), 199-231

    work page 2001

  2. [2]

    Marx, C., Calmon, F., & Ustun, B. (2020). Predictive multiplicity in classification. In International Conference on Machine Learning (pp. 6765-6774). PMLR

    work page 2020

  3. [3]

    Lee, Y., Yao, H., & Finn, C. (2023). Diversify and disambiguate: Out-of- distribution robustness via disagreement. In The 11th International Conference on Learning Representations

    work page 2023

  4. [4]

    Long, C., Hsu, H., Alghamdi, W., & Calmon, F. (2024). Individual arbitrariness and group fairness. Advances in Neural Information Processing Systems , 36

    work page 2024

  5. [5]

    Renard, X., Laugel, T., & Detyniecki, M. (2024). Understanding prediction dis- crepancies in classification. Machine Learning, 113(10), 7997-8026. Investigating the data preprocessing on predictive multiplicity 27

    work page 2024

  6. [6]

    Luengo, J., Fern´ andez, A., Garc ´ ıa, S., & Herrera, F. (2011). Addressing data complexity for imbalanced data sets analysis of SMOTE-based oversampling and evolutionary undersampling. Soft Computing , 15, 1909-1936

    work page 2011

  7. [7]

    Hou, L., Lu, K., & Bi, G. (2024). Predicting the credit risk of small and medium- sized enterprises in supply chain finance using machine learning algorithms. Man- agerial and Decision Economics , 45(4), 2393-2414

    work page 2024

  8. [8]

    S., Abreu, P

    Santos, M. S., Abreu, P. H., Japkowicz, N., Fern´ andez, A., & Santos, J. (2023). A unifying view of class overlap and imbalance: Key concepts, multi-view panorama, and open avenues for research. Information Fusion, 89, 228-253

    work page 2023

Show all 78 references
  1. [9]

    Fern´ andez, A., Garcia, S., Herrera, F., & Chawla, N. V. (2018). SMOTE for learning from imbalanced data: progress and challenges, marking the 15th year anniversary. Journal of Artificial Intelligence Research , 61, 863-905

    2018

  2. [10]

    P., Lai, K

    Zha, D., Bhat, Z. P., Lai, K. H., Yang, F., Jiang, Z., Zhong, S., & Hu, X. (2023). Data-centric artificial intelligence: A survey. arXiv Preprint arXiv:2303.10158

    2023 arXiv

  3. [11]

    Singh, P. (2023). Systematic review of data-centric approaches in artificial intel- ligence and machine learning. Data Science and Management , 6(3), 144-157

    2023

  4. [12]

    Stando, A., Cavus, M., & Biecek, P. (2024). The effect of balancing methods on model behavior in imbalanced classification problems. In Fifth International Workshop on Learning with Imbalanced Domains: Theory and Applications . Pro- ceedings of Machine Learning Research, 16-30

    2024

  5. [13]

    Patil, A., Framewala, A., & Kazi, F. (2020). Explainability of SMOTE-based over- sampling for imbalanced dataset problems. In 2020 3rd International Conference on Information and Computer Technologies (pp. 41-45)

    2020

  6. [14]

    Alarab, I., & Prakoonwit, S. (2022). Effect of data resampling on feature im- portance in imbalanced blockchain data: comparison studies of resampling tech- niques. Data Science and Management , 5(2), 66-76

    2022

  7. [15]

    van den Goorbergh, R., van Smeden, M., Timmerman, D., & Van Calster, B. (2022). The harm of class imbalance corrections for risk prediction models: illus- tration and simulation using logistic regression. Journal of the American Medical Informatics Association, 29(9), 1525-1534

    2022

  8. [16]

    Simson, J., Pfisterer, F., & Kern, C. (2023). Everything, everywhere all in one evaluation: using multiverse analysis to evaluate the influence of model design decisions on algorithmic fairness. arXiv Preprint arXiv:2308.16681

    2023 arXiv

  9. [17]

    Cavus, M., & Biecek, P. (2024). An experimental study on the Rashomon effect of balancing methods in imbalanced classification. arXiv Preprint arXiv:2405.01557

    2024 arXiv

  10. [18]

    Biecek, P., & Samek, W. (2024). Position: Explain to question not to justify. In Forty-First International Conference on Machine Learning

    2024

  11. [19]

    Gøttcke, J. M. N., Bellinger, C., Branco, P., & Zimek, A. (2023). An interpretable measure of dataset complexity for imbalanced classification problems. In Proceed- ings of the 2023 SIAM International Conference on Data Mining (SDM) (pp. 253-261). Society for Industrial and Ap...

    2023

  12. [20]

    C., Dewi, C., Huang, S

    Chen, R. C., Dewi, C., Huang, S. W., & Caraka, R. E. (2020). Selecting critical features for data classification based on machine learning methods. Journal of Big Data, 7(1), 52

    2020

  13. [21]

    P., Lai, K

    Zha, D., Bhat, Z. P., Lai, K. H., Yang, F., & Hu, X.: Data-centric AI: Perspectives and challenges. In Proceedings Of The 2023 SIAM International Conference on Data Mining (SDM), pp. 945-948. Society for Industrial and Applied Mathematics (2023)

    2023

  14. [22]

    Journal of Information and Organizational Sciences , 47(1), pp

    Tanov, V.: Data-centric optimization approach for small, imbalanced datasets. Journal of Information and Organizational Sciences , 47(1), pp. 167-177 (2023). 28 Cavus and Biecek

    2023

  15. [23]

    Business & Information Systems Engineering , pp

    Jakubik, J., V¨ ossing, M., K¨ uhl, N., Walk, J., & Satzger, G.: Data-centric artificial intelligence. Business & Information Systems Engineering , pp. 1-9 (2024)

    2024

  16. [24]

    arXiv Preprint arXiv:2110.03613 (2021)

    Motamedi, M., Sakharnykh, N., & Kaldewey, T.: A data-centric approach for training deep neural networks with less data. arXiv Preprint arXiv:2110.03613 (2021)

    2021 arXiv

  17. [25]

    X., Chukova, S

    Wang, A. X., Chukova, S. S., & Nguyen, B. P.: Data-centric AI to improve churn prediction with synthetic data. In 2023 3rd International Conference on Com- puter, Control and Robotics (ICCCR) , pp. 409-413. *IEEE* (2023)

    2023

  18. [26]

    K., & Sharma, R.: Opportunities and challenges in data-centric AI

    Kumar, S., Datta, S., Singh, V., Singh, S. K., & Sharma, R.: Opportunities and challenges in data-centric AI. IEEE Access (2024)

    2024

  19. [27]

    Ortega V´ azquez, C., vanden Broucke, S., & De Weerdt, J. (2024). Hellinger dis- tance decision trees for PU learning in imbalanced datasets. Machine Learning, 113(7), 4547-4578

    2024

  20. [28]

    Gu, Q., Tian, J., Li, X., & Jiang, S. (2022). A novel random forest integrated model for imbalanced data classification problem. Knowledge-Based Systems, 250, 109050

    2022

  21. [29]

    Li, Y., Adams, N., & Bellotti, T. (2022). A relabeling approach to handling the class imbalance problem for logistic regression. Journal of Computational and Graphical Statistics, 31(1), 241-253

    2022

  22. [30]

    V., Bowyer, K

    Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P.: SMOTE: Synthetic minority over-sampling technique. Journal of Artificial Intelligence Re- search, 16, pp. 321–357 (2002)

    2002

  23. [31]

    A., Li, S.: ADASYN: Adaptive synthetic sampling approach for imbalanced learning

    He, H., Bai, Y., Garcia, E. A., Li, S.: ADASYN: Adaptive synthetic sampling approach for imbalanced learning. In 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence) , pp. 1322–1328. *IEEE* (2008)

    2008

  24. [32]

    Moniz, N., & Monteiro, H. (2021). No free lunch in imbalanced learning. Knowledge-Based Systems, 227, 107222

    2021

  25. [33]

    Hsu, H., & Calmon, F. (2022). Rashomon capacity: A metric for predictive multi- plicity in classification. Advances In Neural Information Processing Systems , 35, 28988-29000

    2022

  26. [34]

    C., & Ustun, B

    Watson-Daniels, J., Parkes, D. C., & Ustun, B. (2023, June). Predictive multi- plicity in probabilistic classification. In Proceedings Of The AAAI Conference On Artificial Intelligence (Vol. 37, No. 9, pp. 10306-10314)

    2023

  27. [35]

    Ferreira, A., & Figueiredo, M. (2012). Efficient feature selection filters for high- dimensional data. Pattern Recognition Letters, 33, 1794-1804

    2012

  28. [36]

    Chandrashekar, G., & Sahin, F. (2014). A survey on feature selection methods. Computers and Electrical Engineering , 40, 16-28

    2014

  29. [37]

    Bommert, A., Sun, X., Bischl, B., Rahnenf¨ uhrer, J., & Lang, M. (2020). Bench- mark for filter methods for feature selection in high-dimensional classification data. Computational Statistics and Data Analysis , 143

    2020

  30. [38]

    B., & Rudnicki, W

    Kursa, M. B., & Rudnicki, W. R. (2010). Feature selection with the Boruta pack- age. Journal of Statistical Software , 36, 1-13

    2010

  31. [39]

    Guyon, I., Weston, J., Barnhill, S., & Vapnik, V. (2002). Gene selection for cancer classification using support vector machines. Machine Learning, 46, 389-422

    2002

  32. [40]

    Zheng, Y., Li, Y., Wang, G., Chen, Y., Xu, Q., Fan, J., & Cui, X. (2018). A novel hybrid algorithm for feature selection. Personal and Ubiquitous Computing , 22, 971 - 985

    2018

  33. [41]

    Solorio-Fern´ andez, S., Carrasco-Ochoa, J., & Mart ´ ınez-Trinidad, J. (2020). A sys- tematic evaluation of filter unsupervised feature selection methods. Expert Sys- tems With Applications , 162, 113745. Investigating the data preprocessing on predictive multiplicity 29

    2020

  34. [42]

    Jebadurai, I., Paulraj, G., Jebadurai, J., & Silas, S. (2022). Experimental analysis of filtering-based feature selection techniques for fetal health classification.Serbian Journal of Electrical Engineering

    2022

  35. [43]

    Bakdash, J., Marusich, L., Kenworthy, J., Twedt, E., & Zaroukian, E. (2020). Statistical significance filtering overestimates effects and impedes falsification: A critique of. Frontiers in Psychology, 11

    2020

  36. [44]

    Gøttcke, J. M. N., Bellinger, C., Branco, P., & Zimek, A.: An interpretable mea- sure of dataset complexity for imbalanced classification problems. In Proceedings Of The 2023 SIAM International Conference on Data Mining (SDM) (pp. 253- 261). Society for Industrial And Applied ...

    2023

  37. [45]

    H., Garcia, L

    Barella, V. H., Garcia, L. P., de Souto, M. C., Lorena, A. C., & de Carvalho, A. C.: Assessing the data complexity of imbalanced datasets. Information Sciences, 553, 83-109 (2021)

    2021

  38. [46]

    Junior, J. D. S. F., & Pisani, P. H. (2022, October). Performance and model com- plexity on imbalanced datasets using resampling and cost-sensitive algorithms. In Fourth International Workshop on Learning With Imbalanced Domains: Theory And Applications (pp. 83-97). PMLR (2022)

    2022

  39. [47]

    Statistic Surveys , 16, pp

    Rudin, C., Chen, C., Chen, Z., Huang, H., Semenova, L., & Zhong, C.: In- terpretable machine learning: Fundamental principles and 10 grand challenges. Statistic Surveys , 16, pp. 1-85 (2022)

    2022

  40. [48]

    Obser- vational Studies , 7(1), pp

    D’Amour, A.: Revisiting Rashomon: A comment on ”The Two Cultures”. Obser- vational Studies , 7(1), pp. 59-63 (2021)

    2021

  41. [49]

    In Proceedings Of The 2022 ACM Conference On Fairness, Accountabil- ity, And Transparency, pp

    Semenova, L., Rudin, C., & Parr, R.: On the existence of simpler machine learning models. In Proceedings Of The 2022 ACM Conference On Fairness, Accountabil- ity, And Transparency, pp. 1827-1858 (2022)

    2022

  42. [50]

    Watson-Daniels, J., Calmon, F. D. P., D’Amour, A., Long, C., Parkes, D. C., & Ustun, B.: Predictive churn with the set of good models. arXiv Preprint arXiv:2402.07745 (2024)

    2024 arXiv

  43. [51]

    In Proceedings Of The 30th ACM SIGKDD Conference On Knowledge Discovery And Data Mining , pp

    Ciaperoni, M., Xiao, H., & Gionis, A.: Efficient exploration of the Rashomon set of rule-set models. In Proceedings Of The 30th ACM SIGKDD Conference On Knowledge Discovery And Data Mining , pp. 478-489 (2024)

    2024

  44. [52]

    I., Cofone, I., & Farnadi, G.: The cost of arbitrariness for individuals: Examining the legal and technical challenges of model multiplicity

    Ganesh, P., Daldaban, I. I., Cofone, I., & Farnadi, G.: The cost of arbitrariness for individuals: Examining the legal and technical challenges of model multiplicity. arXiv Preprint arXiv:2407.13070 (2024)

    2024 arXiv

  45. [53]

    In Proceedings Of The 2022 ACM Conference On Fairness, Accountability, And Transparency, pp

    Black, E., Raghavan, M., & Barocas, S.: Model multiplicity: Opportunities, con- cerns, and solutions. In Proceedings Of The 2022 ACM Conference On Fairness, Accountability, And Transparency, pp. 850-863 (2022)

    2022

  46. [54]

    & Boner, Z.: Amazing things come from having many good models

    Rudin, C., Zhong, C., Semenova, L., Seltzer, M., Parr, R., Liu, J., ... & Boner, Z.: Amazing things come from having many good models. arXiv Preprint arXiv:2407.04846 (2024)

    2024 arXiv

  47. [55]

    F., Machado, C., Paes, L

    Gomez, J. F., Machado, C., Paes, L. M., & Calmon, F. (2024, June). Algorithmic arbitrariness in content moderation. In The 2024 ACM Conference on Fairness, Accountability, and Transparency (pp. 2234-2253)

    2024

  48. [56]

    Hamman, F., Dissanayake, P., Mishra, S., Lecue, F., & Dutta, S. (2024). Quan- tifying prediction consistency under model multiplicity in tabular LLMs. arXiv Preprint arXiv:2407.04173

    2024 arXiv

  49. [57]

    arXiv Preprint arXiv:2402.00728 (2024)

    Hsu, H., Li, G., Hu, S.: Dropout-based Rashomon set exploration for efficient predictive multiplicity estimation. arXiv Preprint arXiv:2402.00728 (2024)

    2024 arXiv

  50. [58]

    Ad- vances in Neural Information Processing Systems , 36 (2024)

    Donnelly, J., Katta, S., Rudin, C., Browne, E.: The Rashomon importance dis- tribution: Getting RID of unstable, single model-based variable importance. Ad- vances in Neural Information Processing Systems , 36 (2024). 30 Cavus and Biecek

    2024

  51. [59]

    In Proceedings Of Workshop on Learning From Imbalanced Datasets, Vol

    Mani, I., & Zhang, I.: kNN approach to unbalanced data distributions: A case study involving information extraction. In Proceedings Of Workshop on Learning From Imbalanced Datasets, Vol. 126, No. 1, pp. 1–7. ICML (2003)

    2003

  52. [60]

    Y., Mao, B

    Han, H., Wang, W. Y., Mao, B. H.: Borderline-SMOTE: A new over-sampling method in imbalanced data sets learning. In International Conference on Intelli- gent Computing , pp. 878–887. Springer, Berlin, Heidelberg (2005)

    2005

  53. [61]

    Applied Intelligence 36, 664– 684 (2012)

    Bunkhumpornpat, C., Sinapiromsaran, K., Lursinsap, C.: DBSMOTE: Density- based synthetic minority over-sampling technique. Applied Intelligence 36, 664– 684 (2012)

    2012

  54. [62]

    Chiang Mai Journal of Science 43(1), 234–246 (2016)

    Siriseriwan, W., Sinapiromsaran, K.: The effective redistribution for imbalance dataset: Relocating safe-level SMOTE with minority outcast handling. Chiang Mai Journal of Science 43(1), 234–246 (2016)

    2016

  55. [63]

    Songklanakarin Journal of Sci- ence and Technology 39(5) (2017)

    Siriseriwan, W., Sinapiromsaran, K.: Adaptive neighbor synthetic minority over- sampling technique under 1NN outcast handling. Songklanakarin Journal of Sci- ence and Technology 39(5) (2017)

    2017

  56. [64]

    In: Advances in Knowledge Discovery and Data Mining: 13th Pacific-Asia Conference, PAKDD 2009, Bangkok, Thailand, April 27–30, 2009, Proceedings, pp

    Bunkhumpornpat, C., Sinapiromsaran, K., Lursinsap, C.: Safe-level-SMOTE: Safe-level-synthetic minority over-sampling technique for handling the class im- balanced problem. In: Advances in Knowledge Discovery and Data Mining: 13th Pacific-Asia Conference, PAKDD 2009, Bangkok, T...

    2009

  57. [65]

    Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: a prac- tical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological) , 57(1), 289-300

    1995

  58. [66]

    In: Breakthroughs in Statistics: Methodology and Distribution , pp

    Wilcoxon, F.: Individual comparisons by ranking methods. In: Breakthroughs in Statistics: Methodology and Distribution , pp. 196–202. Springer, New York, NY (1992)

    1992

  59. [67]

    Ruczy´ nski, H., & Kozak, A. (2024). forester: A Tree-Based AutoML Tool in R. arXiv preprint arXiv:2409.04789

    2024 arXiv

  60. [68]

    MacQueen, J. (1967). Some methods for classification and analysis of multivari- ate observations. In Proceedings of the 5th Berkeley Symposium on Mathematical Statistics and Probability , Berkeley, University of California Press, pp. 281-297

    1967

  61. [69]

    H., & Wallis, W

    Kruskal, W. H., & Wallis, W. A. (1952). Use of ranks in one-criterion variance analysis. Journal of the American Statistical Association , 47(260), 583-621

    1952

  62. [70]

    Friedman, M. (1937). The use of ranks to avoid the assumption of normality im- plicit in the analysis of variance. Journal of the American Statistical Association , 32(200), 675-701

    1937

  63. [71]

    Dunn, O. J. (1964). Multiple comparisons using rank sums. Technometrics, 6(3), 241-252

    1964

  64. [72]

    Spearman, C. (1987). The proof and measurement of association between two things. The American Journal of Psychology , 100(3/4), 441-471

    1987

  65. [73]

    Hollander, M. (2013). Nonparametric Statistical Methods. John Wiley & Sons Inc

    2013

  66. [74]

    C., Costa, I

    Lorena, A. C., Costa, I. G., Spolaˆ or, N., & De Souto, M. C.: Analysis of complexity indices for classification problems: Cancer gene expression data. Neurocomputing, 75(1), pp. 33-42 (2012)

    2012

  67. [75]

    IEEE Transactions on Evolutionary Computation , 14(4), pp

    Orriols-Puig, A., Casillas, A., & Bernad´ o-Mansilla, E.: Exploiting variable se- lection constraints in evolutionary learning algorithms. IEEE Transactions on Evolutionary Computation , 14(4), pp. 503-525 (2010)

    2010

  68. [76]

    K., & Basu, M.: Measuring the complexity of classification problems

    Ho, T. K., & Basu, M.: Measuring the complexity of classification problems. IEEE Transactions on Pattern Analysis and Machine Intelligence , 24(3), pp. 289-300 (2002). Investigating the data preprocessing on predictive multiplicity 31

    2002

  69. [77]

    K., Basu, M., & Mollineda, R

    Ho, T. K., Basu, M., & Mollineda, R. A.: Complexity measures of supervised classification problems. In Proceedings of the 18th International Conference on Pattern Recognition (ICPR), pp. 21-24 (2006)

    2006

  70. [78]

    IEEE Transactions on Knowledge and Data Engineering , 27(2), pp

    Leyva, E., Gonz´ alez, A., & P´ erez, R.: A set of complexity measures designed for applying meta-learning to instance selection. IEEE Transactions on Knowledge and Data Engineering , 27(2), pp. 354-367 (2014). Appendix Data Complexity Measures This section summarizes various ...

    2014

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.