{"id":"2a1edf90-9292-45d1-9cc3-bfede94cb5a5","arxiv_id":"2412.09712","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"Balancing methods increase predictive multiplicity on imbalanced datasets, while filtering effects are inconsistent and largely non-significant in the authors' own tests.","lead":"This paper tests how data balancing and feature filtering change the Rashomon effect, where many different models perform equally well but disagree on predictions. It aims to guide preprocessing choices for trustworthy machine learning, but its main new claims fail its own significance tests.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own Kruskal-Wallis tests find no filtering effect (discrepancy p=0.879, obscurity p=0.779), yet the conclusion calls the Significance Test 'instrumental'; the central filtering claim is contradicted by the reported statistics.","rationale":"We read the paper as attempting to establish, on the basis of 21 datasets, that balancing methods (especially ANSMOTE) inflate predictive multiplicity and filtering methods (especially the Significance Test) reduce it, with complexity moderating both effects. The balancing-side of the claim is at least internally consistent: the KW tests in RQ1 are significant and Dunn post-hoc tests separate the original condition from all balancing methods. The filtering-side, however, collapses under the paper's own statistics. The KW test for filtering conditions is non-significant for both discrepancy and obscurity, and one reported Friedman analysis indicates that filtering significantly increases obscurity, directly contradicting the conclusion. This is not an external challenge to the Rashomon-set construction; it is a logical gap between the reported evidence and the stated finding. The reader's verdict of REJECT is therefore appropriate: the central new contribution is unsupported. We would leave the verdict unchanged. The reader's weakest assumption concerned the representativeness of the small AutoML-generated Rashomon set; that is a legitimate concern, but the more immediate and decisive problem is that the paper's own reported tests fail to support the filtering claim. A re-analysis of the per-dataset metric values would settle whether the contradiction is due to a reporting error or to an actual absence of effect; either way, the authors need to reconcile the conclusion with their tests.","tokens_in":23537,"tokens_out":10312,"duration_ms":92298,"concrete_test":"Recompute the Kruskal-Wallis and Friedman tests, plus a paired post-hoc comparison between the Significance Test and Not Filtered conditions, from the per-dataset discrepancy and obscurity values that underlie Figures 4-6. The raw values should be obtained from the authors' repository (github.com/mcavs/data centric Rashomon paper, noting that the link as printed contains a space and is invalid) or from a corrected release. If the re-analysis confirms that the filtering effect is non-significant or that the Significance Test increases obscurity, the central claim that filtering 'proved instrumental in reducing predictive multiplicity' is not supported and the conclusion must be moderated or retracted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central new claim is that filtering methods, particularly the Significance Test, reduce predictive multiplicity (Conclusion, Section 6). This claim is not supported by the paper's own inferential statistics in Section 5. For RQ2, the Kruskal-Wallis test across the three filtering conditions (Not Filtered, Correlation, Significance) is non-significant for both metrics: discrepancy chi-squared(2)=0.26, p=0.879; obscurity chi-squared(2)=0.50, p=0.779. A non-significant KW test means there is no statistical evidence that filtering changes either disagreement metric. The subsequent Friedman test (with balancing as a block) is significant only for obscurity (chi-squared_F(2)=9.25, p=0.010), and the sentence immediately following states that the 'combined effects of filtering methods and balancing methods significantly increase the obscurity values'—the opposite of the claimed reduction. In RQ4, the analogous Friedman test is non-significant for both metrics (p=0.7165 and p=0.7166). Thus, depending on which reported analysis is accepted, filtering either has no measurable effect or increases obscurity. The conclusion that the Significance Test 'proved instrumental in reducing predictive multiplicity' therefore does not follow from the paper's own results. This is an internal inconsistency, not a matter of external assumptions; even if the Rashomon sets were perfectly representative, the reported statistics contradict the headline claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies how data preprocessing steps—ten balancing methods and two filtering methods (correlation and significance tests)—affect predictive multiplicity, measured by discrepancy and obscurity over empirically constructed Rashomon sets. Experiments are carried out on 21 imbalanced benchmark datasets with the forester AutoML tool, and datasets are grouped into three complexity clusters derived from complexity measures. The paper reports that balancing methods, especially ANSMOTE, inflate predictive multiplicity, that filtering methods, especially the Significance Test, reduce predictive multiplicity, and that data complexity moderates both effects.","tokens_in":23855,"tokens_out":2761,"duration_ms":30022,"significance":"If the reported findings were valid, the paper would offer practically useful guidance for data-centric AI: it would identify preprocessing choices that reduce the variance of predictions among equally accurate models. The study covers a broad set of balancing methods and 21 real-world datasets, uses established predictive-multiplicity metrics, and points to a reproducibility repository. However, the central filtering claim is contradicted by the paper's own statistical tests, and the empirical Rashomon sets are not validated as representative; these issues undermine the main contribution as it stands.","major_comments":[{"comment":"The conclusion states that 'Filtering methods, particularly the Significance Test, proved instrumental in reducing predictive multiplicity,' but this claim is not supported by the paper's own inferential results. For RQ2, the Kruskal-Wallis test across the three filtering conditions is non-significant for both metrics (discrepancy chi-squared(2)=0.26, p=0.879; obscurity chi-squared(2)=0.50, p=0.779). The subsequent Friedman test is significant only for obscurity (chi-squared_F(2)=9.25, p=0.010), and the text immediately after reports that 'the combined effects of filtering methods and balancing methods significantly increase the obscurity values'—the opposite of a reduction. In RQ4, the analogous Friedman test is non-significant for both metrics (p=0.7165 and p=0.7166). Thus no reported test establishes a reduction in predictive multiplicity from filtering; if anything, the only significant effect indicates an increase in obscurity. This internal inconsistency is load-bearing and requires either a reanalysis of the data or a substantial reframing of the paper's claims.","section":"Section 5 (RQ2) and Section 6 (Conclusion)"},{"comment":"The empirical Rashomon set is built with forester using fixed parameters: bayes iter = 5, random evals = 10, and epsilon = 0.05. The paper does not provide any evidence that the ten or fewer models retrieved for each dataset are representative of the true Rashomon set for that dataset. All discrepancy and obscurity values, and hence every comparison across RQ1–RQ6, depend on this sample. If the retrieved models do not span the set of equally accurate models, the reported multiplicity levels and rankings are biased in an unknown direction. The authors should validate coverage, for example by increasing the search budget and checking stability of the metrics, or by using established Rashomon-set exploration methods.","section":"Section 4 (Experiments)"},{"comment":"The manuscript does not state whether the variable filtering is applied before or after the train/test split. If the correlation and significance tests are applied to the full dataset before splitting, the test set informs feature selection, which is a form of target leakage and would bias the filtering comparisons. The text in Section 4 says 'During the balancing phase, adjustments are made to the training set, leaving the test set unchanged,' but no analogous statement appears for filtering. The authors should clarify the exact pipeline order and, if filtering is currently done on the full data, re-run the experiments with filtering inside the training fold only.","section":"Section 4 (Experiments) and Section 3.3 (Filtering Methods)"},{"comment":"The paper interprets the significant Kruskal-Wallis results in RQ3 as evidence that filtering reduces predictive multiplicity, but a significant difference among complexity groups only shows that the metric varies across clusters; it does not establish that filtering lowers the metrics relative to the unfiltered condition. The claims about the moderating role of complexity should be tied to explicit pairwise comparisons or interaction tests, rather than to a single omnibus test.","section":"Section 5 (RQ3) and Section 6"}],"minor_comments":[{"comment":"The roadmap sentence states that 'Sect. 2 covers the essentials of the Rashomon effect and predictive multiplicity, Sect. 3 outlines the experiments; Sect. 4 interprets the findings,' but the actual sections are Methods (3), Experiments (4), Results (5), and Conclusion (6).","section":"Introduction"},{"comment":"Reference [5] appears as a placeholder '[4, ?]' in the first paragraph; the citation should be completed.","section":"Introduction"},{"comment":"The sentence 'Consequently, it is observed that the combined effects of filtering methods and balancing methods significantly increase the obscurity values' is contradictory to the paper's overall conclusion and should either be reconciled or removed.","section":"Section 5 (RQ2)"},{"comment":"In Table 10, MagicTelescope and mammography have NA entries for neighborhood measures. This is presumably due to dataset size, but the reason and the implications for the complexity clustering should be stated in the main text.","section":"Appendix"},{"comment":"The repository link is given as 'github.com/mcavs/data centric Rashomon paper'; it should be a proper URL with the exact repository name to be accessible.","section":"Supplemental Materials"}],"recommendation":"reject","confidential_remarks":"The paper leans heavily on the authors' own prior work (refs 12 and 17) for the benchmark and the definition of the empirical Rashomon set. The central filtering claim is directly contradicted by the manuscript's reported statistics, which is a fundamental issue rather than a local presentation problem. I would not consider this suitable for acceptance at a serious journal without a completely new experimental analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper's headline claim—that filtering, especially the Significance Test, reduces predictive multiplicity—doesn't survive contact with its own statistics. The Kruskal-Wallis tests across filtering conditions are non-significant for both discrepancy (p=0.879) and obscurity (p=0.779), and the only significant Friedman test (obscurity, p=0.010) is described in the text as increasing obscurity. So the conclusion in Section 6 that the Significance Test proved 'instrumental' is not supported by the reported results. That's an internal inconsistency, not an external critique.\n\nWhat's genuinely new is the benchmark: 21 datasets, 10 balancing methods, 2 filtering strategies, 3 complexity clusters, all measured for multiplicity via forester's Rashomon sets. That's a lot of work, and the balancing-inflates-multiplicity finding (especially ANSMOTE) is backed by the KW tests. The complexity correlations in RQ6 (L1, L2, L3, N4) are also a reasonable exploratory contribution.\n\nThe soft spots beyond the filtering contradiction: the Rashomon sets are built with only 10 random evaluations and epsilon 0.05, and there's no validation that this sample spans the true set, so all multiplicity values are conditional on a fairly arbitrary search budget. The paper never clearly states whether filtering happens before or after the train/test split; if before, that's leakage. The code link in the paper is broken (github.com/mcavs/data centric Rashomon paper). And a few complexity metrics are NA for two large datasets, which the clustering just seems to absorb without discussion.\n\nOverall: this is a serious empirical study with a fixable but load-bearing flaw. The target reader is someone working on data-centric AI or predictive multiplicity: they'll get a useful benchmark and a cautionary tale, but not a reliable recipe. I'd send it to peer review, because a good referee can push the authors to either re-analyze properly or rewrite the claims to match their own statistics. But I wouldn't cite the filtering result as-is, and I'd want the methodology around the split and the Rashomon set clarified. It's a 'maybe' for the reading group—useful as a case study in how descriptive plots and inferential tests can tell different stories.","headline":"A substantial benchmark with a filtering claim that the paper's own tests contradict; worth a careful revision rather than a rejection.","tokens_in":24361,"tokens_out":5659,"would_cite":false,"duration_ms":50651,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"On 21 imbalanced datasets, balancing methods such as ANSMOTE enlarge the set of equally accurate but disagreeing models, while significance-test filtering shrinks it, with data complexity moderating both effects.","keywords":["predictive multiplicity","Rashomon effect","imbalanced classification","data-centric AI","balancing methods","filtering methods","data complexity","SMOTE variants"],"falsifier":"Rerun the same 21 dataset and preprocessing combinations with a much larger model search, such as hundreds of random evaluations or exhaustive enumeration on the smallest datasets, and compare the resulting discrepancy and obscurity medians as well as the ranking of balancing methods. If the medians or rankings shift materially, the paper's comparative claims are artifacts of the small Rashomon sample rather than properties of the datasets.","tokens_in":23291,"feed_emoji":"⚖️","tokens_out":7259,"duration_ms":67982,"temperature":0.7,"pith_summary":"The paper is an empirical study of how data preprocessing changes predictive multiplicity, the situation where many models fit a dataset about equally well but disagree on individual predictions. Working with 21 real-world imbalanced datasets, the authors build empirical Rashomon sets with a tree-based AutoML tool and measure disagreement using discrepancy and obscurity. Their central finding is that balancing methods, especially ANSMOTE, inflate both disagreement metrics relative to the original data, while filtering methods, especially the significance test, reduce them. Data complexity moderates both effects: methods like Near Miss and ADASYN become worse on high-complexity datasets, and complexity metrics from the linearity and neighborhood families correlate strongly with predictive multiplicity. The practical claim is that preprocessing choices should be matched to data complexity rather than applied uniformly.","feed_headline":"Balancing data inflates model disagreement; filtering tames it","feed_subtitle":"On 21 imbalanced datasets, preprocessing choices reshape the set of equally good models.","key_machinery":"The empirical Rashomon set $\\hat{R}_{L,\\epsilon}(\\hat{f}_R)$ is the collection of models in the searched hypothesis space whose empirical loss is within $\\epsilon = 0.05$ of the best model found; the paper builds it with the tree-based AutoML tool forester using 5 Bayesian iterations and 10 random evaluations. The two outcome metrics are discrepancy, $\\delta_\\epsilon(\\hat{f}_R) = \\max_{f \\in \\hat{R}} \\frac{1}{n}\\sum_i \\mathbb{1}[f(x_i) \\neq \\hat{f}_R(x_i)]$, and obscurity, which replaces the max with an average over models and observations. These metrics turn 'how many equally good models disagree' into numbers that can be compared across preprocessing pipelines, and the filtering methods are defined by variable selection with correlation and Wilcoxon rank-sum tests, both with Benjamini-Hochberg correction.","core_discovery":"On its own terms, the paper establishes a comparative result: among ten balancing methods, almost all enlarge the Rashomon set's disagreement, with ANSMOTE the worst (average discrepancy rising from about 0.4 to 0.6 and obscurity from 0.1 to 0.2), and filtering can partially reverse this, with the significance test shrinking distributions most consistently. For example, DBSMOTE combined with the significance test lowers discrepancy from about 0.5 to 0.3 and obscurity from 0.15 to 0.1. These effects are not uniform: data complexity changes which balancing method is safest, with Near Miss pushing discrepancy to 0.7 and obscurity to 0.2 on high-complexity datasets while RSLSMOTE and BLSMOTE stay near 0.3 and 0.05 on simpler ones. The paper also reports that linearity metrics (L1, L2, L3) and the neighborhood metric N4 correlate significantly with both disagreement metrics, and that balancing methods improve AUC while filtering alone does not significantly change it.","pith_inferences":["Going beyond the paper's comparisons, the strong correlations of L1, L2, L3, and N4 with multiplicity suggest a cheap screening rule: compute these complexity metrics before choosing a balancing method, and prefer safe-level SMOTE variants when linearity complexity is high.","The Rashomon sets here come from a tree-based AutoML search; an untested but plausible extension is whether the same ranking of balancing methods holds for linear or neural models, whose Rashomon sets have different shapes and sizes.","One testable extension is to check whether the benefit of significance filtering persists under different Rashomon parameters (for example, larger epsilon or many more searched models), since the reported effects are measured on small empirical sets.","If the complexity-multiplicity correlations reflect a causal relationship, then datasets with high L1/L2/L3 may need stronger filtering before any balancing is applied; this is an inference from the observed correlations, not a result the paper proves."],"forward_implications":["Balancing a dataset before training can buy accuracy at the price of wider prediction disagreement, so accuracy-based model selection on balanced data carries hidden arbitrariness.","Significance-test filtering, particularly alongside DBSMOTE or SLSMOTE, is a workable lever for shrinking predictive multiplicity without sacrificing the performance gains of balancing.","The safest balancing method depends on data complexity: RSLSMOTE and BLSMOTE are stable across complexity levels, while Near Miss and ADASYN are risky on high-complexity data.","Complexity metrics from the linearity family (L1, L2, L3) and the neighborhood metric N4 are positively associated with both discrepancy and obscurity, giving practitioners a way to anticipate multiplicity before training many models.","Filtering alone does not significantly change AUC in the experiments, so its demonstrated value is in stabilizing predictions rather than boosting accuracy."],"supporting_citations":[{"why":"Defines predictive multiplicity and supplies the discrepancy-style measurement logic that the paper adapts.","marker":"[2]"},{"why":"Prior study of the Rashomon effect of balancing methods that this paper extends, and the source of the obscurity metric.","marker":"[17]"},{"why":"Provides the imbalanced benchmark datasets and establishes that balancing methods alter model behavior.","marker":"[12]"},{"why":"The forester AutoML tool used to construct the empirical Rashomon sets with the chosen epsilon and search parameters.","marker":"[67]"},{"why":"Introduces SMOTE, the central balancing family that the paper evaluates along with its variants.","marker":"[30]"},{"why":"The Benjamini-Hochberg procedure used to control false discovery rate in both filtering methods.","marker":"[65]"},{"why":"The Wilcoxon rank sum test that defines the significance filtering method.","marker":"[66]"},{"why":"Introduces the two-cultures framing and the Rashomon effect that motivates the paper's research questions.","marker":"[1]"}],"fun_headline_variants":["Balancing data inflates model disagreement; filtering tames it","Data balancing widens Rashomon set, filtering narrows it","Preprocessing: balancing boosts predictive multiplicity, filtering cuts it","On 21 datasets, balancing increases model conflict, filtering reduces it","Balancing hurts, filtering helps: a data-centric view of multiplicity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The empirical Rashomon set produced by the AutoML search (epsilon 0.05, five Bayesian iterations, ten random evaluations) is treated as representative of all equally accurate models, and every discrepancy and obscurity comparison is computed on that small sample.","fun_headline_variants_meta":{"raw":{"variants":["Balancing data inflates model disagreement; filtering tames it","Data balancing widens Rashomon set, filtering narrows it","Preprocessing: balancing boosts predictive multiplicity, filtering cuts it","On 21 datasets, balancing increases model conflict, filtering reduces it","Balancing hurts, filtering helps: a data-centric view of multiplicity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000879,"raw_usage":{"total_tokens":3810,"prompt_tokens":967,"completion_tokens":2843,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":583,"completion_tokens_details":{"reasoning_tokens":2756}},"tokens_in":583,"tokens_out":2843,"duration_ms":20169,"temperature":1.0,"reasoning_tokens":2756,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:48:29.602579+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the same 21 dataset and preprocessing combinations with a much larger model search, such as hundreds of random evaluations or exhaustive enumeration on the smallest datasets, and compare the resulting discrepancy and obscurity medians as well as the ranking of balancing methods. If the medians or rankings shift materially, the paper's comparative claims are artifacts of the small Rashomon sample rather than properties of the datasets.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines predictive multiplicity and supplies the discrepancy-style measurement logic that the paper adapts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the imbalanced benchmark datasets and establishes that balancing methods alter model behavior."},{"cited_title":"forester: A Tree-Based AutoML Tool in R","cited_arxiv_id":"2409.04789","evidence_quote":"The forester AutoML tool used to construct the empirical Rashomon sets with the chosen epsilon and search parameters."},{"cited_title":"V., Bowyer, K","cited_arxiv_id":null,"evidence_quote":"Introduces SMOTE, the central balancing family that the paper evaluates along with its variants."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Benjamini-Hochberg procedure used to control false discovery rate in both filtering methods."},{"cited_title":"In: Breakthroughs in Statistics: Methodology and Distribution , pp","cited_arxiv_id":null,"evidence_quote":"The Wilcoxon rank sum test that defines the significance filtering method."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the two-cultures framing and the Rashomon effect that motivates the paper's research questions."}],"review_version":1}