{"id":"96ba5798-a2c5-4099-bed0-fab95443f33a","arxiv_id":"2506.11315","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A heuristic bilevel optimization wrapper around SVM-SMOTE is claimed to improve minority-class F1 by selecting training samples that increase model-output variance and reduce overlap.","lead":"MOODS is a resampling method for imbalanced binary classification that repeatedly trains a model and keeps or discards candidate training samples based on validation F1. The paper also introduces a metric based on model output spread and claims it explains 1 to 15 percent F1 gains on seven datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The epsilon/delta metric rests on an unvalidated and partly circular 1-D proxy: if the scalar logit does not faithfully represent feature-space diversity, the paper's causal claim that diversity drives the 1-15% F1 gain has no support.","rationale":"The strongest claim in the paper has two parts: (1) MOODS improves F1 over strong baselines, and (2) the epsilon/delta metric demonstrates that this improvement is driven by diversity and reduced overlap. The second part is what elevates the paper beyond yet another sampling heuristic, and it rests entirely on the validity of the 1-D proxy. The paper provides no independent validation of this proxy, and the Def. 4 overlap measure is partly circular because it counts misclassified minority points, exactly what the upper-level objective optimizes. The variance measure is also not obviously tied to feature-space diversity. The paper's own admission that convergence and Pareto optimality are unconfirmed (Sec. 3.1) compounds the problem: the 'optimal training set' language is not supported even by the authors' experiments. A controlled synthetic study with known feature-space diversity and overlap would settle whether the metric measures what it claims; until then, the causal claim is unsubstantiated. This aligns with the reader's weakest-assumption analysis. The F1 improvements themselves may be real, but the central explanatory contribution is not demonstrated, so the REJECT verdict remains appropriate.","tokens_in":14130,"tokens_out":4498,"duration_ms":53966,"concrete_test":"Construct a synthetic binary dataset with known class-conditional Gaussian mixtures, and independently vary the true feature-space diversity (e.g., class covariance) and overlap (e.g., distance between means). Train the fixed architecture on these controlled datasets and compute the Def. 4-6 epsilon/delta values. If the metric does not move monotonically with the controlled feature-space quantities, or if it changes when only training epochs or batch size are varied while the feature distribution is fixed, then the 1-D proxy is invalid and the causal interpretation of Table 2 is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"MOODS's central empirical claim is that the reported 1-15% F1 improvement is driven by increased diversity and decreased overlap, as demonstrated by the epsilon/delta metric. This depends on the assertion in Sec. 4.1 that the scalar output zw*(x) is a faithful 1-D analog of the feature vector x, but no evidence supports this assertion. The overlap component (Def. 4) counts minority training points whose zw* <= 1/2, i.e., points the trained model misclassifies. Since the upper-level objective (Prob. 1) directly maximizes minority and overall F1, any successful F1 improvement reduces this count by construction; the metric therefore tracks the very quantity it is meant to explain, independent of any real change in feature-space overlap. The variance component (Def. 5) uses the spread of scalar logits, which can increase with training dynamics, batch statistics, or softmax saturation even when the input feature distribution is unchanged. The thresholds (26/1.7, 52/0.7) are chosen post hoc in Sec. 5.3, so the S-epsilon/delta labels describe the observed runs rather than test a prediction. The paper itself states in Sec. 3.1 that convergence and Pareto optimality are not confirmed, further weakening the 'optimal training set' claim. If the 1-D proxy is invalid, the causal link between diversity and F1 collapses, leaving only unexplained F1 improvements.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes MOODS, a multi-objective bilevel optimization algorithm for resampling imbalanced training data: an outer loop selects synthetic minority samples (from SVM-SMOTE) and undersamples majority data by optimizing two validation F1 objectives, while an inner loop trains a neural network. The authors also introduce an 'epsilon/delta non-overlapping diversification metric' that measures overlap and diversity of model outputs z_w*(x), and use it to argue that improvements in diversity and overlap drive a 1-15% F1 improvement over state-of-the-art baselines on seven datasets. The paper includes public code and honest statements of limitations, but the central causal claim rests on the validity of the scalar-output proxy.","tokens_in":14413,"tokens_out":5500,"duration_ms":55645,"significance":"If the claims were established, MOODS would be a practical contribution to imbalanced classification, combining bilevel optimization with sampling in a way that could be extended to other objectives. The paper provides a concrete algorithm, evaluates on seven benchmark datasets, and makes code available. However, the contribution's key novelty--the epsilon/delta metric as a validated measure of sampling quality--is not established: the metric is partly circular, thresholds are post hoc, and the proxy from scalar outputs to feature-space diversity is unvalidated. The authors also explicitly state that convergence of the upper-level problem is not confirmed. These issues undermine the paper's main claims.","major_comments":[{"comment":"The overlap measure cSm := {s=(x,1) | z_w*(x) <= 1/2} is exactly the set of minority points misclassified by the trained model under Def. 2. Because Prob. (1) directly maximizes F1_m, any successful optimization reduces this count by construction. Thus the reported decreases in overlap (Table 2, columns Delta(kappa_m)) are not independent evidence of improved feature-space diversity; they are a restatement of the objective being optimized. The causal claim that diversity drives the F1 gains is therefore unsupported by this metric.","section":"Sec. 4.1, Def. 4"},{"comment":"The authors state: 'we did not find a Pareto front so have not yet confirmed convergence nor that the objective is being achieved.' Yet the abstract, Sec. 1, and Sec. 3 refer to 'constructing an optimal training set' and identifying a 'single Pareto optimal point.' Without convergence confirmation or a characterization of the produced point relative to the Pareto set, the optimality claim is not justified and should be either removed or substantially weakened.","section":"Sec. 3.1, Algorithm 1"},{"comment":"No error bars or significance tests are reported for the F1 comparisons. The text reports MOODS variances (2.58e-6 to 2.37e-4) but not those of baselines, and the '1-15% improvement' claim is a point estimate comparison. On Spambase MOODS ties at 0.92 but is 1% below GBO/SSG; without confidence intervals or paired tests it is unclear which differences are meaningful. The table should include standard errors or confidence intervals, and the comparison to SMOTified-GAN uses published results rather than runs on the same infrastructure, which should be flagged as a limitation.","section":"Sec. 5.2, Table 2"},{"comment":"The epsilon/delta thresholds are not specified before evaluation; the paper reports 'We attained five S^{26/1.7} training sets' and 'four S^{52/0.7} ultimate training sets.' These thresholds are chosen post hoc from the observed values (e.g., Table 4), which makes the S^{epsilon/delta} classification descriptive rather than a testable prediction. To be a valid metric, the thresholds should be fixed a priori or derived from an independent criterion.","section":"Sec. 5.3, Def. 6"},{"comment":"The assertion that the scalar output z_w*(x) is 'a 1-D analog' to the feature vector x is not validated. The variance of scalar logits can increase from training dynamics, softmax saturation, or batch statistics even when the input feature distribution is unchanged. No experiment is provided that correlates output-space variance/overlap with feature-space diversity (e.g., using a held-out feature-space diversity measure). Until such validation is provided, the metric's variance component does not support the conclusion that MOODS improves feature diversity.","section":"Sec. 4.1, Def. 5"}],"minor_comments":[{"comment":"The cross-entropy expression for the minority loss uses e^{z_w(x)}/(e^{z_w(x)}+e^{1-z_w(x)}), which corresponds to a two-class softmax with logits z and 1-z, not the standard binary cross-entropy with logit z. This unusual parameterization should be justified or corrected.","section":"Sec. 2, Def. 1"},{"comment":"The rejection step (line 21) halves p(s) for s in \\hat{S}_m^k, but p(s) was initialized over majority training data S_T^M and used to sample majority points in line 5. The role of p(s) for synthetic minority samples is unclear.","section":"Sec. 3.1, Algorithm 1"},{"comment":"The comparison with SMOTified-GAN uses published results instead of running the method on the same infrastructure; this should be acknowledged in the experimental setup and considered when interpreting Table 2.","section":"Sec. 5.1"},{"comment":"The phrase 'Spambase's low F1 scores' is misleading, since 0.92 is among the highest reported in Table 2; the intended point is that MOODS did not surpass the best baseline.","section":"Sec. 5.2"},{"comment":"The typo 'optmization' appears in the paragraph describing the multi-objective bilevel approach.","section":"Sec. 3.1"},{"comment":"The notation z_w* = z_w*(x) is introduced but not used consistently; later definitions use z_w* without specifying dependence on x.","section":"Sec. 4.1"}],"recommendation":"reject","confidential_remarks":"The paper's core metric is circular with respect to the optimized objective, and the authors' own admission that convergence is unconfirmed undermines the 'optimal training set' claim. The empirical F1 improvements may be real but are presented without error bars. These issues are central and would require substantial new validation work; I do not see a way to repair the manuscript within a routine revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: MOODS is a plausible, GAN-free resampling heuristic that extends the authors' earlier MUBO to a multi-objective upper level, and it reports competitive F1 numbers on seven imbalanced datasets with public code. The new epsilon/delta metric is genuinely new, but the paper's own text (Sec. 3.1) admits convergence and Pareto optimality are not confirmed, and the metric as defined rests on an unvalidated 1-D proxy. The stress-test note holds up.\n\nWhat the paper does well: the bilevel formulation is clean, the algorithm is well-specified, and the authors are transparent about the lack of convergence and the post hoc nature of the thresholds. Comparisons include relevant baselines (SMOTE, SVM-SMOTE, GBO, SSG, MUBO, SMOTified-GAN), and the histograms give qualitative support for the claim that resampled training sets look better separated.\n\nThe soft spots are load-bearing. First, the metric's overlap component counts minority points with z_w* <= 1/2, which are exactly the points the trained model misclassifies. Since the outer loop maximizes minority F1, any successful run reduces that count by construction; the overlap measure is a different view of the objective, not independent evidence for diversity. Second, the variance component uses logit spread, which can increase with softmax saturation or batch statistics without any real change in feature-space diversity. Third, the thresholds (26/1.7, 52/0.7) are chosen after seeing the results, so the S-epsilon/delta labels describe the runs; they do not constitute a testable prediction. Fourth, Table 2 shows no error bars for the baselines, and the SMOTified-GAN numbers are taken from the published paper rather than rerun. The paper does report variances for MOODS's own F1 scores, but not for the comparisons.\n\nNone of this makes the method worthless. The F1 improvements are plausible, and a revised version that (i) validates the metric against feature-space measures like nearest-neighbor overlap or actual variance in x, (ii) reports error bars for all methods over the five seeds, and (iii) softens the causal claim to 'improved F1 coincides with increased logit separation' would be a solid contribution.\n\nAs submitted, the core empirical claim—that diversity drives the 1-15% gain—is not supported. I would still send this to reviewers: the method is novel enough, the code is public, and the flaws are identifiable and fixable. A serious referee could turn this into a useful paper.","headline":"A plausible GAN-free resampling heuristic with public code, but the new epsilon/delta metric is an unvalidated proxy and the causal claim about diversity driving F1 gains is not supported; worth sending to reviewers for a major revision.","tokens_in":14977,"tokens_out":2417,"would_cite":false,"duration_ms":27116,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MOODS treats data sampling as a bilevel F1 maximization problem and reports 1-15% F1 gains over state-of-the-art samplers on seven imbalanced datasets.","keywords":["class imbalance","bilevel optimization","multi-objective optimization","data sampling","SVM-SMOTE","F1 score","class overlap","data diversity"],"falsifier":"Train MOODS on a dataset constructed so that the model's scalar output is nearly constant or dominated by one feature while meaningful diversity lies in orthogonal feature directions; if $\\epsilon/\\delta$ still improves but F1 does not, the proxy fails and the claimed mechanism is wrong. Alternatively, if a method that actively worsens $\\epsilon/\\delta$ still matches MOODS's F1, then diversity is not the driver.","tokens_in":13890,"feed_emoji":"⚖️","tokens_out":5741,"duration_ms":46607,"temperature":0.7,"pith_summary":"This paper tries to show that fixing class imbalance is best done by choosing the training set itself, not just reweighting losses or duplicating points. It proposes MOODS, a multi-objective bilevel optimization loop in which an outer problem selects training samples to maximize both minority-class and overall F1 on validation data, while an inner problem trains the model weights on the current sample. To make diversity and overlap measurable, the paper introduces an $\\epsilon/\\delta$ non-overlapping diversification metric that compares scalar model outputs before and after resampling. On seven benchmark datasets, MOODS reaches F1 scores 1-15% higher than state-of-the-art sampling baselines, with the metric pointing to reduced minority overlap and increased output variance as the driver.","feed_headline":"Sampling that maximizes two F1 scores lifts imbalanced-data results","feed_subtitle":"MOODS selects training sets by minority and overall F1; diversity gains drive 1-15% improvement.","key_machinery":"The carrying mechanism is the coupled pair: the MOODS bilevel loop (Algorithm 1) and the $\\epsilon/\\delta$ non-overlapping diversification metric. In MOODS, the outer loop plays the leader choosing which samples enter training, and the inner loop plays the follower that fully trains the network on those samples; acceptance is decided by simultaneous improvement in both validation F1 objectives. The $\\epsilon/\\delta$ metric compares the trained network's scalar outputs $z_{w^*}(x)$ for the original imbalanced data and the resampled set: minority overlap is the fraction of minority outputs on the majority side of $z_{w^*} = 1/2$, and diversity is the order-of-magnitude increase in output variance. A set is declared $S^{\\epsilon/\\delta}$ when overlap falls by at least $\\epsilon$ percentage points and variance rises by at least $\\delta$ orders of magnitude.","core_discovery":"The central claim is that a training set can be optimized directly for imbalanced classification by treating sampling as a bilevel program: the upper level minimizes $(1 - F1_m)$ and $(1 - F1)$ over candidate training subsets built from original majority points, original minority points, and SVM-SMOTE-generated synthetic minority points, subject to the lower-level condition that the model weights minimize binary cross-entropy loss on that subset. MOODS walks through this space by adding or removing one point at a time and accepting a step only when both F1 scores on disjoint validation data improve. In the paper's experiments this yields balanced training sets, and the accompanying $\\epsilon/\\delta$ metric shows minority model outputs moving to the correct side of the $z_{w^*} = 1/2$ boundary and output variance rising by roughly 0.7 to 3.1 orders of magnitude, which the paper links to the 1-15% F1 gains.","pith_inferences":["The paper's causal story, that diversity drives F1, is only as strong as the 1-D proxy; a direct test would compute $\\epsilon/\\delta$ using feature-space distances or a second independent model.","If the proxy holds, the $\\epsilon/\\delta$ metric could serve as a cheap validation signal during training, letting samplers stop when overlap and variance targets are met rather than after full retraining.","The paper reports that it has not yet established convergence of the upper-level problem or a Pareto front; a formal convergence guarantee for the greedy accept/reject scheme would be the natural next step.","Spambase, the least imbalanced dataset, is the one where MOODS does not lead, suggesting that the method's gains concentrate where minority overlap is severe, a testable prediction for other mildly imbalanced datasets."],"forward_implications":["If MOODS is right, sampling methods should be judged by whether they reduce minority overlap and raise output diversity, not by how many synthetic points they generate.","The bilevel formulation gives a template for optimizing other data-level choices, such as which features to keep, by swapping the upper-level objective.","The $\\epsilon/\\delta$ metric can be applied to any pair of training sets, giving researchers a common yardstick to compare sampling algorithms beyond F1.","Because synthetic data are generated by SVM-SMOTE and then filtered by F1, MOODS inherits SMOTE's boundary-focused proposals but discards those that fail to improve validation performance."],"supporting_citations":[{"why":"Supplies the bilevel undersampling baseline that MOODS extends with a multi-objective upper level and synthetic minority data.","marker":"[34]"},{"why":"SVM-SMOTE is the generator of synthetic minority candidates that MOODS filters through its F1-based accept/reject loop.","marker":"[35]"},{"why":"Provides the GBO and SSG state-of-the-art baselines that MOODS is compared against.","marker":"[1]"},{"why":"SMOTified-GAN is a state-of-the-art baseline whose published results are used in the comparison table.","marker":"[39]"},{"why":"Supplies the seven benchmark datasets used for evaluation.","marker":"[22]"},{"why":"Adam is the optimizer used in the inner-loop training of the neural network.","marker":"[24]"},{"why":"Defines the binary cross-entropy loss that appears in the lower-level training problem.","marker":"[5]"}],"fun_headline_variants":["Optimizing sampling via bilevel multi-objective boosts F1","MOODS: Training set optimization lifts minority F1 gains","Bilevel optimization for data sampling improves imbalanced classes","Directly optimize sampling to raise both F1 scores","New metric guides sampling that drives F1 improvements"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the network's scalar output $z_{w^*}(x)$ is a faithful 1-D stand-in for the high-dimensional feature vector $x$, so that variance and overlap of these outputs measure the diversity and overlap of the actual training data.","fun_headline_variants_meta":{"raw":{"variants":["Optimizing sampling via bilevel multi-objective boosts F1","MOODS: Training set optimization lifts minority F1 gains","Bilevel optimization for data sampling improves imbalanced classes","Directly optimize sampling to raise both F1 scores","New metric guides sampling that drives F1 improvements"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000679,"raw_usage":{"total_tokens":3071,"prompt_tokens":917,"completion_tokens":2154,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":2075}},"tokens_in":533,"tokens_out":2154,"duration_ms":15433,"temperature":1.0,"reasoning_tokens":2075,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:11:30.197581+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train MOODS on a dataset constructed so that the model's scalar output is nearly constant or dominated by one feature while meaningful diversity lies in orthogonal feature directions; if $\\epsilon/\\delta$ still improves but F1 does not, the proxy fails and the claimed mechanism is wrong. Alternatively, if a method that actively worsens $\\epsilon/\\delta$ still matches MOODS's F1, then diversity is not the driver.","supporting_citations":[{"cited_title":"A Bilevel Optimization Framework for Imbalanced Data Classification","cited_arxiv_id":"2410.11171","evidence_quote":"Supplies the bilevel undersampling baseline that MOODS extends with a multi-objective upper level and synthetic minority data."},{"cited_title":"Nguyen, Eric W","cited_arxiv_id":null,"evidence_quote":"SVM-SMOTE is the generator of synthetic minority candidates that MOODS filters through its F1-based accept/reject loop."},{"cited_title":"Enhancing and improving the performance of imbalanced class data using novel GBO and SSG: A comparative analysis","cited_arxiv_id":null,"evidence_quote":"Provides the GBO and SSG state-of-the-art baselines that MOODS is compared against."},{"cited_title":"SMOTified-GAN for Class Imbal- anced Pattern Classification Problems","cited_arxiv_id":null,"evidence_quote":"SMOTified-GAN is a state-of-the-art baseline whose published results are used in the comparison table."},{"cited_title":"The uci machine learning repository","cited_arxiv_id":null,"evidence_quote":"Supplies the seven benchmark datasets used for evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the binary cross-entropy loss that appears in the lower-level training problem."}],"review_version":1}