{"id":"f66d2317-b8d7-4c56-a799-20ba3299ef1f","arxiv_id":"1908.03595","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":8,"one_line_summary":"AER is a dynamic ensemble method that reports improved balanced accuracy on seven UCI imbalanced datasets and five GMM-generated variants, with a theoretical complexity claim that is not correctly proved.","lead":"The paper proposes AER, an ensemble method for binary imbalanced classification that combines GMM-based resampling, SGD-trained weights, and test-time likelihood interpolation. It reports accuracy gains over several baselines and claims better time and memory complexity, but the proofs and result tables contain serious errors.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Empirical results are internally inconsistent: Table 8 duplicates Table 4 values across different datasets, so the claimed performance superiority is unsupported.","rationale":"The paper advertises two main contributions: an empirically superior classifier and a theoretical complexity advantage. Of these, the empirical performance claim is the central one, as it appears first in the abstract and drives the practical relevance of AER-XGBoost. The reported tables contain impossible entries: Table 8 reuses the exact recall values from Table 4's Bioassay rows for a GMM-generated dataset, and within Table 8 the log and exp dynamic ensembles have identical F1/G-mean despite different TP-FP ratios. Such internal inconsistencies cannot be explained by tuning or dataset variation; they indicate that the experimental evidence is not trustworthy. The statistical significance claim is equally compromised by mislabeled test statistics and impossible p-values. The reader's weakest_assumption focused on the Section 4 complexity proof, which is also seriously flawed (the step O(mnLt2)=O(mLt2/R) is dimensionally wrong, and the t2 bound does not hide the term). However, the data-integrity issue is more load-bearing because it directly invalidates the abstract's primary claim; even a corrected complexity proof would not salvage a paper with unreliable empirical tables. Since the reader already recommended REJECT and our analysis independently reinforces that conclusion, the verdict should remain unchanged.","tokens_in":31617,"tokens_out":9020,"duration_ms":84005,"concrete_test":"Recompute the F1 and G-mean of the Dynamic Ensemble (log) and (exp) rows in Table 8 from the reported minority/majority recalls and the TP-FP ratio definition in Eq. 22. If the two variants produce identical F1 and G-mean despite different TP-FP ratios, the numbers cannot both be correct, settling the concern that the empirical results are internally inconsistent. This check requires only arithmetic from the published tables and is decisive.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that AER-XGBoost 'outperforms the major existing algorithms based on multiple metrics in most cases, and two hypothesis tests verify the statistical significance further' is not supported by the reported experimental evidence. Table 8 (9-center GMM data) lists Dynamic Ensemble (log) as [83.33% minority recall, 0.47% majority recall, 0.8373 TP-FP, 0.0132 F1, 0.0195 G-mean, 48.35% balanced accuracy] and Dynamic Ensemble (exp) as [83.33%, 78.19%, 3.8225, 0.0132, 0.0195, 48.35%]. The first two rows are exactly the Plain XGBoost and Focal-loss XGBoost rows in Table 4 (Bioassay), and the Dynamic Ensemble rows in Table 4 are themselves identical to those same two rows. A dynamic ensemble on a GMM-generated dataset cannot reproduce the exact per-class recall values of a different dataset. Within Table 8, the log and exp variants share the same F1 and G-mean despite different TP-FP ratios, which is arithmetically impossible unless the numbers were copied. The statistical tests are also unreliable: Wilcoxon tables label the statistic as 'χ2' (e.g., Tables 7 and 12), p-values of exactly 0 appear, and McNemar rows report p<0.01 for chi2=4.17 (actual p≈0.04, Ecoli Table 11). These internal inconsistencies mean the abstract's headline performance assertion has no evidentiary basis. Section 7 further concedes that practical running time is 'usually longer than expected' and that XGBoost is not in the polynomial-time regime, so the theoretical complexity advantage also does not apply to the actual implementation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AER, an adaptive ensemble selection method for binary imbalanced classification. The method fits a GMM to the majority class, generates 2L subsets (global-geometry and local-geometry balanced subsets), trains XGBoost classifiers on each, learns static ensemble weights via SGD, and interpolates these with likelihood-based dynamic weights at test time. The authors claim that the method provides implicit regularization, reduces asymptotic time and memory complexity relative to vanilla implementations, and outperforms existing algorithms on seven UCI datasets and five GMM-generated datasets, with McNemar and Wilcoxon tests verifying significance. The central evidence is a complexity analysis in Section 4 and single-split experimental comparisons in Sections 5-6.","tokens_in":32039,"tokens_out":9699,"duration_ms":83986,"significance":"If the claims were fully supported, the paper would offer a practically relevant contribution: a regularization mechanism for dynamic ensemble selection that is both cheap and effective on high-imbalance problems. The algorithmic idea is clearly laid out, and the choice of modern baselines such as LightGBM and focal-loss XGBoost is reasonable. However, the manuscript's own evidence does not support the headline claims. The complexity improvement is derived under an unjustified iteration bound and contains an algebraic error, and the experimental tables contain internally inconsistent values; the statistical tests are also mislabeled and report erroneous p-values. The paper would need substantial rework of both the theory and the experiments before its contribution can be assessed.","major_comments":[{"comment":"The proof of Theorem 1 contains an algebraic error. In the stochastic-gradient-descent item, the text writes O(mnLt2) = O(mLt2/R), but multiplication by n cannot be replaced by division by R. Substituting the stated assumption t2 ≤ nR/L into O(mnLt2) gives O(m n² R), not the claimed O(mn), so the simplification of TAER(m,n) to O(m^a / L^{a-1} n^b) is not established. The assumption t2 ≤ nR/L is also not justified anywhere in the paper; t2 is the number of SGD iterations, and no bound on it is derived.","section":"§4.1, Theorem 1"},{"comment":"The Dynamic Ensemble rows in Table 8 are internally inconsistent and duplicate values from Table 4. The (log) row has the same minority recall (83.33%), majority recall (0.47%), and TP-FP ratio (0.8373) as the Plain XGBoost row in Table 4, and the (exp) row matches the Focal-loss XGBoost row on the same three metrics; a GMM-generated dataset would not be expected to reproduce these values. Within Table 8, the two Dynamic Ensemble rows share identical F1 (0.0132) and G-mean (0.0195) despite different majority recalls (0.47% vs 78.19%), contradicting the formulas in Eq. (23); the reported balanced accuracy 48.35% also does not equal the average of the listed recalls. These inconsistencies invalidate the experimental evidence for the GMM dataset and the abstract's claim of widespread outperformance.","section":"§6.3, Table 8"},{"comment":"The statistical tests are reported in a way that is not reliable. The Wilcoxon signed-rank test statistic is labeled 'χ2 statistics' in Tables 7, 12, 15, 18, 21, and 24, which is incorrect. More concretely, Table 6 reports p=0.02 for a McNemar χ2 of 54.0, whereas the one-degree-of-freedom chi-square p-value is many orders of magnitude smaller, and Table 11 reports p<0.01 for χ2=4.17, whose correct p-value is approximately 0.04. These errors mean the stated conclusion that 'two hypothesis tests verify the statistical significance' is not supported.","section":"§5.4 and Tables 6, 11"},{"comment":"All performance numbers are single-split point estimates. The datasets are split once into train/validation/test sets at a 3:1:1 ratio, and no standard deviations, confidence intervals, or repeated runs are reported. For the Bioassay dataset, the cost-sensitive SVM and decision-tree results are taken from [59] with an unknown split and unknown parameters, as the paper states in Section 5.2. Given the small test sets (e.g., Ecoli has 336 samples total), these comparisons cannot support the broad claim that AER-XGBoost outperforms existing algorithms 'in most cases'.","section":"§5.1–§5.2, §6"},{"comment":"The paper's own discussion undercuts the theoretical complexity claim for the implemented method. Section 7 states that XGBoost's time complexity is not in the polynomial-time regime assumed by Theorem 1, that the AER-XGBoost running time is 'usually longer than expected,' and that the experiments use full covariance matrices instead of the diagonal approximation on which the Section 4 analysis relies. Consequently, even if the algebraic issue in Theorem 1 were repaired, the complexity advantage would apply only to polynomial-time base classifiers with diagonal-covariance GMMs, not to the AER-XGBoost system actually evaluated.","section":"§7 vs §4"}],"minor_comments":[{"comment":"The cross-reference 'Experimental Framework and and results analysis are demonstrated in Section??' is broken and contains a duplicated 'and'.","section":"§1"},{"comment":"The caption misspells 'Parameters' as 'Parmaeters'.","section":"Table 25"},{"comment":"S(x) is described as a clipping function that floors at 0 and ceilings at 1, but the formula given is a min-max normalization, not a clip; the definition should match the use in Equations (13) and Algorithm 2.","section":"Equation (14)"},{"comment":"Several tables use '–' for undefined metrics without a consistent legend, making it unclear whether the value is 0, undefined, or omitted.","section":"Tables 4, 5, 8"},{"comment":"The F1 and G-mean definitions are stated for a single precision/recall pair, but the text earlier defines class-specific precisions and recalls; the paper should state explicitly whose precision and recall are used in the reported F1 and G-mean values.","section":"Equation (23)"}],"recommendation":"reject","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nThe headline you should know: the method is a real, if incremental, combination of existing ideas, but the paper's central claims don't survive contact with its own tables and proof. Table 8, which reports results on a 9-center GMM dataset, lists Dynamic Ensemble (log) and (exp) with the exact recall values of Plain XGBoost and Focal-loss XGBoost in Table 4 (Bioassay). Within Table 8, the two variants have identical F1 and G-mean despite different TP-FP ratios, which is arithmetically impossible under the metrics they define. The abstract's claim of outperforming major algorithms and significance tests has no evidentiary basis in these numbers.\n\nWhat the paper does well: the AER pipeline—GMM-based two-type resampling, SGD-learned static weights, and on-the-fly likelihood interpolation—is plausible and clearly described. The authors are candid in Section 7, admitting practical running time is longer than expected and that XGBoost doesn't fall into the polynomial-time regime their theory assumes. That honesty is creditable.\n\nThe soft spots beyond the tables: the statistical testing is sloppy. Wilcoxon statistics are labeled χ², several p-values are exactly 0, and Table 11 reports p<0.01 for chi²=4.17 on Ecoli, where the actual p is about 0.04. The complexity proof has a simple algebraic error: O(mnLt2) is not O(mLt2/R), and the assumption t2 ≤ nR/L is ad hoc. Both theorems' conclusions rest on that move.\n\nNet: the empirical and theoretical backbones both have load-bearing flaws. The method isn't worthless—it's a reasonable combination—but this version doesn't support the claims. I would desk-reject it. If the authors fix the tables, rerun the experiments, correct the proof, and tighten the statistics, the idea might deserve a proper review.\n\nBest,\n[Your name]","headline":"A new combination of resampling and weight interpolation with a clear write-up, but copied table entries and an algebraic proof error invalidate the headline claims—desk-reject this version.","tokens_in":852,"tokens_out":1533,"would_cite":false,"duration_ms":61835,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Regularized ensemble beats standard classifiers on skewed data","keywords":["adaptive ensemble","dynamic ensemble selection","imbalanced classification","regularization","stochastic gradient descent","Gaussian mixture model","XGBoost","binary classification"],"falsifier":"Check the proof's displayed identity in Section 4.1: with $t_2$ set to its upper bound $nR/L$, the SGD term $O(mnLt_2)$ becomes $O(mRn^2)$, not $O(mLt_2/R)$, so the hidden-term argument can be tested by substitution; alongside this, measure wall-clock training time of AER-XGBoost versus vanilla XGBoost as dataset size grows, since the paper itself observes that the AER version runs slower in practice.","tokens_in":70,"feed_emoji":"📊","tokens_out":5916,"duration_ms":141808,"temperature":0.7,"pith_summary":"The paper proposes AER, an adaptive ensemble of classifiers with regularization, for binary classification when one class is much rarer than the other. Its central claim is that dynamic ensemble selection can be made robust to overfitting by combining two regularizers: resampling the majority class through a Gaussian mixture model to preserve global geometry, and training the ensemble weights with stochastic gradient descent, which favors minimum-norm solutions. The authors argue that this regularization improves accuracy while also lowering asymptotic time and memory complexity compared with training one classifier on the full data. On seven UCI benchmark datasets and five GMM-generated variations, they report that AER-XGBoost outperforms standard and recent baselines on most metrics, with McNemar's and Wilcoxon tests supporting the difference in most cases.","feed_headline":"Regularized ensemble beats standard classifiers on skewed data","feed_subtitle":"A dynamic ensemble that blends global and local resampling with SGD regularization reports better accuracy on most benchmark sets.","key_machinery":"The load-bearing mechanism is implicit regularization through two routes. First, Gaussian mixture model (GMM) resampling creates two types of subsets: majority-dominated views that force classifiers to respect global geometry, and nearly balanced local views that focus on the minority; Tomek links then remove noisy overlaps from the global subsets. Second, SGD training of the ensemble weights over cross-entropy loss converges to a minimum-norm solution, and the final prediction weight for each test instance interpolates the learned weight vector with the instance's log- or exponential likelihood under each Gaussian component. The complexity argument decomposes the total training cost into GMM fitting, individual classifier training, SGD, and validation, and claims that under a bound on SGD iterations the dominant term becomes $O(m^a L^{1-a} n^b)$, which is asymptotically better than the original $O(m^a n^b)$ when $a \\geq 2$.","core_discovery":"On the paper's own terms, the discovery is that a dynamic ensemble can be regularized implicitly without sacrificing its adaptive power. AER fits a Gaussian mixture model to the majority class, builds 2L balanced sub-datasets that mix global and local views, trains one XGBoost classifier per subset, learns fixed ensemble weights by SGD on cross-entropy loss, and then interpolates those weights with per-test-point likelihoods. The result is a method that, in the authors' experiments, keeps high recall on the minority class while retaining decent majority recall, and performs especially well on highly imbalanced data with complex decision boundaries. The paper also claims that the AER framework reduces the asymptotic training complexity of super-linear classifiers and strictly reduces memory complexity for any polynomial-time classifier.","pith_inferences":["Editorial inference: if the SGD minimum-norm effect is the active regularizer, then swapping SGD for an explicitly norm-constrained optimizer should produce comparable accuracy, and a controlled comparison would isolate the mechanism.","Editorial inference: on minority classes with multiple separated modes, per-Gaussian classifiers should matter more, so AER's advantage over vanilla XGBoost should grow with the number of minority modes; this is testable with GMM-generated data.","Editorial inference: the asymptotic memory bound is the more robust theoretical claim because it does not depend on the SGD iteration bound; even if the time-complexity theorem fails, the strict memory advantage may survive."],"forward_implications":["Dynamic ensemble methods can be regularized by design rather than by explicit norm penalties, opening a direction for overfitting control in classifier selection.","For classifier families whose training time grows super-linearly in the number of instances, wrapping them in AER is claimed to reduce asymptotic training cost, with the benefit growing with the order of m.","AER is most advantageous on data with high imbalance ratios and complex minority-class boundaries; on mildly imbalanced benchmarks its gains shrink and some recent baselines remain competitive.","The exponential-likelihood variant concentrates weights more sharply, so it tends to do better when training/validation and test distributions are similar, while the log-likelihood variant is safer when the test distribution is less predictable.","Regularization from interpolating learned and likelihood-based weights yields better balanced accuracy than either fixed weights or pure dynamic weighting alone."],"supporting_citations":[{"why":"XGBoost is both the base classifier used inside AER and the primary baseline it is compared against.","marker":"[15]"},{"why":"Defines the focal loss neural network baseline that fails on several benchmarks and motivates the comparison.","marker":"[16]"},{"why":"Provides the focal-loss XGBoost baseline, AER's strongest non-AER competitor on several datasets.","marker":"[17]"},{"why":"LightGBM serves as a recent gradient-boosting baseline for comparison.","marker":"[18]"},{"why":"Supplies the dynamic ensemble selection framework and motivation that AER extends.","marker":"[5]"},{"why":"Gaussian mixture modeling is the resampling tool used to preserve global geometry.","marker":"[9]"},{"why":"Supports the claim that SGD converges to a minimum-norm solution, the implicit regularizer for the ensemble weights.","marker":"[47]"},{"why":"Tomek links are used to remove noisy overlapping instances from the global subsets.","marker":"[56]"},{"why":"Supplies the reported Bioassay results for the cost-sensitive SVM and decision tree baselines.","marker":"[59]"}],"fun_headline_variants":["Regularized dynamic ensemble tames skewed data","Ensemble with implicit regularization wins on imbalanced sets","AER: Regularized ensemble for imbalanced data","Dynamic ensemble with SGD regularization beats skewed baselines","SGD-based ensemble regularizes itself for rare classes"],"cache_read_input_tokens":34560,"weakest_assumption_plain":"The theoretical speed-up rests on assuming the ensemble weight training stops after very few iterations and on a step in the proof that may be algebraically wrong; if either gives way, the claimed complexity advantage is not established.","fun_headline_variants_meta":{"raw":{"variants":["Regularized dynamic ensemble tames skewed data","Ensemble with implicit regularization wins on imbalanced sets","AER: Regularized ensemble for imbalanced data","Dynamic ensemble with SGD regularization beats skewed baselines","SGD-based ensemble regularizes itself for rare classes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000766,"raw_usage":{"total_tokens":3398,"prompt_tokens":947,"completion_tokens":2451,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":2378}},"tokens_in":563,"tokens_out":2451,"duration_ms":17888,"temperature":1.0,"reasoning_tokens":2378,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:09:15.695634+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Check the proof's displayed identity in Section 4.1: with $t_2$ set to its upper bound $nR/L$, the SGD term $O(mnLt_2)$ becomes $O(mRn^2)$, not $O(mLt_2/R)$, so the hidden-term argument can be tested by substitution; alongside this, measure wall-clock training time of AER-XGBoost versus vanilla XGBoost as dataset size grows, since the paper itself observes that the AER version runs slower in practice.","supporting_citations":[{"cited_title":"Xgboost: A scalable tree boosting system","cited_arxiv_id":null,"evidence_quote":"XGBoost is both the base classifier used inside AER and the primary baseline it is compared against."},{"cited_title":"Focal loss for dense object detection","cited_arxiv_id":null,"evidence_quote":"Defines the focal loss neural network baseline that fails on several benchmarks and motivates the comparison."},{"cited_title":"Imbalance-xgboost: Leveraging weighted and focal losses for binary label-imbalanced classification with xgboost.Pattern Recognition Letters, 2020","cited_arxiv_id":null,"evidence_quote":"Provides the focal-loss XGBoost baseline, AER's strongest non-AER competitor on several datasets."},{"cited_title":"Lightgbm: A highly efficient gradient boosting decision tree","cited_arxiv_id":null,"evidence_quote":"LightGBM serves as a recent gradient-boosting baseline for comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the dynamic ensemble selection framework and motivation that AER extends."},{"cited_title":"Iterative regularization for learning with convex loss functions","cited_arxiv_id":null,"evidence_quote":"Supports the claim that SGD converges to a minimum-norm solution, the implicit regularizer for the ensemble weights."},{"cited_title":"An experiment with the edited nearest-neighbor rule.IEEE Transactions on Systems, Man, and Cybernetics, 1976","cited_arxiv_id":null,"evidence_quote":"Tomek links are used to remove noisy overlapping instances from the global subsets."},{"cited_title":"Virtual screening of bioassay data.Journal of cheminformatics, 1(1):21, 2009","cited_arxiv_id":null,"evidence_quote":"Supplies the reported Bioassay results for the cost-sensitive SVM and decision tree baselines."}],"review_version":1}