{"id":"ca76d104-f295-4a24-89fb-902891457a19","arxiv_id":"2412.13439","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A MIP-based ensemble weighting scheme jointly selects K classifiers and assigns per-class weights; it reports average balanced accuracy gains of 4.53% over six baselines on four imbalanced CPS datasets.","lead":"This paper proposes a mixed integer programming (MIP) method to assign per-class weights to an ensemble of classifiers, selecting a desired number of classifiers while optimizing their weights. In tests on four imbalanced datasets from cyber-physical systems, it reports consistent but modest improvements in balanced accuracy (0.99% to 7.31%) over six existing weighting schemes.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 0.99–7.31% gains rest on a single 80/20 split with no error bars; with minority classes as small as 11 instances, validation-matrix noise can dominate the comparison, so the central outperformance claim is not yet established.","rationale":"The stress-test pass identifies the evaluation protocol as the load-bearing weakness. The central claim has two parts: the method optimally solves a well-defined MIP, and it outperforms six baselines across four datasets. The first part is internally sound given the stated objective, modulo the inert L1 term: because constraint (5) enforces sum_i w_ij = 1 for each class, the L1 penalty sum_i sum_j w_ij equals m identically, so it cannot promote sparsity or affect the solution. This is a real flaw in the 'elastic net' description and should be corrected, but it does not by itself make the empirical results void. The second part is threatened by the absence of repeated experiments. The validation matrix V is estimated from 5-fold CV on an 80% split, and for minority classes with 11–74 instances the per-class accuracy is extremely noisy; the MIP then optimizes against this noisy V. Since only one 80/20 split is used, the reported improvements could be a favorable draw. No error bars or statistical tests are given, and the tuning procedure for lambda and alpha is not precisely specified. A repeated-split or repeated-CV experiment would settle whether the improvement is systematic. The reader's weakest_assumption already identified the same mechanism, so agreement is 'agree'. The appropriate verdict remains CONDITIONAL: the method is plausible and the code is available, but the empirical claim requires additional validation.","tokens_in":15522,"tokens_out":8652,"duration_ms":79622,"concrete_test":"Run the full pipeline (training/validation, MIP weight calculation, and test evaluation) on 10 or more independent 80/20 stratified splits of D1–D4, or a repeated 5x2 CV; compute the paired difference in balanced accuracy (MIP minus best baseline) per split and report the mean with a 95% bootstrap confidence interval and a Wilcoxon signed-rank test. Additionally, for the rarest classes (D2-A3, D1-F4), record per-class recall per split. If the confidence interval for the average improvement over the best baseline includes zero, or the per-class advantage is not consistent across splits, the claimed 4.53% average improvement is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is that the empirical claim of consistent improvement is based on a single 80/20 split with no repeated trials, error bars, or significance tests. The weights are optimized against the mean validation accuracy matrix V (objective (1)), whose entries for rare classes are high-variance estimates: D2-A3 has 11 instances, so each fold of the 5-fold CV has ~2 instances, and D1-F4 has 74 instances. A small change in V for these classes can change the optimal weights and the selected classifier set, and the test-set balanced accuracy for these classes is itself a coarse Bernoulli-like quantity (the D2 test set contains ~2 A3 instances). Consequently, the reported improvements of 0.99–7.31% (avg 4.53%) may lie within split-to-split noise. The paper reports no confidence intervals and no paired statistical test. This matters more than the (correct) observation that the L1 penalty is inert because constraint (5) fixes sum_i sum_j w_ij = m; that flaw mis-describes the regularization but does not by itself overturn the empirical comparison. The missing repeated-split evidence directly threatens the abstract's central outperformance claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a mixed integer programming (MIP) approach to assign per-class weights to classifiers in a majority-voting ensemble for imbalanced multi-class rare-event detection. Given a validation accuracy matrix V obtained from stratified cross-validation, the method maximizes per-class weighted accuracy with an elastic-net penalty while selecting exactly K classifiers through binary variables. The authors compare the method against six baseline weighting schemes on four datasets (LeakDB, NSL-KDD, SG-MITM, CIC-IDS2017) for ensemble sizes K=2..8, reporting balanced-accuracy improvements of 0.99% to 7.31% (average 4.53%) and corresponding macro-averaged precision, recall, and F1 improvements, together with a computational efficiency study.","tokens_in":15856,"tokens_out":5630,"duration_ms":58211,"significance":"If the reported gains are robust, the paper offers a practically useful contribution: a compact, per-class weighted voting ensemble with explicit cardinality control, backed by a public code repository. The experimental scope is broad for this setting, covering four datasets, seven ensemble sizes, six baselines, and supplementary ablations, and the proposed optimization problem is well posed. The computational efficiency comparison is a clear strength. However, the central empirical claim currently rests on a single data split with no error bars or significance tests, and the advertised elastic-net regularization is mathematically mis-stated; these issues must be addressed before the results can be regarded as establishing the paper's main claims.","major_comments":[{"comment":"The L1 penalty in the objective is constant and therefore has no effect on the optimal solution. Constraint (5) imposes sum_i w_ij = 1 for every class j, so sum_i sum_j w_ij = m identically. Consequently, the term -lambda * alpha * sum_i sum_j w_ij contributes only a constant to the objective and cannot promote sparsity. The actual objective reduces to maximizing (1/m) sum_i sum_j w_ij v_ij - (lambda(1-alpha)/2) sum_i sum_j w_ij^2, i.e., an L2-regularized (ridge) objective. The claims in the Introduction, Abstract, and Related Work that the method 'utilizes elastic net regularization' and that the L1 penalty leads to a sparse and robust weight assignment are therefore not supported. Any sparsity in the solution comes from the hard cardinality constraint via the binary variables x_i, not from the L1 penalty. The authors should either correct the formulation to include a genuine L1 penalty (e.g., by not fixing the total weight sum) or revise the contribution claims to describe ridge regularization plus hard classifier selection.","section":"Problem Formulation, Eq. (1) and Constraint (5)"},{"comment":"The central outperformance claim is based on a single 80/20 split with no repeated trials, confidence intervals, or significance tests. This is especially problematic for the minority classes in Table 1: class F4 in D1 has 74 instances and class A3 in D2 has 11 instances, so the 5-fold validation accuracies for these classes are high-variance estimates and the test sets contain only a handful of instances (about two for A3 in D2). The reported gains of 0.99% to 7.31% in balanced accuracy could therefore lie within split-to-split noise. The authors should repeat the evaluation over multiple stratified splits (or use repeated k-fold cross-validation), report means and standard deviations of the improvements, and apply a paired statistical test across datasets and ensemble sizes (e.g., Wilcoxon signed-rank or at least a paired bootstrap). Without such evidence, the abstract's quantitative claim that MIP 'outperforms all existing approaches' is not established.","section":"Experimental Setup and Results & Discussion, Table 1"},{"comment":"Constraints (8) and (9) force the MIP solution to achieve, on the validation matrix V, a per-class and overall weighted accuracy that exceeds the uniform-weight average by at least epsilon. Because UW-PC and UW-PCC are two of the six baselines, this makes the validation-stage comparison with the uniform baselines partly loaded by construction: the MIP weights are required to beat the uniform baseline on the very matrix used to compute the weights. The test-time comparison is still out-of-sample, so this is not fatal, but it is a fairness concern. The authors should report an ablation without constraints (8)-(9), state how often these constraints are active, and discuss feasibility for small K. They should also give the numerical value of epsilon used in (7)-(9), rather than only saying it is 'sufficiently small.'","section":"Problem Formulation, Constraints (8) and (9)"},{"comment":"The tuning of lambda and alpha is described as incrementing/decrementing each parameter and 'terminating adjustments when performance began to deteriorate,' with no separate validation set and no sensitivity analysis. Since the same V is then used in the MIP objective and constraints, this procedure risks selecting hyperparameters that overfit V. The paper does not report whether the 4.53% average improvement is stable across a range of lambda and alpha values, or whether the conclusion depends critically on the four reported pairs. A sensitivity table or grid study would substantially strengthen the generalization claim.","section":"Experimental Setup, hyperparameter tuning"}],"minor_comments":[{"comment":"The notation E_N and E_A is introduced but never used anywhere else in the paper; it should be removed or put to use in the discussion.","section":"Proposed Weighting Scheme, Phase 1"},{"comment":"All results are reported as percentage improvements over baselines; the absolute balanced accuracy and macro-F1 values of the baselines are never given. Reporting absolute values in a supplementary table would let readers judge the practical magnitude of the gains.","section":"Results & Discussion, Tables 2-5 and Figures 2-5"},{"comment":"The text states that the problem is 'NP-hard' but gives no proof or reference for this specific formulation. Either provide a hardness argument or soften the wording to 'the MIP is solved by a general-purpose solver and may be computationally expensive in the worst case.'","section":"Computational Efficiency"},{"comment":"The constants M and epsilon are not specified numerically; the paper should state the chosen values or explain how they were selected, since both affect the feasibility and the strength of the constraints.","section":"Problem Formulation, Constraints (7)-(9)"},{"comment":"The sentence 'Without loss of generality, we used the mean validation accuracy matrix V as the likelihood matrix in BMA' is not a lossless reduction; using validation accuracy as a Bayesian posterior likelihood is a modeling choice and should be justified as such.","section":"Existing Weighting Schemes"}],"recommendation":"major_revision","confidential_remarks":"The paper's central idea is reasonable and the experimental work is substantial, but the empirical claim needs repeated-split evidence and the elastic-net claim needs to be corrected. The code availability is a plus. The main risk is that the reported improvements are within split noise; this is fixable with additional experiments rather than requiring a change of scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new piece here is the single MIP that jointly assigns per-class weights and selects exactly K classifiers, with code released on GitHub. That formulation is coherent, the experiments are extensive in breadth (4 datasets, 7 ensemble sizes, 6 baselines), and the consistent direction of the gains is at least suggestive. The paper also illustrates the per-class weighting behavior well with the SVM example on NSL-KDD. If the empirical claim held, this would be a useful tool for imbalanced multi-class CPS monitoring.\n\nBut the central claim is not yet established. The biggest problem is evaluation: every reported improvement comes from one fixed 80/20 split, with no repeated trials, no confidence intervals, and no paired significance tests. Minority classes are tiny (D2-A3 has 11 instances, so each validation fold has about 2), making the validation accuracy matrix V noisy and the test balanced accuracy a coarse quantity. The reported 0.99–7.31% improvements could easily be split-to-split noise. This is the load-bearing weakness, and it matters more than the other issues.\n\nThe other issues are real but more contained. The L1 penalty in the elastic net term is inert: constraint (5) fixes the sum of weights per class to 1, so the total sum of all weights is always m, making the alpha-weighted L1 term a constant. The method is ridge-regularized weight assignment, not elastic net, and the paper's methodological claim needs correcting. Constraints (8)–(9) also force the validation solution to beat the uniform-weight baseline by epsilon, which partially builds the UW-PC comparison into the optimization; the test results are still out-of-sample, but that baseline comparison is less informative than the paper implies. The tuning procedure for lambda and alpha is vague (\"incremented/decremented... terminating when performance began to deteriorate\") and it is unclear whether tuning used validation or test performance. The related work also misses earlier optimization-based ensemble selection, though that is a minor omission.\n\nWho should read this: people working on ensemble methods for imbalanced multi-class data, especially in CPS anomaly detection, will find the formulation worth knowing. It is not yet a reliable benchmark result. The code and experimental detail mean a serious referee could get the paper into shape, but only after the authors rerun with repeated stratified CV or multiple random splits, report paired tests and error bars, fix the regularization claim, and clarify the tuning protocol. I would send it to review rather than desk reject, but the revision request needs to be explicit about the single-split problem.","headline":"A legitimate MIP formulation for per-class ensemble weighting with reproducible code, but the headline empirical claim rests on a single 80/20 split with no error bars and a mislabeled elastic net penalty.","tokens_in":16299,"tokens_out":2946,"would_cite":false,"duration_ms":29951,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["90C11","68T05"],"pacs":[],"model":"deepseek-v4-flash","headline":"A single mixed-integer program can select the ensemble and set each class's voting weights, beating six standard schemes by an average of 4.53% in balanced accuracy.","keywords":["ensemble learning","weighted voting","mixed integer programming","class imbalance","rare event detection","cyber-physical systems","elastic net regularization","multi-class classification"],"falsifier":"Run the method and all six baselines on the same four datasets over many random 80/20 splits, re-optimizing the MIP weights inside each training fold; if MIP does not beat the best baseline in balanced accuracy on most splits—especially on the tiny minority classes such as F4 and A3—then the reported 4.53% average edge does not generalize.","tokens_in":15359,"feed_emoji":"🎯","tokens_out":12540,"duration_ms":101225,"temperature":0.7,"pith_summary":"The paper tries to establish that the weighting problem inside a voting ensemble—who votes, and how much each classifier's vote counts for each class—can be solved exactly as one mixed-integer program rather than by the usual heuristics. The method takes the mean per-class validation accuracy of each classifier, treats the weights as continuous decision variables and the choice of $K$ classifiers as binary decisions, and maximizes the mean weighted accuracy minus an elastic net penalty. Across four imbalanced datasets used for rare event detection in cyber-physical systems (water-distribution leaks and network intrusions), and ensemble sizes $K=2$ through $8$, the paper reports balanced-accuracy gains over six established weighting schemes ranging from $0.99\\%$ to $7.31\\%$, with an average of $4.53\\%$. If the claim holds, practitioners get a compact ensemble that is explicitly tuned for rare classes and whose weight calculation cost does not grow with dataset size.","feed_headline":"Per-class voting weights from one optimization beat six baselines","feed_subtitle":"Across four imbalanced datasets, balanced accuracy rises 0.99–7.31 percent, averaging 4.53 percent.","key_machinery":"The load-bearing object is the mixed-integer program with objective (1) and constraints (2)–(9). The decision variables are binary $x_i$ (whether classifier $C_i$ is in the ensemble) and continuous $w_{ij}$ (the vote weight of classifier $C_i$ for class $E_j$); constraint (4) fixes $\\sum_i x_i = K$, constraint (5) normalizes the weights of each class to sum to one, constraints (6)–(7) link selection to weights, and constraints (8)–(9) require the weighted accuracy for every class, and overall, to exceed the uniform-weight average by a small $\\epsilon$. The objective (1) maximizes $\\frac{1}{m}\\sum_i\\sum_j w_{ij} v_{ij}$ minus the elastic net penalty $\\lambda\\left(\\alpha\\sum w_{ij} + \\frac{1-\\alpha}{2}\\sum w_{ij}^2\\right)$, where $v_{ij}$ comes from stratified 5-fold cross-validation. Because the optimization is driven only by the $n\\times m$ accuracy matrix $V$, its size is independent of the training set size.","core_discovery":"The central claim is that an optimal per-class weight matrix $W$ for a weighted voting ensemble can be computed jointly with the selection of the ensemble's $K$ members by solving one optimization problem. The objective maximizes the mean over classes of the weighted sum of validation accuracies $v_{ij}$, penalized by elastic net regularization to keep the weights sparse and stable, and the constraints force the per-class and overall weighted accuracy to beat the uniform-weight baseline on the validation data. The authors report that this formulation, solved with a standard MIP solver, outperformed all six baselines on all four datasets and every tested ensemble size, with the advantage growing as $K$ grows and remaining positive on the highly imbalanced datasets. The worked example on NSL-KDD shows the mechanism: the MIP gives the SVM classifier large weights precisely for the two classes where it is strong, while the baseline schemes spread weight more evenly.","pith_inferences":["Editorial inference: the reported 4.53% average is computed from one fixed 80/20 split per dataset, so repeated re-splitting with weights re-optimized inside each training fold would show how much of the edge is stable; classes with as few as 11 or 74 examples make the validation matrix noisy.","Because the objective consumes only a per-class accuracy matrix, the same formulation can be adapted to cost-sensitive rare event detection by substituting a utility or cost matrix for $v_{ij}$, letting missed leaks or attacks be penalized more heavily than ordinary errors.","The optimization's independence from dataset size suggests the weighting step could be re-run periodically as new validation data arrive, supporting deployment under concept drift; the paper does not test this.","The per-class weight matrix itself is an interpretability artifact: as the SVM example shows, the MIP concentrates weight on the classes where a classifier is strong, which could help operators see which model carries each rare class."],"forward_implications":["Classifier selection and weight assignment are solved in one run, so a user can prescribe any ensemble size $K$ and get the corresponding optimal weights without evaluating all $\\binom{n}{K}$ combinations separately.","The reported advantage over all six baselines grows with $K$: on the tested datasets the improvement in balanced accuracy is consistently larger for $K=7,8$ than for $K=2,3$.","The method remains effective as imbalance worsens: in the LeakDB ablation, balanced accuracy fell by only 0.51–0.92 percentage points when the imbalance ratio was halved or doubled, and stayed at 0.973 or higher when the number of minority classes grew to six.","The weight calculation is cheap enough for practice: on NSL-KDD with $K=3$ and $n=8$ classifiers it takes 0.35 seconds, and the speedup over the baselines grows sharply as $n$ increases to 24."],"supporting_citations":[{"why":"Supplies the elastic net regularization used in the objective to sparsify and stabilize the classifier-class weights.","marker":"Zou and Hastie 2005"},{"why":"Provides the LeakDB water-distribution leakage benchmark (D1), the most imbalanced dataset tested.","marker":"Vrachimis et al. 2018"},{"why":"Provides the NSL-KDD intrusion dataset (D2), which includes a very small minority class.","marker":"Tavallaee et al. 2009"},{"why":"Provides the CIC-IDS2017 intrusion dataset (D4) used in the evaluation.","marker":"Sharafaldin, Lashkari, and Ghorbani 2018"},{"why":"Provides the SG-MITM smart-grid attack dataset (D3) used in the evaluation.","marker":"Elrawy et al. 2023"},{"why":"Defines the weighted majority voting ensemble setting and supports the claim that the classifier count is small in practice.","marker":"Dogan and Birant 2019"},{"why":"Underlies the normalized-accuracy weighted-average baselines WA-PC and WA-PCC.","marker":"Sagi and Rokach 2018"},{"why":"Underlies the differential evolution (DE) baseline that the MIP method is compared against.","marker":"Storn and Price 1997"},{"why":"Underlies the Bayesian model averaging (BMA) baseline that the MIP method is compared against.","marker":"Raftery, Madigan, and Hoeting 1997"}],"fun_headline_variants":["MIP ensemble weighting beats six baselines on imbalanced datasets","Optimal per-class weights lift balanced accuracy by 4.53% on average","Single MIP optimization selects and weights classifiers for rare events","MIP-based ensemble weighting delivers 4.53% average accuracy gain","Sparse MIP weighting outperforms six baselines in multi-class imbalance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the mean validation accuracy of each classifier on each class, computed by 5-fold cross-validation, faithfully predicts that classifier's per-class performance on unseen data; for the smallest classes (F4 with 74 instances, A3 with 11) those estimates are noisy, and the optimized weights can overfit them.","fun_headline_variants_meta":{"raw":{"variants":["MIP ensemble weighting beats six baselines on imbalanced datasets","Optimal per-class weights lift balanced accuracy by 4.53% on average","Single MIP optimization selects and weights classifiers for rare events","MIP-based ensemble weighting delivers 4.53% average accuracy gain","Sparse MIP weighting outperforms six baselines in multi-class imbalance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000238,"raw_usage":{"total_tokens":1500,"prompt_tokens":927,"completion_tokens":573,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":543,"completion_tokens_details":{"reasoning_tokens":480}},"tokens_in":543,"tokens_out":573,"duration_ms":5591,"temperature":1.0,"reasoning_tokens":480,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:08:06.145680+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the method and all six baselines on the same four datasets over many random 80/20 splits, re-optimizing the MIP weights inside each training fold; if MIP does not beat the best baseline in balanced accuracy on most splits—especially on the tiny minority classes such as F4 and A3—then the reported 4.53% average edge does not generalize.","supporting_citations":[{"cited_title":"G.; Kyriakou, M","cited_arxiv_id":null,"evidence_quote":"Provides the LeakDB water-distribution leakage benchmark (D1), the most imbalanced dataset tested."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the NSL-KDD intrusion dataset (D2), which includes a very small minority class."},{"cited_title":"H.; and Ghorbani, A","cited_arxiv_id":null,"evidence_quote":"Provides the CIC-IDS2017 intrusion dataset (D4) used in the evaluation."},{"cited_title":"F.; Hadjidemetriou, L.; Laoudias, C.; and Michael, M","cited_arxiv_id":null,"evidence_quote":"Provides the SG-MITM smart-grid attack dataset (D3) used in the evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the weighted majority voting ensemble setting and supports the claim that the classifier count is small in practice."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Underlies the normalized-accuracy weighted-average baselines WA-PC and WA-PCC."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Underlies the differential evolution (DE) baseline that the MIP method is compared against."},{"cited_title":"E.; Madigan, D.; and Hoeting, J","cited_arxiv_id":null,"evidence_quote":"Underlies the Bayesian model averaging (BMA) baseline that the MIP method is compared against."}],"review_version":1}