{"id":"c7cf656c-ca7a-481a-8bfc-4b311c08fb5b","arxiv_id":"1908.05287","paper_version":6,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"Jointly tuning base-learner hyperparameters and ensemble weights is claimed to lower test MSE versus independently tuned ensembles on 9 of 10 regression datasets, but the evaluation leaks test data into model selection.","lead":"This paper introduces GEM-ITH, an algorithm that optimizes both the hyperparameters of individual models and the weights for combining them in one nested procedure. The authors report lower test error on 9 of 10 public regression datasets, but the evaluation uses the test set while selecting base models, weakening the result.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Base-learner selection in §5.2 is fit on the held-out test set, so the Table 3 MSEs and the GEM-ITH superiority claim rest on leaked test labels.","rationale":"The paper's contribution is an optimization algorithm plus an empirical demonstration. The algorithm itself is coherent: the inner weight problem is convex, and the outer Bayesian search over hyperparameter combinations is a reasonable though expensive nested scheme. What would have to be true for the central claim is that the test MSE numbers in Table 3 compare methods on a test set never used in model construction. That condition is violated in Section 5.2 by the explicit use of 'unseen test observations' to prune and rank candidate base learners. Since every ensemble benchmark in the paper is built from the same selected base learners, the leakage is common to all methods, but it still destroys the validity of the absolute MSEs and makes the relative GEM-ITH advantage unreliable. A second supporting weakness is the unequal search budget—GEM-ITH considers 12^4 combinations while the base learners and GEM are tuned with more limited grid settings—but that is a confounding factor rather than the decisive flaw. The reader's verdict REJECT is consistent with this analysis; my recommendation is UNCHANGED because the central empirical claim remains unsupported by the reported protocol. No machine-checked proof or released code is provided to offset the evaluation issue.","tokens_in":15441,"tokens_out":3271,"duration_ms":35203,"concrete_test":"Re-run the complete pipeline with strict data separation: after the 80/20 split, perform the Section 5.2 heuristic (trial training, performance pruning, correlation ranking, and base-learner selection) using only cross-validated out-of-fold predictions from the 80% training portion, and touch the 20% test labels only once to compute the final MSE for each method. If GEM-ITH still beats GEM and the stacked benchmarks on the same ten datasets by similar margins, the central claim survives; if the advantage shrinks or reverses, the reported Table 3 wins were an artifact of selecting base learners on test labels.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that GEM-ITH achieves better test MSE than GEM and the benchmark ensembles, and that internally tuned hyperparameters differ beneficially—depends on the test MSE in Table 3 being an unbiased comparison. Section 5.2 violates that condition. Step 1 ('Trial training') evaluates trial models 'using unseen test observations'; step 2 prunes models whose prediction error is above average; steps 3–5 then rank by pairwise correlation and select four base learners from the survivors. The same 20% held-out split is subsequently used for every objective value in Table 3. Consequently the reported test labels have already influenced which models enter every ensemble, so Table 3 is not an independent evaluation; it is a selection report. This is not an internal inconsistency—the algorithm is clearly specified—but it invalidates the external comparison. All benchmarks share the same selected base learners, so the leakage affects them too; however, GEM-ITH searches over 12^4 hyperparameter combinations and optimizes the same OOB objective used during selection, so its reported advantage could be amplified by fitting to labels that should have remained untouched. The paper's own statement that '20% of each data set was reserved for testing and the training and optimizing procedure was done on the remaining 80%' is contradicted by the Section 5.2 procedure. Thus the strongest claim is not supported by the reported experiments.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GEM-ITH, a nested optimization algorithm for regression ensembles in which the hyperparameters of each base learner are tuned internally while the ensemble weights are optimized by minimizing MSE. The inner weight problem is a convex quadratic program; Bayesian search limits the hyperparameter search space, and a heuristic selects four diverse, well-performing base learners. The method is evaluated on ten public regression data sets against GEM, BEM, stacked regression, stacked random forest, and stacked k-nearest neighbor, with test MSE reported in Table 3. The paper claims that GEM-ITH achieves the best prediction accuracy among the benchmarks, improves on each base learner, and that its internally tuned hyperparameters differ from those tuned independently.","tokens_in":15706,"tokens_out":3665,"duration_ms":37465,"significance":"The basic idea of jointly tuning hyperparameters and ensemble weights is sensible, and the convexity argument for the inner weight optimization is correct. If the empirical claims were supported by a clean experimental design, the contribution would be practically useful and would extend the GEM framework in a reasonable direction. However, the evaluation protocol has a serious flaw: the held-out test set is used during base-learner selection, and the comparison is not controlled for search budget or variability. As a result, the current paper does not establish its central empirical claim.","major_comments":[{"comment":"The base-model generation heuristic leaks test information. Step 1 of Section 5.2 evaluates trial models 'using unseen test observations', Step 2 prunes models whose prediction error is above average, and Steps 3-5 select four base learners on the basis of those test-set errors. Section 5.1 states that 20% of each data set was reserved for testing and that training and optimization were done on the remaining 80%, but the Section 5.2 procedure uses the same held-out observations to select the models whose predictions are later scored in Table 3. Consequently, Table 3 is a selection report rather than an independent evaluation, and the reported test MSEs are optimistically biased for every method.","section":"Section 5.2"},{"comment":"The comparison is not controlled for search budget. GEM-ITH evaluates 12^4 hyperparameter combinations, while GEM and the other benchmarks use a single independently tuned hyperparameter setting per base learner. Any improvement could therefore be due to the much larger number of evaluated configurations rather than to the joint optimization of weights and hyperparameters. A fair comparison would give the benchmark pipelines the same search budget, e.g., by selecting the best of 12^4 independently tuned ensembles on validation folds, or by reporting the performance of GEM-ITH with only the same number of evaluations as the benchmarks.","section":"Section 5.2 and Section 5.4"},{"comment":"No measure of variability is reported. Section 5.1 says the entire process was repeated 5 times, yet Table 3 and Fig. 2 show only point estimates, with no standard deviations, confidence intervals, or statistical significance tests. Several reported advantages are very small in relative terms, such as Diabetes (2987.23 vs. 3038.89), Wine Quality (3.62 vs. 3.64), and QSAR Fish Toxicity (6.93 vs. 7.04). Without variability information, the claim of 'almost complete dominance' is not supported.","section":"Section 5.4, Table 3 and Fig. 2"}],"minor_comments":[{"comment":"The claim that GEM-ITH finds hyperparameters different from those tuned independently is demonstrated for only one data set in Table 4. The Conclusion generalizes this claim to all data sets, but no supporting aggregate evidence is provided; a table or figure summarizing differences across all ten data sets would be needed.","section":"Section 5.4"},{"comment":"The normalization used to compute 'normalized error rates' is not defined in the text. Please state how each data set's MSE is scaled before averaging or plotting.","section":"Fig. 2"},{"comment":"Table and figure labels are inconsistent: the text uses 'Table.1', 'Table.2', etc., and the bold formatting indicating the best result in Table 3 is not visible in the manuscript. Please standardize the labels and ensure the best values are clearly marked.","section":"Global"},{"comment":"Hyperparameter ranges such as '10^range(-5,0)' and 'linspace(0.01, 5, 20)' are ambiguous about whether endpoints are inclusive and whether values are intended to be log-spaced. Please define the ranges precisely and state the total number of combinations per model.","section":"Section 5.2, Table 2"},{"comment":"Reference [51] (Bergstra and Bengio, 2012) is listed but not cited in the text; the Bayesian search discussion cites Bergstra et al. 2013 and Snoek et al. 2012. Please check that all listed references are cited and that citation numbering is consistent.","section":"References"}],"recommendation":"reject","confidential_remarks":"The test-set leakage in Section 5.2 is a fundamental problem that invalidates the reported comparison, and the unequal search budget between GEM-ITH and the benchmarks means even a corrected comparison would require substantially redesigned experiments. The paper's contribution is modest but could be salvageable if the authors rerun the evaluation with proper nested resampling, matched search budgets, and uncertainty quantification; however, in its current form the central claim is unsupported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper's empirical center does not hold — Section 5.2 selects base learners using the same held-out test set that Table 3 reports. That is a test-label leak, and it invalidates the claimed dominance over GEM and the other benchmarks.\n\nThe algorithm itself, GEM-ITH, is a reasonable composition of existing ideas: nested hyperparameter search over a weighted ensemble objective, using Bayesian search to keep it tractable. The convexity argument for the inner weight problem is correct. The heuristic for choosing diverse, decent base learners is transparent, and the paper gives a clear description of the procedure, including its computational cost. Credit where due: the authors state exactly what they did, which is why the leak is catchable.\n\nBut the claim that joint tuning 'improves the prediction accuracy of each base learner' and 'could achieve better prediction accuracy among all' is not supported by the reported experiments. Step 1 of Section 5.2 evaluates trial models 'using unseen test observations'; step 2 prunes those with above-average error; steps 3–5 rank by correlation and pick four models. Those same test labels then feed every number in Table 3. That is selection on the test set, not evaluation on it. All benchmarks share the selected learners, so the leak is not a comparison artifact, but GEM-ITH's larger search budget and aligned objective make its apparent edge especially suspect.\n\nAlso missing: any error bars, significance tests, or a matched search budget between GEM and GEM-ITH. The comparison is confounded. The paper's own limitations section is honest but does not mention the test-set leakage.\n\nOn the novelty claim: the nested idea is not in the cited references, but the paper overstates the gap by ignoring AutoML/CASH work that already couples hyperparameter search with model selection and stacking.\n\nWho is this for? A reader interested in ensemble weight optimization as a case study in how evaluation leakage can poison a reasonable method. The method may be salvageable with a clean re-run using only training/validation data for model selection, and with error bars. As it stands, I would not cite the empirical results. But it is not a waste of a referee: a serious reviewer could help the authors fix the evaluation and resubmit. My recommendation: reject in current form, but do not desk-reject; send it to review if the venue has room for a revision that redoes the experiments cleanly.","headline":"GEM-ITH is a reasonable nested algorithm, but the reported evaluation leaks test labels in Section 5.2, so the central claim of superiority is not supported.","tokens_in":16238,"tokens_out":2090,"would_cite":false,"duration_ms":21007,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Tuning hyperparameters inside ensemble-weight optimization, rather than separately, yields better regression predictions, and the paper's GEM-ITH algorithm reports wins on 9 of 10 public datasets.","keywords":["ensemble learning","weighted ensemble","stacking","hyperparameter optimization","Bayesian optimization","regression","bias-variance tradeoff"],"falsifier":"Re-run the ten experiments with the test set completely quarantined, selecting base learners and hyperparameters using only training and validation folds, and compare GEM-ITH's test MSE with GEM's; if the margin disappears or reverses, the central claim fails.","tokens_in":15212,"feed_emoji":"🎯","tokens_out":7473,"duration_ms":68244,"temperature":0.7,"pith_summary":"This paper argues that the standard recipe for building a weighted ensemble—tune each model independently, then combine the tuned models—is suboptimal. It introduces GEM-ITH, a nested procedure that treats hyperparameter settings as part of the search for ensemble weights rather than as a preprocessing step. Across ten public regression datasets, GEM-ITH reports the lowest test mean squared error in nine of ten comparisons, beating simple averaging, the Generalized Ensemble Method, and stacked ensembles with linear regression, random forest, and k-nearest neighbors as meta-learners. The practical stakes are that ensembles, already reliable accuracy boosters, could be pushed further without adding new model families, simply by making the combination step aware of each model's settings.","feed_headline":"Joint tuning beats separate tuning in 9 of 10 regressions","feed_subtitle":"Nested search over hyperparameters and ensemble weights beats separately tuned weighted models on ten public datasets.","key_machinery":"The central object is GEM-ITH, a nested optimization algorithm. The outer loop uses Bayesian search to pick b hyperparameter settings for each of k base learners; the inner loop, for each of the b^k combinations, computes out-of-bag predictions from m-fold cross-validation and solves a nonlinear convex program whose decision variables are the ensemble weights, constrained to be nonnegative and sum to one, with mean squared error as the objective. Convexity of that objective over the simplex guarantees that the weight solution is globally optimal for any fixed hyperparameter combination. A separate heuristic builds the base-learner pool: train many models, prune those with above-average error, and keep four diverse, low-correlation models.","core_discovery":"The paper's central claim is that the best ensemble is not generally made from base learners whose hyperparameters are optimal in isolation. In GEM-ITH, Bayesian search proposes candidate hyperparameter configurations for each base learner; for each combination, m-fold cross-validation produces out-of-bag predictions, and a convex optimization problem finds the nonnegative weights, summing to one, that minimize the weighted ensemble's mean squared error. The configuration and weight vector with the lowest objective value are selected. On the ten datasets, GEM-ITH achieves the lowest test MSE in 9 of 10 cases and improves on the individual base learners' predictions. The paper also reports that for the Energy Efficiency dataset, the hyperparameters GEM-ITH selects differ from the independently tuned values (for example, regression tree max_depth 19 rather than 6, Elastic Net alpha 0.76785 rather than 0.00001), which it takes as direct evidence that internal tuning changes what the ensemble needs.","pith_inferences":["If the joint-tuning effect is real, the same nested argument could be tested for classification losses such as log loss or Brier score, since this paper only considers regression with MSE.","An ablation that holds the four base learners fixed across GEM and GEM-ITH would isolate how much of the gain comes from joint hyperparameter tuning versus from the base-model selection heuristic.","Warm-starting Bayesian search with individually tuned hyperparameters or pruning unpromising combinations early could cut the reported run times and make GEM-ITH practical on larger datasets."],"forward_implications":["GEM-ITH reports the lowest test MSE on 9 of the 10 datasets when compared with BEM, GEM, stacked regression, stacked random forest, and stacked k-nearest neighbors.","The hyperparameters GEM-ITH selects differ from independently tuned values, so the best ensemble components need not be the best standalone models.","Joint tuning also improves prediction accuracy relative to each individual base learner on the studied datasets.","The Bayesian-search version trades the global-optimality guarantee of grid search for tractability, and its high computation times make it more suitable for small-to-medium datasets."],"supporting_citations":[{"why":"Supplies the Generalized Ensemble Method, the weight-optimization formulation GEM-ITH extends.","marker":"[26]"},{"why":"Establishes the constrained weight optimization and the diversity and bias-variance argument that motivates joint tuning.","marker":"[27]"},{"why":"Provides the Bayesian optimization strategy the outer loop uses to propose hyperparameter candidates.","marker":"[52]"},{"why":"Provides the hyperopt implementation used to run the Bayesian search.","marker":"[64]"},{"why":"Defines stacked generalization, the paradigm within which GEM-ITH is presented.","marker":"[16]"},{"why":"Supplies stacked regression, one of the benchmarks GEM-ITH is compared against.","marker":"[28]"}],"fun_headline_variants":["Joint tuning of weights and hyperparameters wins 9/10","Nested optimization for weights and hyperparameters beats separate tuning","GEM-ITH joint tuning outperforms separate tuning on 9 of 10 datasets","Internal hyperparameter tuning boosts weighted ensemble performance","Hyperparameters and weights jointly optimized for better regression ensembles"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The procedure assumes the 20 percent holdout test set is reserved for final evaluation only, but the base-model generation step evaluates trial models on those same 'unseen test observations' and prunes weaker models before ensembles are built; if that selection uses the test set, the reported test errors are optimistically biased.","fun_headline_variants_meta":{"raw":{"variants":["Joint tuning of weights and hyperparameters wins 9/10","Nested optimization for weights and hyperparameters beats separate tuning","GEM-ITH joint tuning outperforms separate tuning on 9 of 10 datasets","Internal hyperparameter tuning boosts weighted ensemble performance","Hyperparameters and weights jointly optimized for better regression ensembles"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001057,"raw_usage":{"total_tokens":4428,"prompt_tokens":928,"completion_tokens":3500,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":3416}},"tokens_in":544,"tokens_out":3500,"duration_ms":23934,"temperature":1.0,"reasoning_tokens":3416,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:18:41.264865+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the ten experiments with the test set completely quarantined, selecting base learners and hyperparameters using only training and validation folds, and compare GEM-ITH's test MSE with GEM's; if the margin disappears or reverses, the central claim fails.","supporting_citations":[{"cited_title":"P., & Cooper, L","cited_arxiv_id":null,"evidence_quote":"Supplies the Generalized Ensemble Method, the weight-optimization formulation GEM-ITH extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the constrained weight optimization and the diversity and bias-variance argument that motivates joint tuning."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Bayesian optimization strategy the outer loop uses to propose hyperparameter candidates."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the hyperopt implementation used to run the Bayesian search."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies stacked regression, one of the benchmarks GEM-ITH is compared against."}],"review_version":1}