{"id":"ace066d0-468b-4ab8-8544-5da11a375d9e","arxiv_id":"2412.08526","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SM2 is an energy-aware successive halving optimizer that selects batch size and learning rate using GPU energy per epoch and cyclical learning-rate exploration, cutting HPO energy by 8% to 47% in three test scenarios.","lead":"This paper introduces SM2, a hyperparameter optimization method that tracks GPU energy use while tuning batch size and learning rate, and prunes inefficient configurations early. It reports 8% to 47% lower energy demand than a performance-only version on three model and dataset pairs, at similar or slightly worse accuracy.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported energy savings depend entirely on the unvalidated assumption that one epoch of training on a quarter of the dataset ranks configurations correctly for full 5-10 epoch training; the Transformer's 15pp perplexity loss suggests this proxy can mislead.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: the one-epoch, quarter-dataset exploratory phase must reliably rank configurations for full training, and no evidence is supplied for that reliability. My stress-test of Section 5.1 and Algorithm 1 confirms this is the right place to look. The entire energy-savings claim in Section 5.3 is a comparison of two variants of SM2 that differ only in the objective weight; the pruning decisions themselves are driven by the exploratory signal, so if that signal is noisy, both the savings and the performance parity claims are fragile. I considered other possible concerns: the comparison set is narrow (no Hyperband, random search, or Bayesian optimization on equal budgets), energy appears directly in the optimized objective making part of the reduction definitional, and no code or data are released. These are real limitations, but they are secondary to the proxy-fidelity question because they would not invalidate the paper's internal claim if the proxy were shown to be reliable. Conversely, even with ideal baselines, a broken exploratory ranking would destroy the method's usefulness. The paper does some things well: three different model classes and three hardware setups are tested, the energy tracking is integrated into the training loop, and the reported reductions are internally consistent with the described decision behavior. However, no formal verification, released code, or reproducibility artifacts are provided, so the conditional verdict is appropriate. My recommended verdict is therefore UNCHANGED relative to the reader's CONDITIONAL assessment: the concern is real, concrete, and testable, but it does not, on the current evidence, force a rejection; it demands conditional acceptance pending the proposed fidelity check.","tokens_in":10803,"tokens_out":3030,"duration_ms":33719,"concrete_test":"Re-run SM2 on the same three model/dataset scenarios with a full-fidelity control that spends the same total compute on evaluating all configurations on full data instead of pruning after the one-epoch, quarter-dataset exploratory phase. Then compute rank correlation (e.g., Kendall tau) between the exploratory objective values and the final validation performance/energy ordering at each halving step, and compare the final configurations selected by both procedures. If the exploratory ranking agrees poorly with full-fidelity ranking (tau below about 0.6) or if the full-fidelity control selects different configurations with a better energy-performance trade-off, the savings in Table 1 are not attributable to SM2's proxy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that SM2 reduces energy without meaningful performance loss rests on the fidelity of the exploratory ranking. In Section 5.1, exploratory training is set to '1 epoch trained only on a quarter of the dataset,' and Algorithm 1 uses the objective f(alpha, beta) from Equation (2) evaluated on that short, data-reduced signal to halve the configuration set. If this cheap signal does not preserve the relative ordering of configurations under the full 5-10 epoch thorough training, then successive halving prunes the wrong arms and the reported savings in Table 1 (8%, 47%, and 16% for alpha=0.75 versus alpha=1.0) are an artifact of the specific runs rather than a general property of the method. The paper gives no correlation analysis between exploratory and final rankings, no error bars, and no ablation that varies the exploration length or data fraction. The Transformer result, where energy-based pruning yields roughly 15 percentage points worse perplexity, is consistent with the proxy over-weighting an energy signal that does not align with final performance. Since the exploratory phase is the mechanism that generates all reported savings, this unverified assumption is load-bearing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SM2, an energy-aware hyperparameter optimization method built on successive halving. SM2 alternates between a cheap exploratory phase (one epoch on a quarter of the dataset) and a thorough training phase, using an objective function that combines model performance, measured energy per epoch, and a learning-rate stability score to prune half of the configurations at each exploratory round. The method is evaluated on three scenarios (ResNet-18/CIFAR-10, LSTM/Energy-Household, Transformer/WikiText2) across three Nvidia GPUs. The paper reports total energy reductions of 8%, 47%, and 16% for alpha=0.75 relative to alpha=1.0, and parity factors of 1.76, 1.11, and 1.70 relative to vanilla training, which it interprets as evidence that SM2 cuts HPO energy cost with little or no performance loss.","tokens_in":11038,"tokens_out":3563,"duration_ms":37009,"significance":"The problem addressed—reducing the energy footprint of hyperparameter search—is practically important and timely. The engineering contribution, particularly the integration of real-time GPU energy tracking via Carbontracker into a successive-halving loop, is concrete and could be useful to practitioners. The paper also makes a valuable distinction between exploratory and thorough training and provides a transparent objective function. However, the central scientific claim is not yet established: the reported energy savings partly follow from the construction of the objective, the exploratory ranking that drives pruning is not validated, and the experiments lack repeats and standard HPO baselines. If the authors can supply the missing validation, the work could be a solid empirical contribution to sustainable HPO; as it stands, the evidence is suggestive but not conclusive.","major_comments":[{"comment":"The principal quantitative claim—energy reductions of 8–47% from alpha=0.75 relative to alpha=1.0—is partly a restatement of the objective, since E appears directly in the objective function and alpha weights it. Comparing alpha=0.75 to alpha=1.0 changes the selection rule by design, so the savings do not independently demonstrate that energy awareness is beneficial. The authors should compare SM2 against a standard HPO baseline (e.g., Hyperband/ASHA or random search with an identical total budget) and report the distribution of final performance across repeated trials. The Transformer result, where alpha=1.0 yields roughly 15 percentage points better perplexity than alpha=0.75, directly contradicts the claim of \"no meaningful performance loss\" and needs to be addressed.","section":"§4.3, Eq. (2); §5.3, Table 1"},{"comment":"The entire pruning mechanism rests on the assumption that one epoch of exploratory training on one quarter of the dataset ranks configurations according to their final 5–10 epoch thorough-training performance and energy. No rank-correlation analysis, no sensitivity ablation over the exploration length or data fraction, and no comparison of exploratory versus final configuration rankings are provided. Without this, the reported savings in Table 1 could be artifacts of pruning decisions that happen to work on these three runs, and the Transformer case suggests the proxy can mislead. Please provide evidence for the predictive validity of the exploratory phase, or explicitly characterize the regime where it fails.","section":"§5.1 and Algorithm 1"},{"comment":"All quantitative results are single runs per hardware setup: no seeds, repeats, or error bars are reported, and the text does not make clear whether the full SM2 procedure was executed multiple times or only once per GPU. Furthermore, alpha=0.75 and beta=0.5 were chosen based on \"initial tests\" (§5.1), presumably on the same scenarios, so the reported reductions may reflect tuning to the experimental setup. The authors should provide multiple trials with different seeds, report variance, and perform a sensitivity analysis over alpha (and secondarily beta) to show that the conclusions are robust to the manually selected weights.","section":"§5.1 and §5.2"}],"minor_comments":[{"comment":"The text says the energy per epoch is \"commonly stated in watt per hour\", but the formula divides by 3600 and the Table reports Wh; the unit should be watt-hours (Wh), not \"watt per hour\".","section":"§2.2, Eq. (1)"},{"comment":"The claim that this is \"the first work to optimize the hyperparameters while considering energy consumption\" is too strong and is not supported by the related-work discussion, which already cites Zeus (You et al., 2023) and other energy-aware training methods. Please qualify the novelty claim, e.g., to \"the first SHA-based implementation with real-time GPU energy feedback\", and discuss the differences from existing energy-aware HPO systems.","section":"§1"},{"comment":"The pseudocode is too high-level: the \"Evaluate Exploration\" step is not specified, and it is unclear how the objective function in Eq. (2) is computed from the exploratory runs, how normalization is applied, and how the learning-rate score LR is derived. Adding a few lines describing these steps would greatly improve reproducibility.","section":"§4.4, Algorithm 1"},{"comment":"The figure is dense and the text says \"Each column plots the three acquired attributes\", but the columns are performance, energy, and learning rate; the axes are not always labeled with units, and the vertical exploratory lines are difficult to see. Please increase the figure's legibility and provide unit labels (e.g., Wh for energy).","section":"§5.2, Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful engineering core and addresses a relevant problem, but the current validation is insufficient for the strength of the claims. The major comments ask for evidence that the exploratory ranking is predictive, a comparison with a standard HPO baseline, and repeated runs with error bars and sensitivity analysis. I would be willing to reconsider after these are addressed. Also, the novelty statement in §1 overstates the contribution relative to the cited prior work; this should be corrected during revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is the integration: energy-per-epoch readings fed directly into a successive-halving objective, with cyclical learning-rate exploration and a cheap exploratory pretraining phase to prune configurations. That is a sensible direction, and the reported numbers are internally consistent. The parity metric, quantifying how many manual HPO runs the SM2 overhead would offset, is a useful framing.\n\nThe soft spots are real and load-bearing. The headline comparison is alpha=1.0 versus alpha=0.75, but energy appears directly in the objective, so some of the 8–47% savings are a restatement of the selection rule rather than an independent empirical discovery. The missing baselines matter more: no Hyperband, random search, or Bayesian optimization run on an equal energy budget. Without those, the claim that SM2 is a practical way to cut HPO energy is plausible but not established. The Transformer result, with roughly 15 percentage points worse perplexity under alpha=0.75, also undercuts the blanket claim of no meaningful performance loss.\n\nThe exploratory proxy is the mechanism that generates all the savings, and it is unvalidated. One epoch on a quarter of the dataset is a very cheap signal, and the paper gives no correlation analysis between exploratory rankings and final rankings, no ablation varying exploration length or data fraction, and no error bars despite stating that experiments were run on three hardware setups. Only the A6000 results are shown, so the hardware-independence claim is undersupported. Alpha and beta were tuned after initial tests, and there is no sensitivity analysis.\n\nCitation-wise, the paper covers the standard HPO and energy-tracking literature, but the claim to be the first to optimize hyperparameters while considering energy consumption is too strong given existing work like Zeus and other energy-aware HPO efforts that are cited but not engaged with on that point. The self-citation to their earlier energy-study is fine, but it is not independent evidence.\n\nThat said, I do not think this is a desk-reject. The integration is novel enough, the measurements are plausible, and the problem is worth solving. It deserves a serious referee, but the referee should insist on standard HPO baselines, error bars, a validation of the exploratory proxy, and ideally released code and data. I would not cite it in my own work yet, but I would bring it to a reading group as a useful example of how energy-aware HPO is being approached and where the evidentiary bar needs to be higher.","headline":"SM2 is a reasonable engineering integration of energy tracking into successive halving, but the savings are partly built into the objective and the exploratory proxy is unvalidated, so the empirical claims need stronger support.","tokens_in":11530,"tokens_out":1956,"would_cite":false,"duration_ms":23516,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An energy-aware variant of successive halving cuts hyperparameter-search energy demand by up to 47% while keeping final model quality largely intact.","keywords":["hyperparameter optimization","energy-aware training","successive halving","GPU energy monitoring","sustainable AI","exploratory pretraining","learning rate scheduling"],"falsifier":"Retrain every configuration that SM2 prunes in early rounds for the full 5-10 epochs and compare final performance and total energy against the configuration SM2 selected; if any pruned configuration beats the survivor on both final performance and full-run energy, the exploratory ranking is not a faithful proxy.","tokens_in":10586,"feed_emoji":"⚡","tokens_out":7985,"duration_ms":66632,"temperature":0.7,"pith_summary":"Spend More to Save More (SM2) is an energy-aware version of the successive halving algorithm for hyperparameter optimization. The paper aims to show that tuning can be made sustainable without sacrificing model performance: SM2 prunes configurations using a scalar objective that balances validation performance, energy per epoch, and learning-rate stability, rather than performance alone. In experiments on ResNet/CIFAR-10, LSTM/Energy-Household, and Transformer/WikiText2, SM2 reduced total energy demand by 8%, 47%, and 16% against a performance-only objective, matching final accuracy in the first two scenarios and incurring a roughly 15-point perplexity penalty in the third. The authors argue the brief exploratory training pass costs less than the full runs it avoids, giving a parity factor that pays off after roughly two manual hyperparameter explorations.","feed_headline":"Energy-aware pruning cuts hyperparameter-search power by up to 47%","feed_subtitle":"An exploratory pass plus energy-aware pruning shrinks tuning cost with limited performance impact.","key_machinery":"The load-bearing mechanism is the objective function that rescales performance, energy per epoch, and learning rate into $[0,1]$ and combines them as $f(\\alpha,\\beta)=\\alpha\\cdot P+(1-\\alpha)\\cdot(\\beta\\cdot E+(1-\\beta)\\cdot LR)$, driving each pruning decision in the halving schedule. Around it, the exploratory training phase (one epoch on a quarter of the data) provides a cheap proxy for full training, cyclical learning-rate sweeps identify the largest stable learning rate by measuring loss curvature over a sliding window, and Carbontracker's GPU wattage sampling supplies the energy term. This lets the algorithm pick configurations that match or nearly match performance while operating the GPU in an efficient power state.","core_discovery":"The paper's central claim is that energy efficiency can be folded directly into the hyperparameter optimization objective without degrading the final model. SM2 runs a short exploratory training phase (one epoch on a quarter of the dataset) for each candidate, tracking GPU energy per epoch through the Carbontracker library and sweeping learning rates cyclically. A re-scaled objective $f(\\alpha,\\beta)=\\alpha\\cdot P+(1-\\alpha)\\cdot(\\beta\\cdot E+(1-\\beta)\\cdot LR)$ combines rescaled performance, energy per epoch, and a stable learning rate; with $\\alpha=0.75$ and $\\beta=0.5$, the worse half of configurations is dropped after each exploratory round following successive halving. Only the survivor is trained thoroughly. Empirically, this yields total energy reductions of 8%, 47%, and 16% relative to $\\alpha=1.0$ across the three scenarios, with equal final accuracy/$R^2$ in the first two and a perplexity penalty of about 15 percentage points in the Transformer case.","pith_inferences":["The decisive open question this design passes over is the fidelity of the one-epoch/quarter-data exploratory signal; if that proxy misranks configurations on harder or noisier tasks, the energy savings would come at the cost of final quality.","The method's energy reading is GPU-only, so on clusters where CPU, memory, or cooling dominate the power budget, SM2 could select configurations that look efficient on the GPU while wasting energy elsewhere.","A natural stress test would compare SM2's choices against exhaustive search on a small hyperparameter grid: if the exhaustive best configuration is pruned in the first round across repeated seeds, the proxy assumption fails.","The authors' parity concept could be inverted into an actionable break-even metric that tells practitioners exactly how many skipped manual runs justify adopting SM2 for a given hardware setup."],"forward_implications":["If SM2's exploratory ranking is faithful, hyperparameter optimization can be made noticeably cheaper in energy without changing model selection for tasks where several configurations perform similarly.","The same energy-aware objective can be added to other pruning-based HPO algorithms such as Hyperband or ASHA by feeding energy per epoch as an extra fidelity signal.","The parity analysis implies that any manual HPO workflow doing two or more full training runs would recoup SM2's overhead, making the approach worthwhile for typical iterative tuning.","Because SM2 leaves thorough training untouched, it can be deployed as a wrapper around standard PyTorch training loops without altering the model, optimizer, or loss function.","The reported energy savings are hardware-dependent, but the method's ranking logic transfers across the three tested Nvidia GPUs, suggesting broad applicability within CUDA-based environments."],"supporting_citations":[{"why":"Supplies the Carbontracker library used to sample GPU power and compute energy per epoch.","marker":"[Anthony et al., 2020]"},{"why":"Introduces the successive halving algorithm that SM2 adapts as its pruning backbone.","marker":"[Li et al., 2018]"},{"why":"Provides the cyclical learning-rate scheduling technique used in the exploratory phase to select stable learning rates.","marker":"[Smith, 2017]"},{"why":"Shows batch size and learning rate dominate energy efficiency of neural network training, motivating the two optimized hyperparameters.","marker":"[Geißler et al., 2024]"},{"why":"Documents that GPU efficiency windows lie below maximum utilization, supporting the energy-per-epoch objective in batch-size selection.","marker":"[You et al., 2023]"}],"fun_headline_variants":["Energy-aware successive halving trims tuning power by up to 47%","Exploratory pretraining cuts hyperparameter-search energy by 47%","SM2: energy-aware successive halving cuts tuning power by 47%","Energy-aware HPO trims search energy by up to 47%","Cut tuning power 47% with energy-aware successive halving"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that one epoch of training on a quarter of the dataset, with cyclical learning-rate sweeps and real-time energy readings, ranks hyperparameter configurations the same way full multi-epoch training would; if that proxy misleads, successive halving prunes the wrong configurations and the energy savings vanish.","fun_headline_variants_meta":{"raw":{"variants":["Energy-aware successive halving trims tuning power by up to 47%","Exploratory pretraining cuts hyperparameter-search energy by 47%","SM2: energy-aware successive halving cuts tuning power by 47%","Energy-aware HPO trims search energy by up to 47%","Cut tuning power 47% with energy-aware successive halving"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001058,"raw_usage":{"total_tokens":4437,"prompt_tokens":941,"completion_tokens":3496,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":3400}},"tokens_in":557,"tokens_out":3496,"duration_ms":29175,"temperature":1.0,"reasoning_tokens":3400,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:43:56.284859+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain every configuration that SM2 prunes in early rounds for the full 5-10 epochs and compare final performance and total energy against the configuration SM2 selected; if any pruned configuration beats the survivor on both final performance and full-run energy, the exploratory ranking is not a faithful proxy.","supporting_citations":[{"cited_title":"Cyclical learning rates for training neural networks","cited_arxiv_id":null,"evidence_quote":"Provides the cyclical learning-rate scheduling technique used in the exploratory phase to select stable learning rates."},{"cited_title":"Zeus: Understanding and optimizing \\ GPU \\ energy consumption of \\ DNN \\ training","cited_arxiv_id":null,"evidence_quote":"Documents that GPU efficiency windows lie below maximum utilization, supporting the energy-per-epoch objective in batch-size selection."}],"review_version":1}