{"id":"729e9577-7fdf-4d87-bd2c-26f39529b7db","arxiv_id":"2509.05273","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A systematic Atari benchmark finds that Augmented Random Search delivers the best reward per kilowatt hour, while RecurrentPPO and QR-DQN are the least energy-efficient.","lead":"Seven game-playing AI training methods were compared for electricity use and cost. Simpler methods like ARS used far less energy than heavier methods, though sometimes with weaker game scores.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline NPpkWh ranking is not reproducible from the paper's own stated metric: raw reward/energy puts TRPO above ARS, so the 4.5x ARS claim rests on an undefined normalization.","rationale":"The reader's verdict is CONDITIONAL and I agree it should remain conditional, but the strongest reason is more specific than single-seed variance. The central claim is a quantitative ranking with a headline ratio. That claim is not reproducible from the paper's stated definition of NPpkWh; under the literal definition TRPO outranks ARS. This is an internal inconsistency, not a disagreement with consensus, and it is load-bearing because the top algorithm and the reported 4.5x factor can change with the normalization. Single-seed variance is a real limitation and the paper acknowledges it, but it is secondary: if the metric were well-defined, the broad energy ordering might still hold; if the metric is undefined, the headline comparison cannot be evaluated at all. I also checked the energy/cost tables: the national cost column is evidently scaled by $1.401/kWh instead of $0.1401/kWh, a factor-of-ten error that does not change rankings but confirms the numeric pipeline was not cross-checked. The paper is a useful benchmark effort and the qualitative direction (algorithm choice affects energy) is plausible; the condition should be to specify the metric precisely and release code/data so Table I can be reproduced. This matches the reader's call for conditional acceptance, so no verdict change.","tokens_in":17093,"tokens_out":25894,"duration_ms":190687,"concrete_test":"Recompute NPpkWh from Tables IV-XIII under several definitions: (a) literal reward/energy from Table III; (b) per-game reward normalized by max reward before averaging and dividing by energy; (c) min-max normalized reward; (d) per-game reward/energy normalized to per-game max. Compare each against Table I. If none reproduces all seven entries, ask the authors to release the exact formula and the script that generated Table I; the 'ARS highest / 4.5x' claim should be removed or revised until the metric is specified and reproducible. Also check whether any normalization changes the top-ranked algorithm from ARS to TRPO.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV defines NPpkWh as 'dividing the average episodic reward achieved by each model by its total energy consumption.' Using the reported aggregate values in Table III, this definition yields TRPO = 2459.14/10.96 = 224.4 reward/kWh vs ARS = 606.04/2.83 = 214.2 reward/kWh; TRPO, not ARS, would be the most efficient. Table I instead reports ARS 0.08142 vs TRPO 0.04838 and claims a 4.3-4.5x ARS advantage over QR-DQN. No normalization formula, baseline, or per-game scaling is specified anywhere in Section IV or the table notes, so the headline ranking is not derivable from the paper's own methodology or data. The abstract's '24% less energy (ARS vs. DQN)' is also inconsistent with Table III (ARS uses 75.6% less than DQN), and the National cost column multiplies by $1.401 rather than $0.1401/kWh. These are not cosmetic: the central claim is a ranking/ratio claim, and the only stated computation would put a different algorithm on top. Until the exact NPpkWh formula and reproduction script are provided, the 4.5x efficiency claim is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents an empirical benchmark of seven deep reinforcement learning algorithms (DQN, TRPO, A2C, ARS, PPO, RecurrentPPO, and QR-DQN) trained for one million steps on ten Atari 2600 games, with real-time power measurement used to estimate total energy consumption, CO2-equivalent emissions, and electricity cost. The authors define a metric, NPpkWh (average episodic reward divided by total energy consumption), and use it to rank the algorithms, reporting that ARS is the most energy-efficient (about 4.5x more efficient than QR-DQN) and that RecurrentPPO is generally the most energy-intensive. The paper also analyzes reward stability, training time, and cost trade-offs, and it concludes that algorithmic choice alone can yield substantial differences in energy, emissions, and monetary cost under identical hardware and software. The limitations section acknowledges the use of a single training seed per algorithm-game pair and a fixed hardware configuration.","tokens_in":17339,"tokens_out":5720,"duration_ms":48269,"significance":"If the numerical results were corrected and reproducible, this would be a useful empirical contribution to the Green AI literature for reinforcement learning, an area that has received less systematic study than LLM training. The authors provide detailed per-game tables, use standard external measurement tools (CodeCarbon/pyRAPL and nvidia-smi), and connect their measurements to real carbon-intensity and electricity-price data. The win-summary table is a helpful at-a-glance addition, and the paper does not rely on derived equations or fitted parameters, so circularity is not a concern. However, the central algorithmic ranking and the headline quantitative claims are currently not supported by the paper's own stated metric and data, and several internal arithmetic inconsistencies need to be resolved before the findings can be accepted.","major_comments":[{"comment":"Section IV defines NPpkWh as 'dividing the average episodic reward achieved by each model by its total energy consumption.' Using the aggregate values in Table III, this definition gives TRPO = 2459.14/10.96 ≈ 224.4 reward/kWh and ARS = 606.04/2.83 ≈ 214.2 reward/kWh, so TRPO, not ARS, would be the most efficient under the stated definition. Table I instead reports ARS (0.08142) above TRPO (0.04838) and claims a 4.50x/4.30x advantage over QR-DQN, but no normalization formula, per-game scaling, or baseline is specified in Section IV or in the table notes. Please state the exact NPpkWh formula, provide the per-game normalized values, and reconcile Table I with Table III; the current headline ranking is not derivable from the paper's own definition and data.","section":"Section IV, Table I"},{"comment":"The abstract's '24% less energy (ARS vs. DQN)' is contradicted by Table III, where ARS consumes 2.83 kWh versus DQN's 11.60 kWh, a 75.6% reduction. The abstract's claims of 'nearly 68% less CO2' and '68% lower monetary cost (QR-DQN vs. RecurrentPPO)' are also not supported by Table III: emissions are 11.68 vs. 17.11 kgCO2eq (about 31.7% lower) and national costs are $22.26 vs. $32.57 (also about 31.7% lower). Please correct the abstract or clarify the exact comparison basis for each claimed percentage.","section":"Abstract, Table III"},{"comment":"The Cost (National) column in Table III is inconsistent with the stated U.S. national average electricity price of $0.1401/kWh. For example, A2C should cost 14.16 kWh × $0.1401/kWh = $1.98, not $19.84, and every national cost in Tables III through XIII is larger by a factor of ten than the product of the listed energy and $0.1401/kWh, implying that the authors actually multiplied by $1.401/kWh. This factor-of-ten error propagates to the Section V scaling example (the claimed $2,861 savings over 100 million steps) and to the abstract's cost claims; all cost figures and derived statements should be recomputed.","section":"Table III, Section V"},{"comment":"The paper acknowledges in Section V that it uses 'a single training seed per run,' yet the central claims are algorithm-level rankings, such as 'ARS achieved the highest NPpkWh' and 'RecurrentPPO consistently incurs the highest energy usage.' Given the well-documented variance of deep reinforcement learning training, one seed per algorithm-game pair cannot support these ranking claims without confidence intervals or multiple-seed averages. Either add multi-seed runs, report per-cell variance, or substantially soften the ranking claims to reflect the sampling premise.","section":"Section V, Limitations"},{"comment":"Section IV states that power was 'measured separately for the CPU, RAM, and GPU, and then aggregated,' but it does not specify the sampling frequency, the integration method used to convert power samples to kWh, the treatment of idle or background power, or how the RAPL and nvidia-smi measurements are combined into a single system energy value. Without this measurement protocol, the energy, emission, and cost numbers in Tables III through XIII are not independently reproducible, which is a prerequisite for a benchmarking study of this type.","section":"Section IV, Methodology"}],"minor_comments":[{"comment":"The text states that 'eight' algorithms were benchmarked, but the enumerated list contains seven algorithms (DQN, TRPO, A2C, ARS, PPO, RecurrentPPO, and QR-DQN); please correct the count consistently throughout the paper.","section":"Section I, Section IV"},{"comment":"The phrase 'Tables tables IV to XIII' should be 'Tables IV to XIII'.","section":"Section IV"},{"comment":"Figure 21 (Video Pinball, binned) omits TRPO from the legend even though TRPO has the highest reported average reward for that game in Table XIII; please add the missing series or explain its omission.","section":"Figure 21"},{"comment":"The methodology says that 'default SB3 policy implementations' were used with MlpPolicy for most models, but standard Atari benchmarks typically use convolutional policies and ARS uses a linear policy; please state explicitly which policy architecture was used for each algorithm, since architecture directly affects energy measurements and comparability with prior work.","section":"Section IV, Section III-C"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for a machine learning journal and addresses a genuine gap, but the numerical inconsistencies in the abstract, the undefined normalization in the NPpkWh metric, and the factor-of-ten cost error are load-bearing and must be fixed before the results can be considered reliable. The single-seed limitation further weakens the algorithmic ranking claims. I would support reconsideration after a careful revision that provides the exact metric definition, corrected tables, and a reproducible measurement protocol."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this is a useful first dataset, not a reliable headline claim. The paper measures energy, carbon, and cost for seven DRL algorithms on ten Atari games, which is exactly the kind of benchmarking the Green AI literature has been asking for. But the central NPpkWh ranking is not derivable from the paper's own definition, and several numbers in the text disagree with the tables.\n\nWhat's genuinely new: the systematic cross-algorithm, multi-game energy comparison. The per-game tables (energy, time, emissions) are plausible and will be useful to practitioners. The authors also correctly flag the single-seed limitation, which is honest.\n\nThe soft spots are significant. Section IV defines NPpkWh as dividing average episodic reward by total energy consumption. Using Table III, TRPO gives 2459.14/10.96 = 224.4 reward/kWh and ARS gives 606.04/2.83 = 214.2. So TRPO should rank first, not ARS. Table I reports ARS at 0.08142 and TRPO at 0.04838, with no explanation of the normalization or per-game scaling. The 4.5x ARS-over-QR-DQN claim rests on this undefined metric. The abstract says ARS uses 24% less energy than DQN, but Table III shows 75.6% less (2.83 vs 11.60 kWh). And the 'National' cost column is off by a factor of ten: $19.84 for 14.16 kWh implies $1.401/kWh, not the stated $0.1401/kWh. These are not typos in one cell; they are the paper's main quantitative arguments.\n\nThe broad direction is credible: derivative-free methods are cheap, RecurrentPPO is expensive, and sample efficiency matters. The energy measurements themselves look reasonable. But as written, the headline ratios cannot be checked or trusted. The paper also has no released code or data, and one seed per cell means the ranking could easily change with different seeds.\n\nWho is this for? Anyone who wants rough energy/cost estimates for DRL training on this hardware, and researchers working on sustainability metrics for RL. It deserves a serious referee, but the referee should demand a clear NPpkWh definition, a corrected cost column, consistent abstract percentages, and a release of seeds and raw measurements. If those are fixed, this becomes a solid contribution. Right now, treat it as a useful dataset with an unsupported headline.","headline":"Useful first energy benchmark for DRL, but the headline NPpkWh ranking is unsupported by the paper's own definition and numbers.","tokens_in":17852,"tokens_out":4052,"would_cite":false,"duration_ms":34904,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A seven-algorithm Atari benchmark shows algorithmic choice alone changes a training run's energy use by a factor of about 4.5.","keywords":["deep reinforcement learning","energy efficiency","carbon emissions","Atari benchmark","normalized performance per kilowatt hour","derivative-free optimization","sustainable machine learning"],"falsifier":"Run the same seven algorithms on the same ten games with several random seeds per algorithm-game pair and recompute median NPpkWh on the same hardware: if ARS is not the median top performer, or if the spread between ARS and QR-DQN shrinks dramatically, the paper's central ranking claim is not supported.","tokens_in":16893,"feed_emoji":"⚡","tokens_out":6507,"duration_ms":58450,"temperature":0.7,"pith_summary":"This paper is trying to establish that the choice of deep reinforcement learning algorithm, holding hardware and software fixed, is a first-order driver of energy consumption, CO2 emissions, and electricity cost. It benchmarks seven algorithms on ten Atari 2600 games for one million training steps each, measuring power in real time and converting the readings into energy, emissions, and cost estimates. The headline result is a roughly 4.5x spread in normalized performance per kilowatt hour: derivative-free Augmented Random Search scores 0.08142 while QR-DQN scores 0.01889, and ARS uses about 2.83 kWh per run versus RecurrentPPO's 23.25 kWh. A sympathetic reader would care because, if true, energy-aware algorithm selection is an immediate, no-hardware-change lever for making DRL research cheaper and greener.","feed_headline":"Algorithm choice shifts training energy use 4.5-fold","feed_subtitle":"A seven-algorithm Atari benchmark finds derivative-free random search uses far less electricity than value-based learners.","key_machinery":"The measurement-and-normalization pipeline carries the argument. Power is read in real time per component, aggregated to total energy, converted to emissions using live grid carbon intensity, and priced using the U.S. national average electricity rate. The comparison metric, NPpkWh, divides each algorithm's average episodic reward by its total energy consumption, so a high value means the algorithm buys reward cheaply in energy terms. The paper uses that metric to rank algorithms and to trace efficiency back to algorithmic families: derivative-free random search avoids backpropagation, while trust-region and clipped-ratio policy gradients save energy through sample efficiency.","core_discovery":"On its own terms, the paper's central claim is that algorithm choice alone produces large and consistent differences in energy consumption, CO2-equivalent emissions, and electricity cost across a standard Atari benchmark, with a single fixed workstation and one million training steps per game. ARS achieved the highest Average Normalized Performance per Kilowatt Hour (NPpkWh) at 0.08142, nearly 4.50 times the lowest value, QR-DQN at 0.01889. In aggregate, RecurrentPPO used the most energy (23.25 kWh) and emitted the most CO2 (17.11 kgCO2eq), while ARS used the least (2.83 kWh and 2.08 kgCO2eq). The paper also reports that TRPO won the most games on reward (5 of 10) while ARS won every game on lowest energy and emissions, and that reward and energy do not move together: RecurrentPPO and QR-DQN can reach high rewards while spending disproportionately more energy. The proposed explanation is that sample efficiency and stable learning dynamics, rather than per-step computational simplicity, drive energy efficiency.","pith_inferences":["I infer the exact ranking is hardware- and seed-specific; the robust finding is the existence of a large efficiency spread, not the precise ordering.","A multi-seed, multi-hardware version of this benchmark would likely be more valuable than the single-run result, and would tell whether the spread compresses or grows.","ARS's advantage is probably domain-limited: in sparse-reward or high-dimensional tasks, gradient-based methods' ability to exploit small reward signals likely erodes the gap.","Energy reporting could be inserted into standard RL benchmark workflows at almost no additional cost, making sustainability comparisons routine."],"forward_implications":["Researchers in dense-reward, structured tasks can expect severalfold energy and cost reductions by using derivative-free policy search instead of deep value-based methods, without necessarily sacrificing reward.","Adding NPpkWh to standard reporting would make energy a first-class evaluation axis, allowing cost-per-performance comparisons across papers.","The savings multiply with scale: the paper estimates replacing a RecurrentPPO run with an ARS run over 100 million steps saves about $2,861 in electricity at U.S. national average rates.","Because sample efficiency predicted low energy use better than per-step cost, future greener-DRL work should target sample efficiency as well as network size.","Algorithms with stable reward trajectories tend to need fewer redundant training steps, so stability metrics can serve as proxy indicators of energy efficiency."],"supporting_citations":[{"why":"Introduces the DQN algorithm and its training procedure, one of the seven benchmarked methods.","marker":"[18]"},{"why":"Introduces Augmented Random Search, the algorithm the paper finds most energy-efficient.","marker":"[23]"},{"why":"Introduces PPO and the recurrent variant used in the benchmark.","marker":"[21]"},{"why":"Introduces TRPO, the trust-region policy optimization method benchmarked here.","marker":"[22]"},{"why":"Introduces the actor-critic framework behind A2C.","marker":"[20]"},{"why":"Introduces QR-DQN, the distributional value-based method that ranks least efficient.","marker":"[19]"},{"why":"Supplies the standardized algorithm implementations used for all runs.","marker":"[29]"},{"why":"Supplies the power-measurement integration used to record energy consumption.","marker":"[33]"},{"why":"Supplies live electricity carbon-intensity data for emission estimates.","marker":"[34]"},{"why":"Defines the Atari environment suite used as the benchmark.","marker":"[35]"}],"fun_headline_variants":["Algorithm choice changes DRL energy use 4.5x","Energy-smart DRL: ARS uses 68% less than RecurrentPPO","DRL training energy varies 4.5-fold across algorithms","Green RL: pick ARS to cut energy and CO2 by 68%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that one training run per algorithm and game represents that algorithm's true energy and reward behavior; deep RL results are seed-sensitive, so a different seed could reorder the measured efficiency ranking.","fun_headline_variants_meta":{"raw":{"variants":["Algorithm choice changes DRL energy use 4.5x","Energy-smart DRL: ARS uses 68% less than RecurrentPPO","DRL training energy varies 4.5-fold across algorithms","Green RL: pick ARS to cut energy and CO2 by 68%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000715,"raw_usage":{"total_tokens":3270,"prompt_tokens":1057,"completion_tokens":2213,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":673,"completion_tokens_details":{"reasoning_tokens":2132}},"tokens_in":673,"tokens_out":2213,"duration_ms":14102,"temperature":1.0,"reasoning_tokens":2132,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:23:57.528729+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same seven algorithms on the same ten games with several random seeds per algorithm-game pair and recompute median NPpkWh on the same hardware: if ARS is not the median top performer, or if the spread between ARS and QR-DQN shrinks dramatically, the paper's central ranking claim is not supported.","supporting_citations":[{"cited_title":"The arcade learning environment: An evaluation platform for general agents,","cited_arxiv_id":null,"evidence_quote":"Defines the Atari environment suite used as the benchmark."},{"cited_title":"Simple random search of static linear policies is competitive for reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Introduces Augmented Random Search, the algorithm the paper finds most energy-efficient."},{"cited_title":"Trust region policy optimization,","cited_arxiv_id":null,"evidence_quote":"Introduces TRPO, the trust-region policy optimization method benchmarked here."},{"cited_title":"Stable-Baselines3: Reliable reinforcement learning implementations,","cited_arxiv_id":null,"evidence_quote":"Supplies the standardized algorithm implementations used for all runs."},{"cited_title":"United states of america 2024 carbon intensity data,","cited_arxiv_id":null,"evidence_quote":"Supplies live electricity carbon-intensity data for emission estimates."}],"review_version":1}