{"id":"4414b011-880c-4ee1-980a-d547b70ef02a","arxiv_id":"2506.06484","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":7,"one_line_summary":"Reward shaping penalties for low gas-storage levels and inactivity substantially improve deep reinforcement learning for long-horizon power-to-gas economic dispatch, though results are in-sample on one data year.","lead":"This paper tests whether deep reinforcement learning can learn to run a power-to-gas energy storage plant profitably over long time horizons. The authors show that off-the-shelf algorithms often fail at this task, but simple reward penalties can teach them to use the gas storage, closing much of the gap to an ideal optimizer.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline gains in Table 2 are confounded by in-sample tuning: the same full-year 2022 trajectory is used for training, Optuna selection, and final scoring, so the reported 'Combined' improvements may not transfer to held-out data.","rationale":"The reader's weakest assumption correctly identifies the in-sample evaluation as the load-bearing issue. The strongest evidence for the paper's central claim is Table 2, and the condition that would make that evidence compelling is that the measured gain reflects algorithmic capability rather than selection on the evaluation data. That condition is not secured anywhere in the manuscript. Section 4 describes tuning over roughly 1,000 trials on each case study, and Section 5 reports the best 'Combined' configuration on the same full-year 2022 episodes that were used for training. The synthetic CS1 and CS2 studies do not repair this because they reuse the same generated episodes for both training and reporting. The paper also does not release code or data, so the reported numbers cannot be independently reconstructed. This concern is methodological rather than a demonstration that the result is false; a held-out evaluation could well confirm the improvement. But until that is done, the correct verdict remains conditional, matching the reader's assessment.","tokens_in":14655,"tokens_out":6126,"duration_ms":73212,"concrete_test":"Fix the reported hyperparameters and retrain DQN/PPO Base and Combined on 2022 data only, then evaluate the resulting policies on a held-out year (e.g., 2023 AESO prices and renewables.ninja wind for the same turbines). If the Combined-vs-Base reward difference on 2023 is no longer positive beyond the 5-seed standard deviation, or if the gap to the MIQP optimum widens substantially, the Table 2 improvements are an artifact of in-sample selection. A less expensive variant is an 80/20 temporal split of 2022 with all tuning confined to the training portion.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim rests on Table 2's CS3 comparison, where the 'Combined' variants improve rewards by roughly 12-20% over Base. But the evaluation protocol never separates training from test data. Section 4 states that CS3 is the full year 2022 and that hyperparameters are tuned with around 1,000 Optuna trials per case study; Section 5 then reports the best 'Combined' variant on the same 2022 trajectory. Because the penalty parameters (omega=1000, SOC_p,max=0.01, tau=0.7, alpha=0.02) and the choice of which modification combination is best are selected on the exact episodes used to produce Table 2, the reported gains are upper bounds of what can be expected on unseen conditions. The CS1 and CS2 results are synthetic and likewise reuse the same generated episodes for training and evaluation. Without a temporal holdout, the claim that these modifications 'significantly improve' long-horizon P2G dispatch is not yet established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies economic dispatch of a hybrid power-to-gas (P2G) system with battery storage, wind generation, and a gas turbine, using deep reinforcement learning (DQN and PPO). The main contribution is a set of reward-function modifications—future price forecasts, low-SOC penalties, inactivity penalties, and delayed cost attribution—designed to help DRL agents exploit long-term gas storage. The authors evaluate these modifications on three case studies of increasing horizon (24 hours, 168 hours, and one full year, 2022, for a plant in Southern Alberta) and compare against a mixed-integer quadratic programming (MIQP) optimum. The headline result is that the combined modifications raise CS3 episodic rewards from roughly 19,900 to 22,300 kC$ for DQN and from 18,700 to 22,400 kC$ for PPO, narrowing the gap to the MIQP value of 26,087 kC$.","tokens_in":14855,"tokens_out":3419,"duration_ms":36871,"significance":"If the reported improvements generalize to unseen conditions, the paper is a useful step toward making DRL practical for long-horizon energy storage dispatch, where delayed rewards are a recognized difficulty. The evaluation has notable strengths: the MIQP benchmark is a genuine external optimum, all DRL results are averaged over five seeds with standard deviations, and the reward-shaping modifications are disabled during evaluation so that reported rewards are computed on the original objective. The three case studies with increasing episode length also make the delayed-reward challenge concrete. The main weakness is that the evaluation is entirely in-sample: training, hyperparameter tuning, and final scoring all happen on the same data, and the 'Combined' configuration is selected as the best of many variants on that same data. A held-out evaluation is therefore needed before the central claim of 'significantly improved' long-horizon dispatch can be considered established.","major_comments":[{"comment":"The evaluation protocol does not separate training from test data. For CS3, the full year 2022 is used for training, for the approximately 1,000 Optuna trials per case study, and for the final scoring reported in Table 2; CS1 and CS2 similarly reuse the same generated synthetic episodes for both training and evaluation. Because the penalty parameters (omega=1000, SOC_p,max=0.01, tau=0.7, alpha=0.02) and the DRL hyperparameters are tuned on the exact trajectories used to compute the reported rewards, the 12% and 20% improvements for the 'Combined' rows in CS3 are in-sample upper bounds. A temporal holdout (for example, training on earlier years and scoring on 2022, or a cross-year validation scheme) is needed to support the claim that the proposed modifications 'significantly improve' long-horizon P2G dispatch.","section":"Section 4 and Section 5, Table 2"},{"comment":"The 'Combined' variant is selected as 'the best combination found out of all modifications' after inspecting the results of all variants on the same data. This selection-of-best-on-the-evaluation-set introduces a multiple-comparison bias that is material here: for PPO on CS3, the individual variant rewards range from 18,554 to 22,411 kC$, so choosing the maximum as the headline result inflates the expected performance. The authors should either pre-specify the combined configuration before seeing the test results, use a validation split for model selection, or report all combinations with an appropriate correction for multiple comparisons.","section":"Section 5, Table 2 and accompanying text"},{"comment":"The cost-attribution scheme preserves total episode reward only if all produced fuel is burned by the end of the episode. The paper does not state whether the CS3 episodes end with nonzero remaining SOC in the P2G storage; if fuel remains, the training-time reward signal differs from the evaluation-time objective, which could bias the learned policy. The authors should either verify that fuel is fully consumed at episode end or qualify the conservation claim in Algorithm 2.","section":"Section 3.2.4, Algorithm 2"}],"minor_comments":[{"comment":"There are small typographical errors in the reference list: 'visisted' should be 'visited' in both entries, and the arXiv entry for PPO would benefit from a proper DOI or URL formatting.","section":"References [16] and [18]"},{"comment":"The time-series plots in Figure 3 are dense and hard to read at the printed size; enlarging them or splitting the wind and price panels would improve readability.","section":"Figure 3"},{"comment":"The text notes that 'one of the five independent runs failed' for PPO on CS1, but the seed-level breakdown is not shown anywhere; reporting all five runs in a supplementary table would make the variability more transparent.","section":"Section 5"},{"comment":"The computation of rho divides by SOC_{P2G,t-1}; this is safe under feasible states because the GT can only operate when fuel is available, but adding an explicit guard or note would avoid any ambiguity for readers implementing the algorithm.","section":"Algorithm 2, line 11"}],"recommendation":"major_revision","confidential_remarks":"The in-sample evaluation and the selection of the 'Combined' variant on the evaluation data are the main obstacles to publication. A held-out year or a temporally split validation, plus a pre-specified combined configuration, would resolve the concern. The paper is otherwise within the scope of the venue and the experimental design is largely sound."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid engineering study, not a breakthrough. The genuinely new thing is that it takes P2G dispatch beyond the usual 24-hour horizon, sets up a year-long case with real Alberta price and wind data, and shows that two simple reward penalties—one on low gas-storage SOC, one on P2G inactivity during cheap hours—get DQN and PPO to actually use long-term storage that off-the-shelf versions ignore. That is a useful result for people working on DRL for energy storage, and the comparison against an MIQP optimum is the right gold standard. I read the core claim as plausible.\n\nWhat the paper does well: clean MDP formulation, careful modeling of P2G/GT/BES with degradation and maintenance costs, reasonable parameter choices sourced from the literature, and an honest report that forecasts and cost re-attribution alone do little while the SOC penalty matters most. The synthetic CS2 result, where PPO goes from ignoring P2G to near-optimal after the penalties, is the most convincing part of the paper.\n\nSoft spots: the stress-test note is on target. There is no train/test separation anywhere. CS3 uses the full 2022 year for training, Optuna tuning (~1000 trials), sensitivity analysis, and final scoring; the 'Combined' row is chosen because it was the best of several variants on that same trajectory. So Table 2's 12–20% improvements are upper bounds, not expected transfer. CS1 and CS2 reuse the same generated episodes for training and evaluation. The penalty parameters (omega=1000, SOC_p,max=0.01, tau=0.7) and the choice to combine all modifications are all selected on evaluation data. A held-out year or at least a temporal split would make the headline claim solid. The absence of released code or data also makes it hard to check the reward-shaping details. These are fixable in revision, not fatal.\n\nMinor: the paper's own numbers show the combined modifications still leave a 14–17% gap to MIQP on CS3, and variance across seeds is large for PPO on CS3 (e.g., 20,891±2081). That is worth stating plainly, but it does not undermine the main finding.\n\nWho this is for: researchers in energy dispatch and DRL who want practical reward-shaping ideas for long-horizon storage. I would take it seriously in review; the flaws are methodological cleanliness, not a broken core. Recommendation: send to peer review, with a request for a held-out evaluation and code/data release.","headline":"A useful, incremental DRL-for-P2G dispatch study with a solid MIQP benchmark; the headline gains are real on the data shown, but the lack of any held-out evaluation means the 'significantly improve' claim is not yet proven.","tokens_in":15437,"tokens_out":2126,"would_cite":true,"duration_ms":22240,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reward shaping lets deep reinforcement learning dispatch power-to-gas plants over a full year, nearly matching the ideal optimizer.","keywords":["power-to-gas","deep reinforcement learning","economic dispatch","reward shaping","long-term energy storage","delayed rewards","DQN","PPO"],"falsifier":"Run the identical comparison on a different year of the same market (for instance 2023 prices and wind) without retuning or re-selecting variants; if the combined reward designs do not beat the base agents by a comparable margin on that out-of-sample year, the reported improvement is at least partly an artifact of tuning on the evaluation data.","tokens_in":14410,"feed_emoji":"⚡","tokens_out":9941,"duration_ms":93307,"temperature":0.7,"pith_summary":"The paper tries to establish that deep reinforcement learning can handle the economic dispatch of a power-to-gas plant over a full year, despite the long delay between spending money to make synthetic natural gas and earning money by burning it later. Off-the-shelf DQN and PPO agents barely use the gas system; with reward modifications—penalties for low gas-storage state, penalties for leaving the plant idle in cheap-wind hours, delayed attribution of P2G operating costs, and perfect price forecasts—their year-2022 profits rise from roughly 19.9 and 18.7 million Canadian dollars to 22.3 and 22.4 million, against a mixed-integer quadratic programming optimum of 26.1 million. The point of showing this is that long-term gas storage, the main economic advantage of P2G over batteries, is exactly what prior DRL studies ignored by limiting themselves to 24-hour horizons.","feed_headline":"Reward fixes let AI agents use power-to-gas storage over a full year","feed_subtitle":"Penalties and delayed-cost shaping lift deep-RL profits within ~15% of the optimum on 2022 market data.","key_machinery":"The central mechanism is reward shaping. Three additive modifications reshape the immediate reward so that the future value of stored gas is partially moved forward in time: a low-state-of-charge penalty $p_t=\\omega \\max((SOC_{P2G}^{p,\\max}-SOC_{P2G,t})/SOC_{P2G}^{p,\\max},0)$ that pushes the agent to keep about 0.01 of the gas tank (enough for roughly 22 MWh) available for price spikes; an inactivity penalty that subtracts $\\omega$ when the wholesale price falls below $\\tau=0.7$ times its running mean and the plant stays off despite enough wind; and a cost-attribution algorithm that zeroes the P2G operating cost and the lost-sales compensation at the charging step, banks them in two running sums, and charges them to the later fuel-burning steps through a deduction ratio $\\rho$. Perfect price forecasts for 1 to 24 hours are added to the state but contribute little. The shaping works because the discounted-return objective makes delayed rewards nearly invisible; these terms make part of the gas-storage value visible immediately.","core_discovery":"On its own terms, the paper's claim is that P2G dispatch fails for DRL not because of uncertainty or model complexity but because the reward signal arrives too slowly: buying electricity, paying for CO2, and running an inefficient conversion chain produces no revenue until the gas is burned, possibly many hours later. The paper demonstrates this by building a wind-plus-battery, electrolyzer-methanation, gas-turbine plant and testing DQN and PPO on three horizons. In the hardest setting, a full year of real market data, the base agents barely operate the P2G system and score below the battery-only benchmark, while the combined reward design produces policies that run the P2G system for over 2,500 hours and the gas turbine for about 400 hours, recovering most of the gap to the MIQP optimum. The paper interprets this as unlocking long-term storage: a 12% gain for DQN and 20% for PPO come specifically from getting the agents to fill and later empty the gas storage.","pith_inferences":["Editorial extension: because the 'Combined' variant was chosen as the best of many after experiments on the same year, the reported 12% and 20% gains are likely optimistic; a holdout-year evaluation is the natural check.","Editorial extension: the inactivity penalty is essentially a hand-coded 'buy cheap, store' heuristic, so a reasonable next step is replacing it with a learned opportunity-cost baseline that adapts to seasonal price patterns.","Editorial extension: the cost-attribution scheme leaves total episode reward unchanged and barely helps, which suggests the bottleneck is not accounting timing but the agent's inability to propagate value through hundreds of steps; methods like hindsight relabeling or option-based temporal abstraction may be more effective.","Editorial extension: the SOC-penalty result implies a practical design rule—if a DRL controller cannot see the value of stored gas, add a minimum-reserve constraint as a safety layer; the penalty is essentially a soft version of that constraint."],"forward_implications":["Reward shaping that punishes empty gas tanks and idle conversion lets DRL agents use long-term gas storage, whereas off-the-shelf agents leave the P2G plant almost unused.","The low-SOC penalty is the most influential single modification; its best setting keeps about 22 MWh of fuel in reserve, which the agent can burn during one price spike.","With all modifications combined, DRL reaches roughly 85% of the ideal mixed-integer quadratic programming profit on a full year of real market data.","Perfect short-term price forecasts are not the answer: they improve results only slightly, suggesting the core difficulty is credit assignment over long storage cycles, not near-term price uncertainty.","The same modifications transfer from 24-hour and weekly episodes to a 8760-step year-long episode, indicating the reward-shaping effect is not specific to one horizon."],"supporting_citations":[{"why":"Establishes the earlier DRL dispatch formulation for wind-P2G-GT that this paper extends from a 24-hour horizon to long-term storage.","marker":"[6]"},{"why":"Shows the same hybrid system has been controlled by DRL before but only in short-horizon settings.","marker":"[7]"},{"why":"Supplies the forecasting approach and the earlier evidence that price forecasts improve DRL arbitrage.","marker":"[1]"},{"why":"The prior BES study whose reward-penalty conclusions provide the contrast for the P2G penalty results.","marker":"[2]"},{"why":"Introduces the gas-turbine maintenance cost allocation used in the plant model to discourage frequent start-ups.","marker":"[11]"},{"why":"Defines the DQN algorithm that is one of the two DRL methods tested.","marker":"[14]"},{"why":"Defines the PPO algorithm that is the other DRL method tested.","marker":"[15]"},{"why":"Supplies the wholesale electricity price series used for the year-long case study.","marker":"[16]"},{"why":"Supplies the wind power time series modeled at the plant location for the year-long case study.","marker":"[17]"},{"why":"The mixed-integer quadratic programming solver used to compute the optimal benchmark that DRL results are compared against.","marker":"[26]"}],"fun_headline_variants":["Reward shaping unlocks long-term storage for deep-RL power-to-gas","Delayed rewards fixed: AI runs power-to-gas storage profitably","Deep-RL agents learn to fill gas storage with reward penalties","New reward tricks let DRL utilize power-to-gas over a year","How to make AI wait for gas: reward design for P2G dispatch"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that training and tuning on the same full year that is also used for the reported scores gives a fair performance measure; there is no held-out year to test on.","fun_headline_variants_meta":{"raw":{"variants":["Reward shaping unlocks long-term storage for deep-RL power-to-gas","Delayed rewards fixed: AI runs power-to-gas storage profitably","Deep-RL agents learn to fill gas storage with reward penalties","New reward tricks let DRL utilize power-to-gas over a year","How to make AI wait for gas: reward design for P2G dispatch"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000207,"raw_usage":{"total_tokens":1461,"prompt_tokens":1068,"completion_tokens":393,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":684,"completion_tokens_details":{"reasoning_tokens":300}},"tokens_in":684,"tokens_out":393,"duration_ms":4518,"temperature":1.0,"reasoning_tokens":300,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:55:13.058174+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical comparison on a different year of the same market (for instance 2023 prices and wind) without retuning or re-selecting variants; if the combined reward designs do not beat the base agents by a comparable margin on that out-of-sample year, the reported improvement is at least partly an artifact of tuning on the evaluation data.","supporting_citations":[{"cited_title":"Inte- grated Electricity-Gas System Optimal Dispatch Based on Deep Reinforcement Learning","cited_arxiv_id":null,"evidence_quote":"Shows the same hybrid system has been controlled by DRL before but only in short-horizon settings."},{"cited_title":"Deep reinforce- ment learning for economic battery dispatch: A compre- hensive comparison of algorithms and experiment design choices","cited_arxiv_id":null,"evidence_quote":"The prior BES study whose reward-penalty conclusions provide the contrast for the P2G penalty results."},{"cited_title":"Optimal Economic Gas Turbine Dis- patch with Deep Reinforcement Learning","cited_arxiv_id":null,"evidence_quote":"Introduces the gas-turbine maintenance cost allocation used in the plant model to discourage frequent start-ups."},{"cited_title":"Market and sys- tem reporting","cited_arxiv_id":null,"evidence_quote":"Supplies the wholesale electricity price series used for the year-long case study."},{"cited_title":"Gurobi Optimizer Reference Manual","cited_arxiv_id":null,"evidence_quote":"The mixed-integer quadratic programming solver used to compute the optimal benchmark that DRL results are compared against."}],"review_version":1}