{"id":"7a612aa6-621b-4392-8132-f635efbbfe80","arxiv_id":"2411.16519","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":7,"one_line_summary":"A DDPG agent learns offering curves for a day-ahead electricity seller from historical Italian PUN prices, but the paper only shows training curves and never demonstrates a validated profit improvement.","lead":"This paper applies the Deep Deterministic Policy Gradient algorithm to a simplified day-ahead electricity market, where a price-taking seller learns stepwise offering curves from Italian hourly price data. It reports decreasing training losses but no quantitative profit gains, no comparison with simple baselines, and no out-of-sample validation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (7) rewards every accepted bid as though all K production modes produced it and never uses capacities D_k, so DDPG optimizes a fictitious payoff; the reported loss and reward curves do not support the claimed profit-maximizing bidding.","rationale":"I read the manuscript in good faith. The main claim is that the DDPG policy improves real bidding profit. The weakest point is the reward in Eq. (7), which is the only profit signal the agent sees. As written, it double-counts every accepted volume over all production modes and ignores capacity constraints. This makes the learned policy optimal for a fictitious settlement, so the training curves cannot support the claim. I agree with the reader's weakest_assumption; the evaluation-only-losses issue is secondary. Since the central evidence is invalidated by this misspecification, I would keep the reader's REJECT verdict. No change to the verdict is needed, though a revised version that corrects Eq. (7) and re-runs the experiments could be reconsidered.","tokens_in":10363,"tokens_out":9017,"duration_ms":85550,"concrete_test":"Compute, for a single hour with PUN=50 and a single accepted bid (P=50, V=100), both sides: Eq. (7) yields 5000; a capacity-respecting merit-order dispatch with C=[10,30,60], D=[30,200,800] yields 2600 (30 MWh at cost 10, 70 MWh at cost 30). If the values differ, the reward is not the market payout. To see the impact on the paper's evidence, re-run the reported DDPG protocol with this corrected reward and check whether the normalized reward still increases and whether the resulting optimal offering curve respects D_k.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's claim is that the agent learns to maximize individual payout, which requires the reward to equal the payout of the described settlement. Eq. (7) does not. It sums (P_i*V_i - C_k*V_i) over all K modes for every accepted bid i, so each MWh is paid K times and charged with all K marginal costs. The capacity limits D_k, introduced in Sec. 2.2 and set to [30,200,800] in Sec. 4.1, never enter the reward and never constrain the action; the actor can offer volumes exceeding capacity and Eq. (7) rewards them in full. Concretely, with K=3, C=[10,30,60], one accepted bid P=50, V=100, Eq. (7) gives r=100*(3*50-100)=5000, while the largest feasible dispatch profit is (50-10)*30+(50-30)*70=2600 (the high-cost unit is not used). The normalized reward used in the learning curves is computed from the same misspecified formula, so a normalized reward near 1 does not correspond to high real profit. The decreasing policy/critic losses and increasing normalized reward are therefore evidence only of fitting to a synthetic objective. This is an internal model error, not a difference of market-design opinion.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a Deep Deterministic Policy Gradient (DDPG) agent that learns a stepwise offering curve for a seller in the Italian day-ahead electricity market. The state is a 7-day window of historical PUN prices, the action is an I-step price-volume curve, and the reward (Eq. (7)) is defined as the revenue from accepted bids minus production costs across K production modes. The authors train the agent on four years of hourly PUN data and report policy loss, critic loss, and a normalized reward during training, concluding that the agent learns to operate with greater efficiency and maximize payout.","tokens_in":10634,"tokens_out":3755,"duration_ms":39377,"significance":"The problem is practically relevant, and the use of DDPG for continuous-action bidding on real price data is a reasonable direction that could extend earlier Q-learning approaches. However, the current manuscript does not establish its central claim. The reward function in Eq. (7) is internally inconsistent as a model of profit, and the only reported evidence consists of training curves for that same misspecified objective. There is no baseline, no out-of-sample evaluation, and no test of whether the learned policy yields higher profit under a correct market settlement. The paper is transparent about its assumptions and limitations, which is a positive feature, but the numerical results do not support the headline conclusion as written.","major_comments":[{"comment":"The reward definition double-counts volume across production modes. For each accepted bid i, Eq. (7) sums (Pi*Vi - Ck*Vi) over all k=1..K, so each MWh is paid K times and charged with all K marginal costs. With the paper's values K=3, C=[10,30,60], one accepted bid at P=50, V=100 gives r=100*(3*50 - 100)=5000, whereas the largest feasible dispatch profit, respecting capacities D=[30,200,800] and allocating volume to the cheapest modes first, is (50-10)*30+(50-30)*70=2600. The DDPG agent therefore maximizes a fictitious payoff, not the realized profit described in the text, so the reported reward and loss curves are not evidence of profit-maximizing bidding.","section":"Sec. 2.2, Eq. (7)"},{"comment":"The capacity constraints D_k, set to [30,200,800] in Sec. 4.1, never enter the reward and never constrain the action. The actor network can propose volumes exceeding a mode's capacity and Eq. (7) rewards those volumes in full. A correct profit model must allocate each accepted bid's volume across production modes up to their capacities D_k, e.g., in merit order, and compute profit from the allocated volumes and each mode's cost. Without this, the learned policy has no relationship to the physical or economic constraints of the seller.","section":"Sec. 2.2 and Sec. 4.1"},{"comment":"The normalized reward is defined as r/rmax, where both r and rmax are computed from the same misspecified Eq. (7) and the same PUN series used for training. This makes the reported normalized reward circular as a performance metric. Furthermore, the only reported evidence is training dynamics: decreasing policy loss, decreasing critic loss after an initial spike, and increasing normalized reward. There is no baseline strategy (e.g., bid at marginal cost), no evaluation on a held-out time period, and no comparison of the learned policy's profit under a correct settlement. The abstract's claim that the participant gains the ability to maximize individual payout is therefore unsupported by the presented experiments.","section":"Sec. 4.2"}],"minor_comments":[{"comment":"The caption says the reward is computed using accepted offers with 'offered price less than the registered pun', but Eq. (7) uses Pi <= PUN_{t+1}; the caption should say 'less than or equal to'.","section":"Sec. 2.2, Fig. 5 caption"},{"comment":"The sentence 'The curve of the Actor NN is obtained as the offering that minimizes the reward (in terms of obtained profit)' should read 'maximizes', since the agent optimizes expected cumulative reward.","section":"Sec. 2.2, text near Eq. (7)"},{"comment":"The notation C_i = [10, 30, 60] and D_i = [30, 200, 800] uses the step index i, while Section 2.2 uses i for bid steps and k for production modes. Please use C_k and D_k to avoid confusion.","section":"Sec. 4.1"},{"comment":"The sentence 'Another hypothesis we introduce is that the agent only knows about its expenses...' is an assumption, not a hypothesis; rephrase to 'We assume that the agent knows only its own expenses, available resources, and historical prices.'","section":"Sec. 4.1"},{"comment":"The sentence 'The first type consists of the action space’s partial observability...' reads awkwardly because the list that follows mixes market-design challenges and RL challenges; consider splitting into two explicit enumerations.","section":"Sec. 4.3"},{"comment":"The paper does not mention whether code or trained agents will be released; providing reproducible code and the exact data preprocessing (e.g., timezone handling for PUN data) would strengthen future revisions.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The central flaw in Eq. (7) is a genuine internal inconsistency, not a matter of market-design interpretation. However, it is correctable: the authors can redefine the reward to respect capacities and allocate volume to modes, then rerun the experiments. The manuscript also needs a baseline and an out-of-sample profit evaluation before the central claim can be taken seriously. Given the paper's framing as an application study, I do not see the error as fatal beyond revision, but the revision must be substantive and the numerical evidence must be re-generated. I also note that the paper would benefit from a more thorough literature comparison with recent deep-RL bidding papers beyond [19]."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: this paper should not be published in its current form because the objective the agent optimizes is not the profit the paper claims to maximize. The stress-test holds up: Eq. (7) sums (P_i V_i - C_k V_i) over all K modes for every accepted bid, so each MWh of an accepted offer is paid K times and charged with every mode's cost. The capacities D_k, introduced in Sec. 2.2 and set to [30,200,800] in Sec. 4, never enter the reward or constrain the action. With K=3, one accepted bid at P=50, V=100, C=[10,30,60] gives r=5000 under Eq. (7), whereas the largest feasible dispatch profit is 2600. The normalized reward is computed from the same misspecified formula, so a normalized reward near 1 says nothing about real profitability. This isn't a market-design disagreement; it is an internal inconsistency between the described settlement and the implemented reward.\n\nWhat the paper does well: it is clearly written, the literature review is honest, and the authors are explicit about the simplifications (deterministic costs/capacities, single-agent price-taking). Using four years of Italian PUN data is a reasonable first step, and the DDPG implementation with replay buffer, OU noise, and hyperparameter discussion is standard but competently described. The discussion of limitations in Sec. 4.3 is candid.\n\nThe soft spots beyond Eq. (7): the only evidence is training curves (policy loss, critic loss, normalized reward), all of which measure fit to the training objective. There is no baseline (e.g., a simple rule-based bidder, or the Q-learning of [18]), no out-of-sample profit test, and no error bars across the five simulations. Even if the reward were corrected, the evidence would not support the abstract's claim that the participant gains the ability to maximize payout. The novelty is modest—DDPG for strategic bidding already appears in [19], which the authors cite; the contribution here is a dataset change and a simplified cost structure.\n\nWho this is for: readers interested in a tutorial-style application of DDPG to a day-ahead bidding problem might learn something from the setup, but only after the reward is fixed and the evaluation is strengthened. As is, I would not cite it. It doesn't deserve peer review in this state; I'd desk-reject and invite a resubmission with a corrected reward function, capacity constraints, and a real benchmark.\n\nRecommendation: reject, but the authors seem capable of fixing it. The core idea—continuous action-space bidding via actor-critic on real price data—is fine; the execution is the problem.","headline":"A cleanly written application paper whose reward function is misspecified: Eq. (7) double-counts volumes across production modes, so the learning curves don't support the profit-maximization claim.","tokens_in":11157,"tokens_out":2388,"would_cite":false,"duration_ms":22852,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["91B26","37N40"],"pacs":[],"model":"deepseek-v4-flash","headline":"A Deep Deterministic Policy Gradient agent can learn a profitable stepwise offering curve for a day-ahead electricity seller using only historical PUN prices, with falling policy loss and rising normalized reward as evidence.","keywords":["Day-ahead electricity market","Bidding strategy optimization","Reinforcement learning","Deep Deterministic Policy Gradient","Market clearing price","PUN price","Italian electricity market","EUPHEMIA"],"falsifier":"Settle a held-out week of PUN prices twice for the trained policy: once with the paper's reward as printed, and once with each accepted volume allocated to the cheapest available mode up to its capacity $D_k$. If the two settlements rank alternative policies differently, or if the reported convergence disappears when the reward is corrected, the learning signal is an artifact of the summed reward rather than true profit.","tokens_in":10143,"feed_emoji":"⚡","tokens_out":8452,"duration_ms":74005,"temperature":0.7,"pith_summary":"The paper sets out to show that a price-taking electricity seller can learn a profitable day-ahead bidding strategy without modeling rivals. The agent's state is the previous seven days of Italian PUN prices, its action is a stepwise offering curve of volumes and prices, and its reward is the profit from the steps accepted at the next-day clearing price. The authors adapt Deep Deterministic Policy Gradient to this continuous-action problem and report that, over training, the policy loss falls, the critic loss spikes and then recovers after roughly 300 episodes, and the normalized reward rises toward one. If the claim holds, a market participant could turn public price history into an automated offering schedule that improves over time.","feed_headline":"DDPG learns day-ahead bidding curves from price history alone","feed_subtitle":"Historical Italian prices alone let a deep RL agent learn offering curves that raise its normalized reward.","key_machinery":"The workhorse is Deep Deterministic Policy Gradient with feed-forward actor and critic networks, soft-updated target networks, a replay buffer, and Ornstein-Uhlenbeck exploration noise. The action is a stepwise offering curve with three volume-price steps; the state is a 7-day batch of PUN prices; the reward is Eq. (7), which uses the indicator $\\mathbf{1}_{\\{P_i \\le \\mathrm{PUN}_{t+1}\\}}$ to count accepted steps and subtracts production cost $C_k$ for each mode. The deterministic policy gradient update weaves the actor through the critic's Q-value gradient, which is the mechanism that lets the search run over a continuous action space.","core_discovery":"The central claim is that the deterministic policy gradient framework, adapted to a day-ahead electricity auction, converges to a useful offering policy in a fully observable environment whose state is a 168-dimensional matrix of hourly prices. The reward function in Eq. (7) pays the difference between offered revenue and production cost for every step whose price is at or below the next-day PUN, summed over the three production modes. The paper interprets the decreasing policy loss, the delayed decrease of the critic loss, and the increasing normalized reward as evidence that the agent is moving toward a more profitable strategy over time.","pith_inferences":["A testable extension is to make production costs and capacities stochastic and feed current renewable output into the state; the authors mention this direction, and one would expect the agent to learn a hedging policy rather than a fixed mark-up.","Because the environment treats PUN as an exogenous price process, the learned policy is a best response to history, not to adapting rivals; a multi-agent version with many similar sellers learning simultaneously, which the paper lists as future work, would test whether the strategy remains profitable when other sellers also learn.","Reporting absolute profit in euros per episode alongside the normalized reward would let a reader judge economic significance; the paper currently gives only relative training metrics."],"forward_implications":["A seller can build an offering curve from a 7-day window of historical PUN prices without observing competitors' bids.","The critic's Q-value estimate is unreliable early in training, so stable learning requires many episodes and slow target-network updates.","An episode length below 15 days prevents convergence, while longer episodes improve exploration but show diminishing returns.","The DDPG approach extends the earlier Q-learning supplier-bidding method to continuous action spaces by replacing discrete Q-tables with deep networks and a deterministic policy gradient.","Because production costs and capacities are treated as fixed constants, the current model does not capture renewable intermittency; the paper proposes stochastic production as future work."],"supporting_citations":[{"why":"Defines the EUPHEMIA single price-coupling algorithm that sets the market clearing price used to decide which offers are accepted in the reward.","marker":"[2]"},{"why":"Introduces the Deep Deterministic Policy Gradient algorithm that the paper adapts for continuous-action bidding.","marker":"[9]"},{"why":"Provides the earlier Q-learning supplier-bidding baseline that this paper extends from discrete Q-tables to deep networks.","marker":"[18]"},{"why":"Applies deep reinforcement learning to strategic bidding in electricity markets, serving as the closest related application the method is compared against.","marker":"[19]"}],"fun_headline_variants":["RL agent bids smarter from price history in day-ahead market","Deep RL optimizes energy bidding using past prices only","Price history alone trains agent to win day-ahead auctions","DDPG agent learns profitable bidding strategy from prices","Single-agent RL masters continuous bidding in energy market"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reward function used for training counts every accepted unit of volume as profit for all three production modes at once and never applies the stated capacity limits, so the number the agent maximizes may not be the seller's actual profit.","fun_headline_variants_meta":{"raw":{"variants":["RL agent bids smarter from price history in day-ahead market","Deep RL optimizes energy bidding using past prices only","Price history alone trains agent to win day-ahead auctions","DDPG agent learns profitable bidding strategy from prices","Single-agent RL masters continuous bidding in energy market"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000189,"raw_usage":{"total_tokens":1278,"prompt_tokens":832,"completion_tokens":446,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":448,"completion_tokens_details":{"reasoning_tokens":368}},"tokens_in":448,"tokens_out":446,"duration_ms":4715,"temperature":1.0,"reasoning_tokens":368,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:01:10.119257+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Settle a held-out week of PUN prices twice for the trained policy: once with the paper's reward as printed, and once with each accepted volume allocated to the cheapest available mode up to its capacity $D_k$. If the two settlements rank alternative policies differently, or if the reported convergence disappears when the reward is corrected, the learning signal is an artifact of the summed reward rather than true profit.","supporting_citations":[{"cited_title":"com/globalassets/download-center/single-day-ahead-coupling/ euphemia-public-description.pdf (2020) 23","cited_arxiv_id":null,"evidence_quote":"Defines the EUPHEMIA single price-coupling algorithm that sets the market clearing price used to decide which offers are accepted in the reward."},{"cited_title":"Continuous control with deep reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Introduces the Deep Deterministic Policy Gradient algorithm that the paper adapts for continuous-action bidding."},{"cited_title":"Selecting and ranking individualized treatment rules with unmeasured confounding","cited_arxiv_id":"2002.10436","evidence_quote":"Provides the earlier Q-learning supplier-bidding baseline that this paper extends from discrete Q-tables to deep networks."}],"review_version":1}