{"id":"8144fafa-6d60-4d3d-967a-a0e5aabeccbc","arxiv_id":"2509.02861","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A two-layer distributed RL system with one GNN-observing agent per power line and a learned manager keeps the Grid2Op case14 grid alive far longer than the do-nothing baseline.","lead":"A team from Politecnico di Milano built a distributed reinforcement-learning system for power grid control in which each power line gets its own small agent, a shared graph network summarizes neighborhood information, and a manager picks which line to act on. In a standard Grid2Op simulation benchmark, it keeps the grid running far longer than the do-nothing baseline, and reacts faster than an expert simulation method.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Undocumented DQfD demonstrations may overlap the test chronics, making the headline 6114-step survival an imitation artifact rather than evidence for the distributed GNN architecture.","rationale":"The reader's weakest assumption already identified DQfD demonstration availability/quality as the main soft spot; I agree and sharpen it into a concrete, checkable leakage hypothesis. The table numbers support this: the increment from No-DQfD to Complete is 4236 test steps, i.e., about 69% of the Complete result, so the central performance figure is highly dependent on an undocumented data source. The paper's own Limitations section notes the reliance on expert demonstrations but does not document their composition. I do not see internal mathematical inconsistency or fabrication; the gap is empirical support. Therefore the verdict remains CONDITIONAL: the claim may hold, but it is not demonstrated until the demonstration dataset is shown to be disjoint from the test chronics and sufficiently broad. I chose UNCHANGED rather than a new verdict because the reader already marked CONDITIONAL and my concern is a more precise version of the same premise.","tokens_in":9986,"tokens_out":4113,"duration_ms":48090,"concrete_test":"Inspect the released code/data-generation script (github.com/Carlo000ml/RL4PG) and enumerate the chronic IDs used to generate DQfD demonstrations; intersect them with the 10 test chronics. If any test chronic appears, retrain the full system with those demonstrations excluded and report test survival. If the Complete-vs-No-DQfD gap collapses (e.g., below ~2500 steps), the headline result is leakage-driven; if the gap persists, the DQfD concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 1 attributes most of the gain to DQfD: removing it drops test survival from 6114.4 to 1878, while Do-Nothing is 646.6. The paper never states how many demonstrations the Expert produced, on which chronics, or whether the 10 test chronics were excluded from demonstration collection. Since the environment contains 1004 chronics and only 10 are used for testing, even a small overlap could let the Q-functions memorize expert actions for the exact test scenarios; the reported 6114.4 would then reflect imitation of the Expert on seen time series, not the proposed line-agent/GNN/decomposed-observation design. The no-DQfD result (1878) shows the architecture alone still beats Do-Nothing, so the core 'outperforms baseline' claim survives, but the headline number and the implied superiority of the full system are not established without dataset documentation and a leakage check. The absence of seeds/error bars compounds this: a 10-chronic average with no variance cannot distinguish a robust gain from a few favorable test episodes.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-level distributed RL architecture for Grid2Op topology control. Twenty low-level DQN agents each manage one power line and receive a GNN-enriched local observation, while an RL manager selects which line agent acts. Training combines DQfD from a simulation-based Expert (Marot et al., 2018) with bootstrapped potential-based reward shaping. A homogeneous line-graph representation is introduced to avoid heterogeneous-graph issues. Experiments on l2rpn_case14_sandbox report a test survival time of 6114.4 for the full system versus 646.6 for Do-Nothing (Table 1) and an inference time of 0.187 s versus 2.56 s for the Expert (Table 2). An ablation attributes most of the gain to DQfD and the GNN, with reward shaping having a mixed effect. The Limitations section explicitly acknowledges the manager's global view, the dependence on expert demonstrations, and the unisolated single-step simulation used for action-space reduction.","tokens_in":10203,"tokens_out":4692,"duration_ms":59547,"significance":"If the reported results are reproducible, the paper makes a useful contribution: it combines per-line distributed control, GNN-based observation decomposition, and imitation learning in a way that is modular and potentially scalable, and the code is publicly available. The central narrow claim, that the full system outperforms Do-Nothing, is consistent with Table 1 even without DQfD (1878 vs. 646.6), which is a mitigating fact. However, the evidence is preliminary: averages over only 10 validation and 10 test chronics, no seeds or error bars, undocumented expert demonstrations, no learning-based baseline, and a runtime component that is entangled with the inference-time comparison. The reader's leakage concern lands: the headline number is dominated by the DQfD component, and the demonstration dataset is not described. The paper is therefore a promising research report, but the load-bearing empirical claims need substantial reinforcement before journal acceptance.","major_comments":[{"comment":"The demonstration dataset used for DQfD is not documented: the paper does not state how many demonstrations were collected, on which chronics, whether the 10 validation/test chronics were excluded from demonstration collection, or how expert trajectories were generated. Since DQfD is the largest single contributor to the headline result (removing it drops test survival from 6114.4 to 1878), the reported 6114.4 could in part reflect imitation of the Expert on seen scenarios rather than the distributed GNN architecture. The no-DQfD row attenuates this concern for the narrow baseline claim, but not for the claim of full-system superiority. Please report the demonstration-generation protocol, the dataset size and coverage, and a leakage check (e.g., performance evaluated on chronics provably absent from the demonstration set).","section":"§4.2–§4.3, Table 1"},{"comment":"All performance numbers are averages over 10 validation and 10 test chronics with no standard deviations, no multiple seeds, and no significance tests; the ablation appears to be a single run. With such small evaluation sets, a few favorable episodes can dominate an average. Please provide per-chronic results, repeated-seed statistics (mean±std or confidence intervals), and, where appropriate, a significance test or effect-size measure. This is needed to establish that the improvement over Do-Nothing and the ablation differences are not noise.","section":"§4.3, Table 1 and Fig. 5"},{"comment":"The inference-time comparison is potentially confounded by the single-step simulation used for greedy action-space reduction. The plain simulator step is reported as 0.1097 s, while the proposed model's inference is 0.187±0.145 s; if that simulation is part of the action loop, the 'proposed model' time is not purely a learned-policy time. The Limitations section acknowledges this step but does not isolate its contribution to either runtime or survival performance. Please measure the action loop with and without the single-step simulation, report the number of timing runs and hardware, and clarify what exactly Table 2 measures.","section":"§4.3, Table 2, and Limitations"},{"comment":"The paper compares only against Do-Nothing and the Expert agent; no learning-based baseline (e.g., a single-agent DQN on the same action space, a substation-based multi-agent method, or a GNN-based single-agent method from prior work) is evaluated. The conclusion that the proposed architecture is 'effective' would need such a comparison to support claims about the advantage of line-level distributed control and GNN-based observation decomposition over existing approaches. At minimum, a comparison with a centralized DQN using the same demonstrations and reward shaping would clarify how much of the gain comes from the distributed architecture itself.","section":"§4.3, Table 1"},{"comment":"The bootstrapped potential-based reward-shaping component is presented as a contribution, but the paper does not verify that the shaping preserves optimality in this multi-agent, partially observable setting; it cites Adamczyk et al. (2025) as justification. The ablation is mixed: removing shaping improves validation survival (5667.1 vs. 5452.2) but lowers test survival (5324.3 vs. 6114.4). This ambiguity should be discussed explicitly, and the theory transfer to multi-agent settings should be checked or at least acknowledged as an open question.","section":"§3.1 and Table 1, 'No Reward Shaping' row"}],"minor_comments":[{"comment":"Notation µ_t, φ_t, α_i, β_i is used without definition. A notation table or inline definitions would make the action-selection flow self-contained.","section":"§3, Algorithm 1"},{"comment":"Hyperparameters (GNN dimensions, DQN layer sizes, replay buffer capacities, batch sizes, epsilon half-lives, soft-update and synchronization intervals, gradient-clipping details) are not reported in the text. A hyperparameter table would improve reproducibility even with the code link.","section":"§4.2"},{"comment":"The heatmap symmetry is qualitative and does not by itself demonstrate generalization. Please either present a quantitative generalization measure or soften the claim.","section":"§4.3, Fig. 6"},{"comment":"Report the number of timing runs, the environment/hardware, and whether the timing includes the single-step action-space simulation.","section":"§4.3, Table 2"}],"recommendation":"major_revision","confidential_remarks":"The leakage concern about DQfD demonstrations is legitimate and should be resolved before publication. The paper already includes code, which is good, and the no-DQfD result gives some support to the basic baseline claim. However, the current evidence is too thin for a journal-level acceptance: no seeds, no variance, undocumented demonstrations, and no learning-based baseline. The requested additions (dataset documentation, leakage check, repeated runs, and ablation of the runtime simulation) are within the manuscript's scope and should be feasible. I see no reason to reject outright."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know: this is a cleanly written architecture proposal with one genuinely novel idea—per-line agents over a homogeneous line graph with a shared GNN—but the empirical case is a lot thinner than the numbers suggest. The 6114.4-step test survival rests almost entirely on DQfD pretraining, and the demonstration set is never documented. That is the first thing to fix.\n\nWhat's actually new: the line graph construction (lines as nodes, substations as edges) is a neat way to avoid heterogeneous graphs and the bus-bar asymmetry problem; assigning one agent per line with a shared GNN is a reasonable decomposition of both action and observation spaces; and the manager is learned rather than rule-based CAPA. The authors also ship code and state their limitations honestly—manager global view, reliance on expert demos, the runtime single-step simulation.\n\nWhere it goes soft. The main result is not established. Removing DQfD drops test performance from 6114 to 1878, which still beats Do-Nothing (646), so the architecture alone has some value. But the full-system number could be imitation of the Expert on near-identical chronics. The paper doesn't say how many demonstrations were collected, on which chronics, or whether test chronics were held out. Given 1004 chronics and only 10 held out for testing, even a small overlap could explain the gap. Also, there are no seeds or error bars; 10 test episodes with no variance cannot support a strong claim. No comparison to substation-based MARL baselines, so we don't know if per-line agents actually help. The reward-shaping ablation is mixed and single-run. The inference-time comparison is useful but the action-space reduction step isn't isolated, so the \"0.187s vs 2.56s\" claim conflates the trained policy with a cheaper action filter.\n\nBottom line: if you work on L2RPN, this is worth reading as a modular-design proposal and the code is a starting point, but don't cite the headline number as evidence for the architecture. It deserves peer review—it's not desk-reject material—but a serious referee should send it back for more seeds, full demo documentation, a leakage check, and at least one substation-agent baseline.","headline":"The per-line/graph architecture is new and worth engaging, but the headline result is not established: DQfD drives most of the gain, demos are undocumented, and the evaluation lacks error bars and baselines.","tokens_in":10719,"tokens_out":3279,"would_cite":false,"duration_ms":35133,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"One GNN-observing agent per power line, coordinated by a manager, survives 6114.4 average test steps versus 646.6 for do-nothing, and acts in 0.187 seconds per decision.","keywords":["graph neural networks","distributed reinforcement learning","power grid topology control","Grid2Op","deep Q-learning from demonstrations","multi-agent reinforcement learning","line graph representation","reward shaping"],"falsifier":"Retrain the complete architecture on the same ten test chronics with the DQfD demonstration buffer replaced by an equal-sized buffer of random or heuristic transitions, leaving the GNN, reward shaping, and manager unchanged. If survival time falls toward the reported 1878-step 'No DQfD' ablation rather than staying near 6114, the expert demonstrations—not the graph-based distributed design—carry the result. A second decisive check is to disable the runtime single-step simulation used for greedy action-space reduction; if survival time and inference time barely change, that externality is unimp","tokens_in":9839,"feed_emoji":"⚡","tokens_out":10860,"duration_ms":107580,"temperature":0.7,"pith_summary":"This paper tries to show that power-grid topology control can be handled by a fully distributed reinforcement-learning system: a separate learning agent for each power line, a manager that decides which line acts, and a shared graph neural network that lets each line agent see a compact local view of the surrounding network. On the Grid2Op benchmark, the system survives 6114.4 test steps on average versus 646.6 for the do-nothing baseline, and it makes decisions in 0.187 seconds per step, compared with 2.56 seconds for the simulation-based Expert used to gather demonstrations. The paper argues that the key to this result is decomposing the observation space as well as the action space, which earlier decentralized approaches did not do. If the result holds, the architecture offers a template for controllers that scale with the grid and react quickly enough for real-time operation.","feed_headline":"One learning agent per power line outlasts the passive baseline 9.5x","feed_subtitle":"In the Grid2Op simulator, 20 line-level agents plus a manager keep the grid alive 6114 steps vs 647.","key_machinery":"The load-bearing object is the homogeneous line graph of the power grid: each power line becomes a node, and two nodes are connected when their lines share a substation. Node features are built by concatenating the embeddings of the two terminal substations, each substation embedding being the concatenation of its two bus embeddings, each bus embedding being the sum of the one-hot-like vectors of the elements connected to that bus. A shared GNN runs message passing over this graph, and the embedding of a line's node becomes that line's observation. This single mechanism performs three jobs at once: it gives every line agent a local but informative view, it avoids the bus-bar information asym","core_discovery":"The central claim is that a two-layer distributed architecture can make reinforcement learning practical for topology control in realistic power grids. Twenty low-level agents, one per line, each run a Deep Dueling Double Q-learning policy on a local observation produced by a shared GNN; a twenty-first policy, the manager, selects which low-level agent acts at each time step. The grid is converted into a homogeneous graph in which lines are nodes and substations are edges, so the GNN's message passing encodes neighborhood information directly into each line's observation. Pretraining with Deep Q-Learning from Demonstrations (DQfD) on data from a simulation-based Expert, plus bootstrapped pot","pith_inferences":["The paper leaves the demonstration dataset unspecified; a direct test would be to retrain with a same-sized dataset of random or heuristic transitions, isolating how much of the 6114-step survival is imitation of the Expert versus value learning by the distributed agents.","Because all low-level agents share a single GNN, transfer from a small grid to a larger one is plausible: message passing works on any graph size, so initializing a case14-trained GNN on a larger Grid2Op environment and measuring convergence time would test the stated scalability.","The runtime single-step simulation used for greedy action-space reduction is a hidden dependency; disabling it (for example, replacing it with an N-1 static criterion) would reveal how much of the survival time is learned control and how much is simulator-assisted search.","Reward shaping contributes little on case14; the paper's own hypothesis that it matters in larger, sparser-reward grids is testable by repeating the ablation on a larger Grid2Op case."],"forward_implications":["Adding a line to the grid adds one low-level agent and one graph node, so the per-agent observation size stays fixed; the architecture's training cost scales with the number of lines rather than with the combinatorial topology space.","At 0.187 seconds per decision, the learned policy clears the real-time bar in settings where the 2.56-second simulation-based Expert cannot act quickly enough.","Removing either the GNN or the DQfD pretraining collapses performance, so the reported gain is not marginal but depends on both components.","The homogeneous line-graph representation sidesteps the bus-bar information asymmetry and the need for heterogeneous GNNs identified in earlier power-grid graph work, so it can serve as a standalone input representation for other topology-control methods."],"supporting_citations":[{"why":"Supplies the Grid2Op simulation environment and the l2rpn_case14_sandbox benchmark on which all experiments run.","marker":"Donnot, 2020"},{"why":"Provides the simulation-based Expert whose demonstrations are used for DQfD pretraining of the low-level and manager agents.","marker":"Marot et al., 2018"},{"why":"Defines Deep Q-Learning from Demonstrations, the imitation step that the ablation shows is essential.","marker":"Hester et al., 2018"},{"why":"Provides the Dueling Double Q-learning architecture used for all low-level agents and the manager.","marker":"Wang et al., 2016"},{"why":"Supplies the bootstrapped potential-based reward shaping used to improve credit assignment.","marker":"Adamczyk et al., 2025"}],"fun_headline_variants":["Per-line RL agents with GNN outlast baseline 9.5x in grid control","Graph-based distributed RL: 20 agents beat baseline by 9.5x","Distributed RL per power line keeps grid alive 9.5x longer","GNN-based agent per line outperforms passive baseline in grid sim","Line-level agents and manager: 9.5x more grid survival steps"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The reported gain rests on two unquantified supports: a dataset of expert demonstrations for DQfD pretraining whose size, coverage, and generation protocol are not given, and a runtime single-step simulation that prunes the action space and whose contribution is never measured separately.","fun_headline_variants_meta":{"raw":{"variants":["Per-line RL agents with GNN outlast baseline 9.5x in grid control","Graph-based distributed RL: 20 agents beat baseline by 9.5x","Distributed RL per power line keeps grid alive 9.5x longer","GNN-based agent per line outperforms passive baseline in grid sim","Line-level agents and manager: 9.5x more grid survival steps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0003,"raw_usage":{"total_tokens":1571,"prompt_tokens":746,"completion_tokens":825,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":490,"completion_tokens_details":{"reasoning_tokens":722}},"tokens_in":490,"tokens_out":825,"duration_ms":8356,"temperature":1.0,"reasoning_tokens":722,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T11:19:57.983540+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the complete architecture on the same ten test chronics with the DQfD demonstration buffer replaced by an equal-sized buffer of random or heuristic transitions, leaving the GNN, reward shaping, and manager unchanged. If survival time falls toward the reported 1878-step 'No DQfD' ablation rather than staying near 6114, the expert demonstrations—not the graph-based distributed design—carry the result. A second decisive check is to disable the runtime single-step simulation used for greedy action-space reduction; if survival time and inference time barely change, that externality is unimp","supporting_citations":[{"cited_title":"Expert system for topological remedial action discovery in smart grids","cited_arxiv_id":null,"evidence_quote":"Provides the simulation-based Expert whose demonstrations are used for DQfD pretraining of the low-level and manager agents."},{"cited_title":"Deep q-learning from demonstrations","cited_arxiv_id":null,"evidence_quote":"Defines Deep Q-Learning from Demonstrations, the imitation step that the ablation shows is essential."},{"cited_title":"Dueling network architectures for deep reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Provides the Dueling Double Q-learning architecture used for all low-level agents and the manager."},{"cited_title":"Kulkarni","cited_arxiv_id":null,"evidence_quote":"Supplies the bootstrapped potential-based reward shaping used to improve credit assignment."}],"review_version":1}