{"id":"9d54596f-9653-4b5b-8986-ebedf68f5470","arxiv_id":"2506.02522","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A training-time LLM that refines actions and reshapes rewards, co-trained with an RL controller through a shared buffer, reports large gains over RL and LLM baselines on three L2RPN power grid challenges.","lead":"This paper combines a language model with a reinforcement learning agent for power grid control, using the model to fix poor actions and to re-score rewards during training. It reports large gains on three grid-operation benchmarks while keeping the fast RL agent in charge at deployment time.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported gains may come from the Grid2Op-validated multi-round retry loop rather than from LLM reasoning; an ablation replacing the LLM proposer with random/rule-based proposers under the same validation is needed.","rationale":"The reader's weakest_assumption centers on the framework's structural reliance on a fast, faithful simulator. I agree that is a real premise, but the more load-bearing issue for the paper's central empirical claim is attribution: Algorithm 1's multi-round, simulator-validated retry loop is a confound when we ask whether the LLM's reasoning, rather than the oracle-guided search, produced the reported gains. The reader's rationale does mention the absence of a control isolating the LLM from simulator-backed search, so there is partial agreement, but the stated weakest_assumption emphasizes deployment feasibility rather than the internal validity of the performance comparison. The concern is concrete and testable: replace only the LLM proposer with random or rule-based proposers, keep every other component fixed, and compare final episode rewards. If a non-LLM proposer with five simulator rollouts approaches ACE's numbers, then the state-of-the-art claims and the framework's novelty as an LLM-RL method are substantially weakened. If it does not, the LLM proposer is necessary and the current CONDITIONAL verdict is appropriate. On the positive side, the paper is clearly written, the offline-training/online-deployment separation is sensible, and the reported ablations and hyperparameter studies are detailed; nothing in this critique questions the authors' good faith. The verdict should remain CONDITIONAL pending the proposed control ablation, not because the framework is implausible, but because the marginal contribution of the LLM has not yet been cleanly demonstrated.","tokens_in":17136,"tokens_out":4096,"duration_ms":45411,"concrete_test":"On WCCI 2020 and NeurIPS 2020, run a matched ablation of Algorithm 1 in which fLLM is replaced by (a) random sampling of valid topology actions using the same action vocabulary and constraints as the prompt, and (b) a simple heuristic proposer, e.g., for the most overloaded line, propose bus reassignment at its endpoints. Keep the line-18 Grid2Op validation, the up-to-five-round retry rule, the same D_LLM mixing from Eq. (9), and the same gLLM/SFT settings. Compare 5-seed mean episode rewards against ACE's 69.8 (WCCI 2020) and 145.3 (NeurIPS 2020). If either non-LLM proposer matches ACE within roughly 10%, the LLM-attribution claim fails; if both fall far below, the LLM proposer is load-bearing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim attributes ACE's improvements to LLM reasoning, but the architecture in Algorithm 1 (lines 10-18) and Section 4.3 does not isolate that attribution. Every fLLM action is validated by simulating it in Grid2Op, and if the refined action's reward is inferior to the RL action, the LLM is allowed up to five rounds of new proposals before a transition is accepted into D_LLM. This makes fLLM an action proposer whose output is filtered by a faithful environment oracle. The paper provides no control that replaces fLLM with a non-LLM proposer while keeping the same validation-and-retry protocol. The authors' own Figure 5(b) shows unvalidated LLM suggestions are poor for long-horizon control, which strengthens the possibility that the validation loop, rather than the LLM's domain reasoning, is responsible for the 22%, 13%, and 145% improvements over the RL backbone. As a result, the marginal contribution of the LLM to the headline numbers is unidentified, and the state-of-the-art claim over 'LLM-based methods' is not yet established. This is an attribution/correctness risk, not an objection to using simulators as evaluators.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes ACE, a framework that combines an LLM and an RL agent for large-scale power grid control. During training, the LLM acts as a Policy Actor that refines low-reward actions, with each proposal simulated in Grid2Op and up to five retries, and as a Value Critic that adjusts rewards at selected trajectory points. Refined transitions are pooled with RL transitions in a mixed replay buffer using reward-weighted sampling, and the LLM is occasionally fine-tuned on this buffer. At deployment, only the RL policy acts. Experiments on three L2RPN challenges report substantial improvements over the RL backbone and over LLM-only and LLM4Teach baselines, while requiring only a few hundred LLM refinements.","tokens_in":17210,"tokens_out":9690,"duration_ms":86842,"significance":"If the reported gains are genuinely due to LLM reasoning, this is a practical and timely contribution: it separates offline LLM guidance from online RL execution, addressing the real-time latency problem of LLM-based control in industrial settings. The sample-efficiency numbers (287-682 refinements vs 100K-200K environment steps) are striking, and the ablations show that both the actor and critic components contribute on WCCI 2020. The paper is also honest about the limitations of pure LLM control. However, the central attribution of the improvements to LLM reasoning remains unresolved because no control replaces the LLM proposer with a random or rule-based proposer under the same Grid2Op validation and retry protocol. In addition, the state-of-the-art claim on WCCI 2022 lacks comparison with the actual challenge winner or the cited AlphaZero-based method. These issues do not refute the empirical results, but they currently prevent the paper from establishing its central causal claim.","major_comments":[{"comment":"The attribution of ACE's gains to LLM reasoning is not identified. In Algorithm 1 (lines 10-18) and Section 4.3, every fLLM action is simulated in Grid2Op, and up to five rounds of new proposals are allowed until the refined action beats the RL action. No control replaces fLLM with a random or rule-based proposer under the same validation and retry budget. The 'w/o Multi-round' ablation in Table 2 only removes the retry loop, not the LLM, and Figure 5(b) shows unvalidated LLM suggestions are poor for long-horizon control. The reported improvements over the RL backbone (22%, 13%, 145% in Table 1) could therefore come from the validated retry procedure rather than from LLM reasoning. Please add a control that uses a non-LLM proposer (e.g., random actions or a simple heuristic) with identical Grid2Op validation and the same five-round retry limit, and report the resulting episode rewards.","section":"Algorithm 1, Section 4.3"},{"comment":"The state-of-the-art claim on WCCI 2022 is not supported by the selected baselines. Table 1 compares ACE only to the authors' own expert-guided RL backbone and to a modified LLM4Teach. For WCCI 2022, Section 4.1 cites Dorfer et al. (2022) but the paper never compares ACE to that method or to the actual L2RPN WCCI 2022 challenge winner. 'State-of-the-art' requires comparison with the best known methods on that benchmark; please add such comparisons or restrict the claim to the considered baselines.","section":"Table 1, Section 4.1"}],"minor_comments":[{"comment":"The symbol β denotes the mixing ratio in Eq. (6) and also the temperature in the exponential of Eq. (8); using the same symbol for two quantities in the same derivation is confusing and should be corrected.","section":"Eq. (6) and Eq. (8)"},{"comment":"Table A.2 lists 'Reward threshold (¯r & r) 0.1 & 0.5', while Section 4.5 reports experiments with r ∈ {-0.3, 0, 0.3} and Table 3 shows the default r = 0. Please clarify the default threshold values.","section":"Table A.2, Section 4.5"},{"comment":"The dagger marker † is attached both to ACE (Qwen2-7B + SFT) and to ACE (GPT-4), but the footnote says ACE (GPT-4) does not include SFT; use separate markers for the SFT and no-SFT configurations to avoid ambiguity.","section":"Table 1 footnote"},{"comment":"Algorithm 1 (line 18) stores all refined transitions in D_LLM, whereas Eq. (9) filters them with Iv(τ) at sampling time; please state explicitly whether invalid refinements are discarded at storage time or stored and down-weighted.","section":"Section 3.3, Algorithm 1"},{"comment":"The sentence 'we apply low-rank adaptation fine-tuning on Dmix with reward signals as labels' is under-specified; please define the loss function and describe how the reward labels are used (supervised action prediction, preference ranking, or other).","section":"Section 3.3"},{"comment":"There are typographical errors in Table A.2: 'SFT buffer size 10,00' should be 10,000, and 'Max Token off LLM' should be 'Max Token of fLLM'.","section":"Appendix A.2"},{"comment":"The text refers to Figure 5(a) and 5(b), but the caption describes left/right panels; align the in-text references with the figure labels.","section":"Section 5.1, Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The central issue is the missing attribution control. If the authors can show that a random or rule-based proposer with the same Grid2Op validation loop does not achieve the reported gains, the paper would be significantly stronger. The authors should also be encouraged to compare against the actual WCCI 2022 winner; citing Dorfer et al. without comparison may appear selective. The dagger/SFT notation in Table 1 is likely to confuse readers and should be cleaned up."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Wan et al. propose ACE, an offline-only LLM guider for RL in large discrete-action control tasks like L2RPN. The design is clean: during training, the LLM both proposes corrected actions for low-reward transitions and reshapes rewards for critical trajectories; each proposed action is simulated in Grid2Op and, if it underperforms the RL action, retried up to five times. At deployment the RL policy acts alone. This separation is sensible and the paper's framing is honest about why pure LLM policies are unsuitable for real-time control (Figure 5b).\n\nWhat's genuinely useful here is the concrete architecture: a prioritized mixed buffer, reward-weighted policy updates, and periodic LoRA fine-tuning of the LLM on its own validated outputs. The experimental detail is strong—they report per-module compute overhead, ablation studies, and full prompts in the appendix. The L2RPN results are striking, especially the 145% gain over the expert-guided RL backbone on WCCI 2022.\n\nThe soft spot is exactly what the stress-test note flags: the marginal contribution of the LLM's reasoning is not identified. Every LLM proposal is filtered by a faithful environment simulator with up to five retries. There is no control that swaps the LLM for a random or rule-based proposer under the same validation-and-retry protocol. Without that, the headline improvements could come mostly from the search/validation loop rather than from anything the LLM understands about power grids. The authors' own Figure 5(b) shows unvalidated LLM suggestions are poor for long-horizon control, which makes this concern more acute. This is an attribution problem, not a circularity problem—the rewards come from Grid2Op and the results are genuinely benchmarked.\n\nA couple of smaller issues: hyperparameters (bad-case threshold, K, query intervals, SFT frequency) appear to be tuned on WCCI 2020, the same environment that produces the headline numbers, and the ablation tables lack error bars. The modified LLM4Teach baseline underperforms even plain RL on WCCI 2022, which suggests their reimplementation may be weak. No code or data is released, so exact reproduction is not possible.\n\nStill, this is a real system with thoughtful experiments and a clear, falsifiable claim. The attribution concern is fixable with one control experiment. I'd send it to review—preferably with a request for that control and for error bars—rather than desk reject. It deserves a serious referee.","headline":"Sensible offline LLM-RL integration with impressive L2RPN numbers, but the headline gains may owe more to the Grid2Op-validated retry loop than to the LLM's reasoning; needs a proposer control before the SOTA claim holds.","tokens_in":17948,"tokens_out":2372,"would_cite":false,"duration_ms":22775,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that an LLM used only during training as an action corrector and reward critic lifts RL to the best reported results on three large power-grid control tasks.","keywords":["large language models","reinforcement learning","power grid control","trajectory refinement","reward shaping","co-evolution","sample efficiency","L2RPN"],"falsifier":"Repeat the WCCI 2022 experiment with the Grid2Op validation loop disabled so every LLM-proposed action is accepted un-checked; the claim would be falsified if the 75.9 episode reward and the 145% margin over the expert-guided RL baseline do not fall substantially, because the paper's own Figure 5b shows unvalidated LLM suggestions are poor for long-horizon control. A second check would scramble the text serialization of states and actions while keeping simulation validation intact; if performance stays high, the gains come from the validation filter rather than from LLM reasoning.","tokens_in":16736,"feed_emoji":"⚡","tokens_out":17315,"duration_ms":137775,"temperature":0.7,"pith_summary":"This paper works to establish that a large language model can act as an offline tutor for a reinforcement-learning agent in large-scale industrial control, where LLMs alone are too slow and RL alone is too sample-hungry. In the proposed Agents Co-Evolution (ACE) framework, the LLM operates only during training in two roles: as a Policy Actor that rewrites low-reward actions and as a Value Critic that re-scores important decisions along whole trajectories. Every proposed action is checked by simulating it in the Grid2Op power-grid environment before it enters the replay buffer. On three L2RPN (Learning to Run a Power Network) benchmarks, ACE's RL agent reaches episode rewards of 69.8, 145.3, and 75.9, beating the expert-guided RL backbone by 22%, 13%, and 145% while using far fewer environment samples. Because the LLM is absent at deployment, the learned RL policy keeps the real-time decision speed that industrial loops require.","feed_headline":"Offline LLM coaching lifts RL power-grid scores up to 145 percent","feed_subtitle":"On three L2RPN challenges, the RL agent learns from an LLM's action fixes and reward edits, then runs alone.","key_machinery":"The load-bearing mechanism is the dual-role trajectory-refinement loop that runs while the RL policy trains. The Policy Actor $f^{\\text{LLM}}$ (Eq. 4) is invoked on sampled transitions with reward below $\\bar{r}$; it receives the state, the RL action, and a 'bad action' negative example in text, and proposes a corrected action that the Grid2Op simulator must confirm before the new transition is stored in the separate buffer $\\mathcal{D}_{\\text{LLM}}$. The Value Critic $g^{\\text{LLM}}$ (Eq. 5) is invoked on key trajectory points, selected by reward magnitude or line-flow change, and returns reward adjustments confined to $\\{-2K,-K,+K,+2K\\}$ — a discretized analogue of temporal-credit assignment. The mixed sampling distribution $p_{\\text{mix}}$ (Eq. 9) with importance weights $w_r(\\tau)$ (Eq. 8) then transfers the LLM's improvements into the SAC (Soft Actor-Critic) policy update (Eq. 10), while the same mixed buffer becomes a low-rank adaptation (LoRA) fine-tuning set for the LLM; that two-way flow of data is what the name 'co-evolution' refers to.","core_discovery":"ACE's central claim is that separating LLM reasoning from RL execution — 'think twice, act once' — removes the sample-inefficiency and latency bottlenecks at the same time. During training the Policy Actor takes transitions whose reward $r$ falls below a threshold $\\bar{r}$, converts state and action into text, and proposes a corrected action $\\hat{a}_t$; the correction is accepted only after Grid2Op simulation confirms it, with up to five revision rounds allowed. The Value Critic takes trajectory snippets at key decision points and returns reward adjustments on a discrete grid $\\{-2K,-K,+K,+2K\\}$, acting as an implicit multi-step temporal-difference, TD($\\lambda$), whose credit assignment does not assume exponential decay. Refined transitions live in a separate LLM buffer, and a mixed sampling distribution with reward-based weights $w_r(\\tau)$ feeds the RL updates while also supplying fine-tuning data that improves the LLM coach. At deployment only the RL agent acts, and ACE reports the highest episode rewards on all three L2RPN challenges, with the largest gain (145% over the expert-guided RL backbone) in the WCCI 2022 IEEE-118 environment; ablations on WCCI 2020 show that removing the Actor drops the reward from 69.8 to 48.3 and removing the Critic drops it to 61.5.","pith_inferences":["Beyond the paper: none of ACE's equations is power-grid-specific, so the design should transfer to other control domains that have a fast simulator and textual state-action serialization, such as traffic control or cloud resource management.","Beyond the paper: the paper's evidence that unvalidated LLM suggestions degrade over long horizons suggests a direct stress test — vary simulator fidelity and measure how much of the 22%, 13%, and 145% margins survive.","Beyond the paper: since SFT on the mixed buffer independently improved the LLM's own decisions by 10.9%, a longer co-evolution loop might eventually shrink the coach to a small distilled policy, leaving only the fast RL controller at deployment.","Beyond the paper: the discrete reward adjustments and five-round cap imply a refinement budget; a promising extension is to make that budget adaptive, spending more rounds when the simulator's margin between the RL action and the LLM fix is small."],"forward_implications":["Deployment stays real-time even though the coach is a 7B-parameter LLM, because the LLM never acts online and test-time latency is set by the RL policy alone.","A few hundred targeted refinements substitute for tens of thousands of environment samples: ACE needs 287 to 682 refinements on top of 40–50K samples where baselines need 100–200K.","LLM reward re-scoring is a real source of gain, not a rounding error: without the Value Critic the WCCI 2020 episode reward falls from 69.8 to 61.5, and without the Policy Actor to 48.3.","Fine-tuning the coach on the mixed buffer closes the loop: three rounds of SFT let the Qwen2-7B-based ACE beat its untuned self and the non-tuned GPT-4 variant on refined step rewards and independent decision-making.","Offline selective refinement beats policy-regularization-style LLM guidance: the modified LLM4Teach baseline stays near the RL backbone (64.4 on WCCI 2020) while ACE reaches 69.8."],"supporting_citations":[{"why":"Supplies the Soft Actor-Critic algorithm that ACE uses as its RL backbone; the Q-function and policy updates in Eqs. (2), (3), and (10) are SAC's.","marker":"Haarnoja et al., 2018"},{"why":"The expert-guided winning-solution architecture ACE builds on as its backbone, and the source of the action-space cardinality and state features; its reported results are the primary baseline ACE exceeds.","marker":"Yoon et al., 2021"},{"why":"Provides the Grid2Op simulator that validates every LLM-refined action in Algorithm 1 line 18 and hosts all three L2RPN evaluation environments.","marker":"Donnot, 2020"},{"why":"Defines the WCCI 2020 L2RPN challenge, its reward structure, and test scenarios where ACE reports episode reward 69.8.","marker":"Marot et al., 2020"},{"why":"Defines the NeurIPS 2020 L2RPN adversarial challenge and its test protocol where ACE reports episode reward 145.3.","marker":"Marot et al., 2021"},{"why":"Defines the WCCI 2022 L2RPN environment on the IEEE-118 grid with over 70K actions where ACE reports episode reward 75.9 and its largest margin.","marker":"Marot et al., 2022"},{"why":"LLM4Teach is the LLM-guided-RL baseline; ACE's comparison against its KL-regularized alignment shows offline selective refinement beats policy regularization.","marker":"Zhou et al., 2023"},{"why":"Supplies Qwen2-7B-Instruct, the open LLM used in ACE (Qwen2-7B), whose LoRA fine-tuning demonstrates the co-evolution loop.","marker":"Bai et al., 2023"},{"why":"Supplies GPT-4o-0806, the LLM used in ACE (GPT-4) without fine-tuning; comparing it with the Qwen variant isolates the effect of SFT.","marker":"Achiam et al., 2023"}],"fun_headline_variants":["LLM coach lifts RL grid scores up to 145 percent","ACE: LLM and RL co-evolve for 60K-action grid control","Think twice, act once: LLM refines RL for power grids","Offline LLM boosts RL agent on three grid challenges","LLM as critic and actor trains RL to win grid tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a fast, faithful simulator of the real system is always available during training to validate every LLM-proposed action; if that simulator is missing, slow, or inaccurate, the think-twice loop cannot run, and the paper's own evidence indicates LLM suggestions alone are unreliable over long horizons.","fun_headline_variants_meta":{"raw":{"variants":["LLM coach lifts RL grid scores up to 145 percent","ACE: LLM and RL co-evolve for 60K-action grid control","Think twice, act once: LLM refines RL for power grids","Offline LLM boosts RL agent on three grid challenges","LLM as critic and actor trains RL to win grid tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000184,"raw_usage":{"total_tokens":1355,"prompt_tokens":1019,"completion_tokens":336,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":635,"completion_tokens_details":{"reasoning_tokens":244}},"tokens_in":635,"tokens_out":336,"duration_ms":3667,"temperature":1.0,"reasoning_tokens":244,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:23:06.176530+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the WCCI 2022 experiment with the Grid2Op validation loop disabled so every LLM-proposed action is accepted un-checked; the claim would be falsified if the 75.9 episode reward and the 145% margin over the expert-guided RL baseline do not fall substantially, because the paper's own Figure 5b shows unvalidated LLM suggestions are poor for long-horizon control. A second check would scramble the text serialization of states and actions while keeping simulation validation intact; if performance stays high, the gains come from the validation filter rather than from LLM reasoning.","supporting_citations":[],"review_version":1}