{"id":"eae89fa3-8b7d-421b-84bb-8f592d7f850f","arxiv_id":"2502.00040","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Applying multi-objective RL (DOL plus MOPPO) to power grid topology control yields a set of trade-off policies on a 5-bus Grid2Op testbed, with some robustness and training-efficiency improvements over single-objective RL, though the evaluation has methodological weaknesses.","lead":"This paper tests a multi-objective reinforcement learning approach that produces a menu of power-grid topology control policies, balancing line loading, topological deviation, and switching frequency. It shows on a small 5-bus simulation that such Pareto-optimal policy sets can be generated, though the headline performance gains over single-objective control are not fully supported by the reported numbers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Best-of-set selection of MO policies biases the MO-vs-SO comparison; the claimed 39%/22% gains may be a max-of-N artifact, not a MORL effect.","rationale":"The reader's weakest_assumption is exactly the concern I would raise. The headline percentages in the abstract (30%, 20%) already do not match the tables (39.07/23.86/10.17 and 21.88/5.49/10.17), but that is a reporting inconsistency; the selection asymmetry is a correctness risk. If the comparison is re-run with matched selection effort and the advantage persists, the paper's central claim is supported. If it disappears, the MORL contribution reduces to the more modest, still useful claim that DOL produces a diverse Pareto set (supported by IGD and sparsity, though IGD's reference-set circularity deserves its own caveat). The code repository is a positive: the proposed check can be run from the released implementation. I do not see an internal contradiction in the MORL framework itself; the issue is the evaluative design. Hence CONDITIONAL remains the right verdict, and no change is needed.","tokens_in":10252,"tokens_out":4020,"duration_ms":42167,"concrete_test":"Re-run Tables II and III with a matched selection protocol: for each seed, train K single-objective PPO policies (same architecture, same per-policy training budget, K equal to the average number of non-extreme policies in the DOL set for that seed), select the SO policy with highest E via Algorithm 3, and compare it to the MO policy selected the same way. Report paired per-seed differences with confidence intervals. If the mean delta E drops below the seed-level standard error or changes sign, the claimed advantage is a selection artifact rather than an effect of multi-objective training.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that multi-objective training yields more robust and sample-efficient policies rests on Tables II and III, where the MO policy is chosen per seed by Algorithm 3 as the best (by episode duration E) among the DOL-generated Pareto set, while the SO baseline is a single PPO policy trained on the line-loading reward (Sections II-D, III-C, III-D). This is an asymmetric best-of-N comparison. Episode duration is noisy across seeds and training runs; selecting the maximum over a set of policies can raise the mean even if no individual MO policy is better than SO. The effect is compounded because extreme-weight policies (the ones closest to the SO objective) are excluded from selection, and because only means over 20 seeds are reported, with no paired differences or confidence intervals. The paper therefore does not establish that the 39.07% (moderate contingency) or 21.88% (low-budget) deltas come from multi-objective learning; they could come from picking the lucky policy in a larger pool. The SO policy would need equivalent selection effort (e.g., best of K restarts) or a fixed a priori rule (e.g., a Pareto-center policy) for the comparison to be fair. Since this asymmetry is in the exact metric used for the headline percentages, it is the most load-bearing weakness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multi-policy MORL approach based on deep optimistic linear support (DOL) and multi-objective PPO (MOPPO) for power grid topology control. Custom reward functions are designed for line loading, topological deviation, and switching frequency, and the DOL outer loop generates a set of policies approximating a convex coverage set. Experiments on the RTE 5-bus Grid2Op environment compare DOL with random sampling using hypervolume, sparsity, and IGD; additional case studies compare MO policies with a single-objective (SO) PPO policy under N-1 contingencies and reduced training budgets, reporting improved episode duration of up to 39% and 22% respectively.","tokens_in":10482,"tokens_out":5625,"duration_ms":55663,"significance":"If the quantitative claims were sound, the paper would make a useful contribution: it would be among the first to apply multi-policy MORL to topology control and would provide a decision-support tool that explicitly surfaces trade-offs among operational objectives. The paper is transparent about its scope (5-bus system, limited objectives) and ships publicly available code, which strengthens reproducibility. The qualitative finding that DOL yields a denser coverage of the Pareto front than random sampling is plausible and useful. However, the headline quantitative comparisons are undermined by an asymmetric best-of-set policy selection and by an IGD evaluation whose reference set is constructed from the solutions being scored. These issues need to be resolved before the main claims can be accepted.","major_comments":[{"comment":"The MO policy used in the comparison is selected per seed as the policy with maximum episode duration E among all non-extremal DOL-generated policies, while the SO baseline is a single PPO policy trained on the line-loading reward. This is an asymmetric best-of-N comparison. If E is noisy across policies and seeds, taking the maximum over a policy set inflates the MO mean even when no individual MO policy is better than the SO policy, so the reported gains of 39.07% in Table II and 21.88% in Table III are not attributable to multi-objective training without equalizing selection effort. Please either give the SO baseline the same selection effort (e.g., best of K restarts or seeds), use a fixed a priori rule such as the Pareto-center policy, or report the full distribution of E for all MO policies and show per-seed paired differences with confidence intervals. The exclusion of extreme-weight policies, which are closest to the SO objective, should also be justified because it further tilts the comparison.","section":"Section II-D, Algorithm 3; Tables II and III"},{"comment":"The 'super CCS' used as the IGD reference is constructed as the convex set over all generated solution sets from both DOL and RS runs across seeds, and IGD of each method is then measured against that same set. Since DOL contributes many points to this reference, DOL's lower IGD is at least partly an artifact of the evaluation design. Please measure IGD against a reference that does not contain the evaluated solutions, for example the exact convex hull for the small 5-bus problem, a leave-one-seed-out reference, or a reference built from an independent exhaustive search. Without such a reference, the IGD column in Table I should not be used to support the claim that DOL approximates the true CCS better.","section":"Section III-B, Table I, Figs. 3-5"},{"comment":"The paper reports only means over 20 seeds and percent differences, with no standard deviations, confidence intervals, or significance tests. This is especially important because Algorithm 3 performs a selection step over policies, which can inflate the mean and the variance of the reported E. Please report seed-level distributions, standard errors or bootstrap confidence intervals, and a paired statistical test (e.g., Wilcoxon signed-rank on per-seed episode durations) for both the contingency and training-budget comparisons.","section":"Section III-C and III-D, Tables II and III"},{"comment":"The text does not state whether the E values used in Tables II and III are computed on the same scenarios and episodes that were used to select the best MO policy in Algorithm 3, or whether selection is performed on validation scenarios and the reported values come from held-out test scenarios. If the same E and scenarios are used both for selection and for final evaluation, the reported improvement contains test-set selection bias. Please clarify the protocol, and if selection currently uses test data, move it to validation data or replace it with a fixed a priori policy-selection rule.","section":"Section II-D and Sections III-C/III-D"}],"minor_comments":[{"comment":"The abstract states that MO policies are '30% more successful' under contingencies, but Table II reports 10.17%, 39.07%, and 23.86% for the three contingency levels, and the conclusion says 24%; please make the reported percentages consistent across the abstract, tables, and conclusion.","section":"Abstract vs. Section III-C"},{"comment":"There are several typos: 'choosen' should be 'chosen', 'closet w' should be 'closest w', and Algorithm 1 line 4 contains an unbalanced parenthesis: 'collect samples (MOPPOθ)' should be 'collect samples with MOPPOθ'.","section":"Section II-C and Algorithm 1"},{"comment":"The legend distinguishes 'DOL Points' and 'DOL CCS Points', but the caption says the plots show points from DOL and RS runs; please clarify which markers correspond to which methods and how the 'super CCS' curve is derived from the combined point sets.","section":"Figs. 3-5"},{"comment":"The sentence 'as only little scenarios are available' should be rephrased, and the text should state clearly whether the 16/2/2 scenario split is used consistently for the Pareto-front, contingency, and training-budget case studies, and whether contingency scenarios are applied during training or only at evaluation.","section":"Section III-A"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of an applied ML-for-power-systems venue and the authors are transparent about the small test system and limited objectives. My main concern is the evaluation asymmetry and the circular IGD reference, both of which affect the headline quantitative claims. I believe these can be fixed within the scope of a revision by equalizing the selection effort, adding proper statistics, and using an independent reference set, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Jeroen,\n\nQuick take on arXiv:2502.00040. The paper applies DOL+MOPPO to Grid2Op's 5-bus system, with custom reward functions for line loading, topological deviation, and switching frequency. That is a legitimate first – no prior multi-policy MORL study for topology control that I know of. The trade-off plots in Figs. 3-5 are informative, and the DOL vs random-search comparison for Pareto-front density is sensible. The code is public.\n\nThe qualitative story – multi-objective training gives you a portfolio of policies with interpretable trade-offs – is plausible and probably holds up. The problem is the quantitative headline. The stress-test note is on target: in Tables II and III, the MO policy is selected per seed via Algorithm 3 as the one with the highest episode duration among the DOL-generated set, while the SO policy is a single PPO run trained on the line-loading reward. That is best-of-N versus one draw. Episode duration is noisy across seeds; picking the maximum over a set lifts the mean even if no individual MO policy is genuinely better. The paper gives no paired differences, no confidence intervals, and no equivalent selection effort for the SO baseline. The 39% and 22% deltas likely shrink or vanish under a fair comparison (e.g., best of K SO restarts, or a fixed a-priori policy).\n\nThe IGD evaluation is also circular: the 'super CCS' reference set is built from the union of DOL and RS solution sets, and IGD is measured against that same set. Since DOL contributes many points, its low IGD is partly by construction.\n\nMinor: the abstract says 30% and 20%, but the tables show a range – 39.07% for moderate contingencies, 23.86% for high, 10.17% for none; and 21.88% for low budget, 5.49% for moderate, 10.17% for full. The abstract should quote the range or specify the exact condition. Also, the reward thresholds (rD, rF, dthreshold, Fth) are not reported; they are presumably in the code, but the paper should state them.\n\nOverall: a serious proof-of-concept, not a throwaway. The application is new, the methods are standard, the trade-off findings are useful. The quantitative claims need redone with paired comparisons and a fair selection rule. I'd send to peer review expecting major revision – desk rejection would be wrong.\n\nCheers,\n[Your name]","headline":"Genuine first multi-policy MORL for topology control, but the headline MO-vs-SO gains are a best-of-N artifact and the IGD evaluation is circular.","tokens_in":11050,"tokens_out":3509,"would_cite":true,"duration_ms":29713,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a multi-policy MORL approach can generate Pareto-optimal topology-control policies that survive grid contingencies longer and train more efficiently than single-objective RL.","keywords":["multi-objective reinforcement learning","power grid topology control","Pareto front","deep optimistic linear support","PPO","line loading","switching frequency","transmission congestion management"],"falsifier":"Retrain the single-objective baseline many times per seed and select its best policy by episode duration, or evaluate the multi-objective policy selected by a fixed rule such as the front center, and compare mean episode durations on the same contingency and budget settings. If the gap drops below the reported 22-39%, the central claim of multi-objective advantage is not supported. Rerunning the comparison on a different grid or with a different line-failure distribution would also test whether the effect is specific to the five-bus setup.","tokens_in":10019,"feed_emoji":"⚡","tokens_out":7144,"duration_ms":61272,"temperature":0.7,"pith_summary":"This paper argues that power-grid topology control is inherently multi-objective and that reinforcement learning should reflect that. The authors build a multi-objective RL approach that returns a set of Pareto-optimal policies trading off line loading, topological deviation, and switching frequency, instead of a single policy. In a five-bus benchmark grid, the resulting policies keep the grid from failing longer than a single-objective policy under simulated N-1 line outages (up to roughly 39% longer under moderate contingency frequency) and when training is cut to half (about 22% longer). The paper also reports that the deep optimistic linear support outer loop gives a denser, closer approximation of the Pareto front than random weight sampling. If correct, the approach gives operators an explicit map of operational trade-offs rather than one black-box action policy.","feed_headline":"Multi-objective RL keeps a power grid alive up to 39% longer","feed_subtitle":"Balancing line loading, topology changes and switching beats single-objective policies under N-1 line failures","key_machinery":"The engine is the combination of deep optimistic linear support (DOL) with multi-objective PPO (MOPPO). DOL is an outer-loop algorithm that repeatedly proposes a scalarization weight vector w, prioritized by expected improvement, and MOPPO trains a policy with that weight: a vectorized critic estimates a value vector in R^d, advantages are computed per objective, then scalarized as A_t = w^T A_t before the policy update. The resulting average reward vector V is added to a convex coverage set if it is Pareto-optimal for some weight. The reward vector has three components: line-loading margin (squared thermal margins), topological deviation (piecewise penalty for substations moved away from the default busbar configuration), and switching frequency (piecewise penalty for accumulated switching actions within an interval). The policy selection step later picks, per seed, the non-extremal policy with the highest episode duration E.","core_discovery":"The central claim is that a multi-policy MORL pipeline can solve the topology-control problem better than a single-objective RL baseline. The pipeline trains multiple policies via a multi-objective PPO variant, each scalarized by a different weight vector supplied by deep optimistic linear support, and collects the resulting value vectors into a convex coverage set. On the paper's five-bus case studies, policies from this set outperform a line-loading-only PPO policy in mean episode duration under all tested N-1 contingency frequencies: 94.83% vs 82.66% of the maximum episode with no contingencies, 97.68% vs 58.61% with moderate contingencies, and 90.33% vs 66.47% with high contingency frequency. Under a 50% training budget the multi-objective policies reach 95.27% mean episode duration versus 73.39% for the single-objective policy. Compared with random weight sampling, DOL produces a 50% lower sparsity and a 60% lower inverted generational distance, with similar hypervolume.","pith_inferences":["The headline comparison may overstate the advantage: the multi-objective policy is selected per seed as the best of the generated front, while the single-objective policy is one run. Re-running the single-objective baseline many times and selecting the best, or fixing a selection rule such as the Pareto-front center, would test how much of the 30%/20% gap is due to this selection asymmetry.","A second testable consequence is that multi-objective rewards act as a curriculum or regularizer in low-data regimes; one could verify by ablating each reward component separately to see which one drives early robustness.","On larger grids the trade-off geometry will change; the same DOL+MOPPO pipeline could be evaluated on a bigger benchmark to see whether the Pareto front remains dense and whether selection by episode duration remains a good proxy for operator preference.","Because the method returns a convex coverage set, it could be coupled downstream with an operator utility function or market prices to choose a policy automatically, which the paper does not explore."],"forward_implications":["Operators can be offered a menu of policies with explicit trade-offs (e.g., low switching frequency vs low line loading) instead of one fixed policy.","Multi-objective training appears to act as a regularizer: policies trained to also minimize topology changes and switching survive contingencies longer than single-objective policies.","Under limited compute, multi-objective policies reach usable behavior with half the training interactions, which matters as grid size grows.","DOL's denser Pareto-front approximation makes the method a plausible decision-support layer on top of existing single-objective RL controllers.","The convex coverage set can be extended as new objectives (cost, emissions) are added without retraining from scratch."],"supporting_citations":[{"why":"Provides the line-loading reward and the topology-control challenge formulation the environment rewards build on.","marker":"[3]"},{"why":"Supplies the expert safety rules used alongside the trained policies during contingency testing.","marker":"[8]"},{"why":"Supplies the expert rules and RL methodology that the training setup inherits.","marker":"[14]"},{"why":"Provides the adversarial contingency attack settings and hyperparameter starting points for the case studies.","marker":"[19]"},{"why":"Is the PPO algorithm with generalized advantage estimation that MOPPO extends.","marker":"[22]"},{"why":"Provides the multi-objective PPO implementation and multi-objective RL benchmarking tools.","marker":"[23]"},{"why":"Is the deep optimistic linear support algorithm that constructs the convex coverage set of policies.","marker":"[24]"},{"why":"Defines the MORL framework and the Pareto-front metrics (hypervolume, sparsity, IGD) used for evaluation.","marker":"[25]"},{"why":"Supplies the five-bus simulation environment and the episode-duration metric used to select and compare policies.","marker":"[26]"},{"why":"Provides the optimistic linear support corner-weight computation that DOL relies on to propose weight vectors.","marker":"[27]"}],"fun_headline_variants":["Multi-objective RL prevents grid failure 30% better than single-objective","Pareto-optimal policies beat single-objective RL in grid control","MORL keeps power grids alive 30% longer under N-1 failures","Multi-objective PPO outperforms single-objective in grid topology control"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported advantage depends on comparing the best multi-objective policy chosen per random seed by episode duration against a single single-objective policy from the same seed; if the single-objective baseline had been chosen as the best of several runs or by the same selection rule, the reported advantage could shrink or disappear.","fun_headline_variants_meta":{"raw":{"variants":["Multi-objective RL prevents grid failure 30% better than single-objective","Pareto-optimal policies beat single-objective RL in grid control","MORL keeps power grids alive 30% longer under N-1 failures","Multi-objective PPO outperforms single-objective in grid topology control"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000675,"raw_usage":{"total_tokens":3072,"prompt_tokens":947,"completion_tokens":2125,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":2043}},"tokens_in":563,"tokens_out":2125,"duration_ms":13831,"temperature":1.0,"reasoning_tokens":2043,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T13:49:27.131952+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the single-objective baseline many times per seed and select its best policy by episode duration, or evaluate the multi-objective policy selected by a fixed rule such as the front center, and compare mean episode durations on the same contingency and budget settings. If the gap drops below the reported 22-39%, the central claim of multi-objective advantage is not supported. Rerunning the comparison on a different grid or with a different line-failure distribution would also test whether the effect is specific to the five-bus setup.","supporting_citations":[{"cited_title":"Managing power grids through topology actions: A comparative study between advanced rule-based and reinforcement learning agents,","cited_arxiv_id":null,"evidence_quote":"Supplies the expert safety rules used alongside the trained policies during contingency testing."},{"cited_title":"Exploring grid topology reconfiguration using a simple deep reinforce- ment learning approach,","cited_arxiv_id":null,"evidence_quote":"Supplies the expert rules and RL methodology that the training setup inherits."},{"cited_title":"Hierarchical reinforcement learning for power network topology control","cited_arxiv_id":null,"evidence_quote":"Provides the adversarial contingency attack settings and hyperparameter starting points for the case studies."},{"cited_title":"A toolkit for reliable benchmarking and research in multi-objective reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Provides the multi-objective PPO implementation and multi-objective RL benchmarking tools."},{"cited_title":"A practical guide to multi- objective reinforcement learning and planning,","cited_arxiv_id":null,"evidence_quote":"Defines the MORL framework and the Pareto-front metrics (hypervolume, sparsity, IGD) used for evaluation."},{"cited_title":"Grid2op,","cited_arxiv_id":null,"evidence_quote":"Supplies the five-bus simulation environment and the episode-duration metric used to select and compare policies."},{"cited_title":"Multi-objective decision-theoretic planning,","cited_arxiv_id":null,"evidence_quote":"Provides the optimistic linear support corner-weight computation that DOL relies on to propose weight vectors."}],"review_version":1}