{"id":"2860f563-58a9-4625-a0bc-3536bf016e67","arxiv_id":"2607.28822","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A benchmark of agentic AI for power-system planning finds current-style agents only solve the two simplest task levels on small grids and fail on larger grids or harder tasks.","lead":"This paper tests a self-built AI 'agent' for power-grid planning against six escalating task levels and four grid sizes, finding that it can only complete the two easiest task levels on the smaller grids. The study maps concrete capability gaps — inefficient backend code, token overflow, and missing helper functions — for anyone developing AI copilots for grid operations.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'solvable' result counts workflow completion rather than correct answers; the paper's own admission that the agent misreads solver outputs undermines the central claim until ground-truth verification is added.","rationale":"The reader's weakest assumption—that the self-built 'generic best practice' agent may not represent the actual state of the art—is a legitimate external-scope concern, and it is strongly supported by the paper's own Section V admission that it is 'difficult to predict exactly how those agents would perform under the proposed framework.' However, the more load-bearing problem is internal: even on the runs the authors label 'solved,' the paper concedes the agent did not correctly interpret solver outputs or reason about the numbers. This means the central claim about what is 'solvable' is not backed by a correctness criterion. A reader cannot tell whether 'solved' means 'the workflow executed' or 'the final answer is numerically correct.' The paper could be over-claiming (Levels 1–2 are not actually solved) or the term is just sloppy, but either way the headline result is unverified. This concern does not require access to other agents' code to land, and it can be settled by a concrete reference-solver comparison. I therefore agree partially with the reader: the representativeness issue is real and should be in the manuscript, but the ground-truth-verification issue is more immediate and more damaging to the abstract's central quantitative claim. The reader's CONDITIONAL verdict remains appropriate; the condition should explicitly require either a correctness validator or a guarded restatement of 'solvable.' No verdict change is needed.","tokens_in":7781,"tokens_out":6968,"duration_ms":74323,"concrete_test":"Implement an independent ground-truth verification harness for the exact tiled grids described in Section III.B (the 100-bus base case and its 1,000-bus tiling are in the supplemental material). For each of the 480 logged attempts, compute the correct Level 1 hosting-capacity list and Level 2 reactive-adequacy result using a reference ACOPF solver (e.g., ExaGO or PowerModels.jl) on the same grid and contingency/feasibility definitions. Then compare the agent's final reported answer to this reference with a tolerance on bus selection and numerical values, and recompute the pass rate among attempts currently counted as 'solved.' If the correct-answer pass rate is materially below 100%, the abstract's 'only the two lowest complexity levels are solvable' claim must be revised to specify that completion does not imply correctness, or the claim must be downgraded until verified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—'only the two lowest complexity levels are solvable'—is not supported by the paper's own success metric. In Section IV (Results and Discussion, Table II), OpenAI is reported to have 'solved both prompts in all 10 attempts' and DeepSeek 'solved the Level 1 prompt in 5 of 10 attempts,' yet the same section states: 'Even when the solver sweeps were implemented correctly, the AI did not pick the correct numbers from the solver outputs or did not reason correctly about the solutions.' It also says that extra AI efforts such as 'selecting the lowest-cost solution and making analytical recommendations' were 'not consistent with the numbers produced by the solver, for either vendor or any system.' Thus an attempt labeled 'solved' appears to mean the agent completed the intended sequence of hops and produced some output, not that the output matched a reference solution. Without a ground-truth ACOPF comparator, 'solvable' is ambiguous at best and potentially false: if many apparently successful Level 1/2 runs returned incorrect engineering answers, the accurate headline would be that nothing is solvable, or at least that successful workflow execution does not imply correct results. This is an internal inconsistency, independent of whether the prototype faithfully represents the state of the art, and it directly affects the paper's main quantitative conclusion.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript addresses whether current agentic AI for power systems can support nodal connection-planning studies at scale. The authors survey five grid-specific agentic frameworks, extract a common architecture, and build their own 'generic best practice' agent (AgentiGrid integrated with ExaGO). They evaluate it on four grid sizes (100, 1,000, 10,000, and 100,000 buses) and six prompt levels defined by the number of discrete 'hops' required. The reported results show that Level 1 and Level 2 prompts are completed on 100- and 1,000-bus grids (OpenAI 10/10 on both levels; DeepSeek 5/10 on Level 1 and 10/10 on Level 2), while Levels 3–6 and the 10,000- and 100,000-bus cases produced no results. The authors conclude that only the two lowest complexity levels are solvable on some grid sizes and identify backend and RAG bottlenecks as the main capability gaps.","tokens_in":8074,"tokens_out":4360,"duration_ms":46690,"significance":"If the reported results are verified, the paper provides a valuable cautionary empirical benchmark: it suggests that current agentic AI for power systems is not yet ready for realistic connection studies, and it offers a structured difficulty ladder and a reproducible test methodology. The hop-based prompt hierarchy and the constrained tiled grids are useful contributions, and the availability of code/data for the prototype is a strength. However, the headline claim depends critically on what is meant by 'solvable,' and the manuscript's own text indicates that workflow completion and correct engineering answers were conflated. The significance of the negative result therefore cannot be fully assessed until a correctness check against reference solutions is performed.","major_comments":[{"comment":"The metric labeled 'solved' does not verify correctness. Immediately after Table II the text states: 'Even when the solver sweeps were implemented correctly, the AI did not pick the correct numbers from the solver outputs or did not reason correctly about the solutions,' and that analytical recommendations were 'not consistent with the numbers produced by the solver, for either vendor or any system.' Thus the 10/10 and 5/10 entries indicate that the agent completed the intended workflow, not that its outputs matched a reference solution. Without a ground-truth ACOPF comparator, the abstract's claim that 'only the two lowest complexity levels are solvable' is unsupported; if completed runs returned incorrect numbers, then no level was actually solved. Please add a per-attempt correctness check against a reference solution, or redefine 'solvable' as 'workflow completed' and report correctn","section":"Section IV, Table II"},{"comment":"The paper claims to 'replicate the current state of the art in agentic AI for power systems planning,' but the experiments run only the authors' own AgentiGrid prototype. Section III.A says the prototype 'closely reflects the state of the art described in the literature,' and Section V concedes that 'it is difficult to predict exactly how those agents would perform under the proposed framework.' Because PowerAgent, GridAgent, GridMind, LLM4OPF, and GAIA were not run, the headline conclusion is demonstrated only for this one prototype, not for the state of the art generally. Please narrow the abstract and conclusions to 'our prototype' or, where third-party code is public, run and compare the cited agents.","section":"Abstract; Sections III.A, V"},{"comment":"The 10,000- and 100,000-bus results and the Level 3–6 failures are described only qualitatively and are omitted from Table II. The failures are attributed to specific implementation limitations of one code snapshot (O(N^2) deep-copy sweeps and context-window overflow), which the authors state have already been addressed in a newer AgentiGrid version. As written, the central result is tied to a particular software version and cannot be independently reconstructed from the paper. A supplementary table listing all 480 runs (attempts, tokens, time, and failure mode) would let readers verify the reported coverage and would make the benchmark genuinely reproducible.","section":"Section III.B; Section IV; Table II"}],"minor_comments":[{"comment":"The text introduces 'difficulty level from 0 to 6,' but Table II and the prompt list use only Levels 1–6. Clarify whether Level 0 exists and what it would be.","section":"Section III.B"},{"comment":"The phrase 'The source code is available at [19]' may be ambiguous because reference [19] is the ExaGO v2 software DOI. Please specify whether the AgentiGrid agent code and the grid-generation scripts are included in that DOI or in the supplemental material.","section":"Section III.A"},{"comment":"Using Claude Opus 4.8 for prompt validation and result aggregation is reasonable, but the statement that it 'cannot bias the measured performance of either model' is too strong. A third-party LLM could still misread or misreport solver outputs, which is a reporting risk rather than a model-performance bias.","section":"Section III.B"},{"comment":"The caption says 'state of the art grids and prompts,' but the grids are newly constructed by tiling a 100-bus block and the prompts are newly defined. Rephrase to avoid implying these are the exact grids or prompts used by the cited agents.","section":"Table II caption"},{"comment":"Reference [16] appears to have an unusual volume/issue/page format; verify it against the publisher's style and page range.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is a potentially useful empirical contribution: a negative result about agentic AI for power systems, with a reproducible benchmark and a clear difficulty ladder. The main obstacle is the conflation of workflow completion with correctness; the manuscript's own text admits that even 'successful' attempts did not always produce numbers consistent with the solver. If the authors can add ground-truth verification or honestly reframe the paper as a workflow-completion benchmark, the manuscript would be publishable. The secondary issue is over-generalization from a single self-built prototype to the 'state of the art.' I would not recommend rejection because both issues are addressable within the scope of the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — worth a look. The real contribution is the stress-test itself: six hop-based prompt levels, four tiled grid sizes up to 100k buses, and a concrete failure-mode forensic (cache key omitting reactive load, O(N²) deep-copy sweeps, context-window overflow). That's not present in the cited prior work, and it's the right kind of challenge for the community. The authors are also refreshingly honest about their own prototype's limits.\n\nThe soft spots are real but addressable. First, the abstract claims to 'replicate the current state of the art,' but they built a single generic agent from AgentiGrid/ExaGO and infer the other agents' behavior from descriptions — then concede in Section IV that they can't predict how PowerAgent, GridAgent, or GridMind would do. So the headline finding applies to one prototype, not the literature. Second, and more seriously, the 'solvable' metric is under-defined. Table II counts attempts as solved, yet the same section says the AI didn't pick the correct numbers from solver outputs and its analytical recommendations didn't match solver results. That means 'solved' is at best workflow completion, not engineering-correct answers. Without a ground-truth ACOPF comparator, the central claim — that only Levels 1 and 2 are solvable — is ambiguous. It might be true, but the paper as written doesn't demonstrate it. The authors may have intended a distinction between completing steps and making extra recommendations; if so, they need to draw that line explicitly.\n\nWhat holds up: the tiled grids and adjusted constraints are a reasonable way to test congestion and reactive issues, the hop ladder is a sensible complexity proxy, and the failure modes they document are instructive and specific. The citation pattern looks fine — they cite the five major agent papers and note the code-availability problem correctly.\n\nBottom line: the framework is reusable and the failure analysis is valuable, but the paper needs a ground-truth check or a rewording before the main claim stands. I'd send it to peer review, expecting a conditional accept after revision. Tempering the abstract and adding an explicit definition of 'solved' would fix most of my reservations.","headline":"A candid capability-gap stress test with a useful failure-mode autopsy, but the 'solvable' label overstates what the agent actually got right — the abstract needs a ground-truth comparator or a rewording to 'workflow completion.'","tokens_in":8581,"tokens_out":1729,"would_cite":true,"duration_ms":20462,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Current agentic AI solves only the two simplest grid-planning prompts","keywords":["agentic artificial intelligence","power system planning","grid planning","large language models","capability benchmarking","nodal planning","contingency analysis","scalability"],"falsifier":"A reader could run the same six-level suite on the code released with the paper (or on the current in-house version mentioned in Section IV) and inspect whether Level 3–6 prompts ever produce solver output. The paper states that the OpenAI agent never sent contingencies into the solver and DeepSeek listed but did not evaluate them; if a run of the same agent shows a contingency set being solved and an answer returned, the central claim that these levels are unsolvable would be falsified.","tokens_in":7649,"feed_emoji":"⚡","tokens_out":8118,"duration_ms":76154,"temperature":0.7,"pith_summary":"The paper sets out to test whether agentic AI—AI agents that can chain tool use, run simulations, and iterate—can do realistic power-grid connection studies. The authors replicate the common design of published grid-planning agents and subject their prototype to a structured ladder of six task-complexity levels and four grid sizes. They find that only the two lowest complexity levels are solvable, and only on 100- and 1,000-bus grids; levels 3–6 and grids of 10,000 or more buses produce no results. This matters because grid operators face a wave of new data-center and generation interconnections and had hoped agents could automate the repetitive screening work. The paper concludes that current agentic AI is not yet ready for realistic connection studies and identifies specific backend and retrieval improvements needed.","feed_headline":"Agentic AI tops out at simplest grid-planning tasks","feed_subtitle":"A six-level, four-scale test shows current agents fail on contingency analysis and 10,000-bus grids.","key_machinery":"The key machinery is a structured test suite built around a 'hop-count' difficulty ladder: each prompt's level is the number of discrete computational operations (constructing candidate sets, solving cases, evaluating feasibility, aggregating, selecting) required to go from the base case to the answer. This gives six levels, from single aggregation (Level 1) to N-1 contingency testing (Level 3) and N-2 contingency relief (Level 6). Grids are built by tiling a 100-bus block derived from a reduced IEEE 118-bus system and modified to introduce congestion and reactive-power constraints, then tested at 100, 1,000, 10,000, and 100,000 buses. The difficulty ladder and the tiled, congestible grids t","core_discovery":"On the paper's own terms, its central discovery is that the current state of the art in agentic AI for power systems can handle at most two of six complexity levels on grids up to 1,000 buses. The authors built a generic agent incorporating storage, retrieval-augmented generation, a solver interface, an executor, a context module, and a validator—the elements common to all published grid-planning agents—and tested it on a suite of nodal planning prompts. Levels 1 and 2, which ask for bus hosting-capacity screening and reactive-adequacy reporting, were solved reliably by the OpenAI configuration (10/10 attempts) and partially by the DeepSeek configuration (5/10 on Level 1, 10/10 on Level 2).","pith_inferences":["If the authors' generic agent is a fair proxy for the field, then published claims of agentic AI handling N-1 contingency management or grid control likely overstate maturity; but because the real agents' code and data are not public, this inference depends on that proxy's fidelity.","The hop-count difficulty ladder is a transferable profiling tool: the same methodology could be applied to other engineering AI applications (e.g., water networks, transportation) to locate the boundary between what agents can and cannot do.","The finding that native LLM reasoning with minimal retrieval fails on Level 3+ suggests that agent performance may be gated more by the quality of the tool/retrieval layer than by model intelligence, implying that investments in structured knowledge bases and sparse-data backends could yield larger gains than larger models.","If the cache-key bug is representative, testing protocols should randomize parameter permutations to expose silent result-reuse errors; the paper's proposed stricter protocols would be a good basis for such adversarial testing."],"forward_implications":["Existing agentic AI cannot yet automate realistic connection studies; human engineers must still interpret solver outputs, since even the model's successful sweeps were not always correctly interpreted by the AI.","Grid size is a primary barrier: systems beyond 1,000 buses overflow context windows and face O(N²) sweep costs, so performance does not scale from small test grids to utility-scale systems.","Complexity beyond basic screening requires extensive, detailed retrieval (helper functions and dictionaries), not just a more powerful model or more tokens.","A reproducible test suite of this kind—six prompt levels, four grid scales—can serve as a benchmark for measuring genuine progress in agentic AI for power systems.","The specific failure modes identified (e.g., sweep-cache keys omitting the reactive-load parameter) imply that subtle implementation bugs can invalidate results even for easy prompts."],"fun_headline_variants":["Agentic AI only masters 2 of 6 grid planning complexity levels","Grid-planning AI fails at contingency analysis and large grids","Current agentic AI flunks most power system planning tasks","AI for grid planning: basic tasks only, study finds","Agentic AI capability gap blocks real grid planning"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The conclusion that only the two simplest levels are solvable industry-wide rests on the assumption that the authors' self-built generic agent faithfully represents the actual state of the art in agentic AI for power systems; if the real agents have unreleased fixes or better backend engineering, they might perform higher on the ladder.","fun_headline_variants_meta":{"raw":{"variants":["Agentic AI only masters 2 of 6 grid planning complexity levels","Grid-planning AI fails at contingency analysis and large grids","Current agentic AI flunks most power system planning tasks","AI for grid planning: basic tasks only, study finds","Agentic AI capability gap blocks real grid planning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000672,"raw_usage":{"total_tokens":2876,"prompt_tokens":699,"completion_tokens":2177,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":443,"completion_tokens_details":{"reasoning_tokens":2095}},"tokens_in":443,"tokens_out":2177,"duration_ms":16280,"temperature":1.0,"reasoning_tokens":2095,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T00:16:50.069447+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could run the same six-level suite on the code released with the paper (or on the current in-house version mentioned in Section IV) and inspect whether Level 3–6 prompts ever produce solver output. The paper states that the OpenAI agent never sent contingencies into the solver and DeepSeek listed but did not evaluate them; if a run of the same agent shows a contingency set being solved and an answer returned, the central claim that these levels are unsolvable would be falsified.","supporting_citations":[],"review_version":1}