REVIEW 5 major objections 5 minor 18 references
VeraGrid-Agent: Tool-Augmented LLMs for Distribution Optimal Power Flow at the Grid Edge
T0 review · 5 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash
Pith's one-line read Giving an LLM a power-flow solver raises accuracy on grid-edge questions from around 45% to 97-100%.
desk verdict Practical, reproducible benchmark showing tool access fixes LLM quantitative gaps; the unvalidated solver oracle is the main caveat. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The control loop is the central mechanism: a language model issues read, write, and execute calls against an isolated workspace, writes the feeder and DER description in JSON, runs the VeraGrid solver, and selectively reads the results table of contents before answering. The benchmark is the other half: VeraGrid-MCQ-150, 150 multiple-choice questions generated by 33 expert templates (8 easy, 10 medium, 15 hard) whose ground truth is computed from the solver's record R via formulas such as Ploss = Pf + Pt, with distractors constructed from the same record.
What would settle it
Take a small feeder where the true AC-OPF solution can be verified by independent means (e.g., brute-force search or a second, independently implemented solver), run the 150-question template, and check whether VeraGrid's output and the template formulas (such as Ploss = Pf + Pt) reproduce the independent ground truth; any mismatch in even one question would invalidate the accuracy numbers.
Extended reading notes
Core claim
The central claim is that tool access, not model size or reasoning skill, is the decisive factor for numerically grounded power-flow questions. The paper demonstrates this by comparing the same models in two regimes: answering from the feeder description alone, and acting as VeraGrid-Agent, where the model writes the JSON input, executes the open-source VeraGrid AC-OPF solver, and reads back targeted result tables. Accuracy jumps from 42.7–49.3% to 97.3–100.0%. The few remaining errors trace to wrong interpretation of solver records during multi-step reasoning, not to failures of the solver or the tool calls. The authors frame the benchmark as memorization-resistant because every answer is a
Load-bearing premise
The ground-truth answers are assumed to be correct because they come from the VeraGrid solver plus expert template formulas; if the solver mis-solves the non-convex AC-OPF or a template formula is wrong, the reported accuracy measures agreement with a flawed oracle.
Editorial extensions
If this is right
- The paper's result implies that evaluations of LLMs on engineering questions must include a tool-use regime; scores without tools measure memorization, not computation.
- The choice of LLM backbone matters far less than access to the solver: the best and worst models converge to near-identical accuracy once the solver is available.
- The few failures are semantic misinterpretations of solver records, so improving the agent's query-to-result mapping, not solver accuracy, is the next bottleneck.
- Because ground truth is tied to a specific solved network, the benchmark cannot be gamed by memorizing answers; changes to loads, impedances, or limits change the labels.
- The claimed gains are established for a balanced radial feeder; the same architecture should extend to larger unbalanced three-phase systems, which the paper lists as future work.
Reading between the lines
- The same tool-augmented pattern likely transfers to other non-convex engineering optimization problems with deterministic solvers (gas networks, water distribution), where the LLM's role becomes translating queries into solver calls.
- A testable extension would inject a known bias into the solver's output and check whether agent answers shift accordingly, confirming the agent reads the solver rather than pattern-matching from memory.
- The measured 17–27 seconds per question suggests a distillation target: training a small model on the agent's reasoning traces could produce a fast, tool-free approximate reasoner while retaining much of the accuracy.
- If near-ceiling accuracy holds on harder feeders, the real discriminator between LLMs becomes the fidelity of natural-language-to-solver translation, for example correctly distinguishing 'curtailment' from 'dispatch'.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces VeraGrid-Agent, a tool-augmented LLM that answers distribution optimal power flow (D-OPF) questions by writing a JSON input for the open-source VeraGrid solver, executing it, reading targeted result sections, and then selecting a four-option MCQ answer. To evaluate this, the authors construct VeraGrid-MCQ-150, a generator of 150 multiple-choice questions from 33 expert templates on a single 8-bus radial feeder; the ground-truth label is computed from the solver output via template formulas. Seven LLMs are evaluated in two regimes that differ only in tool access. Without tools, accuracy is 41.3–49.3%; with the agent, accuracy is 97.3–100.0%. The paper concludes that access to external tools improves performance and can solve tasks that cannot be reliably solved using parametric knowledge alone, and the small number of agent failures is attributed to wrong interpretation rather than solver/tool failure.
Significance. If the benchmark is valid, the reported effect is striking and practically relevant: giving an LLM access to a deterministic solver and a targeted read tool brings quantitative grid-edge question answering from near-chance to near-perfect, consistently across seven models. The benchmark design is a useful step toward separating computation from memorization—each label is a deterministic function of a particular solved network, and the agent's answer is traceable to a specific solver record. The open-source repository and template-driven generation are strengths. However, the current evidence base is narrow: one balanced 8-bus feeder, 150 items, no variance estimates or statistical tests, and no independent validation of the solver/template oracle. The central claim is plausible but not yet established at the strength stated in the abstract.
major comments (5)
- [III (VeraGrid-MCQ-150 generation)] The ground-truth labels in VeraGrid-MCQ-150 are computed as f(R), where R is the output of the VeraGrid solver and f is one of 33 template formulas (e.g., Ploss = Pf + Pt in the hard tier, and per-unit current |If| = |Sf|/Vm). The paper reports no convergence tolerances, no comparison with an independent AC-OPF solver, and no audit of the 33 formulas. Because the agent's answer is derived directly from the solver output, a systematic solver or template error would make the agent 'correct' against a defective oracle, while the no-tool baseline could remain wrong. This is load-bearing for the central claim that tool access 'solves tasks that cannot be reliably solved using parametric knowledge alone.' Please provide (i) solver validation (e.g., standard test feeders or agreement with a second solver), (ii) the numerical values of the objective weights lambda_c and lambda_v in Eq. (1), and
- [II.D / III.A] The evaluation protocol fixes the system prompt across the two regimes, but the prompt printed in Section II.D contains a TOOLS section and the instruction 'use tools for every quantitative query if available.' In the no-tool regime the tools are not available, so the model receives a prompt that instructs it to use tools that do not exist. This is a potential confound: the no-tool baseline may be penalized by the contradictory prompt, inflating the measured tool benefit. Please run the no-tool baseline with a prompt that is identical except for removal of the tool descriptions and workflow rules, or otherwise explicitly control for this mismatch.
- [IV, Table I; Conclusion] The empirical evidence is drawn from a single 8-bus radial feeder with 150 questions and no variance estimates or significance tests. The paper itself acknowledges in the Conclusion that 'our experiments use only a single radial feeder modeled in balanced form.' The abstract and conclusion generalize beyond this to 'tasks that cannot be reliably solved using parametric knowledge alone.' This broad claim is not supported by one feeder configuration. Please either add multiple feeders/configurations and repeated runs with standard errors, or restrict the claim to the tested setup.
- [III] The benchmark is described as deterministic, but answer choices are randomly permuted to mitigate position bias; no seed or fixed permutation list is reported. More importantly, the 33 templates are only summarized by difficulty tier, not fully specified. The reproducibility claim ('The benchmark is fully reproducible') is unverifiable from the manuscript without the complete template formulas, the exact solver version and options, and the numerical values of lambda_c and lambda_v. Please add a full template specification and exact generator seed to the appendix or repository.
- [IV.B, Table II] The error decomposition in Table II reports no execution errors or timeouts and attributes all failures to 'wrong interpretation.' The decision rule for this classification is not given, and it presupposes that the solver and template formulas are correct; if a template formula is wrong or a solver run converges to an incorrect local solution, errors would be misclassified. Please define the classification protocol, release per-question traces/logs, and show how the categories are deterministically assigned.
minor comments (5)
- [Eq. (1)] The objective uses 'X_E' in the manuscript, which appears to be a typo for a sum over branches E; please define all symbols before use.
- [III] There is a typo: '50questions per livel' should be '50 questions per level.'
- [IV.B] The sentence 'the model with the full accuracy scorers spends3-4calls' is ungrammatical; clarify which model achieves full accuracy and the associated tool-call count.
- [II.D] The system prompt formatting is inconsistent (e.g., 'VeraGrid agent' vs 'VERAGRID-AGENT'); consider unifying the terminology.
- [Fig. 1] The standalone LLM answer 'Ploss = 5 MW' is likely illustrative, but the figure should state that it is not taken from any evaluated model response.
Circularity Check
No circularity: benchmark labels and agent tool share the same solver by design, but the evaluated task is the LLM's tool-use and interpretation, not a derived result.
full rationale
The paper's derivation chain is an evaluation protocol, not a mathematical derivation. Section III states that 'the VeraGrid-MCQ-150 is produced by a deterministic, template-driven generator whose ground truth is computed directly by the VeraGrid simulator,' and Section II-A says 'Let R=Φ(G) denote the output of the simulator... Our task is to answer queries in natural-language about R without revealing R directly.' The agent then writes the simulator input, executes the same solver, and reads output before answering. This shared oracle is the intended benchmark mechanism: the solver defines the ground-truth labels, and the agent is being tested on whether it can correctly operate the solver and apply the template formulas. High accuracy is not forced by construction; the LLM must still generate valid JSON input, invoke the tool correctly, parse the relevant result records, and perform the specified arithmetic such as Ploss = Pf + Pt. None of these steps is an input to the label-generation procedure. There are no fitted parameters, no imported uniqueness theorems, no self-citation chain carrying the conclusion, and no renamed known result. The legitimate concern—that the VeraGrid solver and the 33 template formulas are not independently validated—is an oracle-validity or correctness-risk issue, not a circularity issue. Under the provided rubric, this is an honest non-finding with score 0.
Assumptions & free parameters
free parameters (2)
- lambda_c (curtailment penalty weight) =
not specified
- lambda_v (voltage penalty weight) =
not specified
assumptions (4)
- domain assumption AC-OPF formulation (Eq. 1) is an appropriate model for the grid-edge D-OPF with DER dispatch.
- domain assumption VeraGrid solver returns the correct solution of Eq. (1).
- domain assumption The template formulas (e.g., Ploss = Pf + Pt, |If| = |Sf|/Vm) correctly compute the queried quantities.
- domain assumption The 150 MCQs are not answerable from parametric memory.
Cite this review
Pith. "Pith review of VeraGrid-Agent: Tool-Augmented LLMs for Distribution Optimal Power Flow at the Grid Edge." pith.science (2026). https://pith.science/paper/CFQOIRQQ
@misc{pith2026260725155,
author = {Pith},
title = {Pith review of: VeraGrid-Agent: Tool-Augmented LLMs for Distribution Optimal Power Flow at the Grid Edge},
year = {2026},
howpublished = {\url{https://pith.science/paper/CFQOIRQQ}},
note = {Machine review of arXiv:2607.25155}
}
abstract
Language models have demonstrated remarkable success in solving a wide range of tasks. However, answering complex scientific questions about the power flow often requires solving the distribution optimal power flow (D-OPF) problem. These questions call for numerical solvers and simulators, as linguistic reasoning from parametric knowledge often gives incorrect answers. In this work, we present VeraGrid-Agent, a tool-augmented LLM that autonomously writes the simulator input, executes the open-source VeraGrid solver, and reads the solver output before answering. To evaluate performance, we introduce VeraGrid-MCQ-150, a set of deterministic, expert template driven, $150$ multiple-choice questions. We evaluate the performance under two regimes: (i) no-tool reasoning and (ii) agent (LLM with simulator access). Without tools, every model performs with an accuracy of $42.7\%$--$49.3\%$. However, with VeraGrid-Agent, accuracy increases to $97.3\%$--$100.0\%$. We also do a failure-mode analysis to show that the few remaining errors arise from wrong interpretations during multi-step reasoning, rather than any failure in the simulators execution.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
M. H. J. Bollen and F. Hassan,Integration of Distributed Generation in the Power System. Wiley-IEEE Press, 2011
2011
-
[2]
Strong NP-hardness of AC power flows feasibility,
D. Bienstock and A. Verma, “Strong NP-hardness of AC power flows feasibility,”Operations Research Letters, vol. 47, no. 6, pp. 494–501, 2019
2019
-
[3]
Toolformer: Language models can teach themselves to use tools,
T. Schick, J. Dwivedi-Yu, R. Dess `ı, R. Raileanu, M. Lomeli, E. Ham- bro, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: Language models can teach themselves to use tools,” inAdvances in Neural Information Processing Systems, vol. 36, 2023
2023
-
[4]
ReAct: Synergizing reasoning and acting in language models,
S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao, “ReAct: Synergizing reasoning and acting in language models,” in Proc. Int. Conf. Learning Representations (ICLR), 2023
2023
-
[5]
E. Karpas, O. Abend, Y . Belinkov, O. Lieber, N. Ratner, Y . Shoham, H. Bata, Y . Levine, K. Leyton-Brown, D. Muhlgay, N. Rozen, E. Schwartz, G. Shachaf, S. Shalev-Shwartz, A. Shashua, and M. Tenenholtz, “MRKL systems: A modular, neuro-symbolic architec- ture that combines large language models, external knowledge sources and discrete reasoning,”arXiv pre...
arXiv 2022
-
[6]
HuggingGPT: Solving AI tasks with ChatGPT and its friends in Hugging Face,
Y . Shen, K. Song, X. Tan, D. Li, W. Lu, and Y . Zhuang, “HuggingGPT: Solving AI tasks with ChatGPT and its friends in Hugging Face,” in Advances in Neural Information Processing Systems, vol. 36, 2023
2023
-
[7]
Gorilla: Large language model connected with massive APIs,
S. G. Patil, T. Zhang, X. Wang, and J. E. Gonzalez, “Gorilla: Large language model connected with massive APIs,” inAdvances in Neural Information Processing Systems, vol. 37, 2024, pp. 126 544–126 565
2024
-
[8]
Large language models for power system applications: A comprehensive literature survey,
M. Sarwar, M. Rizwan, M. Aziz, and A. R. Sudais, “Large language models for power system applications: A comprehensive literature survey,”arXiv preprint arXiv:2512.13004, 2025
arXiv 2025
Show all 18 references
-
[9]
ChatGrid: Intelligent knowledge Q&A for power dispatching control based on large language models and retrieval-augmented generation,
M. Ni, J. Zhang, C. Fu, J. Wang, X. Ning, and S. Li, “ChatGrid: Intelligent knowledge Q&A for power dispatching control based on large language models and retrieval-augmented generation,” in 2024 IEEE 7th Information Technology, Networking, Electronic and Automation Control Co...
2024
-
[10]
ElecBench: A power dispatch evaluation benchmark for large language models,
X. Zhou, H. Zhao, Y . Cheng, G. Liang, G. Liu, W. Liu, Y . Xu, and J. Zhao, “ElecBench: A power dispatch evaluation benchmark for large language models,”arXiv preprint arXiv:2407.05365, 2024
2024 arXiv
-
[11]
GridMind: LLMs-powered agents for power system analysis and operations,
H. Jin, K. Kim, and J. Kwon, “GridMind: LLMs-powered agents for power system analysis and operations,”arXiv preprint arXiv:2509.02494, 2025
2025 arXiv
-
[12]
PowerAgentBench-SS: A benchmark for agentic AI in power system steady-state studies,
C. Mylonas, M. Foti, A. Pomarico, M. Duarte, Q. Zhang, and E. Varvarigos, “PowerAgentBench-SS: A benchmark for agentic AI in power system steady-state studies,”arXiv preprint arXiv:2606.18789, 2026
2026 arXiv
-
[13]
Power systems agent benchmark: Executable eval- uation of AI agents in electric power engineering,
S. Trashchenkov, “Power systems agent benchmark: Executable eval- uation of AI agents in electric power engineering,”arXiv preprint arXiv:2606.20950, 2026
2026 arXiv
-
[14]
ProOPF: Benchmarking and improving LLMs for professional-grade power systems optimization modeling,
C. Shen, Z. Guo, X. Wan, Z. Yang, Y . Zhang, W. Huang, J. Song, Z. Zhang, and M. Sun, “ProOPF: Benchmarking and improving LLMs for professional-grade power systems optimization modeling,”arXiv preprint arXiv:2602.03070, 2026
2026 arXiv
-
[15]
Measuring mathematical problem solving with the MATH dataset,
D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Zhu, D. Song, and J. Steinhardt, “Measuring mathematical problem solving with the MATH dataset,”arXiv preprint arXiv:2103.03874, 2021
2021 arXiv
-
[16]
SciBench: Evaluating college-level scientific problem-solving abilities of large language models,
X. Wang, Z. Hu, P. Lu, Y . Zhu, J. Zhang, S. Subramaniam, A. R. Loomba, S. Zhang, Y . Sun, and W. Wang, “SciBench: Evaluating college-level scientific problem-solving abilities of large language models,” inProc. Int. Conf. Machine Learning (ICML), vol. 235, 2024, pp. 50 622–50 649
2024
-
[17]
Large language models are not robust multiple choice selectors,
C. Zheng, H. Zhou, F. Meng, J. Zhou, and M. Huang, “Large language models are not robust multiple choice selectors,” inProc. Int. Conf. Learning Representations (ICLR), 2024
2024
-
[18]
Optimal power flow using graph neural networks,
D. Owerko, F. Gama, and A. Ribeiro, “Optimal power flow using graph neural networks,”International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp. 5930–5934, 2020
2020
Reviewed August 1, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.