{"id":"a18ec8ea-5d7b-4015-aa20-180872c9c952","arxiv_id":"2607.25155","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Adding a power-flow solver as a tool lifts LLM accuracy on distribution OPF multiple-choice questions from 41–49% to 97–100%.","lead":"This paper connects large language models to a real power-grid solver, letting the model write input files, run the AC-OPF solver, and read the numeric results. On 150 new quiz questions, models with solver access scored 97–100%, while the same models without it scored 41–49%.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-truth labels depend entirely on the correctness of the VeraGrid AC-OPF solver and the 33 template formulas; neither is independently validated, so the reported near-perfect agent accuracy may measure agreement with a flawed oracle.","rationale":"The reader's weakest assumption is precisely the unvalidated solver/template oracle, and I agree. This is the single most load-bearing concern because the entire quantitative evaluation reduces to agreement with the solver output. Without an independent check, the near-perfect agent accuracies could be an artifact of a biased benchmark rather than evidence that tool-augmented LLMs solve real D-OPF questions. However, this is a fixable validation gap, not a demonstrated error: the relative no-tool vs. agent comparison would likely survive even if some labels change, since the agent tracks the solver. Therefore the conditional acceptance is appropriate, with independent oracle validation as the specific condition.","tokens_in":7819,"tokens_out":6848,"duration_ms":65215,"concrete_test":"Run the VeraGrid-MCQ-150 generator on the same 8-bus feeder with an independent, established AC-OPF solver (e.g., MATPOWER or OpenDSS) and compare the ground-truth answers for all 150 questions. Also manually verify each of the 33 template formulas against AC power-flow equations. If any answer differs, recompute agent accuracies with corrected labels to see if the 97–100% range persists; if none differ, the oracle is validated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that tool access lets LLMs solve quantitative grid-edge questions that parametric knowledge cannot. The evidence is agent accuracy of 97.3–100% on VeraGrid-MCQ-150. However, every question's correct answer is computed by the template generator from the output R of the VeraGrid solver (Section III). If the solver does not correctly solve the AC-OPF problem in Eq. (1), or if any of the 33 formulas (e.g., Ploss = Pf + Pt in the hard tier) is wrong, then the observed accuracies measure agreement with a defective oracle. The no-tool baseline would also be scored against the same defective labels, but the agent's answers are derived directly from the solver output, so a systematic solver error could propagate to high agent accuracy while no-tool models remain wrong. The paper provides no independent validation of the solver's correctness, no convergence tolerances, and no audit of the template formulas. Table II's error decomposition attributes failures to interpretation, implicitly assuming the oracle is correct. Thus the benchmark's validity—and the strong version of the central claim—rests on an unverified oracle.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces VeraGrid-Agent, a tool-augmented LLM that answers distribution optimal power flow (D-OPF) questions by writing a JSON input for the open-source VeraGrid solver, executing it, reading targeted result sections, and then selecting a four-option MCQ answer. To evaluate this, the authors construct VeraGrid-MCQ-150, a generator of 150 multiple-choice questions from 33 expert templates on a single 8-bus radial feeder; the ground-truth label is computed from the solver output via template formulas. Seven LLMs are evaluated in two regimes that differ only in tool access. Without tools, accuracy is 41.3–49.3%; with the agent, accuracy is 97.3–100.0%. The paper concludes that access to external tools improves performance and can solve tasks that cannot be reliably solved using parametric knowledge alone, and the small number of agent failures is attributed to wrong interpretation rather than solver/tool failure.","tokens_in":8094,"tokens_out":6644,"duration_ms":63689,"significance":"If the benchmark is valid, the reported effect is striking and practically relevant: giving an LLM access to a deterministic solver and a targeted read tool brings quantitative grid-edge question answering from near-chance to near-perfect, consistently across seven models. The benchmark design is a useful step toward separating computation from memorization—each label is a deterministic function of a particular solved network, and the agent's answer is traceable to a specific solver record. The open-source repository and template-driven generation are strengths. However, the current evidence base is narrow: one balanced 8-bus feeder, 150 items, no variance estimates or statistical tests, and no independent validation of the solver/template oracle. The central claim is plausible but not yet established at the strength stated in the abstract.","major_comments":[{"comment":"The ground-truth labels in VeraGrid-MCQ-150 are computed as f(R), where R is the output of the VeraGrid solver and f is one of 33 template formulas (e.g., Ploss = Pf + Pt in the hard tier, and per-unit current |If| = |Sf|/Vm). The paper reports no convergence tolerances, no comparison with an independent AC-OPF solver, and no audit of the 33 formulas. Because the agent's answer is derived directly from the solver output, a systematic solver or template error would make the agent 'correct' against a defective oracle, while the no-tool baseline could remain wrong. This is load-bearing for the central claim that tool access 'solves tasks that cannot be reliably solved using parametric knowledge alone.' Please provide (i) solver validation (e.g., standard test feeders or agreement with a second solver), (ii) the numerical values of the objective weights lambda_c and lambda_v in Eq. (1), and","section":"III (VeraGrid-MCQ-150 generation)"},{"comment":"The evaluation protocol fixes the system prompt across the two regimes, but the prompt printed in Section II.D contains a TOOLS section and the instruction 'use tools for every quantitative query if available.' In the no-tool regime the tools are not available, so the model receives a prompt that instructs it to use tools that do not exist. This is a potential confound: the no-tool baseline may be penalized by the contradictory prompt, inflating the measured tool benefit. Please run the no-tool baseline with a prompt that is identical except for removal of the tool descriptions and workflow rules, or otherwise explicitly control for this mismatch.","section":"II.D / III.A"},{"comment":"The empirical evidence is drawn from a single 8-bus radial feeder with 150 questions and no variance estimates or significance tests. The paper itself acknowledges in the Conclusion that 'our experiments use only a single radial feeder modeled in balanced form.' The abstract and conclusion generalize beyond this to 'tasks that cannot be reliably solved using parametric knowledge alone.' This broad claim is not supported by one feeder configuration. Please either add multiple feeders/configurations and repeated runs with standard errors, or restrict the claim to the tested setup.","section":"IV, Table I; Conclusion"},{"comment":"The benchmark is described as deterministic, but answer choices are randomly permuted to mitigate position bias; no seed or fixed permutation list is reported. More importantly, the 33 templates are only summarized by difficulty tier, not fully specified. The reproducibility claim ('The benchmark is fully reproducible') is unverifiable from the manuscript without the complete template formulas, the exact solver version and options, and the numerical values of lambda_c and lambda_v. Please add a full template specification and exact generator seed to the appendix or repository.","section":"III"},{"comment":"The error decomposition in Table II reports no execution errors or timeouts and attributes all failures to 'wrong interpretation.' The decision rule for this classification is not given, and it presupposes that the solver and template formulas are correct; if a template formula is wrong or a solver run converges to an incorrect local solution, errors would be misclassified. Please define the classification protocol, release per-question traces/logs, and show how the categories are deterministically assigned.","section":"IV.B, Table II"}],"minor_comments":[{"comment":"The objective uses 'X_E' in the manuscript, which appears to be a typo for a sum over branches E; please define all symbols before use.","section":"Eq. (1)"},{"comment":"There is a typo: '50questions per livel' should be '50 questions per level.'","section":"III"},{"comment":"The sentence 'the model with the full accuracy scorers spends3-4calls' is ungrammatical; clarify which model achieves full accuracy and the associated tool-call count.","section":"IV.B"},{"comment":"The system prompt formatting is inconsistent (e.g., 'VeraGrid agent' vs 'VERAGRID-AGENT'); consider unifying the terminology.","section":"II.D"},{"comment":"The standalone LLM answer 'Ploss = 5 MW' is likely illustrative, but the figure should state that it is not taken from any evaluated model response.","section":"Fig. 1"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the oracle-validation problem: the benchmark's validity is assumed rather than demonstrated. I would not accept without a second-solver cross-check and a complete template audit. The paper also relies on several references with future arXiv identifiers and model versions; please verify all citations. Scope-wise, the paper is a good fit for the journal if the claims are calibrated to the single-feed setup."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"VeraGrid-Agent is a straightforward but well-executed demonstration that giving an LLM access to a power-flow solver fixes its inability to answer quantitative grid-edge questions. The genuinely new asset is VeraGrid-MCQ-150, a deterministic template-driven benchmark of 150 MCQs with answers computed from the VeraGrid solver output. The empirical result is clean: seven LLMs score 41–49% without tools and 97–100% when they can write input files, run the solver, and read the results. That gap is large enough that the lack of repeated runs or error bars doesn't undermine the central claim, though the near-perfect scores are statistically indistinguishable from each other at this sample size.\n\nWhat the paper does well: the design cleanly separates parametric reasoning from solver-grounded answers, the agent's answers are traceable to specific solver output lines, and the failure-mode analysis shows errors come from wrong interpretation of the output, not tool execution failures. The conclusion honestly acknowledges the single balanced 8-bus radial feeder as a limitation.\n\nThe soft spot that matters most is the oracle. All ground-truth labels come from the VeraGrid solver and 33 template formulas; neither is independently validated. The stress-test worry is that if the solver mis-solves the AC-OPF or a formula is wrong, the agent's high accuracy just measures agreement with a defective oracle. That is a real limitation, but it cuts less deeply than it might appear. The paper's claim is about tool access, not physical correctness: the no-tool baseline is scored against the same labels, so the central comparison still holds. Still, the phrase 'verified ground truth of the simulator' overstates things. Adding a validation case against MATPOWER or an analytical solution would materially strengthen the benchmark.\n\nThe agent architecture is ReAct with a solver tool — nothing new there — but the benchmark and the seven-model comparison are the contribution. The promise of a public GitHub repo helps reproducibility, though the exact prompts and full template list are not in the paper.\n\nThis is written for people working on LLM agents for power systems or tool-augmented reasoning more generally. It deserves a serious referee. My advice to the editor: send it out, and ask the authors for variance estimates, a solver validation check, and more transparency about the templates and prompts.","headline":"Practical, reproducible benchmark showing tool access fixes LLM quantitative gaps; the unvalidated solver oracle is the main caveat.","tokens_in":8543,"tokens_out":3927,"would_cite":true,"duration_ms":37827,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Giving an LLM a power-flow solver raises accuracy on grid-edge questions from around 45% to 97-100%.","keywords":["tool-augmented LLM","distribution optimal power flow","AC-OPF","grid edge","simulator grounding","multiple-choice benchmark","VeraGrid","DER dispatch"],"falsifier":"Take a small feeder where the true AC-OPF solution can be verified by independent means (e.g., brute-force search or a second, independently implemented solver), run the 150-question template, and check whether VeraGrid's output and the template formulas (such as Ploss = Pf + Pt) reproduce the independent ground truth; any mismatch in even one question would invalidate the accuracy numbers.","tokens_in":7739,"feed_emoji":"⚡","tokens_out":5329,"duration_ms":44505,"temperature":0.7,"pith_summary":"The paper argues that answering numerical questions about power distribution requires driving a deterministic power-flow solver, not relying on a language model's memory. The authors built VeraGrid-Agent, which writes the solver's input, executes it, and reads back only the relevant results before answering. On 150 expert-generated multiple-choice questions whose answers are computed by the simulator, every tested model scored 41–49% without tool access and 97–100% with the agent loop, including three perfect scores. The conclusion is that access to external computation, not parametric knowledge, is what lets LLMs solve these grid-edge tasks reliably.","feed_headline":"LLM plus solver scores 97-100% on grid-edge quiz","feed_subtitle":"Without tools, seven models all hover near chance on 150 power-flow questions; with them, near perfection.","key_machinery":"The control loop is the central mechanism: a language model issues read, write, and execute calls against an isolated workspace, writes the feeder and DER description in JSON, runs the VeraGrid solver, and selectively reads the results table of contents before answering. The benchmark is the other half: VeraGrid-MCQ-150, 150 multiple-choice questions generated by 33 expert templates (8 easy, 10 medium, 15 hard) whose ground truth is computed from the solver's record R via formulas such as Ploss = Pf + Pt, with distractors constructed from the same record.","core_discovery":"The central claim is that tool access, not model size or reasoning skill, is the decisive factor for numerically grounded power-flow questions. The paper demonstrates this by comparing the same models in two regimes: answering from the feeder description alone, and acting as VeraGrid-Agent, where the model writes the JSON input, executes the open-source VeraGrid AC-OPF solver, and reads back targeted result tables. Accuracy jumps from 42.7–49.3% to 97.3–100.0%. The few remaining errors trace to wrong interpretation of solver records during multi-step reasoning, not to failures of the solver or the tool calls. The authors frame the benchmark as memorization-resistant because every answer is a","pith_inferences":["The same tool-augmented pattern likely transfers to other non-convex engineering optimization problems with deterministic solvers (gas networks, water distribution), where the LLM's role becomes translating queries into solver calls.","A testable extension would inject a known bias into the solver's output and check whether agent answers shift accordingly, confirming the agent reads the solver rather than pattern-matching from memory.","The measured 17–27 seconds per question suggests a distillation target: training a small model on the agent's reasoning traces could produce a fast, tool-free approximate reasoner while retaining much of the accuracy.","If near-ceiling accuracy holds on harder feeders, the real discriminator between LLMs becomes the fidelity of natural-language-to-solver translation, for example correctly distinguishing 'curtailment' from 'dispatch'."],"forward_implications":["The paper's result implies that evaluations of LLMs on engineering questions must include a tool-use regime; scores without tools measure memorization, not computation.","The choice of LLM backbone matters far less than access to the solver: the best and worst models converge to near-identical accuracy once the solver is available.","The few failures are semantic misinterpretations of solver records, so improving the agent's query-to-result mapping, not solver accuracy, is the next bottleneck.","Because ground truth is tied to a specific solved network, the benchmark cannot be gamed by memorizing answers; changes to loads, impedances, or limits change the labels.","The claimed gains are established for a balanced radial feeder; the same architecture should extend to larger unbalanced three-phase systems, which the paper lists as future work."],"fun_headline_variants":["Tool-augmented LLM hits 97-100% on grid quiz, up from 49%","Without solvers, LLMs near chance on power-flow; with them, 97%+","Solver-powered LLM answers grid-edge D-OPF at up to 100% accuracy","Memory alone fails power-flow; tool use lifts LLMs to near-perfect"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The ground-truth answers are assumed to be correct because they come from the VeraGrid solver plus expert template formulas; if the solver mis-solves the non-convex AC-OPF or a template formula is wrong, the reported accuracy measures agreement with a flawed oracle.","fun_headline_variants_meta":{"raw":{"variants":["Tool-augmented LLM hits 97-100% on grid quiz, up from 49%","Without solvers, LLMs near chance on power-flow; with them, 97%+","Solver-powered LLM answers grid-edge D-OPF at up to 100% accuracy","Memory alone fails power-flow; tool use lifts LLMs to near-perfect"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000835,"raw_usage":{"total_tokens":3482,"prompt_tokens":752,"completion_tokens":2730,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":496,"completion_tokens_details":{"reasoning_tokens":2633}},"tokens_in":496,"tokens_out":2730,"duration_ms":17788,"temperature":1.0,"reasoning_tokens":2633,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T03:14:55.029014+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a small feeder where the true AC-OPF solution can be verified by independent means (e.g., brute-force search or a second, independently implemented solver), run the 150-question template, and check whether VeraGrid's output and the template formulas (such as Ploss = Pf + Pt) reproduce the independent ground truth; any mismatch in even one question would invalidate the accuracy numbers.","supporting_citations":[],"review_version":1}