{"id":"52db6201-1184-4ff0-bf5d-3d8036325c43","arxiv_id":"2411.16707","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A feedback-driven multi-agent framework with enhanced retrieval and reasoning lets GPT-4o complete power system simulation tasks with roughly 93 to 97 percent success on the authors' benchmark.","lead":"This paper builds a multi-agent system that helps large language models run power system simulations by retrieving tool documentation, reasoning step-by-step, and correcting errors from the simulator. On 69 self-designed tasks across two power system tools, the system reaches about 93 to 97 percent success, far above plain ChatGPT or fine-tuned models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline success rates are single-run point estimates on 34–35 tasks; with LLM sampling stochasticity, a few flipped task outcomes could move the reported 96.85% by several points, so the quantitative claim needs repetition or confidence bounds.","rationale":"I read the paper as making a quantitative claim: the proposed framework reaches 93.13% and 96.85% on the authors' 69-task benchmark. The design of the framework is coherent and the ablations are thoughtful, and the authors are transparent about components (Remark 2, Remark 4, and the admissions about non-execution-bug failures). The most fragile part is the absence of any estimate of run-to-run variability. LLM-based code generation is stochastic; the benchmark is small; the scoring rule assigns a 100/50/0 per attempt; the paper itself reports the framework sometimes fails on non-execution-bug outputs. One or two non-reproducible successes would move the headline by about 3 points. The reader's verdict already flags the lack of error bars, but the formal weakest_assumption field focuses on the functions/options schema. I do not dispute that schema concern; it is real for adaptability. However, the strongest claim is about the success rates, and the most direct way to falsify or secure that claim is to check whether the success rates reproduce across runs. If they do, the quantitative claim is robust and the CONDITIONAL verdict could move toward ACCEPT (subject to artifact release). If they do not, the central claim should be rephrased as a range or with a variance estimate. I therefore recommend keeping CONDITIONAL but adding repetition as a condition, or setting verdict to CONDITIONAL with an explicit repeated-measures requirement. I set agreement_with_reader to disagree because my selected load-bearing concern is not the reader's weakest_assumption, even though the reader's rationale mentions the absence of error bars.","tokens_in":20373,"tokens_out":11341,"duration_ms":106561,"concrete_test":"Rerun the GPT4o-Full configuration (same prompts, same knowledge base, same temperature=0.1, same API version) on exactly the 69 tasks from Section V, at least 5 independent runs, varying only the request nonce/seed. Report the mean, standard deviation, and min–max of the aggregate success rate and of the first-attempt success rate, per environment. If the 95% confidence interval half-width is at least 3 percentage points in either environment, the headline should be reported as a range, and the CONDITIONAL verdict should require a repeated-measures evaluation before acceptance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is quantitative: 93.13% and 96.85% success on 34 and 35 tasks. The paper reports one run per scheme (temperature 0.1) and no variance, seeds, or confidence intervals. The per-task scoring is effectively ternary (0/50/100), so a single task flipping from 100 to 0 changes the DALINE aggregate by roughly 2.9 percentage points and the MATPOWER aggregate by roughly 2.9 points. GPT-4o API responses are stochastic and model versions change; the main results (Figs. 8 and 12) use gpt-4o-2024-05-13, while the SFT comparison uses gpt-4o-2024-08-06. If 2 of 35 MATPOWER tasks fail on a rerun, 96.85% becomes about 91%, and the abstract's near-perfect claim is materially weaker. This is not an internal inconsistency, but it means the headline number is not established; it is a point estimate from a single sample. The reader's rationale notes 'no error bars,' but the weakest assumption in the verdict concerns the functions/options schema; I regard the missing repeated-run quantification as the more load-bearing issue because it bears directly on the numerical claim.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a feedback-driven multi-agent framework to help LLMs write executable power-system simulation code. The framework combines an enhanced RAG module with adaptive query planning and a triple-based knowledge base, a few-shot chain-of-thought reasoning module, and an environmental acting module that executes code and iteratively corrects it from error messages. The authors evaluate the framework on 34 DALINE tasks and 35 MATPOWER tasks, reporting success rates of 93.13% and 96.85% for GPT4o-Full, compared with much lower rates for ChatGPT-4o, o1-preview, standard RAG variants, and a supervised fine-tuned model. They also report per-task execution time of about 30 seconds and token cost of about 0.014 USD.","tokens_in":20587,"tokens_out":4831,"duration_ms":46502,"significance":"If the reported success rates are reliable, the framework would be a practically useful contribution: it demonstrates that a modular combination of retrieval, structured reasoning, and error feedback can enable a general LLM to operate both a newly released tool (DALINE) and a well-established tool (MATPOWER) at near-perfect accuracy on a diverse task suite. The evaluation is grounded in actual execution in the simulators rather than static code inspection, which is a genuine strength, as are the explicit ablations across framework components and the cost analysis. The promise to release prompts, knowledge bases, tasks, and generated code further supports reproducibility if fulfilled.","major_comments":[{"comment":"The reported \"success rate\" is not a conventional success rate. Because unused attempts are assigned the last executed score, a task solved on the first attempt contributes 100×N_max^t points, whereas the same task solved only on the last attempt contributes only 100 points. With N_max=3 for DALINE and N_max=5 for MATPOWER, early success is weighted 3× and 5× more heavily than late success. Thus the headline \"93.13%\" and \"96.85%\" values conflate correctness with iteration efficiency and are not directly comparable across the two environments. Please report the per-task binary success rate (solved at least once within the attempt budget) as a primary metric, or explicitly rename the current measure an \"attempt-weighted score\" and justify its use for the abstract's claim.","section":"Section V.A, Eqs. (1)-(3)"},{"comment":"All reported success rates are single-run point estimates with no repeated runs, no random seeds, and no confidence intervals. Since per-attempt scores are essentially ternary (0/50/100) and the task counts are only 34 and 35, one task flipping from 100 to 0 changes the aggregate by roughly 2.9 percentage points, and two flips would move the MATPOWER rate from 96.85% below 92%. LLM API outputs are stochastic even at temperature 0.1, and the paper itself shows model-version sensitivity (main results use gpt-4o-2024-05-13, while Section V.D uses gpt-4o-2024-08-06 with a different DALINE success rate for the same scheme). The near-perfect quantitative claim in the abstract is therefore not yet established. Please provide multiple independent runs or bootstrap confidence intervals, and include per-task score tables so readers can recompute the aggregates.","section":"Section V.A, Figs. 8 and 12"},{"comment":"The DALINE SFT comparison uses the full testing dataset as the SFT training dataset (\"all 34 testing simulation tasks, augmented with 16 additional tasks, were employed for fine-tuning\"). This means the SFT model is not being evaluated on unseen tasks. The experiment may show that SFT cannot memorize the training set, but it does not support the broader claim that SFT underperforms the framework on equal terms. In addition, these experiments use gpt-4o-2024-08-06 for all schemes, while the main DALINE results in Figs. 7–9 use gpt-4o-2024-05-13, and Table III reports GPT4o-Full at 95.098% versus 93.13% in Fig. 8. The version sensitivity should be acknowledged and bounded.","section":"Section V.D, DALINE experiments"},{"comment":"The paper states that \"the scoring was conducted manually by human experts during the evaluation process,\" but no inter-annotator agreement, blinded evaluation, or detailed scoring protocol is reported. Because the task set, knowledge bases, prompts, and rubric were all designed by the same group, the quantitative results should be accompanied by the raw per-task score sheets and the generated code and benchmark outputs for independent verification; ideally a second annotator should score a subset of tasks.","section":"Section V.A, scoring"}],"minor_comments":[{"comment":"The name of the DALINE toolbox is inconsistently rendered as \"DALINE\", \"Daline\", and \"D ALINE\" in footnotes, captions, and text; please standardize.","section":"Throughout"},{"comment":"The notation \"CGPT4o R\" is not expanded in the table or its caption; please define it explicitly (presumably ChatGPT with OpenAI's built-in RAG).","section":"Table I"},{"comment":"The scoring equations use N_max^(t) before it is formally introduced; please define all notation before the equations.","section":"Section V.A"},{"comment":"Several hrefs contain unresolved \"[ here]\" placeholders (e.g., the DALINE/MATPOWER links in Section V and the embedding-model link in Table II); provide the actual URLs.","section":"Footnotes and Table II"},{"comment":"Reference [18] duplicates reference [10]; please merge or renumber.","section":"References"},{"comment":"The list of key findings includes two items labeled \"(iv)\"; renumber the items sequentially.","section":"Conclusion"},{"comment":"Table IV shows per-task expenses of 0.014 USD (DALINE) and 0.013 USD (MATPOWER), while the abstract says \"an average cost of 0.014 USD\"; clarify the rounding and averaging method.","section":"Table IV and abstract"}],"recommendation":"major_revision","confidential_remarks":"I would ask the editor to require the authors to deposit the complete per-task scores, generated code, prompts, and knowledge bases as part of the revision, rather than only \"upon acceptance\". The quantitative claims depend on single-run manual scoring, so independent verification of the raw results is essential."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The framework is real engineering—the results come from executing code in actual simulators—and the paper is more honest than most. The genuinely new bit is the adaptive query planning that splits a simulation request into function and option sub-queries, plus the triple-based option document that encodes option-function dependencies. The combination with few-shot CoT and an error-feedback loop is well described, and the ablation is unusually systematic: thirteen schemes, first/final attempt breakdowns, an extended 50-attempt budget, and an SFT comparison. I thought the SFT diagnostic was actually clever—the authors fine-tune on the test tasks and still lose to retrieval, which supports their 'retrieval beats memorization for precise API details' argument. The paper also flags its own limits: Remark 2 admits the whole approach assumes documentation exists, Remark 4 discusses weak error-reporting tools, and the conclusion acknowledges non-execution-bug failures.\n\nThe soft spots are the usual ones for this genre, and one is load-bearing. The headline numbers—93.13% and 96.85%—are single-run point estimates on 34 and 35 tasks, with no variance or confidence intervals. The stress-test note is right: at those task counts a couple of flipped outcomes move the aggregate by several points. I agree that the missing repeated-run quantification is more load-bearing than the functions/options schema worry. Given GPT-4o stochasticity, the exact figures in the abstract should not be treated as established until repeated runs or bounds are reported. Second, the evaluation is self-referential in a way that matters: DALINE is the authors' own toolbox, and the tasks, knowledge documents, prompts, and scoring all come from the same group. The 'unseen tool' claim is about LLM pretraining, not evaluator neutrality. That doesn't invalidate the result, but it does mean the adaptability claim rests on two tools, one of which is in-house. Third, the scoring is manual by the authors, with no inter-rater reliability check. Minor: no code or data released yet, and [10] and [18] are the same paper.\n\nNet: this is a solid engineering contribution for LLM-based scientific computing and power-system assistants. The central argument holds qualitatively; I just wouldn't defend the precise success rates yet. Send it to serious peer review—it deserves referee time—but ask for repeated runs, confidence bounds, and open artifacts. With those, this is a conditional accept.","headline":"A well-engineered, unusually honest agentic-RAG paper whose headline success rates are single-run point estimates on a self-built benchmark; worth serious review.","tokens_in":21146,"tokens_out":5840,"would_cite":true,"duration_ms":50406,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A feedback-driven multi-agent framework lets GPT-4o write executable power-system simulations with 93–97% success on 69 tasks, up from below 30% for baselines.","keywords":["power system simulation","large language models","multi-agent framework","retrieval-augmented generation","chain-of-thought prompting","error feedback","MATPOWER","DALINE"],"falsifier":"Apply the same framework to a simulation tool whose interface is not reducible to a function–option list (for example, a GUI-only or equation-heavy tool) or whose error reporting is vague; if success does not fall far below the reported 93–97%, the paper's claim that documentation structure and error-reporting quality are load-bearing would be called into question. A simpler check: rerun GPT4o-Full on the same 69 tasks with the triple-based document replaced by the raw manual while keeping all other modules, and compare against the reported 81.37% (DALINE) figure for the scheme missing that document.","tokens_in":20107,"feed_emoji":"⚡","tokens_out":5584,"duration_ms":48783,"temperature":0.7,"pith_summary":"Power-system simulations are a bottleneck for LLM-based research assistants because simulation code demands precise function names, option values, and dependencies that models rarely memorize. This paper argues that the fix is not more model parameters or fine-tuning but a feedback-driven multi-agent framework: an enhanced retrieval module that decomposes a natural-language simulation request into function-related and option-related sub-queries, a reasoning module that uses few-shot chain-of-thought and retrieved knowledge to write code, and an acting module that executes the code, reads error messages, and loops corrections back into retrieval and reasoning. On 69 tasks across DALINE (a tool the model has never seen) and MATPOWER (a familiar tool), the framework reports success rates of 93.13% and 96.85%, respectively, while ChatGPT 4o, o1-preview, and fine-tuned GPT-4o all stay below 30% on complex tasks. If the result holds, researchers could delegate simulation implementation to natural-language assistants at roughly 30 seconds and 0.014 USD per task.","feed_headline":"Multi-agent loop lifts LLM simulation success to 96.85%","feed_subtitle":"Feedback-driven retrieval and reasoning let GPT-4o run power-system sims in about 30 seconds for 1.4 cents each.","key_machinery":"The load-bearing object is the triple-based structured option document, built by parsing a tool's manual with an LLM and expert review, in which each entry links an option's name/default format to its dependent function and to its description and value choices. Around this document the framework runs three interacting agents: a retrieval agent that splits a request into function- and option-related sub-queries via few-shot chain-of-thought; a coding agent prompted with static tool knowledge, retrieved details, and a four-step reasoning path; and an environmental acting module that executes code, detects errors, and feeds structured error reports back into retrieval and reasoning. The loop continues until the code runs cleanly or the attempt budget is exhausted.","core_discovery":"The paper's central claim is that the key to LLM-based simulation is correctly identifying and combining functions and options, and that this can be achieved by structured retrieval plus iterative execution feedback. Standard RAG, which retrieves on the whole request, conflates function and option information; supervised fine-tuning compresses away the fine-grained parameter details coding requires. The proposed framework instead splits each request into function-related and option-related sub-queries, maps them to precise keywords, and retrieves from a triple-based option document that records each option's name and default format, the function it depends on, and its description and value choices. The coding agent then follows a universal four-step reasoning path—identify functions, learn syntax, extract option information, generate code—and the environmental acting module runs the code and feeds error reports back into the loop. The reported outcome is that GPT-4o achieves 93.13% success on DALINE and 96.85% on MATPOWER, with complex and standard tasks nearly closing their performance gap.","pith_inferences":["The same function/option decomposition could transfer to other engineering simulation ecosystems, such as circuit or chemical-process simulators, where manuals likewise define functions and options.","A testable extension is to add an LLM-based code-checking agent that inspects successfully executed code for wrong-but-runnable outputs, addressing the non-execution-bug failures the paper flags as open.","The framework's success suggests a general recipe for code generation on specialized tools: retrieve structured relational facts about API elements rather than raw manual prose.","Coupling the feedback loop with an interactive clarification stage could extend the framework to underspecified or ambiguous requests, which the paper identifies as a remaining challenge."],"forward_implications":["A natural-language interface to power-system simulation is feasible now: a researcher can describe a task and receive executable code plus a simulation result.","The success depends on the cumulative combination of query planning, the triple-based knowledge base, few-shot chain-of-thought, and feedback; dropping any one of these visibly lowers success rates.","Supervised fine-tuning is not a substitute: even when trained on the full test set, the fine-tuned model stays below 60%, whereas the retrieval-based framework exceeds 95%.","Simulation tools with weak error reporting will need layered checking or external code-extraction strategies, because the feedback loop's power depends on error-message quality.","Because the function/option decomposition is tool-independent, the framework can be ported to newly released simulation tools without retraining."],"supporting_citations":[{"why":"DALINE toolbox: the unseen simulation environment used to test adaptability to a tool released after the LLM's training cutoff.","marker":"[23]"},{"why":"MATPOWER: the established, pre-training-visible toolbox whose 35 tasks test performance on familiar tools.","marker":"[24]"},{"why":"MATPOWER 8.0 user's manual: the source document parsed into the triple-based option knowledge base.","marker":"[26]"},{"why":"DALINE user manual: supplies the function and option descriptions used to construct the retrieval repository for the unseen tool.","marker":"[25]"},{"why":"Original retrieval-augmented generation formulation, the standard RAG baseline the enhanced RAG module is designed to beat.","marker":"[19]"},{"why":"Chain-of-thought prompting, the technique underlying the few-shot CoT prompts in both retrieval and reasoning modules.","marker":"[21]"},{"why":"Few-shot learning results that justify the few-shot prompting approach used in the retrieval and coding prompts.","marker":"[22]"},{"why":"Prior evidence that LLMs struggle to create OpenDSS distribution grids and simple power-flow code, motivating the simulation-specific framework.","marker":"[6]"},{"why":"Finetuned language models are zero-shot learners, the basis for the supervised fine-tuning comparison scheme.","marker":"[27]"}],"fun_headline_variants":["Multi-agent feedback lifts LLM sim success to 96.85%","LLM power sims: 96.85% success with feedback loop","RAG + feedback: LLM masters power simulation tasks","LLM hits 96.85% on power sims via multi-agent loop","Error-feedback framework makes LLMs power-sim pros"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes that a simulation tool's usability can be captured by a structured document of functions and options derived from its manual, and that the tool returns error messages detailed enough for the feedback loop to correct mistakes.","fun_headline_variants_meta":{"raw":{"variants":["Multi-agent feedback lifts LLM sim success to 96.85%","LLM power sims: 96.85% success with feedback loop","RAG + feedback: LLM masters power simulation tasks","LLM hits 96.85% on power sims via multi-agent loop","Error-feedback framework makes LLMs power-sim pros"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000533,"raw_usage":{"total_tokens":2588,"prompt_tokens":996,"completion_tokens":1592,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":612,"completion_tokens_details":{"reasoning_tokens":1499}},"tokens_in":612,"tokens_out":1592,"duration_ms":12374,"temperature":1.0,"reasoning_tokens":1499,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:09:56.137042+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Apply the same framework to a simulation tool whose interface is not reducible to a function–option list (for example, a GUI-only or equation-heavy tool) or whose error reporting is vague; if success does not fall far below the reported 93–97%, the paper's claim that documentation structure and error-reporting quality are load-bearing would be called into question. A simpler check: rerun GPT4o-Full on the same 69 tasks with the triple-based document replaced by the raw manual while keeping all other modules, and compare against the reported 81.37% (DALINE) figure for the scheme missing that document.","supporting_citations":[{"cited_title":"Daline: A data-driven power flow linearization toolbox for power systems research and education,","cited_arxiv_id":null,"evidence_quote":"DALINE toolbox: the unseen simulation environment used to test adaptability to a tool released after the LLM's training cutoff."},{"cited_title":"Matpower: Steady-state operations, planning, and analysis tools for power systems research and education,","cited_arxiv_id":null,"evidence_quote":"MATPOWER: the established, pre-training-visible toolbox whose 35 tasks test performance on familiar tools."},{"cited_title":"Matpower 8.0 user’s manual,","cited_arxiv_id":null,"evidence_quote":"MATPOWER 8.0 user's manual: the source document parsed into the triple-based option knowledge base."},{"cited_title":"User manual for daline 1.1.5,","cited_arxiv_id":null,"evidence_quote":"DALINE user manual: supplies the function and option descriptions used to construct the retrieval repository for the unseen tool."},{"cited_title":"Retrieval-augmented generation for knowledge-intensive NLP tasks,","cited_arxiv_id":null,"evidence_quote":"Original retrieval-augmented generation formulation, the standard RAG baseline the enhanced RAG module is designed to beat."},{"cited_title":"Language models are few-shot learners,","cited_arxiv_id":null,"evidence_quote":"Few-shot learning results that justify the few-shot prompting approach used in the retrieval and coding prompts."},{"cited_title":"Finetuned language models are zero-shot learners,","cited_arxiv_id":null,"evidence_quote":"Finetuned language models are zero-shot learners, the basis for the supervised fine-tuning comparison scheme."}],"review_version":1}