REVIEW 4 major objections 5 minor 25 references
ZhuLong: Execution-Grounded LLM Agent for EDA Scripting with Offline API Self-Exploration
T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Running LLM code in a sandbox, not static knowledge, makes EDA scripting work: 78.5% Pass@1 vs 23.6% for a plain LLM.
desk verdict Solid systems paper with a useful benchmark and a clear execution-grounded recipe, but the headline attribution of sandbox over retrieval is not supported by the paper's own Table 3. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the closed loop formed by three unified tool calls: search_apis retrieves candidate APIs by embedding similarity; get_api_details returns documentation augmented offline; and run_code executes candidate code in the EDA sandbox, returning outputs, errors, and state changes. Around this loop sits an offline API self-exploration agent that, before runtime, generates minimal counterfactual tests for undocumented parameter ranges, return fields, and error semantics, executes them in the sandbox, and writes the inferred constraints back into the API knowledge base. The argument runs on that loop: accuracy jumps when run_code is added, and the self-exploration enrichment shows up as fewer run_code calls per successful trace.
What would settle it
An independent group could take the released benchmark and agent, write their own assertion oracles from step-by-step design outcomes rather than the provided ones, and rerun the 158 tasks; if Pass@1 falls far below 78.5%, the headline overstates execution grounding and partly measures oracle familiarity. A cheaper check: re-run the self-exploration ablation with exploration applied only to APIs never mentioned in the 158 prompts and see if the +3.2 pp survives.
Extended reading notes
Core claim
The paper claims that an LLM agent for EDA scripting succeeds to the degree that it executes candidate code inside the actual tool environment and iterates on what it observes. On EDA-Eval-PyAether, 158 real-world PyAether tasks scored by assertion-based execution, ZhuLong reaches 78.5% Pass@1; a plain LLM reaches 23.6%, retrieval-augmented generation reaches 32.3%, and the same agent with sandbox execution but no self-exploration reaches 75.3%. The attribution the authors draw is that sandbox execution supplies 43.0 of the 54.9 percentage-point gain over the pure LLM, while offline API self-exploration, which probes undocumented API behavior through counterfactual sandbox experiments and stores the results as enriched documentation, adds a further 3.2 points and reduces tool calls per task by 22.1%. The paper concludes that for long-tail, incompletely documented tool APIs, execution grounding is essential rather than optional.
Load-bearing premise
The whole 78.5% figure rests on the 158 benchmark tasks and their assertion oracles being faithful, non-leaking representations of real PyAether scripting work, since the tasks were built partly from the same API documentation the agent retrieves.
Editorial extensions
If this is right
- Adding sandbox execution to static RAG raises first-attempt accuracy by 43.0 percentage points, meaning most of the system's value comes from seeing real errors and side effects rather than from knowing more API names.
- One re-planning round captures 84.2% of the two-round improvement, so even a small execution-feedback budget buys most of the recoverable correctness.
- Offline API self-exploration adds only 3.2 points of accuracy but cuts run_code calls by 27.0%, so the main near-term payoff of pre-exploration is efficiency, not correctness.
- Vector-based API retrieval outweighs keyword search by 12.7 points, and indexing names plus descriptions beats either alone, so retrieval design still matters inside an execution-grounded agent.
- With retrieval, execution, and self-exploration fixed, the choice of LLM backbone changes results from 55.7% to 83.5%, so the underlying model remains a decisive factor.
Reading between the lines
- If the offline self-exploration cost is amortized over many queries to the same API, the +3.2 pp and 22.1% tool-call reduction may understate its value in high-reuse production settings; the paper measures one benchmark pass, not a long-lived deployment.
- Because 61.4% of benchmark tasks come from API references the agent can index, absolute Pass@1 on tasks invented independently of the documentation would likely be lower; the 78.5% plausibly upper-bounds what this architecture delivers on genuinely unseen tooling.
- The recurring failure mode of hidden C++ binding-level types (such as Task 098's plain Python list versus the internal selection state) suggests that surfacing type constraints, not more prose documentation, is the most promising next lever.
- Interactive single-attempt results (60% PyAether, 50% SKILL) indicate that without the retry loop execution grounding alone is much weaker; supporting safe session rollback could make interactive gains approach the sandbox gains.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes ZhuLong, an LLM-based coding agent for EDA scripting in PyAether and SKILL. It combines three MCP tools (API search, documentation lookup, and sandbox execution) with an offline 'self-exploration' mechanism that probes undocumented APIs via counterfactual experiments. The agent is evaluated on EDA-Eval-PyAether, a new benchmark of 158 assertion-based tasks in the commercial Empyrean Aether environment. The headline results are 78.5% Pass@1 for the full system versus 23.6% for a pure LLM baseline and 32.3% for RAG; ablations attribute 43.0 pp to sandbox execution and 3.2 pp to self-exploration, which also reduces tool calls by 22.1% per trace. A smaller interactive evaluation reports 60.0% and 50.0% Pass@1 for PyAether and SKILL on 20 tasks.
Significance. If the results hold, the paper would make a useful contribution: it is one of the first execution-grounded agents for commercial EDA scripting, provides a concrete assertion-based benchmark for PyAether, and includes a detailed error analysis with trace-level case studies. The efficiency gain in tool calls and the offline exploration mechanism are interesting. However, the central attribution claim is not supported by the paper's own ablations, and the lack of a released benchmark and of variance estimates weakens the quantitative conclusions. The core system is sound and the issues are fixable, so the paper merits revision rather than rejection.
major comments (4)
- [§6.2.2, Table 3, Abstract] The claim that sandbox execution is the 'dominant performance driver' is not supported by the component ablation. In the bottom block of Table 3, removing retrieval from the full system drops Pass@1 from 78.5% to 34.2% (Δ −44.3 pp), which is larger than the −41.2 pp drop caused by removing sandbox execution. The progressive build-up in the top block is also order-dependent: +43.0 pp is the marginal gain of adding sandbox after retrieval is already present, not an attribution of the unique contribution of execution. The paper should either report a symmetric attribution (e.g., a 2x2 design with and without sandbox and retrieval, or Shapley values) or soften the claim to 'both retrieval and execution are essential, with execution providing the larger marginal gain when added to a retrieval-based system.'
- [§6.2.3, Table 3] The claimed +3.2 pp accuracy gain from self-exploration is statistically fragile. With 158 tasks, this corresponds to about 5 tasks changing outcome (124 vs 119 passes), and the standard error of the difference under the null is roughly 3.3–4.9 pp depending on the approximation. No confidence intervals, repeated runs, or paired significance tests are reported, so the difference is within one standard error. The paper should report variance estimates or a McNemar test over the same 158 tasks, or present the efficiency gain (22.1% reduction in tool calls per trace) as the primary evidence for self-exploration.
- [§5.1, §4] All headline numbers depend on EDA-Eval-PyAether, but the benchmark is not released and its construction shares the same API documentation set that the offline self-exploration mechanism enriches. Tasks are drawn from API references, documentation examples, internal training materials, and anonymized CAD cases, with only manual expert review for validation. This creates a risk of distribution overlap between the exploration corpus and the test tasks, which would inflate the self-exploration gain. The authors should release the benchmark with a clear statement of task sources, provide a leakage analysis (e.g., hold-out tasks or a temporal split), and ideally report results on an independently collected test set.
- [§6.1.2, §6.2.1] The default re-planning budget of 2 rounds is selected after observing the ablation on the same benchmark (0 rounds: 62.7%, 1 round: 76.0%, 2 rounds: 78.5%). This makes the headline Pass@1 a tuned configuration rather than a pre-registered protocol. The paper should report the sensitivity of the main comparison (e.g., Pure LLM vs ZhuLong) under each re-planning budget, or justify the budget a priori, to avoid overfitting the evaluation set.
minor comments (5)
- [§1, §8] The contribution list calls EDA-Eval-PyAether the 'first public benchmark,' but Section 8 states code and benchmark will be released upon acceptance; at submission the benchmark is not public. Please rephrase.
- [Table 2] Table 2 does not specify what the 'Pure LLM' baseline can do; clarify whether it has access only to the prompt and describe its generation protocol (temperature, sampling).
- [§5.2] The metric called Pass@1 is defined as the proportion of tasks whose generated program passes all assertions, not the sampling-based Pass@1 of Chen et al.; using a different name or citing the original definition would avoid confusion.
- [§6.2.3] The statement that the 5.0 pp gain of self-exploration without sandbox 'suggests that sandbox execution partially substitutes' for exploration is speculative, since these numbers come from different configurations and baselines; it should be flagged as a hypothesis or tested directly.
- [Table 8] Table 8 reports results on 20 interactive tasks with no confidence intervals; the 10 pp PyAether/SKILL difference is not statistically substantiated and should be presented with appropriate uncertainty or softened.
Circularity Check
No significant circularity: the system's reported gains are empirical ablation measurements, not derivations from their own inputs.
full rationale
The paper's central claims are empirical: a sandbox-execution agent, offline API self-exploration, and a benchmark. No fitted parameter is renamed as a prediction, no equation is defined in terms of the quantity it is supposed to explain, and no load-bearing conclusion rests on a self-citation chain or on an imported uniqueness theorem. The +43.0 pp sandbox gain and +3.2 pp self-exploration gain are measured ablations on an internally constructed benchmark, not values recovered from the inputs by construction. The benchmark's task sources (API references, documentation examples, internal materials, CAD cases) overlap with the API knowledge base that self-exploration enriches, which raises a legitimate benchmark-leakage/validity risk, but that is an evaluation-contamination concern rather than a circular reduction: the tasks still require assertion-checked side effects, and the measured gains could in principle have been zero or negative. The reviewer-flagged inconsistency that removing retrieval costs 44.3 pp while removing sandbox costs 41.2 pp undermines the paper's 'dominant driver' attribution, but that is an internal-evidence/interpretation problem, not a circularity. I therefore find no circular step meeting the required evidentiary bar.
Assumptions & free parameters
free parameters (4)
- Re-planning rounds n =
2
- Retrieval top-K =
5
- Execution timeout =
2500 seconds
- Primary LLM backbone =
DeepSeek-V4-Flash
assumptions (3)
- domain assumption The 158 EDA-Eval-PyAether tasks and their assertions are faithful, representative real-world scripting tasks.
- domain assumption The lightweight sandbox execution mode faithfully reflects the behavior of the commercial interactive EDA environment.
- domain assumption Discovered API behaviors from offline exploration remain valid at evaluation time.
Cite this review
Pith. "Pith review of ZhuLong: Execution-Grounded LLM Agent for EDA Scripting with Offline API Self-Exploration." pith.science (2026). https://pith.science/paper/XT4EADNU
@misc{pith2026260807925,
author = {Pith},
title = {Pith review of: ZhuLong: Execution-Grounded LLM Agent for EDA Scripting with Offline API Self-Exploration},
year = {2026},
howpublished = {\url{https://pith.science/paper/XT4EADNU}},
note = {Machine review of arXiv:2608.07925}
}
read the original abstract
EDA scripting with tool-specific, often undocumented APIs remains a long-tail bottleneck that existing LLMs fail to address. This paper presents ZhuLong, an execution-grounded LLM coding agent for PyAether and SKILL that combines API retrieval, documentation inspection, and sandbox execution via unified MCP tools, augmented by an offline API self-exploration mechanism that infers undocumented API behaviors through counterfactual experimentation. We evaluate ZhuLong on EDA-Eval-PyAether, a benchmark of 158 real-world tasks with assertion-based execution, where the complete system achieves 78.5% Pass@1 in the commercial Empyrean Aether environment, substantially outperforming a pure LLM baseline (23.6%). Ablation studies identify sandbox execution as the dominant performance driver (41.2 pp drop when removed), with the self-exploration mechanism contributing an additional 3.2 pp accuracy gain and a 22.1% reduction in per-task tool calls. On 20 interactive tasks involving unsaved layouts and schematics, ZhuLong achieves 60.0% Pass@1 for PyAether and 50.0% for SKILL.
Figures
Reference graph
Works this paper leans on
-
[1]
Anthropic. 2024. Model Context Protocol (MCP). https://modelcontextprotocol. io/
work page 2024
-
[2]
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program Synthesis with Large Language Models.arXiv preprint arXiv:2108.07732(2021)
arXiv 2021
-
[3]
Jason Blocklove, Siddharth Garg, Ramesh Karri, and Hammond Pearce. 2024. Can EDA Tool Feedback Improve Verilog Generation by LLMs?arXiv preprint arXiv:2411.11856(2024)
arXiv 2024
-
[4]
Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu
-
[5]
Lei Chen, Yiqi Chen, Zhufei Chu, Wenji Fang, Tsung-Yi Ho, Yu Huang, Sadaf Khan, Min Li, Xingquan Li, Yun Liang, Yibo Lin, Jinwei Liu, Yi Liu, Guojie Luo, Zhengyuan Shi, Guangyu Sun, Dimitrios Tsaras, Runsheng Wang, Ziyi Wang, Xinming Wei, Zhiyao Xie, Qiang Xu, Chenhao Xue, Evangeline F Y Young, Bei Yu, Minjie Yuan, Haoyi Zhang, Zuodong Zhang, Yuxiang Zhao...
arXiv 2024
-
[6]
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating Large Language Models Trained on Code.arXiv preprint arXiv:2107.03374(2021)
arXiv 2021
-
[7]
Cline Contributors. 2024. Cline: Autonomous AI Coding Assistant. https://github. com/cline/cline
work page 2024
-
[8]
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. 2024. The Faiss library. (2024). arXiv:2401.08281 [cs.LG]
arXiv 2024
Show all 25 references
-
[9]
Empyrean Technology. [n. d.]. Empyrean PyAether: Full Custom Design Platform Ecosystem. https://empyrean.com.cn/solutions/index-128.html. Accessed: 2026- 06-28
2026
-
[10]
Xue Jiang, Yihong Dong, Zhi Jin, and Ge Li. 2024. SEED: Customize Large Language Models with Sample-Efficient Adaptation for Code Generation.arXiv preprint arXiv:2403.00046(2024)
2024
-
[11]
Rajesh Kumar et al . 2026. AgentForge: Execution-Grounded Multi-Agent LLM Framework for Autonomous Software Engineering.arXiv preprint arXiv:2604.13120(2026). https://arxiv.org/abs/2604.13120
2026 arXiv
-
[12]
Pan, Tian Lan, and Weidong Cao
Qiufeng Li, Rongqian Chen, Quan Cheng, Chengxuan Wang, Sizhe Tang, Wuxi Li, Duo Ding, Chia-Tung Ho, Haoxing Ren, David Z. Pan, Tian Lan, and Weidong Cao. 2026. PDAGENT-BENCH: Characterizing, Grounding, and Architecting LLM Agents for VLSI Physical Design.arXiv preprint arXiv:2...
2026 arXiv
-
[13]
Bingyang Liu, Haoyi Zhang, Xiaohan Gao, Zichen Kong, Xiyuan Tang, Yibo Lin, Runsheng Wang, and Ru Huang. 2025. LayoutCopilot: An LLM-Powered Multiagent Collaborative Framework for Interactive Analog Layout Design.IEEE Transactions on Computer-Aided Design of Integrated Circuit...
2025
-
[14]
Shang Liu, Wenji Fang, Yao Lu, Qijun Zhang, Hongce Zhang, and Zhiyao Xie
-
[15]
Yiyi Lu, Hoi Ian Au, Junyao Zhang, Jingyu Pan, Yiting Wang, Ang Li, Jianyi Zhang, and Yiran Chen. 2025. AutoEDA: Enabling EDA Flow Automation through Microservice-Based LLM Agents.arXiv preprint arXiv:2508.01012(2025). Uses MCP protocol for EDA automation. https://arxiv.org/ab...
2025
-
[16]
2023.GPT-4 Technical Report
OpenAI. 2023.GPT-4 Technical Report. Technical Report arXiv:2303.08774. Ope- nAI
2023 arXiv
-
[17]
Yuan Pu, Zhuolun He, Tairu Qiu, Haoyuan Wu, and Bei Yu. 2024. Customized Re- trieval Augmented Generation and Benchmarking for EDA Tool Documentation QA. InProceedings of the IEEE/ACM International Conference on Computer-Aided Design (ICCAD). arXiv:2407.15353 [cs.CL]
2024 arXiv
-
[18]
Yibo Rui, Yuanhang Li, Rui Wang, Ruiqi Chen, Yanxiang Zhu, Zhixiong Di, Xi Wang, and Ming Ling. 2025. ChaTCL: LLM-Based Multi-Agent RAG Framework for TCL Script Generation. InProceedings of the International Symposium of Electronics Design Automation (ISEDA). 736–742
2025
-
[19]
2024.Virtuoso Studio SKILL Language Reference
Cadence Design Systems. 2024.Virtuoso Studio SKILL Language Reference. https: //www.cadence.com
2024
-
[20]
Shailja Thakur, Baleegh Ahmad, Hammond A Pearce, Benjamin Tan, Brendan Dolan-Gavitt, Ramesh Karri, and Siddharth Garg. 2023. VeriGen: A Large Lan- guage Model for Verilog Code Generation.ACM Transactions on Design Automa- tion of Electronic Systems29, 3 (2023), 1–31
2023
-
[21]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All You Need.Advances in Neural Information Processing Systems30 (2017)
2017
-
[22]
Ning Xu, Zhaoyang Zhang, Senlin Shu, Qi Lei, Jiaqi Lv, Wensuo Wang, Tianhao Zhao, Chao Zhang, Zhaoliang Yang, Xiangyu Li, Zhaorui Su, Jingshan Li, and Xin Geng. 2026. iScript: A Domain-Adapted Large Language Model and Benchmark for Physical Design Tcl Script Generation.arXiv p...
2026
-
[23]
An Yang et al. 2025. Qwen3 Technical Report. arXiv:2505.09388 [cs.CL] https: //arxiv.org/abs/2505.09388
2025 arXiv
-
[2024]
In2024 IEEE/ACM International Conference on Computer Aided Design (ICCAD)
RTLCoder: Outperforming GPT-3.5 in Design RTL Generation with Our Open-Source Dataset and Lightweight Solution. In2024 IEEE/ACM International Conference on Computer Aided Design (ICCAD). Also available at arXiv:2312.08617. https://arxiv.org/abs/2312.08617
-
[2025]
arXiv:2402.03216 [cs.CL] https://arxiv.org/abs/2402.03216
M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. arXiv:2402.03216 [cs.CL] https://arxiv.org/abs/2402.03216
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.