{"id":"32d6bd62-7134-4405-a380-a71111b1f7dc","arxiv_id":"2608.07925","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An agent that executes its code in a sandbox and explores undocumented APIs offline reaches 78.5% Pass@1 on a new 158-task PyAether benchmark, 43 points above static retrieval.","lead":"ZhuLong is an AI programming agent that writes and runs scripts in a safe copy of chip design software, testing and fixing code until it works, and it solves 78.5% of 158 real tasks. The paper's main value is showing that letting the agent actually execute its code, not just read documentation, is what drives accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own Table 3 contradicts the 'sandbox execution is the dominant driver' claim: removing retrieval costs 44.3 pp while removing sandbox costs 41.2 pp, so the central attribution is not supported.","rationale":"I read the paper in good faith and identify the central claim as the attribution of the 78.5% Pass@1 result primarily to sandbox execution over static knowledge, as stated in the Abstract and Section 6.2.2 and echoed in the reader's strongest_claim. The most load-bearing concern is internal: the symmetric ablation in Table 3 shows removing retrieval (44.3 pp drop) hurts slightly more than removing sandbox (41.2 pp drop), so the paper's own evidence does not support calling sandbox the 'dominant performance driver.' The progressive build-up numbers (RAG +8.7 pp, then sandbox +43.0 pp) are order-dependent and do not resolve the attribution. This is not a speculative worry; it is a direct quantitative inconsistency in the reported table, and it affects the paper's main takeaway. The self-exploration contribution of 3.2 pp is also within sampling noise for 158 tasks, compounding the concern. The reader's weakest_assumption focused on benchmark fidelity and leakage, which is a real but separate issue; the reader did note the lack of variance, which overlaps with my statistical point. Because the core result (that the system works and both execution and retrieval are necessary) is solid, but the headline attribution is overstated, the appropriate verdict is CONDITIONAL: the paper should be accepted only after the attribution claims are revised to match the ablations and variance information is supplied. The benchmark-release concern remains valid but is secondary to the internal inconsistency I highlight.","tokens_in":10349,"tokens_out":6919,"duration_ms":75258,"concrete_test":"Re-analyze Table 3 with a symmetric two-proportion test on the 158-task counts: Full (124 passes), w/o Retrieval (54), w/o Sandbox (59). If the difference between the 44.3 pp and 41.2 pp removal deltas is not significant (p>0.05), the correct conclusion is that retrieval and sandbox are equally indispensable, and the paper should not call sandbox 'dominant.' Then run the full system 5-10 times with different random seeds to estimate the variance of the 3.2 pp self-exploration gain; if the 95% confidence interval crosses zero, drop the accuracy claim for self-exploration and retain only the tool-call efficiency claim. Finally, add a configuration with sandbox execution but without search_apis/get_api_details to directly test whether execution alone, without static knowledge, can achieve nontrivial Pass@1.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central scientific message is that execution grounding, not static knowledge, is the decisive factor for EDA scripting (Abstract; §6.2.2; and the reader's strongest_claim). The supporting evidence is Table 3's symmetric ablation: removing sandbox from ZhuLong-Full drops Pass@1 from 78.5% to 37.3% (Δ-41.2 pp), whereas removing retrieval drops it to 34.2% (Δ-44.3 pp). Since the retrieval-removal drop is larger, Table 3 actually shows retrieval is at least as important as sandbox in the full system. The progressive build-up (Pure LLM→RAG +8.7 pp; RAG→+sandbox +43.0 pp) is order-dependent: the 43 pp attributed to sandbox is the marginal gain of adding sandbox after retrieval is already present, not the unique contribution of execution. No configuration with sandbox but no static documentation is reported, so 'execution, not static knowledge' is not directly testable from the provided ablations. In addition, the 3.2 pp self-exploration gain corresponds to roughly 5 tasks out of 158, which is within one standard error (~3.3 pp) of zero, and no repeated runs or variance estimates are reported; the accuracy claim for self-exploration is statistically fragile, although the 22.1% tool-call reduction is more suggestive. The conclusion that execution grounding is 'essential' is well supported, but the stronger 'dominant' / 'not static knowledge' attribution is contradicted or at least not established by the paper's own numbers.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ZhuLong, an LLM-based coding agent for EDA scripting in PyAether and SKILL. It combines three MCP tools (API search, documentation lookup, and sandbox execution) with an offline 'self-exploration' mechanism that probes undocumented APIs via counterfactual experiments. The agent is evaluated on EDA-Eval-PyAether, a new benchmark of 158 assertion-based tasks in the commercial Empyrean Aether environment. The headline results are 78.5% Pass@1 for the full system versus 23.6% for a pure LLM baseline and 32.3% for RAG; ablations attribute 43.0 pp to sandbox execution and 3.2 pp to self-exploration, which also reduces tool calls by 22.1% per trace. A smaller interactive evaluation reports 60.0% and 50.0% Pass@1 for PyAether and SKILL on 20 tasks.","tokens_in":10671,"tokens_out":6699,"duration_ms":71349,"significance":"If the results hold, the paper would make a useful contribution: it is one of the first execution-grounded agents for commercial EDA scripting, provides a concrete assertion-based benchmark for PyAether, and includes a detailed error analysis with trace-level case studies. The efficiency gain in tool calls and the offline exploration mechanism are interesting. However, the central attribution claim is not supported by the paper's own ablations, and the lack of a released benchmark and of variance estimates weakens the quantitative conclusions. The core system is sound and the issues are fixable, so the paper merits revision rather than rejection.","major_comments":[{"comment":"The claim that sandbox execution is the 'dominant performance driver' is not supported by the component ablation. In the bottom block of Table 3, removing retrieval from the full system drops Pass@1 from 78.5% to 34.2% (Δ −44.3 pp), which is larger than the −41.2 pp drop caused by removing sandbox execution. The progressive build-up in the top block is also order-dependent: +43.0 pp is the marginal gain of adding sandbox after retrieval is already present, not an attribution of the unique contribution of execution. The paper should either report a symmetric attribution (e.g., a 2x2 design with and without sandbox and retrieval, or Shapley values) or soften the claim to 'both retrieval and execution are essential, with execution providing the larger marginal gain when added to a retrieval-based system.'","section":"§6.2.2, Table 3, Abstract"},{"comment":"The claimed +3.2 pp accuracy gain from self-exploration is statistically fragile. With 158 tasks, this corresponds to about 5 tasks changing outcome (124 vs 119 passes), and the standard error of the difference under the null is roughly 3.3–4.9 pp depending on the approximation. No confidence intervals, repeated runs, or paired significance tests are reported, so the difference is within one standard error. The paper should report variance estimates or a McNemar test over the same 158 tasks, or present the efficiency gain (22.1% reduction in tool calls per trace) as the primary evidence for self-exploration.","section":"§6.2.3, Table 3"},{"comment":"All headline numbers depend on EDA-Eval-PyAether, but the benchmark is not released and its construction shares the same API documentation set that the offline self-exploration mechanism enriches. Tasks are drawn from API references, documentation examples, internal training materials, and anonymized CAD cases, with only manual expert review for validation. This creates a risk of distribution overlap between the exploration corpus and the test tasks, which would inflate the self-exploration gain. The authors should release the benchmark with a clear statement of task sources, provide a leakage analysis (e.g., hold-out tasks or a temporal split), and ideally report results on an independently collected test set.","section":"§5.1, §4"},{"comment":"The default re-planning budget of 2 rounds is selected after observing the ablation on the same benchmark (0 rounds: 62.7%, 1 round: 76.0%, 2 rounds: 78.5%). This makes the headline Pass@1 a tuned configuration rather than a pre-registered protocol. The paper should report the sensitivity of the main comparison (e.g., Pure LLM vs ZhuLong) under each re-planning budget, or justify the budget a priori, to avoid overfitting the evaluation set.","section":"§6.1.2, §6.2.1"}],"minor_comments":[{"comment":"The contribution list calls EDA-Eval-PyAether the 'first public benchmark,' but Section 8 states code and benchmark will be released upon acceptance; at submission the benchmark is not public. Please rephrase.","section":"§1, §8"},{"comment":"Table 2 does not specify what the 'Pure LLM' baseline can do; clarify whether it has access only to the prompt and describe its generation protocol (temperature, sampling).","section":"Table 2"},{"comment":"The metric called Pass@1 is defined as the proportion of tasks whose generated program passes all assertions, not the sampling-based Pass@1 of Chen et al.; using a different name or citing the original definition would avoid confusion.","section":"§5.2"},{"comment":"The statement that the 5.0 pp gain of self-exploration without sandbox 'suggests that sandbox execution partially substitutes' for exploration is speculative, since these numbers come from different configurations and baselines; it should be flagged as a hypothesis or tested directly.","section":"§6.2.3"},{"comment":"Table 8 reports results on 20 interactive tasks with no confidence intervals; the 10 pp PyAether/SKILL difference is not statistically substantiated and should be presented with appropriate uncertainty or softened.","section":"Table 8"}],"recommendation":"major_revision","confidential_remarks":"The system and benchmark are valuable, but the central narrative that execution grounding outperforms static knowledge is contradicted by the paper's own Table 3, where removing retrieval causes a larger drop than removing sandbox. This is fixable by re-framing the claim and adding the missing ablation (sandbox-only vs retrieval-only) rather than by collecting new data. I would also make benchmark release a condition of acceptance and ask for statistical rigor on the 3.2 pp self-exploration claim before the paper is published."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The paper is a solid systems contribution for LLM-based EDA scripting. What's new: a new benchmark (EDA-Eval-PyAether, 158 tasks with assertion-based execution), the first execution-grounded agent for PyAether and SKILL, and an offline API self-exploration mechanism that probes undocumented APIs. The headline result is credible: adding sandbox execution to a RAG baseline lifts Pass@1 from 32.3% to 75.3%, and the full system reaches 78.5%. The evaluation design is honest in the important way: correctness is judged by execution in a real EDA environment, not by human inspection. The error analysis is concrete; the case studies (e.g., task 098, where a plain Python list fails to populate a hidden C++ selection state) are genuinely instructive. The 22.1% reduction in tool calls per trace from self-exploration is a nice efficiency result, independent of the small accuracy gain.\n\nSoft spots: The paper's central attribution — that sandbox execution is the dominant driver, not static knowledge — is not supported by its own Table 3. Removing retrieval from the full system costs 44.3 points; removing sandbox costs 41.2 points. The progressive build-up (+8.7, +43.0, +3.2) is order-dependent and cannot separate the two components, because no configuration runs sandbox without retrieval. The defensible claim is that both are necessary and the complete loop works; the 'execution, not static knowledge' framing is an overreach.\n\nAlso, all Pass@1 numbers are single-run point estimates with no variance. The 3.2 pp self-exploration gain is about 5 tasks out of 158, within one standard error of zero; I would treat the accuracy contribution as unproven, though the tool-call reduction is suggestive. The benchmark is not yet released, and the two-round retry budget is tuned on the reported evaluation, a minor circularity. The interactive evaluation is a pilot (20 tasks, no retries) — fine as a secondary result.\n\nWho should read this: people building coding agents for tool-specific APIs and anyone in the EDA-ML community who wants a benchmark for scripting. It deserves a serious peer review, but the authors need to correct the attribution claim, add variance or repeated runs, and release the benchmark and code. If they do, this could become a standard reference for PyAether scripting evaluation.","headline":"Solid systems paper with a useful benchmark and a clear execution-grounded recipe, but the headline attribution of sandbox over retrieval is not supported by the paper's own Table 3.","tokens_in":11256,"tokens_out":2956,"would_cite":true,"duration_ms":43458,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Running LLM code in a sandbox, not static knowledge, makes EDA scripting work: 78.5% Pass@1 vs 23.6% for a plain LLM.","keywords":["LLM Agents for EDA","EDA Scripting","Code Generation","Sandbox Execution","API Self-Exploration","execution feedback","PyAether","Retrieval-Augmented Generation"],"falsifier":"An independent group could take the released benchmark and agent, write their own assertion oracles from step-by-step design outcomes rather than the provided ones, and rerun the 158 tasks; if Pass@1 falls far below 78.5%, the headline overstates execution grounding and partly measures oracle familiarity. A cheaper check: re-run the self-exploration ablation with exploration applied only to APIs never mentioned in the 158 prompts and see if the +3.2 pp survives.","tokens_in":10153,"feed_emoji":"⚙️","tokens_out":7008,"duration_ms":70927,"temperature":0.7,"pith_summary":"ZhuLong is an LLM coding agent for EDA scripting that does not stop at retrieving documentation: it runs candidate code in a sandboxed commercial EDA environment, observes errors and side effects, and revises. On a new 158-task benchmark of real PyAether scripting jobs, the full agent passes 78.5% of tasks on the first attempt, versus 23.6% for a plain LLM and 32.3% for static retrieval-augmented generation. The ablation numbers are the paper's central claim: adding sandbox execution to RAG raises accuracy by 43.0 percentage points, while an offline self-exploration mechanism that probes undocumented APIs adds another 3.2 points and cuts tool calls per task by 22.1%. The authors conclude that for long-tail, poorly documented tool APIs, execution grounding is not an optional extra but the main source of working code.","feed_headline":"Sandbox execution beats static RAG for EDA scripts by 43 points","feed_subtitle":"An LLM agent that runs code in the real tool and iterates on errors passes 78.5% of 158 real tasks.","key_machinery":"The load-bearing mechanism is the closed loop formed by three unified tool calls: search_apis retrieves candidate APIs by embedding similarity; get_api_details returns documentation augmented offline; and run_code executes candidate code in the EDA sandbox, returning outputs, errors, and state changes. Around this loop sits an offline API self-exploration agent that, before runtime, generates minimal counterfactual tests for undocumented parameter ranges, return fields, and error semantics, executes them in the sandbox, and writes the inferred constraints back into the API knowledge base. The argument runs on that loop: accuracy jumps when run_code is added, and the self-exploration enrichment shows up as fewer run_code calls per successful trace.","core_discovery":"The paper claims that an LLM agent for EDA scripting succeeds to the degree that it executes candidate code inside the actual tool environment and iterates on what it observes. On EDA-Eval-PyAether, 158 real-world PyAether tasks scored by assertion-based execution, ZhuLong reaches 78.5% Pass@1; a plain LLM reaches 23.6%, retrieval-augmented generation reaches 32.3%, and the same agent with sandbox execution but no self-exploration reaches 75.3%. The attribution the authors draw is that sandbox execution supplies 43.0 of the 54.9 percentage-point gain over the pure LLM, while offline API self-exploration, which probes undocumented API behavior through counterfactual sandbox experiments and stores the results as enriched documentation, adds a further 3.2 points and reduces tool calls per task by 22.1%. The paper concludes that for long-tail, incompletely documented tool APIs, execution grounding is essential rather than optional.","pith_inferences":["If the offline self-exploration cost is amortized over many queries to the same API, the +3.2 pp and 22.1% tool-call reduction may understate its value in high-reuse production settings; the paper measures one benchmark pass, not a long-lived deployment.","Because 61.4% of benchmark tasks come from API references the agent can index, absolute Pass@1 on tasks invented independently of the documentation would likely be lower; the 78.5% plausibly upper-bounds what this architecture delivers on genuinely unseen tooling.","The recurring failure mode of hidden C++ binding-level types (such as Task 098's plain Python list versus the internal selection state) suggests that surfacing type constraints, not more prose documentation, is the most promising next lever.","Interactive single-attempt results (60% PyAether, 50% SKILL) indicate that without the retry loop execution grounding alone is much weaker; supporting safe session rollback could make interactive gains approach the sandbox gains."],"forward_implications":["Adding sandbox execution to static RAG raises first-attempt accuracy by 43.0 percentage points, meaning most of the system's value comes from seeing real errors and side effects rather than from knowing more API names.","One re-planning round captures 84.2% of the two-round improvement, so even a small execution-feedback budget buys most of the recoverable correctness.","Offline API self-exploration adds only 3.2 points of accuracy but cuts run_code calls by 27.0%, so the main near-term payoff of pre-exploration is efficiency, not correctness.","Vector-based API retrieval outweighs keyword search by 12.7 points, and indexing names plus descriptions beats either alone, so retrieval design still matters inside an execution-grounded agent.","With retrieval, execution, and self-exploration fixed, the choice of LLM backbone changes results from 55.7% to 83.5%, so the underlying model remains a decisive factor."],"supporting_citations":[{"why":"Defines the Pass@1 execution metric used to score benchmark tasks.","marker":"[6]"},{"why":"Supplies the underlying autonomous coding-agent runtime that provides the iterative plan-act-observe loop.","marker":"[7]"},{"why":"Supplies the embedding model used for semantic search over API names and descriptions.","marker":"[4]"},{"why":"Supplies the similarity-search library used to index and retrieve APIs.","marker":"[8]"},{"why":"Defines the unified tool protocol through which the agent calls retrieval, documentation, and execution.","marker":"[1]"},{"why":"Is the retrieval-augmented baseline the paper compares against and the representative static-knowledge approach it extends.","marker":"[17]"},{"why":"Supplies prior evidence that EDA tool feedback improves LLM code generation, motivating the execution loop.","marker":"[3]"}],"fun_headline_variants":["Executing code in real tool lifts EDA agent from 23.6% to 78.5%","EDA agents need execution: 78.5% vs 23.6% Pass@1","Sandbox execution adds 43 points to EDA scripting agent","LLM agent passes 78.5% of real EDA tasks by running code","Offline API self-exploration adds 3.2 points, cuts tool calls 22%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole 78.5% figure rests on the 158 benchmark tasks and their assertion oracles being faithful, non-leaking representations of real PyAether scripting work, since the tasks were built partly from the same API documentation the agent retrieves.","fun_headline_variants_meta":{"raw":{"variants":["Executing code in real tool lifts EDA agent from 23.6% to 78.5%","EDA agents need execution: 78.5% vs 23.6% Pass@1","Sandbox execution adds 43 points to EDA scripting agent","LLM agent passes 78.5% of real EDA tasks by running code","Offline API self-exploration adds 3.2 points, cuts tool calls 22%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000836,"raw_usage":{"total_tokens":3663,"prompt_tokens":975,"completion_tokens":2688,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":2573}},"tokens_in":591,"tokens_out":2688,"duration_ms":21296,"temperature":1.0,"reasoning_tokens":2573,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:40:09.110008+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An independent group could take the released benchmark and agent, write their own assertion oracles from step-by-step design outcomes rather than the provided ones, and rerun the 158 tasks; if Pass@1 falls far below 78.5%, the headline overstates execution grounding and partly measures oracle familiarity. A cheaper check: re-run the self-exploration ablation with exploration applied only to APIs never mentioned in the 158 prompts and see if the +3.2 pp survives.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the underlying autonomous coding-agent runtime that provides the iterative plan-act-observe loop."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the unified tool protocol through which the agent calls retrieval, documentation, and execution."}],"review_version":1}