Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Can AI Agents Really Complete RTL-to-GDS? Lessons from Benchmarking Tool-Interactive EDA Workflows

T0 review · 3 major / 4 minor · reviewed 2026-08-01 · deepseek-v4-flash

Pith's one-line read This paper claims that current AI agents can complete an RTL-to-GDS chip-design flow, but that reliable completion is determined by the execution layer controlling tool interfaces and preserving design context, not by foundation-model capab

desk verdict A solid empirical case study showing execution infrastructure beats model choice for reliable RTL-to-GDS, but the headline numbers (141× Token ROI gap, score rankings) are fragile due to single-run evidence. read the letter →

arxiv 2607.17528 v3 pith:77AQ2MJW submitted 2026-07-20 cs.AI cs.ARcs.LG

classification cs.AIcs.ARcs.LG
keywords AgenticEDARTL-to-GDSLLMagentstool-interactiveworkflowsTokenROITclcommandcompatibilityexecutionarchitecturephysicaldesignautomation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether current AI agents can reliably take a chip design from register-transfer level (RTL) to final layout (GDS) using commercial EDA tools, a long, feedback-driven process involving synthesis, placement, clock-tree synthesis, routing, and engineering change orders. It argues that agents can complete such a flow, but that reliable completion is a system property rather than a property of the foundation model: domain-specific skills alone give inconsistent gains, while a structured execution layer that validates tool calls, keeps persistent design state, and returns normalized feedback completed every stage across all four tested models and both timing targets. The paper also shows that similar design progress can cost very different resources, with up to a 141× difference in Token ROI, and that the largest source of physical-design failures is low-level Tcl command incompatibility with the tool version or execution mode. A sympathetic reader would care because the result shifts the bottleneck in Agentic EDA from model intelligence to execution infrastructure and process-level evaluation.

What carries the argument

The central object carrying the argument is the execution harness (called FluxEDA in the paper), which separates agent reasoning from tool execution: it registers EDA capabilities as structured actions, maintains persistent tool sessions and design state, and returns normalized results, thereby removing whole classes of syntactically plausible but operationally invalid commands. The measurement machinery is a gated end-to-end score, S = 0.2·S_stage + 0.8·S_result, where stage completion is gated sequentially so that partial outputs earn no result credit, paired with Token ROI, defined as normalized EDA return divided by token investment times execution-time investment. Together they let the

What would settle it

Repeat each architecture–model pair at least five times under the same timing targets. If a command-line-style agent completes ECO in some repeated runs or the structured harness fails at any stage, the reported 8/8 versus 3/8 stage-completion gap and the architecture rankings become sampling artifacts rather than stable properties. Alternatively, pin the commercial tool version and expose a validated interface to the command-line agents; if the Tcl-compatibility error category remains large, the failure is not caused by interface structure.

Watch

Extended reading notes

Core claim

The paper's central discovery is that end-to-end RTL-to-GDS completion is achievable by current LLM agents, and the decisive factor is the execution layer, not the model. In the reported experiments, every run backed by the structured execution harness completed synthesis, physical implementation, and ECO under both a loose (350 MHz) and a tight (700 MHz) timing target, scoring above 90 on a 0–100 gated scale, while command-line-style agents and their skill-augmented variants completed ECO in at most 3 of 8 runs. The paper attributes this gap to three causes: skills improve local procedural knowledge but do not stabilize long-horizon behavior; efficiency varies by up to 141× even between run

Load-bearing premise

The load-bearing premise is that one run per configuration represents an agent's typical behavior; LLM agents are stochastic, so a single run may misrepresent true performance, and the reported score and stage-completion differences—such as one model scoring 13.33 at 350 MHz and 82.33 at 700 MHz under the same architecture—could be partly sampling noise.

Editorial extensions

If this is right

  • If the central claim is correct, benchmarks for Agentic EDA should report stage completion, runtime, token cost, and Token ROI together with final design quality, because raw cost or runtime alone can reward early termination.
  • Domain-specific skills should be treated as local enhancements, not reliability mechanisms; their effect is model- and target-dependent and can even reduce performance.
  • Commercial EDA toolchains should expose version- and mode-aware validated interfaces rather than raw Tcl shells, which would eliminate the largest observed category of physical-design errors.
  • Agent capability should be evaluated as a complete system—architecture plus model—across multiple constraints, since the same model's score varied by 86 points across architectures in one condition.
  • Efficiency comparisons should be conditioned on verified progress; two runs reaching the same milestone can differ by roughly 141× in Token ROI, so resource use alone is not a valid efficiency signal.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: The 141× Token ROI gap suggests the economics of Agentic EDA may depend more on the execution harness than on model choice, and that cost-efficient deployment could favor structured orchestration even when two systems produce identical design files.
  • Inference: The likely generalization is that any long-horizon, tool-interactive engineering workflow—beyond chip design—will exhibit the same pattern: hidden operational state and interface versioning, not reasoning ability per se, are what make agents fail.
  • Inference: Because the paper ran each configuration once, run-to-run stochasticity is an unresolved confound; repeating runs would likely show that command-line agents occasionally complete ECO and structured harnesses occasionally fail, narrowing the reported architecture gap.
  • Inference: A testable extension would be to add checkpoint and rollback mechanisms to the command-line-style agents while keeping the same tools; if the Tcl-compatibility error category persists, the paper's conclusion about execution support is strengthened, while if it disappears, the failure is better attributed to lack of recovery rather than interface structure.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper reports an empirical benchmark of LLM agents on a PicoRV32 RTL-to-GDS flow using commercial synthesis and physical-design tools. Three agent architectures (Claude Code, Claude Code + EDA Skills, FluxEDA) are crossed with four foundation models under two clock targets. The agents are scored by a gated end-to-end score, stage completion, runtime/cost, and a newly defined Token ROI. The paper's main lessons are that EDA skills do not guarantee flow completion, that execution architecture rather than model alone determines reliability (all FluxEDA runs score above 90), and that Tcl-level interface mismatches dominate physical-design failures. It concludes that robust Agentic EDA requires structured interfaces, persistent context, and process-level evaluation.

Significance. The paper addresses a genuinely important and underexplored problem: evaluating agents on long-horizon, tool-interactive chip-design workflows rather than on RTL generation alone. The gated scoring and the explicit error taxonomy are useful contributions, and the qualitative conclusion that execution architecture and tool-interface control matter is plausible and supported by a large, consistent gap in the reported data. The paper also has a reproducible-by-construction flavor (single design, fixed tools), and it honestly lists its limitations. However, the quantitative claims—model/architecture rankings, the 141x Token ROI gap, and stage-completion rates—rest on one run per configuration and on metrics whose normalization is not fully specified. These deficiencies make the current numerical lessons provisional.

major comments (3)
  1. [§6 and Table 2] The paper explicitly states 'one run per configuration.' For stochastic LLM agents, the observed score differences are not attributable to architecture/model/constraint effects. For example, Kimi K3 under Claude Code scores 13.33 at 350 MHz but 82.33 at 700 MHz, and Claude Sonnet 5 under CC+EDA Skills drops from 87.05 to 11.67. These swings are as large as the architecture effects claimed. The 'all FluxEDA runs above 90' and the 141.2x Token ROI comparison (12.71 vs 0.09) are also computed from single trials. Please provide repeated runs (e.g., at least 5 seeds per configuration) with mean, standard deviation, and per-run scores, and a paired statistical comparison of architectures.
  2. [§2.3, Eq. (1); §3.3, Eq. (4)] The scoring and Token ROI definitions are underspecified. The paper says area and power are 'normalized against frequency-specific reference values' and timing metrics are 'scaled according to predefined lower bounds,' but it never gives these reference values, lower bounds, or the rescale factors used for normalized Token ROI. The exact formula in Eq. (1) is also ambiguous (denominator product? separate penalties?). Without these numbers, the reported scores and ratios cannot be reconstructed or compared across runs; this is load-bearing because the 141x gap and architecture rankings are quantitative conclusions.
  3. [§3.2, §4.3, §4.4] FluxEDA, the winning architecture, is the authors' own system (ref. [3]) and Token ROI is also introduced by the authors, but this is not stated in the main text. The paper should disclose this explicitly and provide an independent or more controlled comparison (e.g., same baseline harness with only the execution layer changed, public artifact) so readers can separate the 'architecture' effect from implementation-specific engineering effort. The error-category percentages in Figure 3 also need denominators and a coding protocol; without these, 31.7% cannot be interpreted.
minor comments (4)
  1. [Table 2] Formatting issue: '0.11' and other cost entries are missing spacing/alignment; also, the table title says 'ROI denotes the normalized Token ROI' but the metric was introduced as 'Token ROI' without normalization until §2.3. Clarify.
  2. [§4.1] Figure 2 is not explicitly discussed in the text; add a sentence connecting the figure to the cost-runtime-quality trade-off.
  3. [§3.3] Equation (4) uses set M with subscripts W_s, T_s, W_h, T_h, but the paper never defines WNS/TNS thresholds or how hold versus setup metrics are combined; a worked example of one score would help.
  4. [§2.4] The phrase 'The present invited paper' is odd in a formal journal submission; either remove it or state the venue explicitly.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the conclusions are empirical measurements, not derivatives of the evaluation metric; the only self-referential element is the authors' own FluxEDA treatment arm, which is not load-bearing.

full rationale

The paper's central claims are empirical measurements. The end-to-end score is explicitly defined in Eqs. (2)-(4) with stage gating, and Token ROI is defined in Eq. (1) as normalized return divided by token and time investment. The qualitative lessons—skills do not guarantee completion, execution architecture correlates with reliable completion, and Tcl-version/interface mismatches dominate physical-design failures—follow from the observed data in Table 2, Table 3, and Figure 3, not from a parameter fitted to the outcome. The only self-referential element is reference [3] (FluxEDA, by overlapping authors), used to describe the structured-execution treatment arm; the evidence for that arm's performance is the experiment reported in this paper rather than the citation itself, so the self-citation is not load-bearing. The paper explicitly admits in Section 6: 'This study is limited to one design, one technology setup, selected commercial-tool versions, and one run per configuration.' This single-run limitation undermines the statistical robustness of quantitative claims such as the 141× Token ROI gap, but that is a reproducibility/validity concern, not a definitional reduction. Eq. (1)'s normalization is underspecified, which also affects interpretability, but nothing in the paper predicts a quantity that was used to construct it. No step reduces by construction to its own inputs, so no significant circularity is present.

Assumptions & free parameters 5 free parameters · 3 assumptions · 1 invented entities

This is an empirical benchmarking paper, so its 'parameters' are evaluation constants and normalization choices rather than fitted physical quantities. The central quantitative results — the score spread, the 141× Token ROI difference, and the 31.7% error share — all depend on these unspecified or hand-chosen evaluation knobs, and the underlying experiment is a single run per cell.

free parameters (5)
  • Stage/result score weights (0.2/0.8) = 0.2 / 0.8
    Eq. (2): S = 0.2*S_stage + 0.8*S_result. The weights are chosen by the authors; no sensitivity analysis is shown, and the weighting determines how much a fully completed run dominates a run that fails ECO.
  • Area/power normalization reference values = unspecified
    §3.3: area and power are 'normalized against frequency-specific reference values'; these references are never given, so the result score cannot be recomputed.
  • Timing score lower bounds = unspecified
    §3.3: negative timing values 'are scaled according to predefined lower bounds'; the bounds are not stated, making the timing sub-scores non-reproducible.
  • Token ROI rescale factors = unspecified
    Eq. (1) and §3.3: token cost and execution time are 'rescaled to comparable ranges' before computing normalized Token ROI. The rescaling is unspecified, and the headline 141× difference depends directly on these scales.
  • Error-category taxonomy = 31.7% Tcl compatibility
    Figure 3: observed events are classified into ad hoc categories (Tcl compatibility, MMMC, etc.) without formal definitions; the 31.7% share and the '0.0%' for FluxEDA depend on this unpublished classification.
assumptions (3)
  • domain assumption A single run per configuration is representative of agent performance
    The entire comparison uses one run per architecture–model–target cell; §6 states 'one run per configuration'. Since LLM agents are stochastic, score differences could be sampling noise.
  • domain assumption The commercial tool setup and PicoRV32 are representative of RTL-to-GDS difficulty
    §3.1 selects one design and one commercial 55-nm setup; the paper acknowledges in §6 that it 'does not capture the scale and signoff complexity of industrial SoCs'.
  • domain assumption The architecture comparison is a fair contest between equally tuned systems
    FluxEDA was built and presumably tuned by these authors, while the Claude Code baselines are used off-the-shelf. If the baselines received less prompt/skill engineering, the gap is partly a tuning artifact.
invented entities (1)
  • Token ROI
    purpose: A cost-efficiency metric relating design quality to token and runtime investment
    Introduced in this paper (Eq. 1) with unspecified normalization; it is a measurement construct, not an independently validated physical quantity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can AI Agents Really Complete RTL-to-GDS? Lessons from Benchmarking Tool-Interactive EDA Workflows." pith.science (2026). https://pith.science/paper/77AQ2MJW

@misc{pith2026260717528,
  author       = {Pith},
  title        = {Pith review of: Can AI Agents Really Complete RTL-to-GDS? Lessons from Benchmarking Tool-Interactive EDA Workflows},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/77AQ2MJW}},
  note         = {Machine review of arXiv:2607.17528}
}
read the original abstract

Large language model (LLM) agents are extending electronic design automation (EDA) beyond static RTL generation toward long-horizon, tool-interactive workflows. Yet it remains unclear whether general-purpose coding agents, even with domain-specific EDA skills, can reliably execute an end-to-end RTL-to-GDS flow encompassing synthesis, physical implementation, and engineering change order (ECO) optimization. We evaluate AI agents on a PicoRV32 RTL-to-GDS flow using commercial EDA tools under two timing targets. Their performance is assessed using end-to-end design score, stage completion, and Token ROI, a cost-efficiency metric relating design quality to runtime and cost. Comparing three agent architectures and four foundation models, we derive three practical lessons. First, domain-specific skills improve agents' understanding of individual subtasks but do not ensure reliable completion of a long-horizon EDA flow. Second, agents that achieve similar design progress can still differ by up to 141 times in Token ROI, revealing substantial differences in runtime and cost efficiency. Third, low-level tool-interface mismatches are a major source of physical design failures, particularly when Tcl commands depend on the tool version or execution mode. These results suggest that robust Agentic EDA requires not only stronger models but also structured tool interfaces, persistent design context, controlled execution, and process-level evaluation.

Figures

Figures reproduced from arXiv: 2607.17528 by the authors.

Figure 1
Figure 1. PicoRV32 RTL-to-GDS evaluation flow. The agent must advance a single design through synthesis, physical imple [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Performance–runtime–cost trade-off across agent [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Distribution of physical-implementation error cate [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. AgenticECO: An Agentic Framework for ECO on 3D Integrated Circuits

    cs.AI 2026-08 conditional novelty 6.0 of 10

    An agentic framework with a minimal-disturbance router and independent verifier clears post-route hybrid-bond spacing defects in 3D-IC designs (7/9 on one backbone, 9/9 on another) with low disturbance and zero clock-...

Reference graph

Works this paper leans on

17 extracted references · 3 linked inside Pith · cited by 1 Pith paper

  1. [3]

    Zhengrui Chen, Zixuan Song, Yu Li, Qi Sun, and Cheng Zhuo. 2026. FluxEDA: A Unified Execution Infrastructure for Stateful Agentic EDA.arXiv preprint arXiv:2603.25243(2026)

  2. [1]

    Anthropic. 2026. Claude Code Documentation. https://docs.anthropic.com/en/ docs/claude-code/overview. Accessed: 2026-06-27

  3. [2]

    Anthropic. 2026. Introducing Claude Sonnet 5. https://www.anthropic.com/ news/claude-sonnet-5. Accessed: 2026-07-09

  4. [4]

    Anyi DeepSeek-AI, Xu, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chenchen Ling, et al

  5. [5]

    ICkylin Team. 2026. IC Agent Hub: IC-Oriented Agent Skills by ICkylin. https:// www.icagenthub.com/skills. Accessed: 2026-07-09. IC-oriented skill marketplace developed by Zhongke Qixin / ICkylin; homepage: https://www.ickylin.com/

  6. [6]

    Pengwei Jin, Di Huang, Chongxiao Li, Shuyao Cheng, Yang Zhao, Xinyao Zheng, Jiaguo Zhu, Shuyi Xing, Bohan Dou, Rui Zhang, et al. 2025. RealBench: Bench- marking verilog generation models with real-world ip designs.arXiv preprint arXiv:2507.16200(2025)

  7. [7]

    Mingjie Liu, Nathaniel Pinckney, Brucek Khailany, and Haoxing Ren. 2023. Ver- ilogEval: Evaluating large language models for verilog code generation. In2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD). IEEE, 1–8

  8. [8]

    Yiyi Lu, Hoi Ian Au, Junyao Zhang, Jingyu Pan, Guanglei Zhou, Yiting Wang, Jingwei Sun, Ang Li, Jianyi Zhang, Hai Li, et al . 2025. AutoEDA: Enabling EDA flow automation through microservice-based LLM agents.arXiv preprint arXiv:2508.01012(2025)

Show all 17 references
  1. [9]

    Yao Lu, Shang Liu, Qijun Zhang, and Zhiyao Xie. 2024. RTLLM: An open-source benchmark for design rtl generation with large language model. In2024 29th Asia and South Pacific Design Automation Conference (ASP-DAC). IEEE, 722–727

  2. [10]

    Moonshot AI. 2026. Kimi K3: Open Frontier Intelligence. https://www.kimi.com/ blog/kimi-k3. Official technical blog, accessed July 21, 2026

  3. [11]

    Nathaniel Pinckney, Chenhui Deng, Chia-Tung Ho, Yun-Da Tsai, Mingjie Liu, Wenfei Zhou, Brucek Khailany, and Haoxing Ren. 2025. Comprehensive Verilog design problems: A next-generation benchmark dataset for evaluating large language models and agents on rtl design and verificat...

  4. [12]

    The Verkor Team, Ravi Krishna, Suresh Krishna, and David Chin. 2026. Design Conductor: An agent autonomously builds a 1.5 GHz Linux-capable RISC-V CPU. arXiv preprint arXiv:2603.08716(2026)

  5. [13]

    Shailja Thakur, Baleegh Ahmad, Hammond Pearce, Benjamin Tan, Brendan Dolan-Gavitt, Ramesh Karri, and Siddharth Garg. 2024. Verigen: A large language model for verilog code generation.ACM Transactions on Design Automation of Electronic Systems29, 3 (2024), 1–31

  6. [14]

    Clifford Wolf. 2019. PicoRV32 - A Size-Optimized RISC-V CPU. https://github. com/YosysHQ/picorv32. Accessed: 2026-07-09

  7. [15]

    Z.AI. 2026. GLM-5.2. https://docs.z.ai/guides/llm/glm-5.2. Developer documen- tation. Accessed: 2026-07-09

  8. [16]

    Ruizhe Zhong, Xingbo Du, Shixiong Kai, Zhentao Tang, Siyuan Xu, Hui-Ling Zhen, Jianye Hao, Qiang Xu, Mingxuan Yuan, and Junchi Yan. 2023. Llm4eda: Emerging progress in large language models for electronic design automation. arXiv preprint arXiv:2401.12224(2023). 7

  9. [2026]

    arXiv preprint arXiv:2606.19348(2026)

    DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv preprint arXiv:2606.19348(2026)

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.