{"id":"0ea5fa21-8fbf-40ce-bd2e-bf46ba16b789","arxiv_id":"2607.17528","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"LLM agents can complete an RTL-to-GDS chip flow, but reliable completion depends on the execution infrastructure, not the foundation model alone.","lead":"Researchers tested four AI models and three agent setups on a real chip-design flow from RTL to GDS. They found that a structured execution harness, not the smartest model, was what let agents finish the job reliably.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-run evidence undercuts architecture comparisons and the 141× Token ROI claim; repeated runs are needed before rankings are cited.","rationale":"The paper's central claim is system-level: reliable RTL-to-GDS completion depends on the execution layer more than on the foundation model alone. The evidence for this is the consistent FluxEDA advantage across models and timing targets, plus the error-category analysis. I considered other potential concerns — that the Token ROI normalization is unspecified, that FluxEDA's zero Tcl-compatibility errors are by design, or that the 141× claim pairs runs differing in both model and architecture — but these are either secondary or derivative. The deepest vulnerability is that every configuration is run exactly once (§6). The paper's strongest quantitative statements (score spreads, 141× ROI gap, model-dependent skill effects) are all point estimates from single stochastic trajectories. The stage-completion table aggregates 8 independent runs per architecture, but each of those runs is itself a single sample, so the 7/8 vs. 8/8 difference is not statistically meaningful without variance information. This does not falsify the qualitative thesis; it does mean the paper's specific numbers should not be treated as benchmarks until repeated runs are available. The reader's CONDITIONAL verdict already reflects this; my recommended test would strengthen or weaken that verdict accordingly, not change it.","tokens_in":8811,"tokens_out":6175,"duration_ms":51013,"concrete_test":"Run each of the 24 configurations at least 5 times (ideally 10) with varied random seeds while holding tool versions and prompts fixed. Report mean ± standard deviation for score, stage completion, cost, runtime, and normalized Token ROI. Then use a permutation or bootstrap test to compare FluxEDA against CC and CC+Skills on stage completion and score, and a paired test for the Token ROI ratio between CC+DeepSeek-V4 and CC+Skills+Kimi K3 at 350 MHz to see whether the 141× gap persists (e.g., remains above 10×). If the variance is large or the differences fail significance, the architecture ranking and the headline ROI claim collapse.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim — that execution architecture, not foundation-model capability, determines reliable RTL-to-GDS completion — rests on comparative evidence in Table 2 and Table 3. But §6 states there is only 'one run per configuration.' LLM agents are stochastic (even at fixed temperature, sampling and tool interactions introduce variance), so the reported score spreads — e.g., Claude Sonnet 5 dropping from 87.05 (CC, 350 MHz) to 11.67 (CC+Skills, 350 MHz), or Kimi K3 scoring 13.33 at 350 MHz but 82.33 at 700 MHz under Claude Code — could be sampling noise rather than architecture or model effects. The 141× Token ROI gap in §5.2 is computed between two single runs that happen to achieve the same 13.33 score, and its normalization (Eq. 1) is unspecified; the precise magnitude is fragile. The stage-completion aggregate (8 runs per architecture) is also built from single-shot configurations: a 7/8 vs. 8/8 ECO completion difference is not statistically meaningful without variance information. These issues do not invalidate the qualitative thesis, but they do mean the quantitative lessons — model-dependent skill effects, the 141× efficiency gap, and score rankings — are not established. Repeated runs and statistical comparison are required before these numbers can serve as benchmarks.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports an empirical benchmark of LLM agents on a PicoRV32 RTL-to-GDS flow using commercial synthesis and physical-design tools. Three agent architectures (Claude Code, Claude Code + EDA Skills, FluxEDA) are crossed with four foundation models under two clock targets. The agents are scored by a gated end-to-end score, stage completion, runtime/cost, and a newly defined Token ROI. The paper's main lessons are that EDA skills do not guarantee flow completion, that execution architecture rather than model alone determines reliability (all FluxEDA runs score above 90), and that Tcl-level interface mismatches dominate physical-design failures. It concludes that robust Agentic EDA requires structured interfaces, persistent context, and process-level evaluation.","tokens_in":9100,"tokens_out":5547,"duration_ms":49548,"significance":"The paper addresses a genuinely important and underexplored problem: evaluating agents on long-horizon, tool-interactive chip-design workflows rather than on RTL generation alone. The gated scoring and the explicit error taxonomy are useful contributions, and the qualitative conclusion that execution architecture and tool-interface control matter is plausible and supported by a large, consistent gap in the reported data. The paper also has a reproducible-by-construction flavor (single design, fixed tools), and it honestly lists its limitations. However, the quantitative claims—model/architecture rankings, the 141x Token ROI gap, and stage-completion rates—rest on one run per configuration and on metrics whose normalization is not fully specified. These deficiencies make the current numerical lessons provisional.","major_comments":[{"comment":"The paper explicitly states 'one run per configuration.' For stochastic LLM agents, the observed score differences are not attributable to architecture/model/constraint effects. For example, Kimi K3 under Claude Code scores 13.33 at 350 MHz but 82.33 at 700 MHz, and Claude Sonnet 5 under CC+EDA Skills drops from 87.05 to 11.67. These swings are as large as the architecture effects claimed. The 'all FluxEDA runs above 90' and the 141.2x Token ROI comparison (12.71 vs 0.09) are also computed from single trials. Please provide repeated runs (e.g., at least 5 seeds per configuration) with mean, standard deviation, and per-run scores, and a paired statistical comparison of architectures.","section":"§6 and Table 2"},{"comment":"The scoring and Token ROI definitions are underspecified. The paper says area and power are 'normalized against frequency-specific reference values' and timing metrics are 'scaled according to predefined lower bounds,' but it never gives these reference values, lower bounds, or the rescale factors used for normalized Token ROI. The exact formula in Eq. (1) is also ambiguous (denominator product? separate penalties?). Without these numbers, the reported scores and ratios cannot be reconstructed or compared across runs; this is load-bearing because the 141x gap and architecture rankings are quantitative conclusions.","section":"§2.3, Eq. (1); §3.3, Eq. (4)"},{"comment":"FluxEDA, the winning architecture, is the authors' own system (ref. [3]) and Token ROI is also introduced by the authors, but this is not stated in the main text. The paper should disclose this explicitly and provide an independent or more controlled comparison (e.g., same baseline harness with only the execution layer changed, public artifact) so readers can separate the 'architecture' effect from implementation-specific engineering effort. The error-category percentages in Figure 3 also need denominators and a coding protocol; without these, 31.7% cannot be interpreted.","section":"§3.2, §4.3, §4.4"}],"minor_comments":[{"comment":"Formatting issue: '0.11' and other cost entries are missing spacing/alignment; also, the table title says 'ROI denotes the normalized Token ROI' but the metric was introduced as 'Token ROI' without normalization until §2.3. Clarify.","section":"Table 2"},{"comment":"Figure 2 is not explicitly discussed in the text; add a sentence connecting the figure to the cost-runtime-quality trade-off.","section":"§4.1"},{"comment":"Equation (4) uses set M with subscripts W_s, T_s, W_h, T_h, but the paper never defines WNS/TNS thresholds or how hold versus setup metrics are combined; a worked example of one score would help.","section":"§3.3"},{"comment":"The phrase 'The present invited paper' is odd in a formal journal submission; either remove it or state the venue explicitly.","section":"§2.4"}],"recommendation":"major_revision","confidential_remarks":"The paper's strongest result—that structured execution support matters more than model choice—is plausible but is currently supported by single runs and by metrics developed by the same group that built the winning architecture. The authors do not disclose in the main text that FluxEDA is their own system. This is not a reason to reject, but it raises the burden for statistical evidence and transparency. Please ensure the editor sees the conflict-of-interest note before sending for review or publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is worth reading and citing for its central lesson: for long-horizon RTL-to-GDS, structured execution architecture matters more than which LLM backend you pick. The evidence for that is a consistent gap: every FluxEDA run scores above 90 across four models and two timing targets, while the shell-based baselines mostly sit in the 10–87 range. That's a real result, not a manufactured one.\n\nWhat's new: it's the first systematic comparison of four foundation models across three agent architectures on a full tool-interactive RTL-to-GDS flow, and it introduces Token ROI as a cost-efficiency measure. The error-category analysis—31.7% of physical-design errors coming from Tcl tool-version/mode compatibility issues—is a useful, concrete takeaway. The paper also writes clearly and explicitly states its limitations in the conclusions.\n\nThe soft spots are mostly quantitative. There is one run per configuration, as §6 admits. LLM agents are stochastic; some of the score swings (e.g., Claude Sonnet 5 dropping from 87.05 to 11.67 with skills; Kimi K3 going from 13.33 to 82.33 between targets under Claude Code) could be sampling noise. The 141× Token ROI gap in §5.2 is computed between two single runs that happen to land at the same 13.33 score, and the normalization behind Eq. (1) is underspecified. The stage-completion table (8 runs per architecture) is built from those same single-shot runs, so the 7/8 vs. 8/8 ECO difference isn't statistically meaningful. Area/power reference values and timing lower bounds are also not specified.\n\nNone of that sinks the qualitative thesis—the gap between FluxEDA and shell baselines is large and consistent—but it does mean the precise numbers should not be treated as benchmarks until repeated runs, full normalization details, and ideally released artifacts are provided. The paper is also a self-evaluation of the authors' own FluxEDA system, which raises the bar for transparency, but that's not disqualifying.\n\nMy recommendation: send it to peer review. It's a legitimate empirical contribution with a clear, testable claim. Require repeated runs, statistical reporting, and full metric definitions before publication; without those, the quantitative claims should be labeled preliminary.\n\nFor your reading group, it would spark good discussion about evaluating agentic systems.","headline":"A solid empirical case study showing execution infrastructure beats model choice for reliable RTL-to-GDS, but the headline numbers (141× Token ROI gap, score rankings) are fragile due to single-run evidence.","tokens_in":9632,"tokens_out":2349,"would_cite":true,"duration_ms":18744,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that current AI agents can complete an RTL-to-GDS chip-design flow, but that reliable completion is determined by the execution layer controlling tool interfaces and preserving design context, not by foundation-model capab","keywords":["Agentic EDA","RTL-to-GDS","LLM agents","tool-interactive workflows","Token ROI","Tcl command compatibility","execution architecture","physical design automation"],"falsifier":"Repeat each architecture–model pair at least five times under the same timing targets. If a command-line-style agent completes ECO in some repeated runs or the structured harness fails at any stage, the reported 8/8 versus 3/8 stage-completion gap and the architecture rankings become sampling artifacts rather than stable properties. Alternatively, pin the commercial tool version and expose a validated interface to the command-line agents; if the Tcl-compatibility error category remains large, the failure is not caused by interface structure.","tokens_in":8673,"feed_emoji":"⚙️","tokens_out":4521,"duration_ms":42295,"temperature":0.7,"pith_summary":"This paper asks whether current AI agents can reliably take a chip design from register-transfer level (RTL) to final layout (GDS) using commercial EDA tools, a long, feedback-driven process involving synthesis, placement, clock-tree synthesis, routing, and engineering change orders. It argues that agents can complete such a flow, but that reliable completion is a system property rather than a property of the foundation model: domain-specific skills alone give inconsistent gains, while a structured execution layer that validates tool calls, keeps persistent design state, and returns normalized feedback completed every stage across all four tested models and both timing targets. The paper also shows that similar design progress can cost very different resources, with up to a 141× difference in Token ROI, and that the largest source of physical-design failures is low-level Tcl command incompatibility with the tool version or execution mode. A sympathetic reader would care because the result shifts the bottleneck in Agentic EDA from model intelligence to execution infrastructure and process-level evaluation.","feed_headline":"Execution layer, not model, decides reliable chip-design completion","feed_subtitle":"Benchmarking three agent architectures shows same design progress can differ 141× in cost efficiency.","key_machinery":"The central object carrying the argument is the execution harness (called FluxEDA in the paper), which separates agent reasoning from tool execution: it registers EDA capabilities as structured actions, maintains persistent tool sessions and design state, and returns normalized results, thereby removing whole classes of syntactically plausible but operationally invalid commands. The measurement machinery is a gated end-to-end score, S = 0.2·S_stage + 0.8·S_result, where stage completion is gated sequentially so that partial outputs earn no result credit, paired with Token ROI, defined as normalized EDA return divided by token investment times execution-time investment. Together they let the","core_discovery":"The paper's central discovery is that end-to-end RTL-to-GDS completion is achievable by current LLM agents, and the decisive factor is the execution layer, not the model. In the reported experiments, every run backed by the structured execution harness completed synthesis, physical implementation, and ECO under both a loose (350 MHz) and a tight (700 MHz) timing target, scoring above 90 on a 0–100 gated scale, while command-line-style agents and their skill-augmented variants completed ECO in at most 3 of 8 runs. The paper attributes this gap to three causes: skills improve local procedural knowledge but do not stabilize long-horizon behavior; efficiency varies by up to 141× even between run","pith_inferences":["Inference: The 141× Token ROI gap suggests the economics of Agentic EDA may depend more on the execution harness than on model choice, and that cost-efficient deployment could favor structured orchestration even when two systems produce identical design files.","Inference: The likely generalization is that any long-horizon, tool-interactive engineering workflow—beyond chip design—will exhibit the same pattern: hidden operational state and interface versioning, not reasoning ability per se, are what make agents fail.","Inference: Because the paper ran each configuration once, run-to-run stochasticity is an unresolved confound; repeating runs would likely show that command-line agents occasionally complete ECO and structured harnesses occasionally fail, narrowing the reported architecture gap.","Inference: A testable extension would be to add checkpoint and rollback mechanisms to the command-line-style agents while keeping the same tools; if the Tcl-compatibility error category persists, the paper's conclusion about execution support is strengthened, while if it disappears, the failure is better attributed to lack of recovery rather than interface structure."],"forward_implications":["If the central claim is correct, benchmarks for Agentic EDA should report stage completion, runtime, token cost, and Token ROI together with final design quality, because raw cost or runtime alone can reward early termination.","Domain-specific skills should be treated as local enhancements, not reliability mechanisms; their effect is model- and target-dependent and can even reduce performance.","Commercial EDA toolchains should expose version- and mode-aware validated interfaces rather than raw Tcl shells, which would eliminate the largest observed category of physical-design errors.","Agent capability should be evaluated as a complete system—architecture plus model—across multiple constraints, since the same model's score varied by 86 points across architectures in one condition.","Efficiency comparisons should be conditioned on verified progress; two runs reaching the same milestone can differ by roughly 141× in Token ROI, so resource use alone is not a valid efficiency signal."],"fun_headline_variants":["Execution layer, not model size, makes RTL-to-GDS work","Chip-design agents: structured tools beat bigger brains","141× cost gap in AI chip design: it's the harness","Skills don't finish chips; structured execution does","Tool interface, not model, decides AI chip design success"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that one run per configuration represents an agent's typical behavior; LLM agents are stochastic, so a single run may misrepresent true performance, and the reported score and stage-completion differences—such as one model scoring 13.33 at 350 MHz and 82.33 at 700 MHz under the same architecture—could be partly sampling noise.","fun_headline_variants_meta":{"raw":{"variants":["Execution layer, not model size, makes RTL-to-GDS work","Chip-design agents: structured tools beat bigger brains","141× cost gap in AI chip design: it's the harness","Skills don't finish chips; structured execution does","Tool interface, not model, decides AI chip design success"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000171,"raw_usage":{"total_tokens":1128,"prompt_tokens":785,"completion_tokens":343,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":529,"completion_tokens_details":{"reasoning_tokens":260}},"tokens_in":529,"tokens_out":343,"duration_ms":3632,"temperature":1.0,"reasoning_tokens":260,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T17:42:27.533154+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat each architecture–model pair at least five times under the same timing targets. If a command-line-style agent completes ECO in some repeated runs or the structured harness fails at any stage, the reported 8/8 versus 3/8 stage-completion gap and the architecture rankings become sampling artifacts rather than stable properties. Alternatively, pin the commercial tool version and expose a validated interface to the command-line agents; if the Tcl-compatibility error category remains large, the failure is not caused by interface structure.","supporting_citations":[],"review_version":1}