{"id":"53ac7b08-fefa-4dea-af9d-b84ca0d4027d","arxiv_id":"2608.12751","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"SynAct, a closed-loop LLM agent with retrieval and latent-space Bayesian optimization, reduces average WNS to 27.03% of bootstrap synthesis across 14 designs.","lead":"SynAct is an AI agent that repeatedly reads a chip's synthesis reports and issues new optimization commands, closing the loop between diagnosis and action. It cuts worst negative slack to 27% of the baseline across 14 designs on a commercial tool, suggesting LLM agents can adaptively tune EDA flows.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing no-feedback control leaves the central claim that closed-loop adaptation drives SynAct's gain unproven; the self-reproduced ChatLS baseline is a secondary but real fairness risk.","rationale":"The reader's weakest_assumption (ChatLS self-reproduction) is legitimate and is explicitly acknowledged in Sec. VI-B, but I find a more load-bearing issue upstream: the evidence does not isolate the closed-loop mechanism that is the paper's stated first contribution and the source of the claimed advantage. Every SynAct iteration evaluates ten candidates and commits the best safe one; the one-shot baseline and bandit do not have an equivalent selection budget. The existing ablations (Sec. VI-F) hold the iterative loop fixed and vary only RAG and BO, so they cannot rule out that the gain comes from evaluating about 50 candidates and keeping the best, with diagnosis playing a minor or negligible role. The ChatLS port, even if faithful, would not fix this attribution gap. I nonetheless would not reject: the system is well described, the self-identified limitations are stated, the runtime cost is reported, and the empirical headline could survive better controls. I would keep the CONDITIONAL verdict and make the frozen-state control and release of prompts/per-run data explicit conditions for acceptance. This is a partial agreement with the reader: same baseline concern, but the primary stress is the missing no-feedback ablation.","tokens_in":18629,"tokens_out":10626,"duration_ms":121736,"concrete_test":"Add a frozen-state control: for all 14 designs, run SynAct for five iterations exactly as in Sec. VI, but feed the Optimization Agent the bootstrap-state reports and the initial diagnosis at every iteration instead of the latest post-command reports, while retaining the same candidate generation, ten-candidate evaluation, safety filter, and BO experience log. Compute the WNS Ratio Avg.; if it is close to 27.03% (e.g., within 5 percentage points), the reported gain is driven by candidate evaluation and selection rather than by adaptive state feedback, and the closed-loop claim must be weakened. In the same run, record per-iteration WNS and per-run standard deviations to also address the missing variance information.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The weakest point is not only the ChatLS reproduction (which the authors disclose): the paper never isolates the mechanism it claims to be central. In each of five iterations, SynAct generates ten candidates, evaluates all of them with the synthesis tool, and commits the best safe command (Sec. III-D). Candidate evaluation is 84.85% of runtime (Fig. 6). Thus the headline 27.03% WNS ratio could largely come from selecting the best among roughly 50 evaluated commands while a one-shot baseline gets a single script. The ablations in Sec. VI-F vary BO and retrieval, but every variant keeps live state feedback and multi-candidate selection, so they do not test whether the updated diagnosis and closed-loop adaptation, rather than candidate breadth, is the source of the gain. The cross-LLM consistency (DeepSeek 27.03%, GPT-5.2 20.37%) also does not resolve this confound because both LLMs run the same candidate-evaluation loop. An official ChatLS reproduction would address the baseline fairness issue, but it would not by itself prove the adaptive component is load-bearing.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents SynAct, a closed-loop LLM reasoning-acting agent for logic synthesis optimization on the commercial tool AltiSyn. At each of up to five iterations, an Analysis Agent parses live timing reports and proposes probes; an Optimization Agent generates ten candidate commands, conditioned on the diagnosis, GraphRAG-retrieved tool knowledge, and a Bayesian-optimization seed decoded from a GrammarVAE latent space; Candidate Selection evaluates all candidates, filters unsafe ones, and commits the best-scoring safe command. Across 14 OpenCores designs with DeepSeek V3.1, SynAct reports an average WNS ratio of 27.03% relative to bootstrap synthesis while holding area near 99.28% and power near 98.9% of bootstrap; a GPT-5.2 configuration reaches 20.37% WNS. These results are compared with a reproduced one-shot LLM baseline (ChatLS, 71.73%) and a bounded contextual bandit (CBTune, 66.67%), with ablations for the BO and GraphRAG modules.","tokens_in":18782,"tokens_out":5872,"duration_ms":63068,"significance":"If confirmed, SynAct is a substantive advance: it replaces one-shot script generation with a state-adaptive loop that produces explicit rationales, and it takes a step beyond fixed-action search methods on a commercial command vocabulary. The manuscript has several genuine strengths: the reward is a user-specified weighted violation metric computed from external tool reports, the GrammarVAE is pretrained on commands rather than on outcome labels, the main experiments are repeated five times per design, the cross-LLM replication is a concrete check on model-dependence, and the authors candidly disclose that the ChatLS comparison is a controlled reproduction rather than the official implementation. The principal weakness is experimental: the ablation suite never isolates the closed-loop adaptive mechanism from the effect of evaluating and selecting among many candidates, so the central claim that iterative adaptation is the source of the gain is not yet demonstrated.","major_comments":[{"comment":"The ablation study does not establish the paper's central claim that closed-loop adaptation is the source of the gain. Every variant in Fig. 8, including 'w/o BO' and 'vanilla RAG', generates ten candidates per iteration, evaluates all of them by synthesis, and commits the best safe command; candidate evaluation is 84.85% of runtime (Fig. 6). The headline 27.03% WNS ratio versus a one-shot baseline could therefore substantially reflect a best-of-fifty selection effect rather than the agent's updated diagnosis and iterative adaptation. I request two controls: (a) a single-iteration 'best-of-N' variant that generates the same total number of candidates from the initial state, evaluates all of them, and commits the best safe command with no subsequent feedback; and (b) a closed-loop variant that retains candidate evaluation but conditions each iteration's generation only on the set of previously executed commands and their results, without re-diagnosing the latest detailed reports. If either control reaches comparable WNS ratios, the adaptive contribution claimed in Section VII must be restated.","section":"§VI-F (Ablation Study on BO and Retrieval Modules), Fig. 8 and Fig. 6"},{"comment":"The comparison to ChatLS rests on a self-reproduced port, as the authors state: 'Since its source code is unavailable, we reproduce ChatLS following the published methodology' and 'the resulting comparison represents our controlled reproduction rather than the official implementation.' Matching reported results within 5% on the original seven-benchmark 45 nm FreePDK setup does not guarantee parity after porting to 14 designs, ASAP7, and AltiSyn, where prompt structure, retrieval fidelity, and command syntax can change behavior. I ask for (a) an official or independently verified ChatLS run on the new setup, or at least per-benchmark evidence that the ported five-step script is not systematically weaker on AltiSyn; and (b) a sensitivity analysis for the manually bounded seven-arm CBTune action space in Table V, since a narrow arm set may understate a fixed-action baseline on AltiSyn's richer command language.","section":"§VI-B (Comparison setup and method reproduction), Table III, Table V"},{"comment":"No variance or statistical testing is reported for the central comparisons. Tables III and VI report five-run averages only, and the ablation text reports aggregate ratios such as 27.0% vs. 37.5% without standard deviations or paired tests across the 14 designs. This matters because some contrasts are small: 27.03% (full SynAct) versus 28.6% (vanilla RAG), and the DeepSeek versus GPT-5.2 difference could fall within run-to-run noise. I request per-design standard deviations, error bars in Fig. 8, and a paired significance test for SynAct against each baseline and each ablation variant.","section":"§VI-A, §VI-B, Tables III and VI, Fig. 8"}],"minor_comments":[{"comment":"There are typographic and formatting issues: the JSON snippets in Fig. 2 use mixed curly and straight quotation marks, and the Table I header 'CPD. 1' with the footnote markers below Table II is confusing. These should be cleaned before publication.","section":"Fig. 2 and Table I"},{"comment":"The paper does not report the concrete values of several hyperparameters that drive the method, including kappa and kappa_acq, lambda, w_max, the RBF length scale, k_intra, the top-k retrieval count, and the safety-filter thresholds (-20 ps and 0.2 x r_boot). These values are needed for reproducibility, and for the most sensitive ones a short sensitivity analysis would strengthen the ablation claims.","section":"§IV-A and §VI-A"},{"comment":"The comparison budget is described in terms of committed actions rather than synthesis evaluations. Since SynAct evaluates ten candidates per iteration for five iterations while ChatLS commits a single five-step script and CBTune evaluates its seven arms, the paper should state the total number of synthesis runs per method explicitly, even though end-to-end runtime is already reported in Table IV.","section":"§III-D and §VI-A"},{"comment":"The trajectory analysis in Fig. 7 shows only four of fourteen benchmarks; the selection criterion for these representative designs should be stated, and a one-line version covering all designs would be more convincing.","section":"§VI-C, Fig. 7"},{"comment":"Reference [15] cites the Synopsys Design Compiler User Guide, but the experiments are run on AltiSyn from ZeniSyn Design Systems. If the command documentation used for GraphRAG is AltiSyn's own manual, the citation should be to that manual or the relationship should be clarified.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the scope of the journal and should interest the EDA community. My main concern is the missing no-feedback control, because without it the central 'adaptive closed-loop' claim is confounded with the benefit of evaluating many candidates. Adding the requested controls and variance reporting is feasible within the manuscript's scope, so I recommend major revision rather than rejection. The ChatLS reproduction disclosure is honest, but official validation of that baseline would materially increase confidence in the head-to-head comparison."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"SynAct is a serious, well-built system: a closed-loop LLM agent that diagnoses live synthesis reports, retrieves tool knowledge via GraphRAG, and uses latent-space BO to reuse experience on a commercial synthesis tool. The 27.03% WNS ratio relative to bootstrap across 14 designs is a big number, and the assembly is genuinely new—no one has coupled those pieces into a single synthesis agent. The across-LLM replication (DeepSeek 27.03%, GPT-5.2 20.37%) and the ablations isolating BO and GraphRAG are the right kind of evidence. The authors also validate their ChatLS reproduction against the original setup within 5%, which is more than most self-reproductions do.\n\nThe soft spot is not the baseline reproduction; it is the missing control for the central interpretive claim. SynAct evaluates ten candidates per iteration and commits the best safe one; candidate evaluation is 84.85% of runtime. A one-shot method that simply samples, say, 50 commands and picks the best from the same distribution could plausibly capture most of the same WNS gain without any reasoning, diagnosis, or state adaptation. The ablations in Fig. 8 keep the multi-candidate loop and live feedback in every variant, so they measure the marginal value of BO and GraphRAG, not the value of the closed loop itself. The paper states that iterative adaptation is the source of the gain, but that claim is underdetermined by the data.\n\nThere are smaller issues: no variance reported on the five-run averages, a manually bounded seven-arm action space for CBTune, and several control constants (reward weights, exploration coefficients, retrieval k) unreported. None is fatal.\n\nMy bottom line: this is a plausible and well-engineered contribution worth serious referee time, but the load-bearing causal claim needs a no-feedback control (best-of-k with equal evaluation budget, or a single-shot version with the same candidate pool). I would not desk-reject; I would send it out and request that control, plus code/prompts/parameters and per-run statistics. If the control collapses the margin, the contribution shrinks to a useful engineering system with a smaller, still real advantage. If the control holds, this is a notable result either way. The paper is aimed at EDA automation researchers and anyone studying closed-loop decision making under expensive evaluation, and it deserves a careful peer review rather than a quick reject.","headline":"A genuinely new closed-loop LLM synthesis agent with a large reported timing improvement, but the paper does not isolate whether the loop itself or just multi-candidate evaluation is doing the work.","tokens_in":19427,"tokens_out":3797,"would_cite":true,"duration_ms":41158,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An LLM agent that iteratively diagnoses live synthesis reports and issues targeted commands reduces average worst negative slack to 27.03% of the bootstrap value across 14 designs, with area and power nearly unchanged.","keywords":["logic synthesis","LLM agent","timing optimization","worst negative slack","retrieval-augmented generation","Bayesian optimization","grammar variational autoencoder","closed-loop optimization"],"falsifier":"Run the same five-iteration, ten-candidate protocol against the original implementation of the one-shot script baseline on the same 14 designs and the same commercial tool session; if the WNS-ratio gap nearly vanishes or reverses, the claim that closed-loop adaptation beats one-shot scripts collapses. A second check: disable only the Bayesian-optimization module while still exposing all historical commands as prompt context; if WNS matches the full system, the experience-reuse mechanism is not load-bearing.","tokens_in":18343,"feed_emoji":"🤖","tokens_out":10590,"duration_ms":94449,"temperature":0.7,"pith_summary":"SynAct is an attempt to establish that logic-synthesis tuning belongs to closed-loop LLM agents rather than one-shot scripts or fixed-action search. The paper claims that an agent which alternates diagnosis of live synthesis reports with targeted command generation, grounded by graph-based retrieval of tool documentation and by Bayesian optimization over past commands, cuts the remaining worst negative slack (WNS) to 27.03% of the bootstrap value on average across 14 designs, while keeping area at 99.28% and power near 98.9% of bootstrap. The same loop run with a different, stronger LLM reaches 20.37% WNS, which the paper reads as evidence that the adaptive mechanism, not the underlying model, drives the gain. The payoff, if the claim holds, is a general pattern for turning expensive stateful tool sessions into adaptive optimization.","feed_headline":"LLM agent cuts chip timing violations to 27 percent of baseline","feed_subtitle":"Across 14 designs, iterative diagnosis and action beats one-shot scripts and fixed-action search while holding area and power steady.","key_machinery":"The load-bearing object is the reasoning-acting loop: an Analysis Agent converts raw tool reports into a structured diagnosis and follow-up probe commands, an Optimization Agent converts that diagnosis plus retrieved knowledge and a Bayesian-optimization seed into candidate commands, and a selection step evaluates, filters, and commits the best safe candidate before feeding new reports back. Two auxiliary mechanisms carry the adaptation. First, a three-layer GraphRAG module organizes tool documentation into scenarios, commands, and configurable variables connected by intra- and inter-layer edges, so retrieval is scenario-driven rather than flat similarity search. Second, a GrammarVAE encodes commands into a continuous latent space, where a reward-weighted radial-basis surrogate with a shrinking or expanding trust region proposes nearby commands from historical experience. The explicit score function $score(C)=r(C)+\\kappa\\sigma(z)-\\rho(C)$ ties the two together by rewarding objective satisfaction, encouraging exploration in uncertain regions, and penalizing recent repetition.","core_discovery":"The central discovery, stated on the paper's own terms, is that synthesis optimization can be cast as an interactive Markov decision process and profitably driven by an LLM agent that reasons before every action and observes the tool's response before choosing the next. The Analysis Agent parses raw PPA reports, issues its own analyze_* probes when the reports are under-specified, and refines a structured diagnosis of the critical path; the Optimization Agent turns that diagnosis, retrieved manual sections, and a BO-seeded command suggestion into ten candidate commands; Candidate Selection executes all of them, discards unsafe ones, and commits the best safe command by a score that balances reward, exploration, and anti-repetition. Across 14 designs with five iterations, average remaining WNS falls to 27.03% of bootstrap (TNS to 17.86%), with area at 99.28%, dynamic power at 98.92%, and static power at 98.64% of bootstrap; a second LLM configuration improves the WNS ratio to 20.37%. The paper's conclusion is that the closed loop itself, not the particular LLM, is the source of the improvement.","pith_inferences":["If the loop rather than the LLM is the source of gain, the same architecture should transfer to any EDA stage whose commands mutate a state visible through textual reports, such as floorplanning, placement, clock-tree synthesis, or high-level synthesis, given a command grammar and documentation.","A sharper ablation than the paper's would give a one-shot LLM script generator feedback after each of its five steps and let it revise its own commands; if that closes most of the gap, the marginal contribution of the graph retrieval and experience reuse is smaller than the contribution of feedback itself.","A complementary test would replace the LLM's candidate generation with a fixed library of commands and let Bayesian optimization alone select among them with the same evaluation budget; this would separate experience-guided search from the LLM's ability to invent new commands."],"forward_implications":["One-shot LLM script generation leaves timing on the table; interactive tool sessions should be treated as the primary interface for synthesis tuning.","Historical commands are reusable: commands close in the grammar-latent space produce more similar rewards (mean $\\Delta r$ of 0.188 for near pairs versus 0.238 for far pairs), so past experience can seed future search.","The agent mechanism transfers across LLMs; with a stronger LLM the WNS ratio improves from 27.03% to 20.37%, indicating the loop sets the mechanism and the model sets the ceiling.","WNS continues to improve with more iterations, so the five-iteration budget is a runtime-quality trade-off rather than an intrinsic limit of the method.","The framework is portable to other synthesis tools that expose a scripting interface, command documentation, and structured PPA and timing reports, by replacing tool-specific adapters and the knowledge base."],"supporting_citations":[{"why":"Defines the one-shot LLM script-generation baseline that SynAct is compared against; the paper reproduces it because the source code is unavailable.","marker":"[11]"},{"why":"Defines the contextual-bandit baseline whose fixed action space is manually bounded to seven commands for the SynAct comparison.","marker":"[8]"},{"why":"Supplies the reasoning-acting pattern that the Analysis Agent and Optimization Agent loop is built on.","marker":"[13]"},{"why":"Supplies the multi-agent coordination style used to organize the Analysis and Optimization roles.","marker":"[14]"},{"why":"Supplies the EDA-customized text embedding model and retrieval approach used inside the graph-based retrieval module.","marker":"[12]"},{"why":"Supplies the grammar variational autoencoder that encodes synthesis commands into the continuous latent space where Bayesian optimization operates.","marker":"[20]"},{"why":"Supplies the retrieval-augmented generation foundation that grounds LLM output in retrieved tool documentation.","marker":"[16]"},{"why":"Establishes Bayesian optimization for logic synthesis, the prior approach that the BO-guided experience refinement extends.","marker":"[5]"}],"fun_headline_variants":["LLM agent adaptively tunes chip synthesis, cuts timing to 27%","Reasoning-acting LLM agent cuts chip timing to 27% of baseline","Adaptive LLM agent iterates on chip synthesis, hits 27% WNS","Closed-loop LLM agent improves chip timing to 27% of baseline","SynAct LLM agent reasons and acts to cut chip timing to 27%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reproduced one-shot LLM script baseline faithfully instantiates that method on the new tool setup; if the reproduction is weaker than the intended pipeline, the reported advantage over one-shot scripts is overstated.","fun_headline_variants_meta":{"raw":{"variants":["LLM agent adaptively tunes chip synthesis, cuts timing to 27%","Reasoning-acting LLM agent cuts chip timing to 27% of baseline","Adaptive LLM agent iterates on chip synthesis, hits 27% WNS","Closed-loop LLM agent improves chip timing to 27% of baseline","SynAct LLM agent reasons and acts to cut chip timing to 27%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00027,"raw_usage":{"total_tokens":1620,"prompt_tokens":933,"completion_tokens":687,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":598}},"tokens_in":549,"tokens_out":687,"duration_ms":6031,"temperature":1.0,"reasoning_tokens":598,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:00:15.828475+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same five-iteration, ten-candidate protocol against the original implementation of the one-shot script baseline on the same 14 designs and the same commercial tool session; if the WNS-ratio gap nearly vanishes or reverses, the claim that closed-loop adaptation beats one-shot scripts collapses. A second check: disable only the Bayesian-optimization module while still exposing all historical commands as prompt context; if WNS matches the full system, the experience-reuse mechanism is not load-bearing.","supporting_citations":[{"cited_title":"ChatLS: Multimodal retrieval-augmented generation and chain-of-thought for logic synthesis script customiza- tion,","cited_arxiv_id":null,"evidence_quote":"Defines the one-shot LLM script-generation baseline that SynAct is compared against; the paper reproduces it because the source code is unavailable."},{"cited_title":"CBTune: Contextual bandit tuning for logic synthesis,","cited_arxiv_id":null,"evidence_quote":"Defines the contextual-bandit baseline whose fixed action space is manually bounded to seven commands for the SynAct comparison."},{"cited_title":"ReAct: Synergizing reasoning and acting in language models,","cited_arxiv_id":null,"evidence_quote":"Supplies the reasoning-acting pattern that the Analysis Agent and Optimization Agent loop is built on."},{"cited_title":"AutoGen: Enabling next-gen LLM applications via multi-agent conversations,","cited_arxiv_id":null,"evidence_quote":"Supplies the multi-agent coordination style used to organize the Analysis and Optimization roles."},{"cited_title":"Customized retrieval aug- mented generation and benchmarking for EDA tool documentation QA,","cited_arxiv_id":null,"evidence_quote":"Supplies the EDA-customized text embedding model and retrieval approach used inside the graph-based retrieval module."},{"cited_title":"Grammar varia- tional autoencoder,","cited_arxiv_id":null,"evidence_quote":"Supplies the grammar variational autoencoder that encodes synthesis commands into the continuous latent space where Bayesian optimization operates."},{"cited_title":"BOiLS: Bayesian optimisation for logic synthesis,","cited_arxiv_id":null,"evidence_quote":"Establishes Bayesian optimization for logic synthesis, the prior approach that the BO-guided experience refinement extends."}],"review_version":1}