{"id":"97f557dd-2a08-4fd6-b929-045c8c6097f6","arxiv_id":"2608.02915","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"LACE uses multiple LLM agents to translate RISC-V instruction intentions into hardware modifications, raising end-to-end generation success from near zero to 72.8 percent on four embedded cores.","lead":"LACE is an LLM-powered multi-agent system that turns natural-language descriptions of new RISC-V instructions into working hardware edits across four different processor cores. It reports cutting the failure rate of instruction extension generation from near-total to about 73 percent, which could lower the expert effort needed to add custom instructions to open-source chips.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 72.8% pass@1 likely measures success of the full agentic repair loop, not one-shot generation, because the non-agentic baseline is not given the same Verilator/riscv-formal feedback; the gain may be driven by iterative repair rather than the IR/RAG design.","rationale":"Reading in good faith, LACE is a serious systems paper: the two-level IR is a sensible abstraction, the agentic RAG is concretely tooled, the PPA comparison is useful, and the code is public. The arithmetic of Table III is internally consistent with the stated 72.8% average. The load-bearing issue is not arithmetic but attribution. The configuration section explicitly makes a complete LACE run include Verilator and riscv-formal feedback, while the direct baseline is described only as non-agentic generation. Without a repair-enabled control that lacks the IR and RAG, the reported improvement cannot be assigned to the claimed mechanisms. The reader's RVFI concern is real but is a disclosed scoping limitation; the baseline asymmetry is a measurement concern that the current text does not disclose or bound. The appropriate outcome remains CONDITIONAL: the claims are promising, but the headline metric needs a matched control and an explicit iteration budget before the 72.8% figure should be used as evidence for the IR/RAG contributions.","tokens_in":11059,"tokens_out":10510,"duration_ms":100019,"concrete_test":"Add a repair-enabled non-agentic control to the Table III benchmark: use the same GPT-4o, same target files, same ISAX descriptions, same 2-shot examples, and the same Verilator plus riscv-formal feedback loop, but remove the two-level IR translation and the agentic RAG, replacing them with flat retrieval or full-file context and a single 'fix the errors' prompt. Recompute the 20 cells and also report the distribution of repair iterations and LLM calls per successful run for LACE and for the control. If the control reaches a comparable average pass rate, the 72.8% headline should be attributed to the feedback loop rather than to the IR/RAG architecture; if the control stays near the original roughly 2%, the stated attribution is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section VI.A defines pass@1 as the fraction of nine independent 'complete LACE runs' whose final RTL compiles with Verilator and passes riscv-formal. A complete LACE run includes the iterative syntax and functional checks and Agent⑥ feedback loop shown in Figure 2 and Section V.B. Section VI.B compares this against 'direct (non-agentic) LLM generation using identical target files, ISAX descriptions, 2-shot examples, and model settings,' without stating whether the direct baseline also receives the same Verilator/riscv-formal repair loop. If the direct baseline is a single generation with no feedback, LACE is allowed many repair iterations while the baseline is allowed none. The paper reports neither the number of LLM calls per successful run nor a repair-enabled direct control. Consequently the central quantitative claim, a jump from roughly 2% to 72.8%, does not isolate the two-level IR and agentic RAG, which are the paper's main claimed contributions. The term pass@1 is also used in a nonstandard way: standard pass@1 accepts exactly one generated solution per trial, whereas here it counts a multi-step agent trajectory that may be repaired until the formal check passes. This is a measurement and attribution concern, not a claim of misleading intent; it can be resolved with a matched control.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents LACE, a multi-agent LLM framework that takes natural-language RISC-V ISA extension (ISAX) descriptions, translates them into a two-level IR (operation-level and HDL task-level), performs retrieval-augmented code localization and modification on large processor repositories, and closes the loop with riscv-formal verification (assuming RVFI availability). The authors evaluate LACE on four open-source RISC-V cores (PicoRV32, e203 hbirdv2, ibex, cv32e40x) and five ISAX instructions, reporting an average pass@1 of 72.8% versus essentially zero for direct non-agentic LLM generation, together with ASIC synthesis overheads and RAG search quality metrics. The central claims are that LACE makes ISAX integration across cores significantly more automated, reliable, and efficient.","tokens_in":11331,"tokens_out":5426,"duration_ms":46775,"significance":"If the results hold under a fair comparison, LACE would be a valuable contribution to the emerging area of LLM-assisted hardware design, combining instruction abstraction, agentic repository navigation, and formal verification in a way that goes beyond isolated RTL generation. The paper is honest about the RVFI dependency, makes its code and evaluation setup publicly available, and provides a concrete benchmark across multiple cores and instruction types. However, the quantitative headline depends heavily on the pass@1 comparison being apples-to-apples, and on the RAG benchmark being constructed independently of the system's own design artifacts; these issues are addressable with additional experiments but are load-bearing for the stated conclusions.","major_comments":[{"comment":"The pass@1 metric for LACE counts a complete agentic run that includes iterative syntax and functional checks and the Agent⑥ feedback loop (Figure 2), whereas the 'direct (non-agentic) LLM generation' baseline is described only as using 'identical target files, ISAX descriptions, 2-shot examples, and model settings,' with no mention of any repair loop. If the baseline is a single generation without feedback, the 72.8% versus near-zero gap conflates the contribution of iterative repair with that of the two-level IR and agentic RAG. To support the attribution, the authors should report the distribution of LLM calls per successful run and add a control where the non-agentic generator is also allowed to iterate against Verilator/riscv-formal failures, or explicitly justify why such a control is infeasible.","section":"§VI.A and §VI.B"},{"comment":"The RAG benchmark is constructed by 'collecting the given meta-prompts and their corresponding code segments to be modified' (50 tests). Because the meta-prompts are the authors' own design artifacts and the queries are exactly the ones LACE was built around, this evaluation partly rewards LACE's own choices and does not demonstrate superiority for unseen natural-language queries. The MRR comparison in Figure 5 should be repeated on a held-out set of ISAX descriptions or on code segments from a core not used in prompting, or the authors should discuss the inherent circularity.","section":"§VI.C"},{"comment":"The end-to-end claim of a 'working integration and verification flow' is conditional on the stated assumption that RVFI is 'either already available or can be added with engineering effort,' with fully automatic RVFI insertion outside scope. Since riscv-formal requires RVFI, the 72.8% pass@1 results implicitly include whatever manual RVFI instrumentation was performed for the four cores. The paper should quantify or at least characterize this manual effort (e.g., person-hours or lines of instrumentation per core) so that the level of automation claimed is not overstated.","section":"§V.A and §VII"}],"minor_comments":[{"comment":"The average direct-baseline pass@1 is exactly 2.2% (Table III) rather than 'near-zero'; the text should state the exact number or use a precise qualifier such as 'low single digits.'","section":"§VI.B"},{"comment":"The term 'pass@1' is used in a nonstandard way: it measures the fraction of nine independent multi-step agent trajectories that converge, not the probability that a single generated solution is correct. Although the definition is clear in Section VI.A, the abstract's 'pass@1 generation accuracy' may mislead readers; consider renaming it to 'run success rate' or explicitly contrasting it with one-shot pass@1.","section":"§VI.A and Abstract"},{"comment":"The value for ibex sbox is shown as 88.8% while the corresponding percentage for other cores is 88.9%; please make the rounding consistent.","section":"Table III"},{"comment":"The y-axis label 'Hops Count' should be defined in the caption or text; it appears to mean the number of verification-and-repair iterations, but this is never stated explicitly.","section":"Figure 6"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an interesting and timely problem, and the code release is a plus. The main concern is the asymmetry in the pass@1 comparison: if the direct baseline is not given the same repair loop, the headline result is not an attribution of the IR/RAG design. The RAG benchmark circularity and the unquantified RVFI effort further temper the claims. With a matched control and a fairer RAG benchmark, the paper would be much stronger; I believe these are achievable within a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: this is a genuinely useful systems paper, but read the 72.8% as end-to-end pipeline success, not one-shot generation accuracy, and not as a clean ablation of the IR/RAG design. The pass@1 definition includes the Verilator/riscv-formal repair loop, and the direct baseline is not described as receiving that same feedback.\n\nWhat is actually new: the integration. Combining a two-level IR, agent-guided graph RAG, and riscv-formal feedback into a single workflow and evaluating it on four embedded cores with five ISAXes is beyond what SCAIE-V and AGON did, even though the individual components are reused. The stepwise verification idea—check baseline, then extension—is sound. The paper ships code and reports synthesis numbers; that is real evidence. The limitation on RVFI is stated plainly in Section V.A, and future-work mentions automating RVFI insertion. Citation pattern is fair; the reuse of [31] and prior operator-based work is acknowledged.\n\nSoft spots, in proportion. The main one is measurement attribution. The reported pass@1 counts a complete LACE run, which includes iterative syntax and functional checks with feedback, while the non-agentic baseline is only described as direct generation. If the baseline gets no repair loop, the comparison measures the whole feedback architecture, not the two-level IR or the RAG. There is no count of LLM calls per successful run, and the term pass@1 is nonstandard for a multi-step trajectory. The fix is easy: give the baseline the same repair loop and report success after matched repair budgets. Second, n=9 per cell with no error bars makes differences like 44.4% versus 55.6% uninterpretable. Third, the RAG benchmark in Section VI-C builds its 50 queries from the system's own meta-prompts, so its MRR gain is partly self-referential. Fourth, the end-to-end automation claim is conditional on RVFI availability or manual instrumentation; that is an assumption the paper owns, but it means the \"automatic\" flow leaves out a nontrivial step.\n\nNone of these undermine the core demonstration. As a proof that a multi-agent LLM workflow with formal feedback can integrate real ISAXes into real cores, the paper holds up. The central quantitative claim just needs a matched control and more careful wording.\n\nThis paper is for people building LLM-assisted hardware design flows. I would send it to peer review, with a request for the repair-enabled baseline and error bars.","headline":"The 72.8% pass@1 is an end-to-end pipeline number, not a clean test of the IR/RAG design — but the integrated workflow is real and deserves peer review.","tokens_in":11880,"tokens_out":3329,"would_cite":true,"duration_ms":30611,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LACE claims that a team of LLM agents, guided by a two-level instruction IR and a graph-based RAG retriever, can turn a natural-language RISC-V instruction extension into localized RTL edits that pass formal verification, reaching an…","keywords":["RISC-V","instruction set extension","large language models","multi-agent systems","RTL generation","formal verification","retrieval-augmented generation","riscv-formal"],"falsifier":"Run the published workflow on an open-source core that has no RVFI support and do not add any instrumentation; if riscv-formal cannot check the new instruction without that manual effort, the end-to-end automation claim fails. Alternatively, rerun the four-core, five-instruction evaluation with nine runs per cell under the stated Verilator-plus-riscv-formal criterion and check whether the average pass@1 reproduces 72.8%.","tokens_in":10878,"feed_emoji":"🔧","tokens_out":7522,"duration_ms":62654,"temperature":0.7,"pith_summary":"LACE is a multi-agent workflow that claims to automate the slowest part of adding custom RISC-V instructions to an existing processor: turning a natural-language description of the new instruction into correct, localized edits of the processor's Register-Transfer Level (RTL) hardware code, and then proving the edit is right. The paper reports that across four embedded open-source cores and five instruction extensions, the workflow achieves an average 72.8% pass@1 rate, where a run counts as successful only if the modified core compiles with Verilator and passes riscv-formal checks for both the original instructions and the new one, while direct LLM generation succeeds only on the simplest case. The reason this matters is that ISA extensions are currently adapted per core by hand, so an automated, verification-closed pipeline could let workload experts propose instructions in plain language and get working hardware implementations across the RISC-V ecosystem.","feed_headline":"LLM agents add RISC-V instructions at 72.8% first-try pass rate","feed_subtitle":"Multi-agent workflow turns plain-language instruction ideas into verified RTL edits across four processor cores.","key_machinery":"The load-bearing mechanism is the two-level IR paired with an agentic retrieval-and-edit loop. Level one, operation-level IR, expresses an instruction's semantics with predefined interface operators (read instruction, read or write register, read or write PC, read or write memory, write rd) and arithmetic operators; this keeps the specification reusable across cores. Level two, HDL task-level IR, turns those operations into repository-specific edit instructions, guided by meta-prompts that encode common integration patterns and by architecture hints such as whether the pipeline needs flushing. The agentic RAG subsystem indexes modules and always-blocks as graph nodes with hierarchy and dataflow edges, so a supervisor can dispatch searches that return file paths and line ranges for precise edits. Verification closes the loop: riscv-formal with RVFI-based assertions replaces differential testing, and stepwise checking of baseline then extended behavior lets the feedback agent catch regressions before they propagate.","core_discovery":"The paper's central claim is that the abstraction gap between an ISA specification and HDL implementation can be bridged by a two-level intermediate representation plus retrieval-guided editing and formal-verification feedback. LACE first translates the natural-language ISAX intent into operation-level IR, small reusable operators such as reading a register, accessing the PC, or writing back to rd, which are microarchitecture-agnostic. A second translation maps these operations into concrete HDL modification tasks using core-specific hints about pipeline depth and control paths, so the same intent can be realized in very different cores. The agents locate the relevant code through an agentic graph-based retrieval system, edit the RTL, lint with Verilator, and then run riscv-formal in two stages: first on the unmodified baseline core, then on the core with the new instruction. With this loop, LACE reports an average pass@1 of 72.8% across PicoRV32, e203 hbirdv2, ibex, and cv32e40x, versus near-zero for non-agentic generation, and comparable area and frequency overhead to an expert-designed extension framework.","pith_inferences":["If the operator-level IR is as reusable as the paper suggests, it could serve as a portable front-end for ISAX proposals: one natural-language spec could feed standardization discussions and generate implementations across many cores at once, which the paper itself only gestures at.","The RVFI precondition means the reported automation is not end-to-end for cores without per-instruction trace hooks; a natural test is whether the same loop can be closed by automatically synthesizing RVFI instrumentation, which the paper lists as future work.","The same agentic RAG plus formal-checking loop could generalize beyond ISAX to other repository-scale RTL modifications, such as adding security hardening, changing memory interface timing, or migrating a core to new technology-specific primitives, though the paper only evaluates instruction extension.","A concrete extension would be to benchmark against human expert effort measured in wall-clock hours, not just pass rates and PPA, since the paper's value proposition is reduced manual labor but it does not directly measure that labor."],"forward_implications":["A natural-language instruction description can be reused across heterogeneous cores: the operator-level IR stays the same while the HDL task translation adapts to pipeline depth and control style.","Formal, compiler-agnostic verification means LACE does not need a custom compiler front-end or a trusted golden model for each new instruction, removing two bottlenecks of differential testing.","The stepwise baseline-then-extension check turns the integration process into an iterative loop where early errors are caught before they accumulate, which the paper shows reduces verification hops for complex extensions like load mul.","Because area and frequency overheads stay within about 10% for most instructions and match an expert framework on PicoRV32, the automated route is not obviously paying a large hardware cost for automation.","The harder cases are memory-coupled and long-latency instructions (sincos and load mul), where pass@1 drops to 44-67%, indicating that multi-cycle control and cross-module coordination remain the frontier."],"supporting_citations":[{"why":"Supplies the open-source SCAIE-V custom-interface framework that LACE compares against on PicoRV32 for area and frequency overhead.","marker":"[4]"},{"why":"Supplies the CoreDSL-based Longnail alternative whose per-core adaptation bottleneck LACE targets.","marker":"[5]"},{"why":"Provides the ReAct reasoning-and-acting technique that structures the LLM agents' tool use.","marker":"[6]"},{"why":"Source of the cv32e40x core used in evaluation and of the large-repository example motivating agentic RAG.","marker":"[22]"},{"why":"Provides the riscv-formal RVFI-based formal checks that define the pass@1 success criterion.","marker":"[24]"},{"why":"Provides Verilator compilation as the first-stage gate of the pass@1 criterion.","marker":"[29]"},{"why":"Supplies the graph-based HDL RAG backend that LACE adapts for repository localization.","marker":"[31]"},{"why":"PicoRV32 is one of the four evaluation cores and the host for the SCAIE-V comparison.","marker":"[36]"},{"why":"e203 hbirdv2 is the two-stage pipelined core used in the convergence-efficiency study.","marker":"[37]"},{"why":"ibex is one of the four evaluation cores.","marker":"[38]"}],"fun_headline_variants":["LLM agents spin up RISC-V instructions, 72.8% first-try","Multi-agent AI writes RISC-V RTL, hits 72.8% on first go","LACE: LLM agents convert specs to verified RISC-V RTL","AI agents automate RISC-V extension with 72.8% pass rate","From intent to RTL: LLM agents ace 72.8% on RISC-V"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The flow's end-to-end automation assumes the target core already exposes per-instruction execution traces (RVFI) or that someone will manually add those hooks, because the formal verification that defines success cannot run without them; the paper states this assumption explicitly.","fun_headline_variants_meta":{"raw":{"variants":["LLM agents spin up RISC-V instructions, 72.8% first-try","Multi-agent AI writes RISC-V RTL, hits 72.8% on first go","LACE: LLM agents convert specs to verified RISC-V RTL","AI agents automate RISC-V extension with 72.8% pass rate","From intent to RTL: LLM agents ace 72.8% on RISC-V"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001102,"raw_usage":{"total_tokens":4597,"prompt_tokens":947,"completion_tokens":3650,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":3538}},"tokens_in":563,"tokens_out":3650,"duration_ms":21084,"temperature":1.0,"reasoning_tokens":3538,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:54:59.646365+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the published workflow on an open-source core that has no RVFI support and do not add any instrumentation; if riscv-formal cannot check the new instruction without that manual effort, the end-to-end automation claim fails. Alternatively, rerun the four-core, five-instruction evaluation with nine runs per cell under the stated Verilator-plus-riscv-formal criterion and check whether the average pass@1 reproduces 72.8%.","supporting_citations":[{"cited_title":"Scaie-v: an open- source scalable interface for isa extensions for risc-v processors,","cited_arxiv_id":null,"evidence_quote":"Supplies the open-source SCAIE-V custom-interface framework that LACE compares against on PicoRV32 for area and frequency overhead."},{"cited_title":"Longnail: High-level synthesis of portable custom instruction set extensions for risc-v processors from descriptions in the open-source coredsl language,","cited_arxiv_id":null,"evidence_quote":"Supplies the CoreDSL-based Longnail alternative whose per-core adaptation bottleneck LACE targets."},{"cited_title":"riscv-formal,","cited_arxiv_id":null,"evidence_quote":"Provides the riscv-formal RVFI-based formal checks that define the pass@1 success criterion."},{"cited_title":"Verilator,","cited_arxiv_id":null,"evidence_quote":"Provides Verilator compilation as the first-stage gate of the pass@1 criterion."},{"cited_title":"Hdlxgraph: Bridging large language models and graph- based hdl representation,","cited_arxiv_id":null,"evidence_quote":"Supplies the graph-based HDL RAG backend that LACE adapts for repository localization."},{"cited_title":"Picorv32 – a size-optimized risc-v cpu core,","cited_arxiv_id":null,"evidence_quote":"PicoRV32 is one of the four evaluation cores and the host for the SCAIE-V comparison."},{"cited_title":"Hummingbirdv2 e203 core and soc,","cited_arxiv_id":null,"evidence_quote":"e203 hbirdv2 is the two-stage pipelined core used in the convergence-efficiency study."},{"cited_title":"Ibex: An embedded 32-bit RISC-V CPU core,","cited_arxiv_id":null,"evidence_quote":"ibex is one of the four evaluation cores."}],"review_version":1}