{"id":"14ab2765-8b61-4aaa-b279-4ad0613e70f2","arxiv_id":"2412.20954","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LLM-generated nano-operator functions allow automated synthesis of customized out-of-order RISC-V processors from ISA documents, with simulated 2.35x average speedups over BOOM.","lead":"AGON is a new automated flow that uses a large language model to turn processor instruction descriptions into working out-of-order processor designs. It reports up to 2.35x average speedups over an expert-designed CPU on eight applications, with only a few lines of manual debugging per processor.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 2.35x speedup claim uses simulator features that Section V-E says are not implemented in the synthesized RTL, so the reported performance and area may describe different designs.","rationale":"The reader's weakest assumption was that AGON's cycle-accurate simulator is faithful for both AGON and BOOM. That is a valid calibration concern, but the manuscript's own Section V-E exposes a sharper, more specific problem: the simulator reports performance for optimization choices that the synthesis flow explicitly does not implement. This makes the performance and area numbers in Figure 8 refer to different microarchitectures, which is an internal inconsistency rather than a reliance on an unvalidated external baseline. The claim that the framework itself works, including LLM generation of nOP functions and the DSE machinery, is reasonably supported by the pass-rate data and the Pareto-front plots, and I would not reject the paper on this basis. However, the headline quantitative comparison to BOOM should be conditional on either (a) restricting reported cycles to the synthesizable optimization subset and re-plotting Figure 8, or (b) validating the simulated cycles against RTL simulation of the actually synthesized processor. The concrete test above settles which version of the claim is true. I partially agree with the reader because the simulator-faithfulness issue is real, but the more load-bearing and actionable concern is the synthesis/simulation scope mismatch, which the reader did not identify as the primary weakest assumption.","tokens_in":19014,"tokens_out":1614,"duration_ms":19614,"concrete_test":"Re-run the Section VII-C comparison after restricting the optimization search space to only the fusion behavior that Section V-E says synthesize() actually implements: fuse all AL-related nOPs within each instruction, support only instructions with at most two register inputs, and disable any cross-instruction fusion. Compare the resulting cycle counts and area against the currently reported 2.35x/79.5% numbers. If the restricted design is slower or larger than reported, the headline claim must be revised; alternatively, run RTL simulation of the synthesized processors on the same benchmarks and compare with the AGON simulator's cycle counts.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section V-E explicitly states: \"For ease of implementation, we temporarily only support fusing all AL-related nOPs within each instruction when synthesis. ... We defer the hardware implementation of other fusing strategies and instructions with three or more inputs for future work.\" Yet the headline comparison in Section VII-C relies on cycle counts from AGON's simulator after isa.auto_fuse and processor.auto_config, and Figure 8 reports cycles for auto_fuse(gain_th=0) designs. Those simulated designs can include cross-instruction fusion patterns and instructions with three or more register inputs, neither of which the current synthesize() primitive can turn into RTL. The area reported in Figure 8, by contrast, comes from synthesized RTL (Section VII-A: \"we implement our processor and synthesize the generated circuit to report its accurate area\"). The result is that the performance arm of the comparison is for a more aggressively optimized design than the area arm: the 2.35x speedup and 79.5% area cannot both be attributed to the same generated processor. This is an internal inconsistency between the experiment methodology and the stated synthesis capability, not merely a question of simulator calibration. Even if the simulator were perfectly accurate for the simulated microarchitecture, the simulated microarchitecture is not the one AGON generates. The framework contribution (nOP IR, LLM generation, DSE primitives) can still be valuable, but the central claim \"AGON-designed processors are 2.35x faster than BOOM at 79.5% area\" is not supported by the evidence as presented.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"AGON is a framework that takes RISC-V ISA extension documents and, with the help of an LLM and a custom intermediate representation called nOP functions, generates out-of-order processor RTL with minimal human intervention. The paper's contributions are an executable nOP-function IR, multi-level verification primitives (instruction, ISA, processor), scheduling/fusion primitives, a Bayesian DSE auto-configuration primitive, and an LLM-based few-shot generation flow with self-debugging and clustering. The headline experimental claim, in Section VII-C, is that AGON-designed customized processors are on average 2.35x faster than BOOM across 24 scenarios while using only 79.5% of BOOM's area. The paper also reports high LLM pass rates for nOP-function generation relative to direct Chisel generation, and lower reported human debugging effort than prior LLM-based CPU generation works.","tokens_in":19227,"tokens_out":5229,"duration_ms":56507,"significance":"If the performance and area claims are substantiated, AGON would be a meaningful step toward practical LLM-assisted processor customization: the nOP IR is a plausible mechanism for turning ISA documents into executable functional descriptions, the multi-level verification approach is sensible, and the reported pass@1 numbers (e.g., 95.9% for RV64I, and 18 extended instructions generated without human intervention) are concrete evidence that the IR improves LLM reliability over direct HDL generation. The comparison to BOOM is an ambitious and useful target for an automated flow. However, the central quantitative claim currently rests on an unvalidated internal simulator and on a synthesis path that implements only a subset of the simulated optimizations, which prevents the reader from attributing the reported speedup and area to the same generated design. These issues are fixable but require additional experiments or a significant recasting of the claims.","major_comments":[{"comment":"The headline comparison is internally inconsistent about what design is being measured. Section V-E states that synthesis 'temporarily only support[s] fusing all AL-related nOPs within each instruction' and that 'instructions with three or more inputs' are deferred to future work, yet Section VII-C uses isa.auto_fuse(gain_th=0) and processor.auto_config to produce the cycle counts in Figure 8, and auto_fuse(gain_th=0) is explicitly described in Section VII-C as fusing all nOPs into one OP. The simulated design can therefore include cross-instruction fusion patterns and instructions with three or more inputs, none of which the current synthesize() primitive can turn into RTL. The area reported in Figure 8, by contrast, comes from synthesized RTL per Section VII-A. Consequently, the 2.35x speedup and 79.5% area cannot both be attributed to the same processor as generated by the current framework. The authors should either implement the full fusion and multi-input instruction support in RTL, or report performance for the synthesized subset, and clearly label the two designs separately.","section":"§V-E, §VII-C"},{"comment":"The BOOM baseline is not independently measured. The paper states, 'Applying BOOM configurations, we use our simulator to report the performance of benchmark applications compiled to RV64I running on BOOM as the baseline.' Both the AGON cycle counts and the BOOM cycle counts therefore come from the same AGON cycle-accurate simulator, with no calibration against BOOM RTL simulation, FPGA prototyping, or silicon. If the simulator is optimistic for AGON's microarchitecture or pessimistic for BOOM's, the 2.35x figure becomes an artifact of the model. The authors should validate the simulator against actual BOOM RTL simulation on the RV64I baseline, and report the resulting error bars or correction factors before using it to support the central performance claim.","section":"§VII-A"},{"comment":"The area comparison mixes two different measurement methodologies. For AGON processors, Section VII-A says the generated circuit is synthesized with a commercial 65nm flow to obtain 'accurate area.' For BOOM, Section VII-C says the area limit is 'the area of BOOM estimated by McPAT.' McPAT estimates of BOOM's area are not the same as a synthesis result (McPAT uses analytical models and requires calibration), so the reported 79.5% area ratio is not a like-for-like comparison. The authors should synthesize BOOM with the same 65nm flow, or at least validate the McPAT estimate against a synthesis of BOOM, before reporting the area ratio as a headline result.","section":"§VII-C, §VII-A"},{"comment":"The performance comparison conflates ISA-extension gains with microarchitectural optimization. The AGON processors execute the domain-specific extended instructions (for example, SHA256/SHA512 and the packed-SIMD instructions in Table V), while the BOOM baseline is reported with the same benchmarks 'compiled to RV64I.' A large part of the speedup is then attributable to the added custom instructions themselves, not to the auto_fuse/auto_config tuning primitives that Section VII-C is intended to demonstrate. To isolate the contribution of AGON's optimization primitives, the authors should also report an AGON processor that runs the same benchmark compiled to RV64I only, or otherwise control for the ISA difference, so that the reader can see how much of the 2.35x comes from the microarchitecture flow rather than from the instruction set extensions.","section":"§VII-C"}],"minor_comments":[{"comment":"The pass@1 rates are reported for only three trials per instruction, and several instructions (sha512sig1, kslra32, clz32, kwmmul) show 0% pass@1. The text says AGON 'succeeds in generating 18 instructions without human intervention' but it is not clear exactly which instructions are counted and how the 0% cases interact with the debugging LoC in Table VII. Please clarify the definition of 'without human intervention' and report the number of trials or confidence intervals.","section":"§VI, Table VI"},{"comment":"The abstract and Section V-D use the phrase 'correctness guarantees,' but the described verification is test-based (test IOs and test programs), not a formal proof. Suggest softening 'guarantees' to something like 'multi-level verification' to match what is actually implemented.","section":"Abstract, §V-D"},{"comment":"The Pareto-front markers ('darker dots') are difficult to distinguish in the grayscale printing, and the figure does not indicate the number of simulated design points per benchmark. Please add a legend and report the size of the explored design space for each benchmark.","section":"Figure 7"},{"comment":"The experimental setup does not state the compiler used to generate the benchmark binaries, nor whether the same compiler and optimization flags were used for the BOOM RV64I baseline and the AGON extended-ISA binaries. Please specify the toolchain and flags, since this can materially affect cycle counts.","section":"§VII-A"},{"comment":"The DSE cost function in Equation 4 uses area(x) from the simulator (McPAT), while the final reported area comes from synthesis. The paper should state explicitly that the optimization target and the reported metric are produced by different estimation methods, and discuss the resulting discrepancy.","section":"§V-C"}],"recommendation":"major_revision","confidential_remarks":"The framework idea is genuinely interesting and the nOP IR is a credible contribution, but the central experimental claim currently rests on a comparison whose two arms are produced by different design representations and different estimation tools. The authors can address this either by extending the RTL flow to cover the simulated optimizations, or by limiting the performance claims to the synthesizable subset and calibrating the simulator against BOOM RTL simulation. If the latter route is taken, the paper would still be valuable as an automated design flow for a restricted but practical fusion scheme, though the headline would be substantially weaker than the current 2.35x claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: AGON is a real systems contribution. The nOP-function IR is a sensible answer to the fact that LLMs are bad at RTL but okay at describing instruction semantics. It lets the LLM write Pythonic dataflow functions, and AGON's compiler turns those into an executable ISA model and, eventually, BOOM-based RTL. The instruction/ISA/processor primitives, multi-level verification, and DSE hooks are coherent, and the pass-rate comparison against direct Chisel generation is credible evidence that the IR helps. For the framework alone, this paper deserves a serious referee.\n\nThe soft spot is the experimental claim. Both arms of the BOOM comparison are simulated in AGON's own simulator, with no calibration against BOOM RTL, FPGA, or silicon. That alone makes the headline number provisional. The stress-test concern is stronger and, reading the paper, it lands: Section V-E says the synthesized RTL only supports fusing AL-related nOPs within each instruction and defers instructions with three or more inputs, while Section VII-C's comparison uses isa.auto_fuse(gain_th=0) in the simulator, which can include cross-instruction fusion and three-input instructions. So the simulated performance in Figure 8 is for a design that synthesize() does not currently generate, while the area is from actual synthesized RTL. The 2.35x speedup and 79.5% area therefore describe different designs. This is an internal inconsistency, not just a calibration question. The authors are honest that they defer the unsupported optimizations, but then the paper should not present the simulated-only performance as the design's result.\n\nTwo smaller issues: per-benchmark auto-tuning against a single BOOM baseline conflates ISA-extension gains with microarchitecture quality, so the comparison reads as much as \"custom ISA beats general-purpose ISA\" as \"AGON beats BOOM.\" And no code or artifacts are released, so none of this can be checked independently.\n\nWho this is for: anyone tracking LLM-assisted hardware design. The framework is a credible stepping stone, and the writing is clear enough that the inconsistency is visible. Recommendation: send to peer review with a request for released code, simulator calibration against BOOM RTL, and a revised comparison that reports performance and area for the same synthesized design. Conditional accept would be reasonable after that.","headline":"A genuine framework contribution with a real IR idea, but the headline 2.35x/79.5% comparison is undermined by an uncalibrated simulator and a mismatch between what is simulated and what is synthesized.","tokens_in":19921,"tokens_out":2261,"would_cite":true,"duration_ms":24012,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AGON turns ISA documents into customized out-of-order processors that beat an expert-designed CPU by an average of 2.35×.","keywords":["processor design automation","LLM-based hardware generation","intermediate representation","nOP function","out-of-order processor","RISC-V customization","design space exploration","PPA optimization"],"falsifier":"Run the same eight benchmarks on BOOM's RTL (or an FPGA prototype) at the small, large, and giga configurations and compare the measured cycle counts with the cycle counts AGON's simulator reports; if the real BOOM cycles are substantially lower than simulated, the claimed 2.35× speedup would shrink or disappear.","tokens_in":18741,"feed_emoji":"⚙️","tokens_out":9016,"duration_ms":81470,"temperature":0.7,"pith_summary":"AGON is a framework for automatically turning natural-language instruction-set architecture (ISA) documents into working out-of-order (OoO) processor designs. The paper claims that, using an intermediate representation built from nano-operator functions (nOP functions), an LLM can generate instruction functionality accurately enough that a human only fixes a few lines of code per processor. AGON then schedules and optimizes the nOP operations into hardware, and its auto-tuning searches over microarchitecture configurations. The claimed result is that the resulting customized processors beat BOOM, an expert-designed general-purpose RISC-V CPU, by an average of 2.35× in performance while using about 79.5% of its area. This matters because it points toward a practical path from specifications to high-performance processors without months of manual RTL design.","feed_headline":"Auto-designed CPUs beat expert-designed BOOM by 2.35x","feed_subtitle":"LLMs plus a custom intermediate representation turn ISA documents into faster out-of-order processors.","key_machinery":"The load-bearing mechanism is the nano-operator function (nOP function) intermediate representation: a small fixed vocabulary of storage, program-counter, and arithmetic-logic operations combined as a dataflow graph. It makes instruction semantics expressible in a few lines of Python-like code, executable immediately for verification, and independent of how the hardware executes them. That decoupling is what lets AGON apply rule-based functional-equivalence transformations, fuse nOPs into functional units via instruction.fuse and isa.auto_fuse, and explore over 179 million microarchitecture configurations with processor.auto_config without touching the functional description. The same nOP graph is compiled into a cycle-accurate simulator and, for a subset of fusion strategies, synthesized into Chisel-based RTL that plugs into the BOOM core.","core_discovery":"The paper's central claim is that the bottleneck in LLM-based processor design is not the LLM itself but the representation it is asked to produce. AGON defines a nano-operator function (nOP function) intermediate representation: each instruction is written as a Python-like dataflow of small operations such as register reads, shifts, XORs, and PC updates. Because nOP functions are executable and hide port connections, sequential structures, and bit-width management, LLMs can generate them accurately, with a pass@1 of 95.9% over the 49 RV64I instructions and high success on most tested crypto and packed-SIMD extensions. AGON compiles these functions into a cycle-accurate out-of-order processor simulator, applies rule-based fusion and microarchitecture search, and synthesizes the result into RTL in the style of the BOOM core. Across eight domain-specific benchmarks and three BOOM configurations, the customized AGON processors are faster in 23 of 24 scenarios, with an average 2.35× speedup and an average area of 79.5% of BOOM's.","pith_inferences":["The functionality/optimization decoupling could extend beyond CPUs: if the nOP vocabulary were broadened, the same pipeline could generate accelerators or vector units, which share the HDL difficulties that trip up LLMs.","The current auto-tuning objective is cycle count times area; optimizing for energy or peak power would likely produce different Pareto fronts and could be evaluated with the same simulator and estimator.","Because nOP functions are executable and rule-based, they could support formal equivalence checking against later hand-written RTL, strengthening the correctness claim beyond simulation-based verification.","The 2.35× figure is tied to simulator fidelity; if the simulator were calibrated against BOOM RTL simulation or FPGA prototyping, the comparison would test how the advantage carries to real hardware."],"forward_implications":["A new instruction-set extension described in natural language can be prototyped as an out-of-order processor with only a handful of human-fixed lines; the paper reports average debug effort of 2 to 9.67 lines across its eight benchmark processors.","LLM-based generation becomes practical at processor scale: with the nOP representation, the LLM succeeded on 95.9% of RV64I instructions and on 18 of the more complex crypto and packed-SIMD instructions without human fixes.","PPA-aware auto-tuning produces a genuine design-space frontier: changing the fusion gain threshold shifts designs between high-performance/larger-area and lower-performance/smaller-area regions.","In area-constrained comparisons, AGON's customized processors beat the expert-written BOOM in 23 of 24 tested scenarios, including a 17.40× speedup on the matrix-multiplication benchmark under the small-BOOM area budget."],"supporting_citations":[{"why":"SonicBOOM: the expert-designed general-purpose CPU used as the baseline and the RTL core into which AGON's synthetic units are integrated.","marker":"[25]"},{"why":"RISC-V cryptography extension specification: supplies the SHA256/SHA512 instructions and ISA documents used as benchmark inputs.","marker":"[17]"},{"why":"RISC-V packed-SIMD extension specification: source of the matrix and math benchmark instructions AGON is asked to implement.","marker":"[41]"},{"why":"Nuclei software suite: provides six of the eight benchmark applications used in the evaluation.","marker":"[45]"},{"why":"Demonstrates few-shot learning in LLMs, the basis for the three-example prompting scheme used to generate nOP functions.","marker":"[42]"},{"why":"Supplies the self-debugging idea: syntax-check feedback is looped back to the LLM until the generated nOP function compiles.","marker":"[43]"},{"why":"Provides the sample-cluster-select mechanism used to pick a correct nOP function from multiple stochastic LLM outputs.","marker":"[44]"},{"why":"Chisel is the hardware construction language targeted by AGON's RTL synthesis stage.","marker":"[8]"}],"fun_headline_variants":["LLM + custom IR yield CPUs 2.35x faster than BOOM","AGON: auto-design CPUs that beat expert BOOM by 2.35x","CPU design automation: 2.35x speedup over BOOM with LLMs","From ISA docs to OoO CPUs: AGON's 2.35x win over BOOM","New framework turns ISA docs into 2.35x faster CPUs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that AGON's simulator reports BOOM's performance truthfully, because the paper measures the baseline with that simulator instead of running BOOM's actual RTL or physical hardware.","fun_headline_variants_meta":{"raw":{"variants":["LLM + custom IR yield CPUs 2.35x faster than BOOM","AGON: auto-design CPUs that beat expert BOOM by 2.35x","CPU design automation: 2.35x speedup over BOOM with LLMs","From ISA docs to OoO CPUs: AGON's 2.35x win over BOOM","New framework turns ISA docs into 2.35x faster CPUs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000618,"raw_usage":{"total_tokens":2918,"prompt_tokens":1048,"completion_tokens":1870,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":664,"completion_tokens_details":{"reasoning_tokens":1760}},"tokens_in":664,"tokens_out":1870,"duration_ms":10841,"temperature":1.0,"reasoning_tokens":1760,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:06:43.433643+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same eight benchmarks on BOOM's RTL (or an FPGA prototype) at the small, large, and giga configurations and compare the measured cycle counts with the cycle counts AGON's simulator reports; if the real BOOM cycles are substantially lower than simulated, the claimed 2.35× speedup would shrink or disappear.","supporting_citations":[{"cited_title":"Sonicboom: The 3rd generation berkeley out-of-order machine,","cited_arxiv_id":null,"evidence_quote":"SonicBOOM: the expert-designed general-purpose CPU used as the baseline and the RTL core into which AGON's synthetic units are integrated."},{"cited_title":"[Online]","cited_arxiv_id":null,"evidence_quote":"RISC-V cryptography extension specification: supplies the SHA256/SHA512 instructions and ISA documents used as benchmark inputs."},{"cited_title":"[Online]","cited_arxiv_id":null,"evidence_quote":"RISC-V packed-SIMD extension specification: source of the matrix and math benchmark instructions AGON is asked to implement."},{"cited_title":"[Online]","cited_arxiv_id":null,"evidence_quote":"Nuclei software suite: provides six of the eight benchmark applications used in the evaluation."}],"review_version":1}