{"id":"89a68658-f720-4464-8147-3e30209c7530","arxiv_id":"2507.06376","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"SLDB provides a benchmark for evaluating LLMs on full SoC integration, showing that current LLMs pass syntax checks but fail most functional integration tasks.","lead":"This paper introduces SLDB, a benchmark suite of ten complete System-on-Chip designs for testing how well large language models handle system-level hardware integration. It also reports a case study showing that top LLMs often produce syntactically correct code but fail at functional DMA handshakes and system configuration.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The synthetic library's 'guaranteed correctness' for every Table III configuration is the key unverified assumption: only ten fixed 2x2 baselines are tested, so the exponential scalability claim rests on an untested universal property of ESP combinations.","rationale":"The reader's weakest assumption is the same one I identify: the synthetic library's scalability claim rests on an unverified universal correctness guarantee. The paper's central contribution is the benchmark and its ability to scale beyond ten fixed designs, so this assumption is load-bearing. If it fails, the exponential design space is not usable as advertised, and LLM evaluation results on synthetic combinations could be measuring broken reference designs rather than LLM integration skill. I do not escalate to rejection because the ten baseline SoCs are real, synthesized, and reported in Table II, and the guarantee is testable in principle. The small number of LLM inference runs (three) and the deferred release of the prompt also weaken the case study, but those are secondary to the correctness of the reference designs. A conditional verdict is appropriate: the paper should supply direct evidence for at least several non-baseline combinations before the synthetic library is treated as a validated benchmark feature.","tokens_in":10035,"tokens_out":5506,"duration_ms":65515,"concrete_test":"Generate a non-baseline SoC via the synthetic library, e.g., a 4x4 configuration with all ten accelerator tiles, NoC/DMA bitwidth 1024, 4096 KB shared local memory, caches enabled, and an Ibex or SPARC CPU instead of Ariane. Simulate the provided testing application for each accelerator with golden outputs and run Vivado implementation for the smallest listed FPGA (e.g., xilinx-vc707). If any accelerator fails simulation or the design fails implementation, the Section IV-D universal correctness claim is false; if all pass, it provides missing evidence for the synthetic library's usability.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV-D asserts that 'any SoC combination offers guaranteed correctness' for all configurations in Table III, citing ESP's correct-by-construction design. This is the load-bearing bridge from the ten verified 2x2 baseline SoCs to the paper's central claim of an exponential synthetic library. The paper reports implementation metrics only for those ten baselines (Table II); no non-baseline combination is simulated, synthesized, or functionally tested. ESP's correct-by-construction property concerns socket and NoC integration for ESP-compatible tiles; it does not automatically extend to application-level functional correctness when system parameters vary over the ranges in Table III. Changing NoC/DMA bitwidth (32-1024), shared local memory size (64-4096 KB), cache configuration, CPU type, or tile dimensions can alter address maps, DMA burst and packetization behavior, driver and linker assumptions, and FPGA resource feasibility. The qualifying clause 'assuming a correctly integrated baseline accelerator and correct accelerator-specific configuration' is precisely what must be demonstrated across the claimed configuration space, not assumed. Without at least one generated, simulated, and synthesized non-baseline example, the universal guarantee is unsupported and the exponential scalability claim is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SLDB, a benchmark suite for evaluating LLMs on system-level SoC integration and configuration tasks in the ESP open-source platform. It contributes ten baseline 2x2 heterogeneous SoCs covering ten accelerators from RTL-Repo and Koios, with full integration wrappers, device drivers, system configurations, and testing applications; a synthetic library is claimed to generate exponentially many SoC variants; and a case study evaluates four LLMs (o1, o3-mini-high, GPT-4o, DeepSeek-R1) on wrapper and driver generation for these accelerators, reporting syntactic and functional pass rates as well as error breakdowns. The paper also discusses the design flow used to build the benchmark and the results of an in-context-learning experiment.","tokens_in":10243,"tokens_out":5917,"duration_ms":64623,"significance":"If the claimed scalability and correctness of the synthetic library hold, SLDB would be a valuable resource: it extends LLM evaluation beyond component-level RTL to realistic system integration, is built through real FPGA synthesis (Vivado 2023.2), includes diverse accelerators and full-stack communication parameters, and is open-sourced. The case study provides concrete baselines for LLM performance in DMA wrapper generation, with findings (high syntax success, low functional success) that are plausible and useful for the LLM-aided EDA community. The main limitation is the unsupported universal correctness guarantee for the synthetic library, which currently weakens the exponential-scalability claim.","major_comments":[{"comment":"The paper claims that 'any SoC combination offers guaranteed correctness' for all configurations listed in Table III, but this claim is asserted, not demonstrated. Only the ten 2x2 baseline SoCs from Table II are synthesized and functionally validated, each containing a single accelerator with default configuration parameters. The configuration space in Table III (NoC/DMA bitwidth up to 1024, shared local memory up to 4096 KB, SoC dimensions up to 8x8, different CPU types, and cache settings) can alter address maps, DMA burst and packetization behavior, driver and linker assumptions, and FPGA resource feasibility; none of these combinations is generated, simulated, or synthesized in the paper. Before claiming an exponential synthetic library, the paper should (i) provide at least one generated and validated non-baseline configuration, or (ii) explicitly scope the correctness guarantee to ESP's socket/NoC integration and state that application-level functional correctness for arbitrary combinations must be verified by the user.","section":"Section IV-D, Tables II and III"},{"comment":"The functional validation of the ten baseline SoCs is not reported in a way that supports the benchmark's use as ground truth. The text states that the designs were 'first evaluated in simulation, then synthesized and implemented,' and Table II lists only PPA metrics. No simulation tests, pass/fail criteria, or test counts are given. The description in Step 4 that 'we use randomized values to test the accelerators, since their functionality has already been verified' is ambiguous: it does not specify whether the test application compares accelerator outputs against golden values computed from a reference model or computed offline. Since the LLM evaluation's correctness labels depend entirely on these baselines being correct, the paper should provide explicit evidence that each baseline SoC passes its intended functional tests, along with a clear description of the testing methodology.","section":"Section IV-C, Section IV-B (Step 4), Table II"}],"minor_comments":[{"comment":"The phrase 'suggestedpass rates' is missing a space ('suggested pass rates').","section":"Section II, footnote 1"},{"comment":"The sentence 'The pass@3 results consider non-overlapping successful design results across different LLM generation runs' is ambiguous; standard pass@k counts a task as successful if any of k independent runs passes, and the current phrasing could be misread as requiring distinct successful runs. Please clarify the exact computation.","section":"Section V-B, Table IV"},{"comment":"The 'Code length' column does not specify the unit (lines, characters, or tokens). Please define it in the caption.","section":"Table I"},{"comment":"The row 'Multicast - (up to 16)' is unclear; specify the selected value in the baseline and the allowed range, just as is done for the other parameters.","section":"Table III"},{"comment":"The full prompt used for the LLM evaluation is deferred to the open-source release; since the paper already has an appendix, including the exact prompt there would improve reproducibility.","section":"Section V-B"},{"comment":"The caption says 'Number of correct designs per SoC type,' but the figure's y-axis scale is not described in the text; please state what the plotted values represent (e.g., total successes across all models and runs).","section":"Figure 7"}],"recommendation":"major_revision","confidential_remarks":"This is a useful benchmark contribution, and the case study is a reasonable demonstration of SLDB's use. The central concern is the synthetic library's 'guaranteed correctness' claim, which goes beyond what is evidenced; the revision should provide a concrete non-baseline validation or carefully scope the claim. The self-citation to ESP is appropriate given that SLDB is built on that platform. The title's 'End-To-End' is slightly stronger than the presented fine-grain LLM case study, but this is a minor framing issue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"SLDB is a real artifact, not just a proposal. The authors integrated ten accelerators into full ESP SoCs, ran simulation and Vivado synthesis, and report PPA numbers. That is the kind of groundwork too many benchmark papers skip. The system-level framing is genuinely new relative to RTLLM, VerilogEval, RTL-Repo, and Met-Rex, which stop at component-level RTL. Anyone working on LLM-assisted integration or configuration will find it useful.\n\nThe case study is a reasonable first use: four LLMs asked to produce DMA wrappers, driver code, and configuration parameters, scored by accelerator processing stages. The finding that syntactic correctness is high but functional correctness collapses at the DMA handshake is worth having, and the in-context learning appendix is a nice addition.\n\nThe soft spot is Section IV-D. The claim that 'any SoC combination offers guaranteed correctness' is the bridge from the ten verified baselines to the exponential synthetic library, and it is asserted rather than demonstrated. Only the ten 2x2 baselines were tested. No non-baseline combination is simulated or synthesized. ESP's correct-by-construction property legitimately covers socket and NoC integration for ESP-compatible tiles, but it does not automatically extend to application-level functional correctness when you vary NoC bitwidth from 32 to 1024 bits, shared local memory from 64 to 4096 KB, or cache and CPU configurations. Those parameters affect address maps, DMA burst behavior, driver assumptions, and FPGA feasibility. The qualifying clause about 'correctly integrated baseline accelerator and correct accelerator-specific configuration' is exactly what needs evidence across the configuration space. This is a real gap, but it is a section-level gap, not a fatal one: the ten baselines stand on their own, and the synthetic library can be reframed as a collection of plausible configurations rather than a proven universal guarantee.\n\nTwo smaller issues: pass@k is computed from three runs, which is thin, and the full prompt and scripts are deferred to the open-source release. Both are fixable in revision.\n\nI disagree with the circularity concern in the reader's report. Building on ESP is a legitimate domain choice, and self-citation is not a flaw when the platform is the actual integration substrate. The citation pattern looks fine.\n\nWho should read this: researchers in LLM-aided EDA, benchmark builders, and people evaluating AI-assisted hardware integration. It deserves a serious referee. My recommendation: send it to review, with a request that the authors either provide at least one generated, simulated, and synthesized non-baseline SoC combination or soften the universal correctness claim, and release the evaluation artifacts.","headline":"SLDB is a genuinely useful system-level benchmark for LLM-aided SoC design, but the synthetic-library scalability claim rests on an untested correctness guarantee.","tokens_in":10792,"tokens_out":2055,"would_cite":true,"duration_ms":21458,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SLDB benchmark tests LLMs on full-system chip integration, not just RTL blocks.","keywords":["System-on-Chip","Large Language Models","LLM-aided design","Benchmark suite","Heterogeneous architectures","Accelerator integration","DMA handshake","Design-space exploration"],"falsifier":"Select an SoC configuration from Table III that is not one of the ten baselines, for example an 8x8 configuration with several distinct accelerators and a non-default cache or NoC bitwidth, synthesize it with the provided flow, and run the bundled testing application in simulation and on FPGA; a failure in any of the four accelerator processing stages would refute the scalability guarantee.","tokens_in":9837,"feed_emoji":"🧩","tokens_out":8175,"duration_ms":78054,"temperature":0.7,"pith_summary":"The paper introduces SLDB, a benchmark dataset for evaluating large language models on system-level design of heterogeneous system-on-chip architectures. Existing LLM hardware benchmarks concentrate on single RTL components, so they miss the integration, communication, and configuration constraints that dominate real SoCs. SLDB supplies ten complete, synthesized SoC designs built on the ESP platform, an open-source tile-based SoC design flow, plus a synthetic library that combines accelerators and tiles into exponentially many distinct configurations. A reported case study applies SLDB to four LLMs on accelerator-wrapper generation tasks, scoring functional correctness stage by stage. The result is a reusable resource for measuring whether LLM-aided design can move beyond individual blocks.","feed_headline":"SLDB benchmark scores LLMs on full-chip integration, not just RTL code","feed_subtitle":"SLDB supplies ten complete SoC designs plus a synthetic library, and shows DMA handshakes are the main failure point.","key_machinery":"The central object is the SLDB dataset itself, organized around the ESP tile-and-socket architecture for loosely coupled accelerators. The argument is carried by the four-stage Accelerator Processing Stages (APS) model, which turns system-level integration correctness into a granular, checkable chain: a wrapper is fully correct only if every stage completes in order. The synthetic library then scales the ten curated accelerators into an exponential family of SoC configurations, relying on the ESP socket interfaces to preserve correctness when tiles are combined.","core_discovery":"The central claim is that SLDB provides a system-level counterpart to component-level RTL benchmarks, letting LLM performance be measured on tasks such as integrating an accelerator into an ESP SoC, generating DMA wrappers, extracting accelerator-specific configuration parameters, and producing driver and testing code. Each of the ten baseline designs is a full 2x2 tile-based SoC containing an Ariane RISC-V CPU tile, a memory tile, an auxiliary tile, and one loosely coupled accelerator from hardware benchmarks such as Koios and RTL-Repo, with post-synthesis performance, power, and area numbers reported. The synthetic library makes any combination of the ten accelerators with configurable CPU, memory, auxiliary tile, cache, NoC bitwidth, and SoC dimension available without modifying the integration code. The paper also demonstrates the suite with a fine-grain LLM evaluation in which integration wrappers are judged against the four accelerator processing stages (configuration, load, compute, store); a wrapper is fully functionally correct only if all four stages complete in order.","pith_inferences":["A natural extension is to use SLDB-generated wrappers as supervised training data for fine-tuning open models on DMA integration, since the failure analysis already pinpoints the error modes.","The synthetic library could enable a self-checking benchmark loop: generate a non-baseline SoC, simulate the testing application, and use pass or fail as automated feedback to the LLM, though this depends on the correctness assumption named below.","If the correct-by-construction guarantee for arbitrary tile combinations is not independently verified, SLDB's scalability claim would be limited to the ten validated baseline SoCs."],"forward_implications":["LLM-aided design evaluation can move from pass rates on standalone Verilog components to functionally grounded, system-level correctness on complete SoCs.","The synthetic library lets future benchmarks be scaled by configuration rather than by hand-building each new SoC design.","Stage-by-stage scoring identifies where LLMs fail first, pointing to DMA handshake and state-transition logic as the next bottleneck for model improvement.","Because SLDB includes bare-metal and Linux software paths, LLM outputs can be evaluated end-to-end, from RTL wrapper to running application."],"supporting_citations":[{"why":"Supplies the ESP platform, tile/socket architecture, and the correct-by-construction integration flow that SLDB is built on.","marker":"[1]"},{"why":"Provides deep-learning accelerators (LSTM, SPMV) adopted into SLDB, chosen for long-context complexity.","marker":"[11]"},{"why":"Provides the repository-level RTL accelerators (AES, SHA-256, SOBEL, FFT, FCDNN, SIMPLEDNN, CONVOLUTION) selected for SLDB.","marker":"[18]"},{"why":"Represents the component-level Verilog benchmark generation approach that SLDB argues is insufficient at the system level.","marker":"[13]"},{"why":"Is the component-level RTL benchmark that defines the baseline for LLM hardware design evaluation that SLDB extends.","marker":"[17]"},{"why":"Is a post-synthesis benchmark that motivates SLDB's use of synthesis-based performance metrics.","marker":"[19]"},{"why":"Defines the Ariane RISC-V core used in the CPU tile of every baseline SLDB SoC.","marker":"[24]"},{"why":"Supplies the task-decomposition methodology used to structure the fine-grain LLM evaluation.","marker":"[25]"}],"fun_headline_variants":["New benchmark tests LLMs on full SoC integration, not just RTL","SLDB dataset scales SoC benchmarks exponentially for LLM tests","LLM-aided design benchmark exposes DMA handshake failures","System-level SoC benchmark suite for LLM integration tasks","SLDB: system-level LLM benchmark reveals SoC integration pitfalls"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The synthetic library's exponential-scaling promise depends on ESP's correct-by-construction guarantee for arbitrary tile combinations, for which the paper supplies no independent verification.","fun_headline_variants_meta":{"raw":{"variants":["New benchmark tests LLMs on full SoC integration, not just RTL","SLDB dataset scales SoC benchmarks exponentially for LLM tests","LLM-aided design benchmark exposes DMA handshake failures","System-level SoC benchmark suite for LLM integration tasks","SLDB: system-level LLM benchmark reveals SoC integration pitfalls"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000736,"raw_usage":{"total_tokens":3292,"prompt_tokens":953,"completion_tokens":2339,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":2251}},"tokens_in":569,"tokens_out":2339,"duration_ms":18167,"temperature":1.0,"reasoning_tokens":2251,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:05:40.131476+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Select an SoC configuration from Table III that is not one of the ten baselines, for example an 8x8 configuration with several distinct accelerators and a non-default cache or NoC bitwidth, synthesize it with the provided flow, and run the bundled testing application in simulation and on FPGA; a failure in any of the four accelerator processing stages would refute the scalability guarantee.","supporting_citations":[{"cited_title":"Agile SoC development with open ESP,","cited_arxiv_id":null,"evidence_quote":"Supplies the ESP platform, tile/socket architecture, and the correct-by-construction integration flow that SLDB is built on."},{"cited_title":"Koios: A deep learning benchmark suite for FPGA architecture and CAD re- search,","cited_arxiv_id":null,"evidence_quote":"Provides deep-learning accelerators (LSTM, SPMV) adopted into SLDB, chosen for long-context complexity."},{"cited_title":"RTL-Repo: A benchmark for evaluating LLMs on large-scale RTL design projects,","cited_arxiv_id":null,"evidence_quote":"Provides the repository-level RTL accelerators (AES, SHA-256, SOBEL, FFT, FCDNN, SIMPLEDNN, CONVOLUTION) selected for SLDB."},{"cited_title":"VerilogEval: evaluating Large Language Models for Verilog code generation,","cited_arxiv_id":null,"evidence_quote":"Represents the component-level Verilog benchmark generation approach that SLDB argues is insufficient at the system level."},{"cited_title":"RTLLM: An open-source benchmark for design RTL generation with Large Language Model,","cited_arxiv_id":null,"evidence_quote":"Is the component-level RTL benchmark that defines the baseline for LLM hardware design evaluation that SLDB extends."},{"cited_title":"MetRex: A Benchmark for Verilog Code Metric Reasoning Using LLMs","cited_arxiv_id":"2411.03471","evidence_quote":"Is a post-synthesis benchmark that motivates SLDB's use of synthesis-based performance metrics."},{"cited_title":"The cost of application-class processing: Energy and performance analysis of a linux-ready 1.7-ghz 64-bit risc- v core in 22-nm fdsoi technology,","cited_arxiv_id":null,"evidence_quote":"Defines the Ariane RISC-V core used in the CPU tile of every baseline SLDB SoC."},{"cited_title":"Chain-of-thought prompting elicits reasoning in large language models,","cited_arxiv_id":null,"evidence_quote":"Supplies the task-decomposition methodology used to structure the fine-grain LLM evaluation."}],"review_version":1}