Pith. sign in

REVIEW 2 major objections 6 minor 2 cited by

SLDB: An End-To-End Heterogeneous System-on-Chip Benchmark Suite for LLM-Aided Design

T0 review · 2 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read SLDB benchmark tests LLMs on full-system chip integration, not just RTL blocks.

desk verdict SLDB is a genuinely useful system-level benchmark for LLM-aided SoC design, but the synthetic-library scalability claim rests on an untested correctness guarantee. read the letter →

arxiv 2507.06376 v1 pith:K5DYDIVX submitted 2025-07-08 cs.AR

classification cs.AR
keywords System-on-ChipLargeLanguageModelsLLM-aideddesignBenchmarksuiteHeterogeneousarchitecturesAcceleratorintegrationDMAhandshakeDesign-spaceexploration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces SLDB, a benchmark dataset for evaluating large language models on system-level design of heterogeneous system-on-chip architectures. Existing LLM hardware benchmarks concentrate on single RTL components, so they miss the integration, communication, and configuration constraints that dominate real SoCs. SLDB supplies ten complete, synthesized SoC designs built on the ESP platform, an open-source tile-based SoC design flow, plus a synthetic library that combines accelerators and tiles into exponentially many distinct configurations. A reported case study applies SLDB to four LLMs on accelerator-wrapper generation tasks, scoring functional correctness stage by stage. The result is a reusable resource for measuring whether LLM-aided design can move beyond individual blocks.

What carries the argument

The central object is the SLDB dataset itself, organized around the ESP tile-and-socket architecture for loosely coupled accelerators. The argument is carried by the four-stage Accelerator Processing Stages (APS) model, which turns system-level integration correctness into a granular, checkable chain: a wrapper is fully correct only if every stage completes in order. The synthetic library then scales the ten curated accelerators into an exponential family of SoC configurations, relying on the ESP socket interfaces to preserve correctness when tiles are combined.

What would settle it

Select an SoC configuration from Table III that is not one of the ten baselines, for example an 8x8 configuration with several distinct accelerators and a non-default cache or NoC bitwidth, synthesize it with the provided flow, and run the bundled testing application in simulation and on FPGA; a failure in any of the four accelerator processing stages would refute the scalability guarantee.

Watch

Extended reading notes

Core claim

The central claim is that SLDB provides a system-level counterpart to component-level RTL benchmarks, letting LLM performance be measured on tasks such as integrating an accelerator into an ESP SoC, generating DMA wrappers, extracting accelerator-specific configuration parameters, and producing driver and testing code. Each of the ten baseline designs is a full 2x2 tile-based SoC containing an Ariane RISC-V CPU tile, a memory tile, an auxiliary tile, and one loosely coupled accelerator from hardware benchmarks such as Koios and RTL-Repo, with post-synthesis performance, power, and area numbers reported. The synthetic library makes any combination of the ten accelerators with configurable CPU, memory, auxiliary tile, cache, NoC bitwidth, and SoC dimension available without modifying the integration code. The paper also demonstrates the suite with a fine-grain LLM evaluation in which integration wrappers are judged against the four accelerator processing stages (configuration, load, compute, store); a wrapper is fully functionally correct only if all four stages complete in order.

Load-bearing premise

The synthetic library's exponential-scaling promise depends on ESP's correct-by-construction guarantee for arbitrary tile combinations, for which the paper supplies no independent verification.

Editorial extensions

If this is right

  • LLM-aided design evaluation can move from pass rates on standalone Verilog components to functionally grounded, system-level correctness on complete SoCs.
  • The synthetic library lets future benchmarks be scaled by configuration rather than by hand-building each new SoC design.
  • Stage-by-stage scoring identifies where LLMs fail first, pointing to DMA handshake and state-transition logic as the next bottleneck for model improvement.
  • Because SLDB includes bare-metal and Linux software paths, LLM outputs can be evaluated end-to-end, from RTL wrapper to running application.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to use SLDB-generated wrappers as supervised training data for fine-tuning open models on DMA integration, since the failure analysis already pinpoints the error modes.
  • The synthetic library could enable a self-checking benchmark loop: generate a non-baseline SoC, simulate the testing application, and use pass or fail as automated feedback to the LLM, though this depends on the correctness assumption named below.
  • If the correct-by-construction guarantee for arbitrary tile combinations is not independently verified, SLDB's scalability claim would be limited to the ten validated baseline SoCs.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper introduces SLDB, a benchmark suite for evaluating LLMs on system-level SoC integration and configuration tasks in the ESP open-source platform. It contributes ten baseline 2x2 heterogeneous SoCs covering ten accelerators from RTL-Repo and Koios, with full integration wrappers, device drivers, system configurations, and testing applications; a synthetic library is claimed to generate exponentially many SoC variants; and a case study evaluates four LLMs (o1, o3-mini-high, GPT-4o, DeepSeek-R1) on wrapper and driver generation for these accelerators, reporting syntactic and functional pass rates as well as error breakdowns. The paper also discusses the design flow used to build the benchmark and the results of an in-context-learning experiment.

Significance. If the claimed scalability and correctness of the synthetic library hold, SLDB would be a valuable resource: it extends LLM evaluation beyond component-level RTL to realistic system integration, is built through real FPGA synthesis (Vivado 2023.2), includes diverse accelerators and full-stack communication parameters, and is open-sourced. The case study provides concrete baselines for LLM performance in DMA wrapper generation, with findings (high syntax success, low functional success) that are plausible and useful for the LLM-aided EDA community. The main limitation is the unsupported universal correctness guarantee for the synthetic library, which currently weakens the exponential-scalability claim.

major comments (2)
  1. [Section IV-D, Tables II and III] The paper claims that 'any SoC combination offers guaranteed correctness' for all configurations listed in Table III, but this claim is asserted, not demonstrated. Only the ten 2x2 baseline SoCs from Table II are synthesized and functionally validated, each containing a single accelerator with default configuration parameters. The configuration space in Table III (NoC/DMA bitwidth up to 1024, shared local memory up to 4096 KB, SoC dimensions up to 8x8, different CPU types, and cache settings) can alter address maps, DMA burst and packetization behavior, driver and linker assumptions, and FPGA resource feasibility; none of these combinations is generated, simulated, or synthesized in the paper. Before claiming an exponential synthetic library, the paper should (i) provide at least one generated and validated non-baseline configuration, or (ii) explicitly scope the correctness guarantee to ESP's socket/NoC integration and state that application-level functional correctness for arbitrary combinations must be verified by the user.
  2. [Section IV-C, Section IV-B (Step 4), Table II] The functional validation of the ten baseline SoCs is not reported in a way that supports the benchmark's use as ground truth. The text states that the designs were 'first evaluated in simulation, then synthesized and implemented,' and Table II lists only PPA metrics. No simulation tests, pass/fail criteria, or test counts are given. The description in Step 4 that 'we use randomized values to test the accelerators, since their functionality has already been verified' is ambiguous: it does not specify whether the test application compares accelerator outputs against golden values computed from a reference model or computed offline. Since the LLM evaluation's correctness labels depend entirely on these baselines being correct, the paper should provide explicit evidence that each baseline SoC passes its intended functional tests, along with a clear description of the testing methodology.
minor comments (6)
  1. [Section II, footnote 1] The phrase 'suggestedpass rates' is missing a space ('suggested pass rates').
  2. [Section V-B, Table IV] The sentence 'The pass@3 results consider non-overlapping successful design results across different LLM generation runs' is ambiguous; standard pass@k counts a task as successful if any of k independent runs passes, and the current phrasing could be misread as requiring distinct successful runs. Please clarify the exact computation.
  3. [Table I] The 'Code length' column does not specify the unit (lines, characters, or tokens). Please define it in the caption.
  4. [Table III] The row 'Multicast - (up to 16)' is unclear; specify the selected value in the baseline and the allowed range, just as is done for the other parameters.
  5. [Section V-B] The full prompt used for the LLM evaluation is deferred to the open-source release; since the paper already has an appendix, including the exact prompt there would improve reproducibility.
  6. [Figure 7] The caption says 'Number of correct designs per SoC type,' but the figure's y-axis scale is not described in the text; please state what the plotted values represent (e.g., total successes across all models and runs).

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: SLDB is a self-contained benchmark; self-citations to ESP reflect a normal platform choice, and the one load-bearing 'guaranteed correctness' assertion is an unverified assumption rather than a circular derivation.

full rationale

The paper's central contribution is a curated dataset (ten baseline SoCs plus a synthetic library) and an LLM case study. No derivation in the paper reduces to its own inputs: the baseline SoCs are built from external accelerators (RTL-Repo, Koios) and synthesized with Vivado; the LLM evaluation measures pass rates against human-validated integrations rather than fitting parameters to the benchmark. Self-citations to ESP [1] are used to describe the platform and to justify the synthetic-library scalability claim. This is a normal choice of infrastructure, and the cited ESP work is an open-source platform with an independent existence; it is not a 'uniqueness theorem' invoked to forbid alternatives. The one load-bearing assertion, Section IV-D's 'any SoC combination offers guaranteed correctness,' relies on ESP's correct-by-construction methodology and is not verified for non-baseline configurations; however, this is an unproven assumption or omitted proof rather than a circular step, because the claim is not equivalent to an input by construction and no fitted parameter is renamed as a prediction. Accordingly the circularity score is low (2) for self-citation without demonstrated circular reduction.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The benchmark's validity rests on the correctness of the ESP platform and of the 10 accelerators taken from prior work; both are assumed, not independently verified here.

assumptions (3)
  • domain assumption The ESP platform's correct-by-construction methodology guarantees functional correctness for any tile combination.
    Invoked in Section IV-D to support the synthetic library claim.
  • domain assumption The 10 accelerators from RTL-Repo and Koios are functionally correct as retrieved.
    Stated in Section IV-C: 'assumed correctness at the accelerator level.'
  • domain assumption LLM pass@k with k=3 independent runs is a sufficient evaluation metric.
    Used in Section V-B for the case study.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SLDB: An End-To-End Heterogeneous System-on-Chip Benchmark Suite for LLM-Aided Design." pith.science (2026). https://pith.science/paper/K5DYDIVX

@misc{pith2026250706376,
  author       = {Pith},
  title        = {Pith review of: SLDB: An End-To-End Heterogeneous System-on-Chip Benchmark Suite for LLM-Aided Design},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K5DYDIVX}},
  note         = {Machine review of arXiv:2507.06376}
}
read the original abstract

Over the last few years, Large Language Models (LLMs) have emerged as a valuable tool for Electronic Design Automation (EDA). State-of-the-art research in LLM-aided design has demonstrated the ability of LLMs to generate syntactically correct RTL code, showcasing encouraging prospects for integrating AI into the hardware design process. A key enabler of these advancements is the availability of high-quality benchmarks to evaluate new approaches. However, existing datasets and benchmarks fall short of system-level design, as they focus primarily on component-level information and low-complexity designs. To address this gap, we introduce the System-Level Design Benchmark (SLDB), a dataset tailored for evaluating LLMs in system-level integration and configuration tasks. SLDB includes a curated benchmark suite of 10 baseline SoC designs, whose components can be combined into an exponential number of distinct tile-based SoCs through a synthetic library. The dataset provides full SoC configurations, accelerator integration code, communication parameters, and accelerator-aware system configurations, along with testing-application code, compatible with the ESP platform[1].

Figures

Figures reproduced from arXiv: 2507.06376 by the authors.

Figure 1
Figure 1. The ESP tile-based architecture [1]. between performance and design effort. In the context of accelerator integration, these tradeoffs have resulted in two dominating paradigms: loosely coupled and tightly coupled accelerators [22]. Tightly coupled accelerators integrate directly with pro￾cessor cores and memory hierarchies, significantly reducing latency. However, this tight integration limits flexibility, com￾plic… view at source ↗
Figure 2
Figure 2. SLDB design flow. Steps marked with a black-filled circle require manual configuration. Steps marked with a white-filled [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. SLDB synthetic library functionality [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Distribution of errors per Accelerator Processing Stage for 3 LLM generation runs. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 6
Figure 6. Figure 6: categorizes the types of errors encountered for each LLM across all runs. Prominent error types include incorrect handling of DMA signals, DMA state transitions, and specifi￾cation adherence issues. Notably, advanced models like o1 and DeepSeek-R1 present fewer DMA sig…
Figure 7
Figure 7. Figure 7: Number of correct designs per SoC type handling complex DMA handshakes and state transitions, suggesting areas for targeted improvements [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Summary of correctly generated DMA stages, for 7 [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CHIA: An open-source framework for principled, agentic AI-driven hardware/software co-design research

    cs.AR 2026-06 unverdicted novelty 7.0 of 10

    CHIA is an open-source framework for agentic AI-driven hardware/software co-design using CHIA loops as directed cyclic graphs, a tool library, and features for reliable experimentation, shown via five case studies.

  2. GateTruth: Auditing the Rigor of RTL Design Benchmarks via Mutation Testing

    cs.AR 2026-08 conditional novelty 6.0 of 10

    A mutation-testing audit of RTL benchmark testbenches finds RTLLM v2.0 mostly below a 95% fault-kill floor and shows that an output-token cap can reorder a leaderboard.

Reference graph

Works this paper leans on

27 extracted references · 25 canonical work pages · cited by 2 Pith papers

  1. [1]

    Agile SoC development with open ESP,

    P. Mantovani, D. Giri, G. Di Guglielmo, L. Piccolboni, J. Zuckerman, E. G. Cota, M. Petracca, C. Pilato, and L. P. Carloni, “Agile SoC development with open ESP,” inProceedings of the International Conference on Computer-Aided Design (ICCAD), Nov. 2020

  2. [2]

    Invited: A modular digital VLSI flow for high-productivity SoC Design,

    B. Khailany, E. Krimer, R. Venkatesan, J. Clemons, J. S. Emer, M. Fo- jtik, A. Klinefelter, M. Pellauer, N. Pinckney, Y . S. Shao, S. Srinath, C. Torng, S. L. Xi, Y . Zhang, and B. Zimmer, “Invited: A modular digital VLSI flow for high-productivity SoC Design,” inDesign Automation Conference (DAC), pp. 1–6, June 2018

  3. [3]

    Rashinkar, P

    P. Rashinkar, P. Paterson, and L. Singh,System-on-a-chip Verification: Methodology and Techniques. Springer Science & Business Media, 2007

  4. [4]

    Chipyard: Integrated design, simulation, and implementation framework for custom SoCs,

    A. Amid, D. Biancolin, A. Gonzalez, D. Grubb, S. Karandikar, H. Liew, A. Magyar, H. Mao, A. Ou, N. Pemberton, P. Rigge, C. Schmidt, J. Wright, J. Zhao, Y . S. Shao, K. Asanovi´c, and B. Nikoli ´c, “Chipyard: Integrated design, simulation, and implementation framework for custom SoCs,”IEEE Micro, vol. 40, pp. 10–21, July/August 2020

  5. [5]

    Does SoC hardware development become agile by saying so: A literature review and mapping study,

    A. Rautakoura and T. H ¨am¨al¨ainen, “Does SoC hardware development become agile by saying so: A literature review and mapping study,” ACM Transactions on Embedded Computing Systems, vol. 11, pp. 1–27, June 2023

  6. [6]

    A scalable methodology for agile chip development with open-source hardware components,

    M. C. Dos Santos, T. Jia, M. Cochet, K. Swaminathan, J. Zuckerman, P. Mantovani, D. Giri, J. J. Zhang, E. J. Loscalzo, G. Tombesi, K. Tien, N. Chandramoorthy, J.-D. Wellman, D. Brooks, G.-Y . Wei, K. Shepard, L. P. Carloni, and P. Bose, “A scalable methodology for agile chip development with open-source hardware components,” inProceedings of the Internati...

  7. [7]

    Automatically improving LLM-based Verilog generation using EDA tool feedback,

    J. Blocklove, S. Thakur, B. Tan, H. Pearce, S. Garg, and R. Karri, “Automatically improving LLM-based Verilog generation using EDA tool feedback,” 2025

  8. [8]

    RTLFixer: Automatically fixing RTL syntax errors with Large Language Model,

    Y . Tsai, M. Liu, and H. Ren, “RTLFixer: Automatically fixing RTL syntax errors with Large Language Model,” inProceedings of Design Automation Conference (DAC), DAC ’24, 2024

Show all 27 references
  1. [9]

    Verilogcoder: Autonomous Verilog coding agents with graph-based planning and Abstract Syntax Tree (AST)-based waveform tracing tool,

    C.-T. Ho, H. Ren, and B. Khailany, “Verilogcoder: Autonomous Verilog coding agents with graph-based planning and Abstract Syntax Tree (AST)-based waveform tracing tool,”AAAI, vol. abs/2408.08927, 2025

  2. [10]

    RTLCoder: Outperforming GPT-3.5 in design RTL generation with our open-source dataset and lightweight solution,

    S. Liu, W. Fang, Y . Lu, Q. Zhang, H. Zhang, and Z. Xie, “RTLCoder: Outperforming GPT-3.5 in design RTL generation with our open-source dataset and lightweight solution,” in2024 IEEE International Workshop on LLM-Aided Design, 2024

  3. [11]

    Koios: A deep learning benchmark suite for FPGA architecture and CAD re- search,

    A. Arora, A. Boutros, D. Rauch, A. Rajen, A. Borda, S. A. Damghani, S. Mehta, S. Kate, P. Patel, K. B. Kent, V . Betz, and L. K. John, “Koios: A deep learning benchmark suite for FPGA architecture and CAD re- search,” in2021 31st International Conference on Field-Programmable ...

  4. [12]

    Benchmarking Large Language Models for au- tomated Verilog RTL code generation,

    S. Thakur, B. Ahmad, Z. Fan, H. A. Pearce, B. Tan, R. Karri, B. Dolan- Gavitt, and S. Garg, “Benchmarking Large Language Models for au- tomated Verilog RTL code generation,”Design, Automation & Test in Europe Conference & Exhibition (DATE), pp. 1–6, 2022

  5. [13]

    VerilogEval: evaluating Large Language Models for Verilog code generation,

    M. Liu, N. Pinckney, B. Khailany, and H. Ren, “VerilogEval: evaluating Large Language Models for Verilog code generation,” inInternational Conference on Computer-Aided Design (ICCAD), 2023

  6. [14]

    A dataset for Large Language Model- driven AI accelerator generation,

    M. Nazzal, D. Vungarala, M. Morsali, C. Zhang, A. Ghosh, A. Khreishah, and S. Angizi, “A dataset for Large Language Model- driven AI accelerator generation,”ArXiv, vol. abs/2404.10875, 2024

  7. [15]

    Gemmini: Enabling systematic deep-learning architecture evaluation via full-stack integration,

    H. Genc, S. Kim, A. Amid, A. Haj-Ali, V . Iyer, P. Prakash, J. Zhao, D. Grubb, H. Liew, H. Mao, A. Ou, C. Schmidt, S. Steffl, J. Wright, I. Stoica, J. Ragan-Kelley, K. Asanovic, B. Nikolic, and Y . S. Shao, “Gemmini: Enabling systematic deep-learning architecture evaluation vi...

  8. [16]

    MG-Verilog: multi- grained dataset towards enhanced LLM-assisted Verilog generation,

    Y . Zhang, Z. Yu, Y . Fu, C. Wan, and Y . C. Lin, “MG-Verilog: multi- grained dataset towards enhanced LLM-assisted Verilog generation,” inThe First IEEE International Workshop on LLM-Aided Design (LAD’24), 2024

  9. [17]

    RTLLM: An open-source benchmark for design RTL generation with Large Language Model,

    Y . Lu, S. Liu, Q. Zhang, and Z. Xie, “RTLLM: An open-source benchmark for design RTL generation with Large Language Model,” 2024 29th Asia and South Pacific Design Automation Conference (ASP- DAC), pp. 722–727, 2023

  10. [18]

    RTL-Repo: A benchmark for evaluating LLMs on large-scale RTL design projects,

    A. Allam and M. Shalan, “RTL-Repo: A benchmark for evaluating LLMs on large-scale RTL design projects,” in2024 IEEE LLM Aided Design Workshop (LAD), pp. 1–5, 2024

  11. [19]

    MetRex: A benchmark for Verilog code metric reasoning using LLMs,

    M. Abdelatty, J. Ma, and S. Reda, “MetRex: A benchmark for Verilog code metric reasoning using LLMs,”ArXiv, vol. abs/2411.03471, 2024

  12. [20]

    Kulal, P

    S. Kulal, P. Pasupat, K. Chandra, M. Lee, O. Padon, A. Aiken, and P. Liang,SPoC: search-based pseudocode to code. Red Hook, NY , USA: Curran Associates Inc., 2019

  13. [21]

    Domain-specific hardware accelerators,

    W. J. Dally, Y . Turakhia, and S. Han, “Domain-specific hardware accelerators,”Communication of the ACM, vol. 63, p. 48–57, June 2020

  14. [22]

    An analysis of accelerator coupling in heterogeneous architectures,

    E. G. Cota, P. Mantovani, G. Di Guglielmo, and L. P. Carloni, “An analysis of accelerator coupling in heterogeneous architectures,” in2015 52nd ACM/EDAC/IEEE Design Automation Conference (DAC), pp. 1–6, 2015

  15. [23]

    Invited: The case for embedded scalable platforms,

    L. P. Carloni, “Invited: The case for embedded scalable platforms,” in Proceedings of the 53th Annual Design Automation Conference (DAC), 2016

  16. [24]

    The cost of application-class processing: Energy and performance analysis of a linux-ready 1.7-ghz 64-bit risc- v core in 22-nm fdsoi technology,

    F. Zaruba and L. Benini, “The cost of application-class processing: Energy and performance analysis of a linux-ready 1.7-ghz 64-bit risc- v core in 22-nm fdsoi technology,”IEEE Transactions on Very Large Scale Integration (VLSI) Systems, vol. 27, pp. 2629–2640, Nov. 2019

  17. [25]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V . Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” inProceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, (Red ...

  18. [26]

    Efficient streaming language models with attention sinks,

    G. Xiao, Y . Tian, B. Chen, S. Han, and M. Lewis, “Efficient streaming language models with attention sinks,”arXiv, 2023

  19. [27]

    Language models are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. C...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.