REVIEW 7 cited by
RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Model
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Inspired by the recent success of large language models (LLMs) like ChatGPT, researchers start to explore the adoption of LLMs for agile hardware design, such as generating design RTL based on natural-language instructions. However, in existing works, their target designs are all relatively simple and in a small scale, and proposed by the authors themselves, making a fair comparison among different LLM solutions challenging. In addition, many prior works only focus on the design correctness, without evaluating the design qualities of generated design RTL. In this work, we propose an open-source benchmark named RTLLM, for generating design RTL with natural language instructions. To systematically evaluate the auto-generated design RTL, we summarized three progressive goals, named syntax goal, functionality goal, and design quality goal. This benchmark can automatically provide a quantitative evaluation of any given LLM-based solution. Furthermore, we propose an easy-to-use yet surprisingly effective prompt engineering technique named self-planning, which proves to significantly boost the performance of GPT-3.5 in our proposed benchmark.
Forward citations
Cited by 7 Pith papers
-
PDAGENT-BENCH: Characterizing, Grounding, and Architecting LLM Agents for VLSI Physical Design
PDAGENT-BENCH is a new benchmark suite with 353 curated problems and an agentic workflow framework for evaluating LLM/VLM agents across five capability dimensions in VLSI physical design.
-
Alpha-RTL: Test-Time Training for RTL Hardware Optimization
TTT-RTL performs per-design test-time RL on an LLM policy with EDA-derived PPA rewards and an adaptive KL controller, reducing geometric-mean PPA product by 65.1% on RTLLM v2.0 and ADP by 59.4% on an industrial FPU unit.
-
UVMarvel: an Automated LLM-aided UVM Machine for Subsystem-level RTL Verification
UVMarvel automatically constructs subsystem-level UVM testbenches for mainstream bus protocols using LLMs, an IR, and supporting libraries, reaching 95.65% average code coverage in 4.5 hours of automated runtime.
-
Verilog-Evolve: Feedback-Driven and Skill-Evolving Verilog Generation
Verilog-Evolve uses executable feedback from simulation, synthesis, timing, and GEMM metrics to refine LLM-generated Verilog and evolves skills across tasks, improving functional success and downstream hardware qualit...
-
From Concept to Practice: an Automated LLM-aided UVM Machine for RTL Verification
UVM^2 is an LLM-driven system that generates and refines UVM testbenches for RTL verification, reporting up to substantial time savings and average code/function coverage of 87.44%/89.58% on designs up to 1.6K lines, ...
-
HYPERHEURIST: A Simulated Annealing-Based Control Framework for LLM-Driven Code Generation in Optimized Hardware Design
HYPERHEURIST uses simulated annealing to refine functionally validated LLM-generated RTL designs, producing more stable PPA optimization than single-pass LLM generation across eight benchmarks.
-
Agentic Hardware Design as Repository-Level Code Evolution
HORIZON applies repository-level self-evolution to hardware design artifacts and reports 100% completion on ChipBench, RTLLM, Verilog-Eval, and nine CVDP categories using a hands-free agent loop.
Discussion (0). Sign in to comment.