REVIEW 7 cited by
Comprehensive Verilog Design Problems: A Next-Generation Benchmark Dataset for Evaluating Large Language Models and Agents on RTL Design and Verification
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
We present the Comprehensive Verilog Design Problems (CVDP) benchmark, a new dataset and infrastructure to advance LLM and agent research in hardware design and verification. CVDP includes 783 problems across 13 task categories, covering RTL generation, verification, debugging, specification alignment, and technical Q&A authored by experienced hardware engineers. Problems are offered in both non-agentic and agentic formats. The benchmark introduces more realistic and challenging contexts than prior work, with state-of-the-art models achieving no more than 34% pass@1 on code generation. Agentic tasks$\unicode{x2013}$especially those involving RTL reuse and verification$\unicode{x2013}$are particularly difficult. Evaluation uses open-source tools and model scoring infrastructure, with comprehension tasks assessed via BLEU and LLM-based judging. CVDP reveals substantial gaps in current model capabilities, underscoring the need for continued research toward robust, real-world hardware design automation.
Forward citations
Cited by 7 Pith papers
-
CHIA: An open-source framework for principled, agentic AI-driven hardware/software co-design research
CHIA is an open-source framework for agentic AI-driven hardware/software co-design using CHIA loops as directed cyclic graphs, a tool library, and features for reliable experimentation, shown via five case studies.
-
Interpretable and Verifiable Hardware Generation with LLM-Driven Stepwise Refinement
Framework uses LLM-driven stepwise application of transformation rules to generate verifiable RTL hardware designs from specifications.
-
FinHardBench: Can LLMs Generate Latency-Aware Hardware for Financial Computing?
FinHardBench, 33 financial FPGA tasks, finds LLMs pass functional tests 19-61% of the time and produce many routed-but-wrong designs, while top models tune a 6-stage pipeline to optimal latency more reliably than thre...
-
Can AI Agents Really Complete RTL-to-GDS? Lessons from Benchmarking Tool-Interactive EDA Workflows
LLM agents can complete an RTL-to-GDS chip flow, but reliable completion depends on the execution infrastructure, not the foundation model alone.
-
When LLMs Over-Answer: Measuring and Mitigating Quality Issues in LLM-Based Hardware Description Language Question Answering
LLM answers to HDL questions are often redundant and verbose; a task-aware multi-agent framework cuts redundancy by 37% and padding by 31% while raising judge-based quality scores.
-
ChipVerilog: A Large-Scale OpenCores-Derived Benchmark for LLM-Based Verilog RTL Generation
On 64 large OpenCores-derived Verilog tasks, top LLMs reach 23.6% functional pass@1, 37.5% pass@5, and 0% on designs with two or more submodules, showing hierarchical RTL generation remains unsolved.
-
Revolution or Hype? Seeking the Limits of Large Models in Hardware Design
Large models can help early-stage hardware design and verification, but their reliability, data, and precision limits mean traditional EDA algorithms and formal verification remain necessary.
Discussion (0). Sign in to comment.