Pith. sign in

REVIEW 4 major objections 4 minor 39 references

The paper claims that LLM-generated answers to HDL questions systematically over-answer—burying correct content under redundant alternatives and verbose padding—and that a task-aware multi-agent refinement framework measurably fixes this, r

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 19:07 UTC pith:GAJMSMUV

load-bearing objection Useful over-answering measurement for HDL Q&A, but the mitigation's quality gains likely reflect judge bias more than substantive improvement. the 4 major comments →

arxiv 2607.17063 v2 pith:GAJMSMUV submitted 2026-07-19 cs.AI

When LLMs Over-Answer: Measuring and Mitigating Quality Issues in LLM-Based Hardware Description Language Question Answering

Ziteng Hu , Jiachi Chen , Wenhao Lv , Huan Zhang , Yingjie Xia This is my paper
classification cs.AI
keywords hardware description languageLLM question answeringover-answeringanswer redundancyanswer verbositymulti-agent frameworkLLM-as-Judgeanswer quality
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish a two-part claim: (1) LLM answers to hardware description language questions systematically over-answer—they contain correct content but bury it under redundant alternative solutions (65.7% of responses) and verbose padding (69.1%), and nearly half fail to fully cover the human expert's accepted answer; (2) a task-aware multi-agent refinement framework, which splits each answer into core and non-core parts, prunes redundant core candidates via a proposer–challenger–reflector debate, and compresses non-core content by task type, measurably fixes this over-answering. Across four backbone LLMs, the framework raises judged core quality from 3.71 to 4.67 and non-core quality from 3.72 to 4.23, while reducing core answer count by 37% and non-core length by 31%. If correct, this means concise, expert-style HDL answers can be produced without sacrificing substance, and the paper provides a reusable dataset and dual-view evaluation protocol for measuring communicative quality.

Core claim

On the paper's own terms, the central discovery is a quality profile gap: LLMs are not simply worse than human experts in HDL Q&A; they are more readable and structured but pervasively over-answer—65.7% of responses flag as redundant, 69.1% as verbose, and 49.0% do not fully align with accepted expert answers. The paper's constructive discovery is that this over-answering is correctable: a task-aware multi-agent pipeline that decomposes a draft into core answers and non-core content, debates the core candidates jointly to eliminate invalid and derivative variants, and task-specifically compresses or deletes non-core padding, increases judged core quality from 3.71 to 4.67 and non-core qualit

What carries the argument

The central mechanism is the explicit decomposition of a draft answer into two streams—core answers (independent blocks that directly address the question) and non-core units (background, restatement, summaries)—followed by two specialized operations: a multi-role debate among proposer, challenger, and reflector agents that examines all core candidates jointly so that subsumed or derivative solutions are pruned alongside incorrect ones, and a task-conditioned compressor that assigns each non-core unit keep, compress, or delete based on the inferred task type. Evaluation is dual-view: an LLM-as-Judge scores core quality, core count, content quality, and length, while accepted-answer-based met

Load-bearing premise

The load-bearing premise is that the LLM-as-Judge scores—validated on only 40 answers—capture true technical quality rather than stylistic preferences for concise, well-structured text, since the framework is explicitly built to produce exactly such text.

What would settle it

A concrete check: have a panel of human HDL experts, blinded to which answers were refined, re-score a random sample of 100 raw-vs-refined pairs on technical correctness and completeness only. If the refined answers do not win (or if human scores show no gain comparable to the judge's +0.96 core-quality lift), the mitigation claim is refuted. A second check: inspect a sample of pruned core candidates and deleted non-core units to see whether any contained information absent from the accepted answer; deletion precision near 100% would support the paper's claim, but any substantive deleted item

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If the results hold, LLM-based HDL assistants can be made substantially more concise without losing substantive content, reducing the reading burden on engineers by a measured 37% fewer core answers and 31% shorter non-core text.
  • The improvement is consistent across four different backbone LLMs and all four task categories, indicating the gains come from the structured decomposition and debate rather than from any single model.
  • The framework's reductions do not come at the cost of alignment: core consistency with accepted answers rises slightly after refinement, and deletion precision (97–98%) shows pruned content is absent from the expert reference.
  • The released 6,246-question dataset with expert-validated accepted answers and the dual-view evaluation protocol provide a reusable benchmark for communicative quality, complementing functional-correctness benchmarks.
  • Because generation and debugging tasks show the worst over-answering, targeted refinements for those categories may yield the largest practical gains for engineers.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The improvement may be partly an artifact of the evaluator: the LLM-as-Judge likely rewards conciseness and structure, so a blind human re-scoring of technical completeness alone could show smaller quality gains; this is an editorial inference beyond the paper's claims.
  • The core/non-core decomposition with accepted-answer references is portable to other expert Q&A domains (e.g., software engineering, legal or medical advice) where accepted-answer corpora exist, since over-answering appears to be a general LLM behavior; this is an editorial extension.
  • Because the paper uses consistency with accepted answers rather than simulation or synthesis as the correctness check, a stronger future test would verify refined answers' synthesizability where the question admits it; this is an editorial suggestion.
  • The 49% consistency gap indicates pruning alone may not fix solution-selection divergence; category-aware fine-tuning on accepted answers is a plausible next step, which the authors do not claim.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper studies 'over-answering' in LLM responses to hardware description language (HDL) questions. It constructs a dataset of 6,246 Stack Overflow HDL Q&A pairs with accepted answers, organizes them into a four-category taxonomy, and conducts a user study with 19 engineers. The study reports that LLM answers are redundant in 65.7% of cases, verbose in 69.1%, fail to fully cover the accepted answer in 49.0%, yet are preferred for readability in 58.3% of cases. The paper then proposes a task-aware multi-agent framework (M1 generation, M2 decomposition/debate/compression, M3 dual-view evaluation) and evaluates it on 363 questions across four LLMs, reporting average core-answer quality gains from 3.71 to 4.67 (+0.96), non-core content quality from 3.72 to 4.23 (+0.51), a 37% reduction in core-answer count, and a 31% reduction in non-core content length. The evaluation uses an LLM-as-Judge as the primary quality metric, with a 40-answer human validation, plus accepted-answer-based metrics (CC/CP/DP).

Significance. The problem is timely and important: HDL answers can directly affect hardware correctness, and 'over-answering' is a plausible and understudied failure mode. The released dataset of 6,246 expert-annotated HDL Q&A pairs and the taxonomy are useful resources for the community, and the paper's code release is a concrete strength. The user-study findings, if robust, are a meaningful empirical contribution. However, the central mitigation claim — that the multi-agent framework improves substantive answer quality — rests primarily on an LLM-as-Judge whose scoring criteria (concision, single-answer focus) coincide with the framework's design objectives. The accepted-answer metrics provide only partial, modest corroboration. The paper therefore needs additional independent evidence before the quality-improvement claim can be accepted.

major comments (4)
  1. [§V-B4 and Algorithm 1, Eq. (3)] The load-bearing claim that refinement raises core quality from 3.71 to 4.67 and non-core quality from 3.72 to 4.23 is measured by M3.1, an LLM-as-Judge. The judge's scoring likely rewards concise, single-solution, well-structured output — exactly what M2 is engineered to produce. The human validation of the judge is limited to 40 sampled answers and reports only MAE/Spearman/adjacent agreement; it does not separately validate judge behavior on unrefined vs. refined answers, and the distribution of human-vs-judge scores is not shown. The +0.96/+0.51 gains may therefore partly reflect judge stylistic preference rather than technical quality. I recommend reporting judge–human agreement separately for unrefined and refined outputs, or supplementing with a blinded human evaluation on a larger sample, and presenting CC/CP as co-primary outcomes.
  2. [§III-B3] The user study was not blinded: participants were shown which answer was the LLM's and which was the human expert's. Because redundancy, conciseness, and readability are subjective judgments, this can bias the headline percentages (65.7%, 69.1%, 49.0%, 58.3%). Although inter-annotator agreement on a subset is reported, the primary study uses a single annotator per question with identities revealed. A blinded or at least partially masked design would strengthen the descriptive claim. If full blinding is impractical, the authors should at least quantify the potential bias (e.g., by re-annotating a sample without revealing identities).
  3. [§V-C, Eq. (4)] The accepted-answer-based metrics do not independently establish that refinement improves technical quality. Core Consistency and Core Purity remain modest in absolute terms (48.1–59.4% and 55.3–65.5% after refinement) and improve by only +5 to +7 points. Deletion Precision is high (97–98%), but it only verifies that deleted content is absent from the accepted answer; it does not establish that the retained core is technically correct or complete. The paper explicitly acknowledges that CC/CP measure 'non-divergence from a human reference, not correctness per se.' This is an honest limitation, but it means the 'quality improvement' claim is not yet anchored by an independent correctness-oriented metric.
  4. [§VI-A, Table VI] The single-prompt ablation is reported for only two backbone models on a subset of questions, with no statistical significance tests or confidence intervals. Since the paper argues that the multi-agent structure is superior to a single redundancy-removal prompt, this comparison should be extended to all four backbones and the full evaluation set, with variance estimates. As written, the ablation is suggestive but not conclusive.
minor comments (4)
  1. [§III-B2] The description of the participant/question allocation is unclear: 19 questionnaire versions, 363 questions, each question evaluated by a single annotator, and 'balanced across the four categories' — please clarify how many questions each participant rated and how the versions were constructed.
  2. [§V-B4] The claim that M3 'matches or exceeds human–human agreement' would be easier to verify if the human–human agreement values were reported alongside the judge–human values. Please provide the human–human MAE/Spearman on the same 40 answers.
  3. [§III-C, Fig. 3] The figure's 'Readability' values (36%, 39%, 44%, 28%) appear inconsistent with the 58.3% overall LLM preference in Table II. Clarify whether these values are issue rates, preference rates, or something else, and adjust the caption/legend accordingly.
  4. [Abstract and §I] Minor wording issues: 'curate them into a dataset' and 'we release our dataset consists of' are ungrammatical. Also, the phrase 'with accepted answers' is used repeatedly where 'containing accepted answers' or 'with their accepted answers' would read more naturally.

Circularity Check

0 steps flagged

No significant circularity; evaluation biases are acknowledged validity threats, not equation-level reductions.

full rationale

The paper's claimed chain is empirical rather than derivational. The over-answering rates (65.7% redundancy, 69.1% verbosity, 49.0% non-coverage) come from a user study (Section III-B/III-C) and are observations, not consequences of a prior definition. The mitigation claim (3.71→4.67 core, 3.72→4.23 non-core) is an evaluation of the proposed pipeline, not a fitted prediction: no parameter is tuned to the judge, and the same judge scores both unrefined and refined answers (Section IV-D-1). The paper explicitly acknowledges that the LLM-as-Judge 'may favor concise, well-structured outputs' (Section VI-B1), which is a real internal-validity limitation, but a biased metric is not an equation that makes the improvement true by construction. The independent anchors—human validation on 40 answers (MAE 0.113/0.131, adjacent agreement 100%, Section V-B4) and accepted-answer CC/CP/DP metrics (Eq. 4; Tables IV-V)—provide out-of-sample checks, even if the absolute CC/CP levels are moderate. Self-citations [19,20] appear only in the security-optimization taxonomy cell and do not carry the load-bearing argument. No uniqueness theorem or prior-work ansatz is imported as a forcing argument. Thus the derivation is self-contained; the residual concerns are correctness/validity threats, not circularity.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

The central claims rest on the validity of the user-study measurements and the LLM-as-Judge; these are empirical protocols rather than mathematical axioms, but they carry domain assumptions about reference quality and metric validity. No new physical or formal entities are introduced; 'over-answering', core/non-core streams, and role-playing agents are methodological constructs grounded in the paper's user study, not postulated objects requiring independent evidence.

free parameters (1)
  • Maximum debate rounds = 3
    Set to 3 (citing [27]) with the Reflector deciding on disagreement; no sensitivity analysis is reported, so results may depend on this hand-chosen cap.
axioms (4)
  • domain assumption Stack Overflow accepted answers are a valid expert reference for answer quality
    Used as the ground truth for consistency (CC/CP) and as the 'human expert' answers in the user study; the paper concedes an accepted answer reflects one engineer's choice and may not be uniquely optimal (Section VI-B).
  • domain assumption LLM-as-Judge scores generalize from 40 human-validated answers to all 363 evaluations
    M3.1 is the primary quality metric; its validation is on 40 sampled answers with adjacent agreement 100%, but no breakdown by task/model and no error bounds are given (Section V-B4).
  • domain assumption Keyword-filtered Stack Overflow sample is representative of real HDL Q&A
    Dataset built by keyword filtering of a Stack Overflow snapshot; contamination from off-topic hits or LLM-generated posts is not analyzed (Section III-A).
  • domain assumption LLM-based task classification is reliable enough for downstream task-conditioned analysis
    Task taxonomy underlies the M1 profiler and category-specific compression; category-level accuracy is 93.94% with near-miss subcategory confusions, which is treated as sufficient (Section III-A4).

pith-pipeline@v1.3.0-alltime-deepseek · 16184 in / 15298 out tokens · 150526 ms · 2026-08-01T19:07:50.430737+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of When LLMs Over-Answer: Measuring and Mitigating Quality Issues in LLM-Based Hardware Description Language Question Answering." pith.science (2026). https://pith.science/paper/GAJMSMUV

@misc{pith2026260717063,
  author       = {Pith},
  title        = {Pith review of: When LLMs Over-Answer: Measuring and Mitigating Quality Issues in LLM-Based Hardware Description Language Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GAJMSMUV}},
  note         = {Machine review of arXiv:2607.17063}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

The rapid advancement of large language models (LLMs) has led practitioners to increasingly rely on them for answering questions about hardware description languages (HDLs). Because HDL is ultimately synthesized into physical hardware, an imprecise or redundant answer can propagate into timing violations or non-synthesizable logic that surface only late in the design flow, making the quality of HDL answers especially consequential. However, the quality of LLM-generated responses, particularly in comparison with answers provided by human experts, remains unclear. To investigate this question, we collect 6,246 HDL Q&A posts with accepted answers from Stack Overflow and curate them into a dataset, organized into a taxonomy of four main categories (Conceptual, Debugging, Generation, and Optimization) and ten subcategories. Using this dataset, we design a user study conducted with 19 HDL engineers with one to three years of experience. Our findings reveal a pervasive over answering tendency: LLMs supply correct content but bury it under redundant alternatives (65.7%) and verbose padding (69.1%), while nearly half of answers (49.0%) fail to fully align with expert answers yet participants still preferred LLM responses for readability (58.3%). Motivated by these findings, we propose a multi-agent framework for improving LLM-based HDL question answering. We evaluate answer quality using an LLM-as-Judge and two structural metrics: the number of core answers, which reflects redundancy since LLMs often provide multiple alternative solutions, and the length of non-core content, which reflects verbosity. Evaluated on the four mainstream LLMs, our framework increases the average core-answer quality score from 3.71 to 4.67 (+0.96) and the non-core content quality from 3.72 to 4.23 (+0.51), on a five-point scale.

Figures

Figures reproduced from arXiv: 2607.17063 by Huan Zhang, Jiachi Chen, Wenhao Lv, Yingjie Xia, Ziteng Hu.

Figure 1
Figure 1. Figure 1: An example of the quality gap between LLM-generated answers [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Pipeline for constructing and annotating the HDL Q&A dataset. We [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Task-dependent severity of LLM answer quality issues. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Pipeline of the multi-agent framework. M1 profiles the question (task type, intent, verification needs) and drafts an answer; M2 decomposes it into core and non-core content (M2.1), prunes redundant core candidates via multi-role debate (M2.2), and compresses non-core content by task type (M2.3); M3 scores the refined answer with an LLM-as-Judge (M3.1) and validates it against the accepted answer (M3.2). E… view at source ↗
Figure 5
Figure 5. Figure 5: Case study showing how multi-role debate prunes a redundant [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

39 extracted references · 1 canonical work pages

  1. [1]

    Ieee standard for verilog register transfer level synthesis,

    “Ieee standard for verilog register transfer level synthesis,”IEEE Std 1364.1-2002, pp. 1–108, 2002

    2002

  2. [2]

    Stack overflow is not dead yet: Crowd answers still matter,

    D. Helic and T. Santos, “Stack overflow is not dead yet: Crowd answers still matter,”Journal of Systems and Software, p. 112988, 2026

    2026

  3. [3]

    Assessing and advancing benchmarks for evaluating large language models in software engineering tasks,

    X. Hu, F. Niu, J. Chen, X. Zhou, J. Zhang, J. He, X. Xia, and D. Lo, “Assessing and advancing benchmarks for evaluating large language models in software engineering tasks,”ACM Transactions on Software Engineering and Methodology, 2025

    2025

  4. [4]

    A survey of research in large language models for electronic design automation,

    J. Pan, G. Zhou, C.-C. Chang, I. Jacobson, J. Hu, and Y . Chen, “A survey of research in large language models for electronic design automation,” ACM Transactions on Design Automation of Electronic Systems, vol. 30, no. 3, pp. 1–21, 2025

    2025

  5. [5]

    Verilogcoder: Autonomous verilog coding agents with graph-based planning and abstract syntax tree (ast)- based waveform tracing tool,

    C.-T. Ho, H. Ren, and B. Khailany, “Verilogcoder: Autonomous verilog coding agents with graph-based planning and abstract syntax tree (ast)- based waveform tracing tool,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 1, 2025, pp. 300–307

    2025

  6. [6]

    Rtl- coder: Fully open-source and efficient llm-assisted rtl code generation technique,

    S. Liu, W. Fang, Y . Lu, J. Wang, Q. Zhang, H. Zhang, and Z. Xie, “Rtl- coder: Fully open-source and efficient llm-assisted rtl code generation technique,”IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 44, no. 4, pp. 1448–1461, 2024

    2024

  7. [7]

    Verilogeval: Evaluating large language models for verilog code generation,

    M. Liu, N. Pinckney, B. Khailany, and H. Ren, “Verilogeval: Evaluating large language models for verilog code generation,” in2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD). IEEE, 2023, pp. 1–8

    2023

  8. [8]

    Rtllm: An open-source benchmark for design rtl generation with large language model,

    Y . Lu, S. Liu, Q. Zhang, and Z. Xie, “Rtllm: An open-source benchmark for design rtl generation with large language model,” in2024 29th Asia and South Pacific Design Automation Conference (ASP-DAC). IEEE, 2024, pp. 722–727

    2024

  9. [9]

    Betterv: controlled verilog generation with discriminative guidance,

    Z. Pei, H.-L. Zhen, M. Yuan, Y . Huang, and B. Yu, “Betterv: controlled verilog generation with discriminative guidance,” inProceedings of the 41st International Conference on Machine Learning, ser. ICML’24. JMLR.org, 2024

    2024

  10. [10]

    I. O. for Standardization, S. Technical Committee ISO/IEC JTC 1, Information technology. Subcommittee SC 7, and systems engineer- ing,Systems and Software Engineering: Systems and Software Quality Requirements and Evaluation (SQuaRE): System and Software Quality Models. ISO, 2011

    2011

  11. [11]

    Llm4eda: Emerging progress in large language models for electronic design automation,

    R. Zhong, X. Du, S. Kai, Z. Tang, S. Xu, H.-L. Zhen, J. Hao, Q. Xu, M. Yuan, and J. Yan, “Llm4eda: Emerging progress in large language models for electronic design automation,”arXiv preprint arXiv:2401.12224, 2023

    Pith/arXiv arXiv 2023

  12. [12]

    Palnitkar,Verilog HDL: a guide to digital design and synthesis

    S. Palnitkar,Verilog HDL: a guide to digital design and synthesis. Prentice Hall Professional, 2003, vol. 1

    2003

  13. [13]

    Ieee standard for verilog hardware description language,

    “Ieee standard for verilog hardware description language,”IEEE Std 1364-2005 (Revision of IEEE Std 1364-2001), pp. 1–590, 2006

    2005

  14. [14]

    Ieee standard for vhdl language reference manual,

    “Ieee standard for vhdl language reference manual,”IEEE Std 1076- 2019, pp. 1–673, 2019

    2019

  15. [15]

    Rtlfixer: Automatically fixing rtl syntax errors with large language models,

    Y . Tsai, M. Liu, and H. Ren, “Rtlfixer: Automatically fixing rtl syntax errors with large language models,” in2024 61st ACM/IEEE Design Automation Conference (DAC), 2024, pp. 1–6

    2024

  16. [16]

    Veridebug: A unified llm for verilog debugging via contrastive embed- ding and guided correction,

    N. Wang, B. Yao, J. Zhou, Y . Hu, X. Wang, Z. Jiang, and N. Guan, “Veridebug: A unified llm for verilog debugging via contrastive embed- ding and guided correction,” in2025 IEEE International Conference on LLM-Aided Design (ICLAD), 2025, pp. 61–67

    2025

  17. [17]

    Codev: Empowering llms with hdl generation through mul- tilevel summarization,

    Y . Zhao, D. Huang, C. Li, P. Jin, M. Song, Y . Xu, Z. Nan, M. Gao, T. Ma, L. Qi, Y . Pan, Z. Zhang, R. Zhang, X. Zhang, Z. Du, Q. Guo, and X. Hu, “Codev: Empowering llms with hdl generation through mul- tilevel summarization,”IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 45, no. 4, pp. 1893–1906, 2026

    1906

  18. [18]

    Nonblocking assignments in verilog synthesis, coding styles that kill!

    C. E. Cummingset al., “Nonblocking assignments in verilog synthesis, coding styles that kill!”SNUG (Synopsys Users Group) 2000 User Papers, 2000

    2000

  19. [19]

    Secv: Llm-based secure verilog generation with clue-guided exploration on hardware-cwe knowledge graph,

    F. Fan, Y . Xia, and L. Kuang, “Secv: Llm-based secure verilog generation with clue-guided exploration on hardware-cwe knowledge graph,” inProceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25, J. Kwok, Ed. International Joint Conferences on Artificial Intelligence Organization, 8 2025, pp. 8049–8057, main Tr...

    work page doi:10.24963/ijcai.2025/895 2025

  20. [20]

    Secfsm: Knowledge graph-guided llms for generating secure verilog code of finite state machine in socs,

    Z. Hu, Y . Xia, J. Chen, L. Kuang, and Y . Wan, “Secfsm: Knowledge graph-guided llms for generating secure verilog code of finite state machine in socs,”IEEE Transactions on Dependable and Secure Com- puting, pp. 1–15, 2026

    2026

  21. [21]

    Comprehensive verilog design problems: A next-generation benchmark dataset for evaluating large language models and agents on rtl design and verification,

    N. Pinckney, C. Deng, C.-T. Ho, Y .-D. Tsai, M. Liu, W. Zhou, B. Khailany, and H. Ren, “Comprehensive verilog design problems: A next-generation benchmark dataset for evaluating large language models and agents on rtl design and verification,”arXiv preprint arXiv:2506.14074, 2025

    Pith/arXiv arXiv 2025

  22. [22]

    Chatgpt outperforms crowd workers for text-annotation tasks,

    F. Gilardi, M. Alizadeh, and M. Kubli, “Chatgpt outperforms crowd workers for text-annotation tasks,”Proceedings of the National Academy of Sciences, vol. 120, no. 30, p. e2305016120, 2023

    2023

  23. [23]

    Chatgpt-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning,

    P. T ¨ornberg, “Chatgpt-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning,”arXiv preprint arXiv:2304.06588, 2023

    Pith/arXiv arXiv 2023

  24. [24]

    W. G. Cochran,Sampling techniques. john wiley & sons, 1977

    1977

  25. [25]

    Probable inference, the law of succession, and statistical inference,

    E. B. Wilson, “Probable inference, the law of succession, and statistical inference,”Journal of the American Statistical Association, vol. 22, no. 158, pp. 209–212, 1927

    1927

  26. [26]

    The measurement of observer agreement for categorical data,

    J. R. Landis and G. G. Koch, “The measurement of observer agreement for categorical data,”biometrics, pp. 159–174, 1977

    1977

  27. [27]

    Improving factuality and reasoning in language models through multiagent debate,

    Y . Du, S. Li, A. Torralba, J. B. Tenenbaum, and I. Mordatch, “Improving factuality and reasoning in language models through multiagent debate,” arXiv preprint arXiv:2305.14325, 2023

    Pith/arXiv arXiv 2023

  28. [28]

    Benchmarking large language models for auto- mated verilog rtl code generation,

    S. Thakur, B. Ahmad, Z. Fan, H. Pearce, B. Tan, R. Karri, B. Dolan- Gavitt, and S. Garg, “Benchmarking large language models for auto- mated verilog rtl code generation,” in2023 Design, Automation & Test in Europe Conference & Exhibition (DATE), 2023, pp. 1–6

    2023

  29. [29]

    Origen: Enhancing rtl code generation with code-to-code augmentation and self-reflection,

    F. Cui, C. Yin, K. Zhou, Y . Xiao, G. Sun, Q. Xu, Q. Guo, D. Song, D. Lin, X. Zhang, and Y . E. Liang, “Origen: Enhancing rtl code generation with code-to-code augmentation and self-reflection,” in2024 ACM/IEEE International Conference On Computer Aided Design (IC- CAD), 2024, pp. 1–9

    2024

  30. [30]

    What makes a good code example?: A study of programming q&a in stackoverflow,

    S. M. Nasehi, J. Sillito, F. Maurer, and C. Burns, “What makes a good code example?: A study of programming q&a in stackoverflow,” in2012 28th IEEE International Conference on Software Maintenance (ICSM). IEEE, 2012, pp. 25–34

    2012

  31. [31]

    Mining successful answers in stack overflow,

    F. Calefato, F. Lanubile, M. C. Marasciulo, and N. Novielli, “Mining successful answers in stack overflow,” in2015 IEEE/ACM 12th Working Conference on Mining Software Repositories. IEEE, 2015, pp. 430–433

    2015

  32. [32]

    Is stack overflow obsolete? an empirical study of the characteristics of chatgpt answers to stack overflow questions,

    S. Kabir, D. N. Udo-Imeh, B. Kou, and T. Zhang, “Is stack overflow obsolete? an empirical study of the characteristics of chatgpt answers to stack overflow questions,” inProceedings of the 2024 CHI Conference on Human Factors in Computing Systems, 2024, pp. 1–17

    2024

  33. [33]

    Metagpt: Meta programming for a multi-agent collaborative framework,

    S. Hong, M. Zhuge, J. Chen, X. Zheng, Y . Cheng, J. Wang, C. Zhang, S. Yau, Z. Lin, L. Zhouet al., “Metagpt: Meta programming for a multi-agent collaborative framework,” inInternational Conference on Learning Representations, vol. 2024, 2024, pp. 23 247–23 275

    2024

  34. [34]

    Agentverse: Facilitating multi- agent collaboration and exploring emergent behaviors,

    W. Chen, Y . Su, J. Zuo, C. Yang, C. Yuan, C.-M. Chan, H. Yu, Y . Lu, Y .-H. Hung, C. Qianet al., “Agentverse: Facilitating multi- agent collaboration and exploring emergent behaviors,” inInternational Conference on Learning Representations, vol. 2024, 2024, pp. 20 094– 20 136

    2024

  35. [35]

    Chateval: Towards better llm-based evaluators through multi- agent debate,

    C.-M. Chan, W. Chen, Y . Su, J. Yu, W. Xue, S. Zhang, J. Fu, and Z. Liu, “Chateval: Towards better llm-based evaluators through multi- agent debate,” inInternational conference on learning representations, vol. 2024, 2024, pp. 9079–9093

    2024

  36. [36]

    Verimind: Agentic llm for automated verilog generation with a novel evaluation metric,

    B. Nadimi, G. O. Boutaib, and H. Zheng, “Verimind: Agentic llm for automated verilog generation with a novel evaluation metric,”arXiv preprint arXiv:2503.16514, 2025

    Pith/arXiv arXiv 2025

  37. [37]

    Rtlsquad: Multi- agent based interpretable rtl design,

    B. Wang, Q. Xiong, Z. Xiang, L. Wang, and R. Chen, “Rtlsquad: Multi- agent based interpretable rtl design,”arXiv preprint arXiv:2501.05470, 2025

    Pith/arXiv arXiv 2025

  38. [38]

    Judging llm-as-a-judge with mt-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xinget al., “Judging llm-as-a-judge with mt-bench and chatbot arena,”Advances in neural information processing systems, vol. 36, pp. 46 595–46 623, 2023

    2023

  39. [39]

    G-eval: Nlg evaluation using gpt-4 with better human alignment,

    Y . Liu, D. Iter, Y . Xu, S. Wang, R. Xu, and C. Zhu, “G-eval: Nlg evaluation using gpt-4 with better human alignment,” inProceedings of the 2023 conference on empirical methods in natural language processing, 2023, pp. 2511–2522

    2023