Pith. sign in

REVIEW 5 major objections 5 minor 16 references

This paper claims that selecting 80% of a synthetic RTL corpus by a behavior-aware alignment score, calibrated on 10% of pairs and balanced by coverage and structural richness, yields a better fine-tuning set for hardware LLMs than the full

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 09:59 UTC pith:QQRYZEWT

load-bearing objection A useful, mostly sound paper on label-efficient curation of synthetic RTL corpora; the headline gains are plausible, but the behavior-aware score may be reading itself and needs an ablation before the central claim is fully established. the 5 major comments →

arxiv 2607.29283 v1 pith:QQRYZEWT submitted 2026-07-31 cs.AR cs.LG

RTLCurator: Label-Efficient Data Curation for RTL Generation

Siyang Cai , Cangyuan Li , Wenjing Chang , Kun Wang , Haoyu Gao , Yinhe Han , Ying Wang This is my paper
classification cs.AR cs.LG
keywords RTL generationdata curationsynthetic datafunctional verificationcontrastive learningactive learningalignment scorelarge language models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Training LLMs to write register-transfer level (RTL) code depends on large paired spec–code datasets, which are increasingly synthetic and mostly wrong; in two widely used synthetic corpora only about a quarter and a half of pairs pass functional tests. RTLCurator is a method for deciding which pairs to keep under a fixed budget, rather than which to discard. It scores every pair for behavioral alignment using an encoder trained to contrast correct implementations against implementations that fail simulation, corrects that score with a small number of actively validated pairs, and then distributes the retention budget across score levels, representation clusters, and structural complexity. On CodeV and RTLCoder, keeping 80% of the pool this way outperforms fine-tuning on the full corpus across all reported metrics, while validating only 10% of pairs; ranking purely by the score or filtering by full-pool simulation does not. The paper's case is that behavioral correctness is a better input signal for curation than it is a filtering criterion.

Core claim

The central claim is that a compatibility prior learned from source-side functional failures carries enough behavioral signal that, after calibrating on at most 10% of a new corpus, it orders the entire pool for curation. RTLCurator trains an encoder by progressively contrasting each specification with randomly paired RTL, with same-cluster cross-specification RTL, and finally with same-specification implementations that fail simulation, producing a cosine-similarity score q for every pair. A lightweight calibrator, fit on actively chosen validated pairs with at most B labels, turns q into an alignment score p for all records. The retained subset is then built by allocating the budget k acro

What carries the argument

The paper's load-bearing machinery is a three-stage contrastive encoder (retrieval initialization, in-cluster cross-spec negatives, and same-spec functional-failure negatives) that yields a compatibility prior q(s,r) = cosine similarity of the spec and RTL embeddings; a budgeted active-learning loop that validates at most B pairs and fits a small MLP calibrator to produce an alignment score p; and a staged selection policy that distributes the retention budget across score strata, fixed clusters over the preprocessed spec embeddings, and a structural-richness bonus computed from token, line, port, control, FSM, and clock features. The encoder supplies a corpus-wide ordering before any target

Load-bearing premise

The load-bearing assumption is that a compatibility prior trained on the source corpus's functional-failure modes, after calibration with only 10% of target labels, orders the whole target corpus by true behavioral alignment; if the source failure types and specification styles are not representative, or if the small label set is noisy, the ranking can misorder the unvalidated majority.

What would settle it

Take a synthetic RTL corpus with full simulation labels for every pair, run RTLCurator's 1%-budget calibration, and measure the agreement between its ranking and the oracle ranking (e.g., rank-correlation between p and the true pass/fail over the full pool). If the correlation at 1% labels is close to zero or the top 1,000 records' accept rate does not separate from the bottom 1,000, the paper's label-efficiency claim is falsified. Alternatively, compare the 80% subset's downstream pass rate to an oracle that exhaustively searches small subsets for the best fine-tuning set; a large gap would i

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Keeping 80% of a noisy synthetic RTL corpus with RTLCurator improves downstream pass rates over using the full corpus, so curators need not discard the majority of generated data to gain quality.
  • A single trained alignment encoder transfers across synthetic corpora with only 10% target labels, suggesting one curated model can serve multiple data pipelines.
  • Score-only ranking underperforms random selection, so any deployment of behavior-aware scores must also preserve coverage and structural diversity in the retained subset.
  • Full-pool simulation filtering yields no better downstream performance, so label-efficient validation can replace exhaustive functional testing for curation purposes.
  • The non-monotonic budget sweep (peak at 80%, decline at 90%) implies that optimal retention is a property of the corpus and the selection policy, not a fixed rule.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct extension is that synthetic RTL pipelines could deliberately over-generate and then curate, rather than trying to generate correctly the first time; the curation step absorbs the noise.
  • The calibration-by-small-validation-set mechanism might transfer to other code-generation domains where a compatibility score from one distribution is available and ground-truth alignment is expensive, such as general-purpose instruction data.
  • Because the paper isolates the contributions of coverage versus structural richness as future work, a natural test is to ablate each component on a third corpus to see whether the balancing term or the richness term carries most of the gain.
  • The large gains on RTLLM's design-level problems relative to module-level benchmarks hint that the biggest benefit of faithful curation appears when the corpus contains many larger, harder designs, which correctness-based filtering would otherwise prune.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. This paper presents RTLCurator, a framework for curating synthetic RTL datasets under a validation budget B and a retention budget K. It trains a spec–RTL compatibility encoder on a source corpus using contrastive negatives that include same-specification functional failures, then applies the frozen encoder to a target corpus. To bridge distribution shift, it rewrites each target specification into an auxiliary 'source-style' specification using the original spec and the candidate RTL (Eq. 2). A small validation set (≤10% of the pool) is actively acquired and used to calibrate a scorer; the final subset is selected by combining the calibrated alignment score with representation coverage and RTL structural richness. On CodeV and RTLCoder, retaining 80% of the corpus with RTLCurator is reported to outperform full-corpus fine-tuning on all ten metrics on VerilogEval and RTLLM, while score-only selection and full-pool simulation filtering do not.

Significance. The paper addresses a real and under-studied problem: cheap synthetic RTL corpora contain large fractions of behaviorally wrong pairs, and simulation-based filtering is expensive and biased toward short modules. The proposal to spend a small validation budget on calibrating a corpus-wide score and to select by balancing quality, coverage, and structural richness is plausible, and the target benchmarks are external, with benchmark-duplicate pairs removed. The empirical pattern across two corpora and ten metrics is striking. However, the current evidence does not isolate the behavior-aware score as the causal ingredient: the auxiliary-spec rewrite is conditioned on the candidate RTL, the selection policy is never ablated without the score, and no variance or sensitivity analysis is reported. The method's practical value would be well established if these gaps are addressed.

major comments (5)
  1. [Method — Training the Compatibility Prior (Eq. 2)] The auxiliary specification e_si = R(s_i, r_i) is generated from both the original spec and the candidate RTL, and the score q_i is then computed against the same r_i. A wrong-but-coherent RTL will tend to produce an auxiliary spec that describes the wrong behavior, so q_i can be high even when the original spec and RTL disagree. The fact that score-only selection (Align. top) is no better than Random in Table 1 is consistent with this confound. Because the behavior-aware score is a central contribution, please add (i) an ablation that generates e_si from s_i alone or otherwise masks r_i during rewriting, and (ii) a selection-policy ablation that removes the alignment score while keeping the coverage and structural components. Without these, the causal role of the learned score is not established.
  2. [Experiments — Table 1, Figure 3] All headline results are five-run means with no measures of uncertainty. Several of the key gaps are small (e.g., RTLCoder VerilogEval Machine pass@1: 66.24 vs 64.20 for Random; Human pass@1: 40.42 vs 39.71). Please report per-run values, standard deviations or confidence intervals, and a paired significance test across the five runs (or increase the number of runs). This is necessary for the paper's central claim that RTLCurator 'improves on training with the full corpus across all reported metrics.'
  3. [Experiments — Table 3 (Full-V al)] The comparison with Full-V al is not retention-matched: Full-V al keeps only the ~20% of RTLCoder that passes simulation, while RTLCURATOR retains 80%. The claim that 'filtering the whole pool by simulation does no better' therefore conflates subset quality with retention budget. Please add a matched-budget control (e.g., Full-V al augmented to the same 80% budget, or RTLCURATOR at 20% retention) so the two policies are compared at equal retention.
  4. [Experiments — Figure 5 (Calibration)] The top–bottom accept-rate gap is measured on records ranked by a calibrator that was fitted on the same B labels used to compute the gap. This is an in-sample diagnostic and does not show how well the score generalizes to unlabeled pairs. Please evaluate the gap on a held-out queried set or via cross-validation over the queried pairs, and report calibration performance on records not used for fitting.
  5. [Method — Retention-Budgeted Selection Policy] The pipeline contains many free design parameters (margin µ in Eq. 3, stage weights λ in Eq. 5, cluster count C, structural weights w in Eq. 8, strata quotas, cell bonus, acquisition-route quotas, calibration evidence weights η_i). None are reported, and no sensitivity analysis is given. To rule out tuning on the target pools or benchmarks, please provide the chosen values and a sensitivity sweep for the most influential parameters.
minor comments (5)
  1. [Experiments — setup] The technical appendix is referenced but not present, and two section references appear as empty 'Sec.'. Please include the appendix and fix the cross-references.
  2. [Method — Eq. (2)] Specify the rewriting model R used in Eq. (2) and the prompt/template; the current text simply calls it 'the rewriting model.'
  3. [Figure 4] The caption contains a stray '- ator' artifact. Also clarify whether the functional-proxy ranking on OriGen uses original or rewritten specifications.
  4. [Experiments — duplicate removal] Please describe how benchmark-duplicate pairs are detected before removal (string match, embedding similarity, etc.), since the quality of this step affects the leakage argument.
  5. [Experiments — baselines] For DS2 and LESS, report the hyperparameters and whether their selection uses any target labels or reference examples; this matters for interpreting their lower performance.

Circularity Check

0 steps flagged

No construction-level circularity; central result is benchmark-external, though Eq. (2) introduces a score-leakage correctness risk.

full rationale

The headline result is not forced by construction. Retention subsets are evaluated by fine-tuning on VerilogEval and RTLLM, and the paper removes pairs that duplicate these problems; the source encoder is trained on a disjoint VeriCoder/OriGen split; calibration labels are a 10% subset of CodeV/RTLCoder and are not the evaluation labels; the full-pool simulation baseline in Table 3 is an independent external check. The one genuinely self-referential part is Eq. (2), where e_si = R(s_i,r_i) is generated from the very RTL being scored, so q_i = cos(f(e_si), f(r_i)) can partly measure self-consistency rather than true spec-RTL alignment. This is a real leakage/correctness risk for the claim that the score is behavior-aware, and the paper should ablate it (e.g., score with the original s_i or withhold r_i from the rewrite). It is not, however, a construction-level circularity in the derivation of the downstream result: the final metrics are external, the score-only policy underperforms, and the retained subset quality is judged by independent pass@k, not by the score itself. The only overlapping-author citation (Chang et al. 2024, DAC) is background for data scarcity/augmentation and is not load-bearing; no uniqueness theorem or ansatz is imported from prior work by the same authors. Therefore the circularity score is low, with the caveat that the Eq. (2) leakage should be treated as a correctness risk.

Axiom & Free-Parameter Ledger

8 free parameters · 4 axioms · 0 invented entities

The central result rests on the reliability of source-side simulation labels, the transferability of the learned prior, and the accuracy of a small set of target validation labels. The selection policy contributes several unspecified free parameters (cluster count, structural weights, strata quotas, calibration weights) that are not reported in the main text.

free parameters (8)
  • Margin μ (Eq. 3)
    Contrastive margin in the curriculum loss; value not reported.
  • Curriculum stage weights λ (Eq. 5)
    Weights for retrieval replay, in-cluster, and functional contrastive losses; values not reported.
  • Cluster count C
    Fixed clusters for coverage routing; number not reported.
  • Structural richness weights w (Eq. 8)
    Fixed weighted combination of token/line/port/control/FSM features; weights not specified.
  • Strata quotas and cell bonus
    Selection policy distributes the retention budget over strata and clusters and adds a quantile-based structural bonus; exact shares and bonus magnitude not specified.
  • Calibration evidence weights η_i (Eq. 7)
    Per-sample confidence weights in calibrator loss; not specified.
  • Acquisition route quotas
    Five routing routes divide each batch; exact quotas not specified.
  • Validation revision rounds R_tb
    Maximum testbench revision rounds in validation protocol; not specified.
axioms (4)
  • domain assumption Source-side simulation and testbench validation correctly identify verified pairs and functional failures used to train the prior.
    Stage 2B negatives are defined by source-side functional validation; noisy labels would distort the compatibility prior.
  • domain assumption The rewritten specification e_s = R(s_i, r_i) preserves the behavioral contract of s_i for the encoder pathway.
    Eq. (2) rewrites target specs in source-corpus style; if the rewrite drops behavioral constraints, the embedding geometry and clustering are distorted.
  • domain assumption Target validation labels are accurate, including LLM-judge or expert adjudication of unresolved cases.
    The calibrator in Eq. (7) directly depends on these binary labels; errors propagate into the Alignment Score.
  • domain assumption VerilogEval and RTLLM are valid proxies for downstream utility J in Eq. (1).
    The retained subset is optimized for SFT utility; benchmark pass rates are used to measure that utility.

pith-pipeline@v1.3.0-daily-deepseek · 14101 in / 13569 out tokens · 136757 ms · 2026-08-03T09:59:09.049510+00:00 · methodology

0 comments
read the original abstract

Training large language models (LLMs) to write register-transfer level (RTL) requires large corpora of paired specifications and code, and such data is scarce enough that most public corpora are now synthesized. Synthesis provides scale but not correctness, and in two widely used RTL datasets only 24.4% and 53.5% of pairs pass generated functional tests. This raises the question of how much of such a corpus to keep and which part of it. Correctness alone is a poor answer. A pair that misbehaves in one corner case still shows valid syntax and interface conventions, and complex sequential designs are both harder to generate and harder to validate, so filtering by correctness leaves a corpus of short and simple modules. Correctness is also hard to obtain, since behavior leaves little trace on the surface in RTL, and validating an entire corpus only sorts pairs into passed and failed. We present RTLCurator, which learns a behavior-aware compatibility prior by contrasting each specification with implementations that fail simulation, and calibrates it to a new corpus using a small number of validated pairs. It then constructs the retained subset by balancing alignment, representation coverage, and RTL structural richness. On CodeV and RTLCoder, keeping 80% of the corpus this way improves on training with the full corpus across all reported metrics while validating only 10% of the pool, whereas ranking by the score alone falls below random selection and filtering the whole pool by simulation does no better.

Figures

Figures reproduced from arXiv: 2607.29283 by Cangyuan Li, Haoyu Gao, Kun Wang, Siyang Cai, Wenjing Chang, Ying Wang, Yinhe Han.

Figure 1
Figure 1. Figure 1: A one-operator edit can preserve syntax and struc [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 1
Figure 1. Figure 1: Validation does reveal behavior but is expensive [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of RTLCURATOR. An encoder is first trained on the source corpus with a progressive curriculum whose negatives move from random responses to responses that fail functional validation, which yields a compatibility prior for every pair of a target corpus before any pair is validated. A budgeted acquisition loop then queries at most B pairs for validation and uses the resolved labels to calibrate this… view at source ↗
Figure 3
Figure 3. Figure 3: Budget sensitivity of downstream supervised fine-tuning on CodeV and RTLCoder. Curves show five-run mean [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: RTLCoder target-domain calibration across cumu [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

16 extracted references · 4 linked inside Pith

  1. [2]

    doi: 10.18653/v1/ 2024.findings-acl.137

    Asso- ciation for Computational Linguistics. doi: 10.18653/v1/ 2024.findings-acl.137. URL https://aclanthology.org/2024. findings-acl.137/. Fan Cui, Chenyang Yin, Kexing Zhou, Youwei Xiao, Guangyu Sun, Qiang Xu, Qipeng Guo, Demin Song, Dahua Lin, Xingcheng Zhang, and Yun Liang. OriGen: Enhancing RTL code generation with code-to-code augmentation and self-...

    doi:10.18653/v1/ 2024

  2. [3]

    Wenji Fang, Shang Liu, Jing Wang, and Zhiyao Xie

    doi: 10.1145/3676536.3676830. Wenji Fang, Shang Liu, Jing Wang, and Zhiyao Xie. Cir- cuitFusion: Multimodal circuit representation learning for agile chip design. InThe Thirteenth International Con- ference on Learning Representations,

    arXiv

  3. [8]

    org/abs/2505.12697

    URL https://arxiv. org/abs/2505.12697. Mingjie Liu, Nathaniel Pinckney, Brucek Khailany, and Haoxing Ren. VerilogEval: Evaluating large language mod- els for Verilog code generation. InProceedings of the IEEE/ACM International Conference on Computer-Aided Design, pages 1–8,

    Pith/arXiv arXiv

  4. [9]

    10323812

    doi: 10.1109/ICCAD57390.2023. 10323812. Mingjie Liu, Yun-Da Tsai, Wenfei Zhou, and Haoxing Ren. CraftRTL: High-quality synthetic data generation for Verilog code models with correct-by-construction non- textual representations and targeted code repair. InThe Thirteenth International Conference on Learning Repre- sentations, 2025a. URL https://openreview.n...

    arXiv 2023

  5. [10]

    Yichuan Ma, Yunfan Shao, Peiji Li, Demin Song, Qipeng Guo, Linyang Li, Xipeng Qiu, and Kai Chen

    doi: 10.1109/ASP-DAC58780.2024.10473904. Yichuan Ma, Yunfan Shao, Peiji Li, Demin Song, Qipeng Guo, Linyang Li, Xipeng Qiu, and Kai Chen. UnitCoder: Scalable iterative code synthesis with unit test guidance,

    arXiv 2024

  6. [11]

    Yue Wang, Weishi Wang, Shafiq Joty, and Steven C. H. Hoi. CodeT5: Identifier-aware unified pre-trained encoder- decoder models for code understanding and generation. In Proceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 8696–8708. As- sociation for Computational Linguistics,

    2021

  7. [12]

    URL https://aclanthology

    18653/v1/2021.emnlp-main.685. URL https://aclanthology. org/2021.emnlp-main.685/. Anjiang Wei, Huanmi Tan, Tarun Suresh, Daniel Mendoza, Thiago S. F. X. Teixeira, Ke Wang, Caroline Trippel, and Alex Aiken. VeriCoder: Enhancing LLM-based RTL code generation through functional correctness validation,

    2021

  8. [13]

    Rethinking data selection at scale: Random selection is almost all you need

    Tingyu Xia, Bowen Yu, Kai Dang, An Yang, Yuan Wu, Yuan Tian, Yi Chang, and Junyang Lin. Rethinking data selection at scale: Random selection is almost all you need. InFindings of the Association for Computational Linguis- tics: EMNLP 2025, pages 2698–2711, Suzhou, China,

    2025

  9. [14]

    doi: 10.18653/ v1/2025.findings-emnlp.146

    Association for Computational Linguistics. doi: 10.18653/ v1/2025.findings-emnlp.146. URL https://aclanthology.org/ 2025.findings-emnlp.146/. Zhangchen Xu, Yang Liu, Yueqin Yin, Mingyuan Zhou, and Radha Poovendran. KodCode: A diverse, challenging, and verifiable synthetic dataset for coding,

    2025

  10. [15]

    URL https://arxiv.org/abs/2407.10424. Yaoyu Zhu, Di Huang, Hanqi Lyu, Xiaoyun Zhang, Chongx- iao Li, Wenxuan Shi, Yutong Wu, Jianan Mu, Jinghua Wang, Yang Zhao, Pengwei Jin, Shuyao Cheng, Shengwen Liang, Xishan Zhang, Rui Zhang, Zidong Du, Qi Guo, Xing Hu, and Yunji Chen. CodeV-R1: Reasoning-enhanced verilog generation,

    Pith/arXiv arXiv

  11. [16]

    URL https://arxiv.org/abs/2505.24183

    arXiv

  12. [2020]

    URL https://aclanthology.org/2020.findings- emnlp.139/

    doi: 10.18653/v1/2020.findings- emnlp.139. URL https://aclanthology.org/2020.findings- emnlp.139/. Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y . Wu, Y . K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. DeepSeek-Coder: When the large language model meets programming—the rise of code intelligence,

    doi:10.18653/v1/2020.findings- 2020

  13. [2022]

    Chaofan Li, Jianlyu Chen, Yingxia Shao, Defu Lian, and Zheng Liu

    URL https://arxiv.org/abs/2211.15533. Chaofan Li, Jianlyu Chen, Yingxia Shao, Defu Lian, and Zheng Liu. Towards a generalist code embedding model based on massive data synthesis,

    Pith/arXiv arXiv

  14. [2023]

    En-Ming Huang, Yu-Hung Kao, Ren-Hao Deng, Wei-Po Hsin, Yao-Ting Hsieh, Cheng Liang, Hsiang-Yu Tsou, Mu- Chi Chen, Yu-Kai Hung, Shao-Chun Ho, Po-Hsuang Huang, Shih-Hao Hung, and H

    URL https://arxiv.org/abs/2306.01250. En-Ming Huang, Yu-Hung Kao, Ren-Hao Deng, Wei-Po Hsin, Yao-Ting Hsieh, Cheng Liang, Hsiang-Yu Tsou, Mu- Chi Chen, Yu-Kai Hung, Shao-Chun Ho, Po-Hsuang Huang, Shih-Hao Hung, and H. T. Kung. Structured testbench generation for LLM-driven HDL design and verification- oriented data curation,

    Pith/arXiv arXiv

  15. [2024]

    Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu

    doi: 10.1145/3649329.3657356. Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. M3-embedding: Multi- linguality, multi-functionality, multi-granularity text em- beddings through self-knowledge distillation. InFind- ings of the Association for Computational Linguistics: ACL 2024, pages 2318–2335, Bangkok, Thailand,

    arXiv 2024

  16. [2025]

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xi- aocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou

    URL https: //openreview.net/forum?id=rbnf7oe6JQ. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xi- aocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. CodeBERT: A pre- trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1536–1547. Association...

    2020