{"id":"af9ed0a2-8088-4225-bcdd-48ba591103c3","arxiv_id":"2505.06096","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A new Verilog dataset with license and copyright filtering, a similarity-based copyright violation benchmark, and a fine-tuned Llama model (FreeV) with a 3% reported violation rate and VerilogEval pass@10 gains.","lead":"The authors built a large Verilog code dataset filtered for licenses and copyright signs, then fine-tuned a Llama model on it. They report a 3% rate of generating protected code and improved functional code generation, but the measurement relies on a small, quickly assembled benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 3% copyright-violation rate only tests files whose headers the Section III-C filter would catch; unmarked protected Verilog in FreeSet is never probed, so the 'smallest infringement risk' claim is not established.","rationale":"The paper is an honest engineering contribution: it releases a large corpus, a reproducible curation pipeline, and a first benchmark for a real risk; those are useful regardless of the overclaim. The strongest claim, however, is a safety guarantee, and safety claims need evidence about the hardest cases, not the easiest. The pipeline's repo-license allowlist plus header-keyword scan cannot certify absence of protected code, because copyright subsists without notice and proprietary RTL frequently appears inside permissively licensed repositories (the paper's own 2K-file finding confirms this). The benchmark compounds the problem by testing only files that carry explicit copyright declarations, so the 3% result is measured on the class most likely to be removed. A contamination audit against unmarked protected files, and a violation-rate estimate on prompts from those files, would settle whether the concern lands. If it does land, the appropriate fix is to soften the claim to 'reduced measured risk on explicitly-noticed protected code' and to report the filter's false-negative rate. This is consistent with the reader's conditional verdict: the contribution is worth publishing with caveats and release requirements, but the central headline as phrased is not supported.","tokens_in":10592,"tokens_out":4745,"duration_ms":54444,"concrete_test":"Run a contamination audit on FreeSet: collect a hold-out set of Verilog files from the gray-area repos excluded for lacking a license, plus files with known proprietary provenance whose headers contain none of the filter keywords; strip comments, then use the same MinHash/LSH pipeline (§III-D2) and an embedding similarity search to test whether any of these files are near-duplicates of the 222,624 FreeSet files. Also run the §III-A benchmark using first-20% prompts from these unmarked protected files on FreeV. If near-duplicates are found, or if the violation rate on unmarked prompts is materially above 3% with a confidence interval that excludes 3%, the Section III-C filtering assumption fails and the infringement-free claim should be withdrawn.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central safety claim—FreeV has the smallest potential for copyright infringement at 3% (Abstract, §IV-B)—rests on two under-inclusive mechanisms. First, §III-C.2 filters the dataset by keeping only repos with a chosen open-source license and then scanning file headers for explicit phrases such as 'proprietary', 'confidential', and 'all rights reserved'. A permissively licensed repo can contain proprietary RTL, and copyright protection does not require a header or any of those keywords; protected code can be identified by body comments, register/port names, file names, or simply by provenance from a non-free source. The paper itself reports finding over 2K proprietary files inside 'reportedly open-source' repos, showing repo-level licenses are insufficient. Second, the §III-A benchmark builds its test set from files with copyright declarations, i.e., the same class the keyword filter is designed to catch. Prompting with first-20% fragments of those files and measuring cosine similarity only estimates how often the model reproduces explicitly-noticed protected code that was already excluded from FreeSet. It does not estimate generation of unmarked protected code that may have survived into the 222k-file training set. The 3% number is therefore a lower bound conditional on the filter being complete, not evidence that FreeV was trained without copyrighted code. This does not invalidate the dataset's practical value, but it does invalidate the 'infringement-free' framing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an end-to-end pipeline for building a copyright-filtered Verilog training dataset (FreeSet, over 220k files), a cosine-similarity benchmark for estimating how often Verilog-tuned LLMs reproduce protected code, and a continually pre-trained model (FreeV, based on Llama-3.1-8B-Instruct). The main reported results are that FreeV has a 3% copyright-violation rate, lower than prior fine-tuned Verilog models, and that it improves VerilogEval-Human pass@10 from 25.9% to 36.0% compared with its 4-bit base model.","tokens_in":10932,"tokens_out":4463,"duration_ms":50711,"significance":"If the claims were fully established, the contributions would be useful: FreeSet appears to be the largest open Verilog corpus at 16.5 GB / 222,624 files, the curation pipeline addresses a real gap in prior datasets, and the VerilogEval gains are concrete and measured rather than asserted. The authors also release the dataset and describe their filtering framework, which supports reproducibility. The main significance is therefore in the dataset and pipeline, not in the legal claim: the 3% violation rate is a preliminary estimate on a narrow benchmark, and the paper itself acknowledges in Section V that the benchmark is preliminary. The central safety claim, however, is currently stated much more strongly than the evidence supports.","major_comments":[{"comment":"The headline 3% violation rate is not statistically grounded as presented. The benchmark uses only 100 prompts drawn from a 2K-file corpus, and a 3/100 result has a 95% binomial confidence interval of roughly 0.6% to 8.5%, which overlaps plausible rates for several other models and for the base model. The paper does not specify how cosine similarity is computed (token-level? embedding? over which fields?), how the 20% file prefixes are truncated to 64 words, or whether the 100 prompts are independent samples. Please report the full experimental protocol, include confidence intervals or exact per-model counts, and soften the claim 'smallest potential for hardware copyright infringement' accordingly.","section":"Section IV-B and Section III-A"},{"comment":"The filtering pipeline is under-inclusive for the strong claim that FreeSet is free of copyright-protected material. The file-by-file check inspects only header comments for a fixed keyword list ('proprietary', 'confidential', 'all rights reserved'), and the benchmark's test corpus is built from files that carry detectable copyright declarations. This means the 3% rate estimates leakage of the same class of explicitly marked files that the filter is designed to remove; it does not estimate leakage of protected Verilog whose notices appear in the body, contain no such keywords, or have no notice at all. The paper itself reports finding over 2K proprietary files inside reportedly open-source repositories, which shows that repository-level licenses are insufficient, but it does not validate the header-keyword filter against unmarked protected code. Please either add a validation study (e.g., manual or provenance-based audit of a random sample of surviving files) or explicitly reframe the claim as a reduction in risk for detected, explicitly marked copyrighted files.","section":"Section III-C.2 and Section IV-B"},{"comment":"The paper's own Discussion section calls the benchmark 'preliminary' and lists open problems such as additional prompting methods and expanded test corpora, yet the Abstract and Conclusion state as a result that FreeV has 'the smallest potential for hardware copyright infringement among prior works.' Since the benchmark cannot currently establish a legal or even a robust empirical infringement rate, the abstract and conclusion should be aligned with the provisional nature of the benchmark. At minimum, 'infringement-free' and 'smallest risk' should be replaced with a statement about measured similarity-based violation rates on the proposed benchmark.","section":"Section V and Abstract"},{"comment":"The construction of the 2K-file copyrighted benchmark corpus is not described precisely. The text says this dataset is 'collected in Section III-B,' but Section III-B describes the general FreeSet curation pipeline, not the curation of the copyrighted benchmark set. It is also unclear whether the 100 benchmark prompts are drawn from the same 2K proprietary files found during FreeSet filtering or from a separate corpus, and how the modules are isolated after comment stripping. Please clarify the provenance of the benchmark files, whether they overlap with files removed from FreeSet, and how the 100 prompts were selected from the 2K files; without this, the benchmark's coverage and independence cannot be assessed.","section":"Section III-A and Section IV-A"}],"minor_comments":[{"comment":"There is a typo in 'only epositories with the fair-use modules are utilized'; it should read 'only repositories.'","section":"Section III-C.2"},{"comment":"The concluding paragraph states that the curation framework 'results in a curated and open-source dataset FreeV'; the dataset is named FreeSet, not FreeV, throughout the rest of the paper.","section":"Section VI"},{"comment":"Reference [12] has a typo in the title: 'Cerilog' should be 'Verilog.'","section":"References"},{"comment":"The phrase 'with a limit of 64 words per prompt' is ambiguous when combined with 'the first 20% of a copyrighted code file'; please specify whether the 20% prefix is truncated to 64 words or whether 64 words is a separate cap, and report the distribution of prompt lengths.","section":"Section III-A"},{"comment":"The paper uses 'fair-use data' and 'fair-use modules' to describe the dataset, but fair use is a legal defense, not a property of a corpus. This wording should be changed to 'properly licensed' or 'reusable' to avoid implying a legal determination that the paper does not make.","section":"Section I and III-B"},{"comment":"The dataset is described as open-source, but the Hugging Face link contains the suffix 'LabUse'; if the dataset is restricted to lab or research use, the 'open-source' characterization should be qualified in Table I and the abstract.","section":"Table I and dataset link"}],"recommendation":"major_revision","confidential_remarks":"The paper's core dataset and benchmark are potentially valuable, but the editorial framing ('Infringement-Free', 'smallest potential') goes beyond what the measurements support; please ensure the authors calibrate the claims to the preliminary benchmark and clarify the dataset's actual license before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nIf you work on RTL generation or on dataset curation for code LLMs, this paper is worth your time. The dataset is the real contribution: FreeSet, 222k Verilog files with repo-level license filtering plus file-by-file header scanning for copyright keywords. That combination is genuinely new among the cited Verilog datasets—BetterV checks repo licenses but none do file-level scanning. They also report finding 2k+ proprietary files inside supposedly open-source repos, which alone justifies the extra step. The copyright benchmark, crude as it is, is a first attempt at measuring a real risk that the EDA community is starting to care about.\n\nThe VerilogEval numbers are honest and appropriately modest: FreeV improves pass@10 from 25.9% to 36.0% over its Llama-3.1-8B base, and the authors admit it lags SOTA because they only do continual pre-training. Fine.\n\nThe soft spots are real but not fatal, and the stress-test note gets the big one right. The 3% violation rate is measured on a test set built from files with explicit copyright declarations—the same class the keyword filter catches. It does not probe unmarked protected code that could have slipped into FreeSet through a permissively licensed repo with proprietary RTL inside. So the paper does not establish \"infringement-free\" training; it establishes low measured risk for one class of infringement. The authors actually hedge most of the time, calling it \"potential for infringement\" and a \"preliminary benchmark,\" but the abstract and conclusion lean on \"infringement-free\" and \"guarantees.\" That framing should be softened.\n\nOther issues are minor. The benchmark uses 100 prompts, a 0.8 cosine threshold, and no confidence intervals, with the similarity computation underspecified. The dataset is hosted under a \"LabUse\" tag on HuggingFace, which conflicts with the \"open-source\" description in the abstract and Table I. These are fixable with more transparency and a license clarification.\n\nNet: this is a solid engineering paper with a reusable artifact and a useful first-cut benchmark. The central defense claim needs a correction, not a rejection. I'd bring it to reading group, and I'd cite the dataset. A serious journal or conference should send it to peer review; given the gap between the claim and the evidence, I'd expect major-ish revisions focused on the benchmark's coverage and the dataset's license.\n\nBest,\n\n[You]","headline":"A genuinely useful copyright-filtered Verilog dataset and a first-of-its-kind infringement benchmark, but the 'infringement-free' claim outruns the evidence.","tokens_in":11437,"tokens_out":988,"would_cite":true,"duration_ms":12325,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that filtering Verilog training data for open licenses and copyright markers nearly eliminates a fine-tuned model's tendency to reproduce protected code, while still improving code-generation accuracy over its base model.","keywords":["LLM copyright","Verilog generation","hardware design","dataset curation","license filtering","fine-tuning","intellectual property","RTL code generation"],"falsifier":"Scan every file in FreeSet for copyright notices that do not contain any of the filtered keywords (for example, a bare '© 2020 Example Corp' header); if even one such file exists, the dataset's copyright-cleanness claim is not established. Alternatively, run the paper's benchmark with full-file prompts on FreeV and check whether the violation rate exceeds 3%.","tokens_in":10387,"feed_emoji":"🔒","tokens_out":4381,"duration_ms":38501,"temperature":0.7,"pith_summary":"The paper argues that copyright risk in Verilog-generating LLMs comes largely from uncurated training data, and that a dataset filtered for open licenses and copyright markers can nearly eliminate that risk without sacrificing code quality. To show this, the authors assemble FreeSet, 222,624 Verilog files from GitHub repositories that carry open-source licenses, with files containing words like 'proprietary', 'confidential', or 'all rights reserved' removed. They then fine-tune Llama-3.1-8B-Instruct on FreeSet to produce FreeV, reporting a 3% copyright violation rate on their similarity benchmark, the lowest among compared models, alongside a 10.1 percentage-point improvement in VerilogEval Human pass@10 over the base model. The paper also contributes the benchmark itself: prompting models with the first 20% of copyrighted Verilog files and flagging outputs whose cosine similarity exceeds 0.8.","feed_headline":"Copyright-checked Verilog dataset cuts infringement risk to 3 percent","feed_subtitle":"A 220k-file open-source dataset and a fine-tuned Llama model keep functional gains while nearly removing protected-code reproduction.","key_machinery":"The load-bearing machinery is the dataset curation pipeline. It combines GitHub API queries granularized by creation date and license to collect about 1.3 million Verilog files, repository-level license allowlisting (MIT, Apache 2.0, GPL, LGPL, Mozilla, Creative Commons, Eclipse, BSD), file-by-file header scanning for 'proprietary', 'confidential', and 'all rights reserved' (which removed over 2,000 protected files), MinHash-based Locality Sensitive Hashing duplication removal with a 0.85 similarity threshold, and syntax verification with Icarus Verilog. The evaluation pipeline is equally central: a copyright benchmark that prompts models with the first 20% (64 words) of each of 100 protected modules and counts a violation when a generated output has cosine similarity at least 0.8 to the protected code.","core_discovery":"The central claim is that a hardware-code LLM can be fine-tuned on a large dataset scrubbed of copyrighted material, and that this scrubbing is the reason the fine-tuned model FreeV reproduces protected designs only 3% of the time, versus 9-15% for prior Verilog-tuned models. The scrubbing operates in two layers: repository-level license screening that keeps only repositories under recognized open-source licenses, and file-level scanning of header comments for copyright markers. The paper further claims that this safer training does not cost functionality: FreeV improves over its base model on VerilogEval Human, with pass@10 rising from 25.9% to 36.0%, though it remains below state-of-the-art Verilog-tuned models that also use instruction tuning.","pith_inferences":["The curation pipeline likely transfers to other hardware description languages such as SystemVerilog and VHDL, and to software code domains, whenever header comments are the primary copyright markers.","The benchmark's design (first 20% of a file, 64-word prompt) probably undercounts memorization; prompting with larger fractions or with the model's own completions of long files could raise violation rates for all models, including FreeV.","Repository-level license filtering excludes unlicensed repositories entirely, which may discard legally reusable code (e.g., code placed in the public domain without a formal license); a provenance-aware filter could recover some of that data.","The paper's own suggestion to use structural similarity measures like GNN4IP hints that text-cosine benchmarks miss designs that are structurally copied but textually rewritten."],"forward_implications":["FreeSet's license and copyright filters can serve as a template for future hardware datasets; it is the largest open-source Verilog dataset reported, at 16.5 GB and 222,624 files.","Fine-tuning on copyright-cleaned data does not inherently reduce functional performance: FreeV's pass@5 and pass@10 improve by 7.9 and 10.1 percentage points over its base.","The benchmark can be applied to other Verilog-tuned models to estimate their copyright risk before deployment.","Because 3% of FreeV's outputs still clear the similarity threshold, even a carefully filtered dataset and model retain some infringement risk."],"supporting_citations":[{"why":"Supplies the MinHash/Jaccard deduplication method and serves as a comparison dataset and model in both the copyright benchmark and VerilogEval.","marker":"[12]"},{"why":"Provides a comparison model and dataset for the copyright-infringement evaluation and the VerilogEval table.","marker":"[15]"},{"why":"Offers a comparison model and dataset, and its lack of license checks motivates the proposed curation framework.","marker":"[16]"},{"why":"The only prior work with repository-level license checks; used as a comparison model and dataset.","marker":"[17]"},{"why":"Supplies a comparison model with state-of-the-art VerilogEval scores, used to contextualize FreeV's performance.","marker":"[33]"},{"why":"Adds another comparison model and dataset for both the copyright benchmark and VerilogEval.","marker":"[34]"},{"why":"Provides the legal motivation by citing a lawsuit against OpenAI and Microsoft over open-source code training.","marker":"[21]"},{"why":"Proposed as a future structural similarity metric for the copyright benchmark, showing an acknowledged limitation of cosine similarity.","marker":"[35]"}],"fun_headline_variants":["Scrubbed Verilog dataset yields 3% copyright risk in LLM","Copyright-safe Verilog LLM: 220k files, 3% infringement","FreeV: Verilog LLM cuts protected-code output to 3%","Open-source Verilog data trains LLM with 3% copyright risk"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The filter catches copyrighted files only when they carry the scanned keywords or a recognizable open-source license marker, so protected code that lacks 'proprietary', 'confidential', or 'all rights reserved' in its header could remain in FreeSet and make the 3% violation rate an underestimate.","fun_headline_variants_meta":{"raw":{"variants":["Scrubbed Verilog dataset yields 3% copyright risk in LLM","Copyright-safe Verilog LLM: 220k files, 3% infringement","FreeV: Verilog LLM cuts protected-code output to 3%","Open-source Verilog data trains LLM with 3% copyright risk"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000191,"raw_usage":{"total_tokens":1325,"prompt_tokens":909,"completion_tokens":416,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":333}},"tokens_in":525,"tokens_out":416,"duration_ms":4023,"temperature":1.0,"reasoning_tokens":333,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:48:22.046417+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Scan every file in FreeSet for copyright notices that do not contain any of the filtered keywords (for example, a bare '© 2020 Example Corp' header); if even one such file exists, the dataset's copyright-cleanness claim is not established. Alternatively, run the paper's benchmark with full-file prompts on FreeV and check whether the violation rate exceeds 3%.","supporting_citations":[{"cited_title":"VeriGen: A Large Language Model for Cerilog Code Generation,","cited_arxiv_id":null,"evidence_quote":"Supplies the MinHash/Jaccard deduplication method and serves as a comparison dataset and model in both the copyright benchmark and VerilogEval."},{"cited_title":"RTLCoder: Outperforming GPT-3.5 in Design RTL Generation with Our Open- Source Dataset and Lightweight Solution,","cited_arxiv_id":null,"evidence_quote":"Offers a comparison model and dataset, and its lack of license checks motivates the proposed curation framework."},{"cited_title":"OpenAI, Microsoft want court to toss lawsuit accusing them of abusing open-source code,","cited_arxiv_id":null,"evidence_quote":"Provides the legal motivation by citing a lawsuit against OpenAI and Microsoft over open-source code training."},{"cited_title":"GNN4IP: Graph Neural Network for Hardware Intellectual Property Piracy Detec- tion,","cited_arxiv_id":null,"evidence_quote":"Proposed as a future structural similarity metric for the copyright benchmark, showing an acknowledged limitation of cosine similarity."}],"review_version":1}