{"id":"ed71f626-fcbb-4e82-8714-f3e3be8de8fd","arxiv_id":"2506.07239","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A method that predicts line-level timing and congestion issues directly from Verilog code using CL-Verilog embeddings and gradient-boosted classifiers.","lead":"VeriLoC uses embeddings from a Verilog-specialized language model to flag individual lines of RTL code that will cause timing or routing problems, without running slow synthesis and layout tools. If it works in practice it could give chip designers fast feedback during coding.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unspecified split unit in §4.1 can leak project/module identity into test; the reported F1/MAPE may not measure generalization to new designs.","rationale":"The reader's weakest assumption identifies data leakage in the random split as a central threat, and that is also the most load-bearing concern in my read. The paper's stated goal is early-stage prediction for new RTL designs, so the evaluation must prove transfer across designs, not just across sampled lines or modules. Since Section 4.1 does not specify the split unit, and the dataset is heavily project-structured, the reported F1/MAPE numbers may reflect memorization of project-level patterns rather than transferable line-level semantics. I also note the secondary issues the reader raised: the abstract's 4% MAPE is not directly supported by Tables 3-5, and the novelty claim of being first at line-level timing is in tension with reference [1]'s title. However, both are secondary to the split problem: even if the 4% figure is reconciled and the novelty claim is narrowed, the empirical central claim still requires a leakage-free evaluation. I do not move the verdict because the reader's CONDITIONAL verdict already correctly requires this re-evaluation; my read confirms that condition rather than replacing it.","tokens_in":20776,"tokens_out":5133,"duration_ms":64021,"concrete_test":"Re-run the full pipeline with a group-wise split by OpenABCD top-level project (e.g., leave-one-project-out cross-validation or an 80/20 project split), keeping the §3 architecture and §4 hyperparameters unchanged. Report line-level F1 for congestion/timing and module-level WNS MAPE, plus the number of train/test modules and any module overlap. If project-level F1 drops materially (e.g., by more than 0.05) or MAPE rises above 0.10, the random split leaked project-specific information and the reported numbers overstate generalization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that VeriLoC predicts line- and module-level design quality for RTL code. That claim is only as strong as the evaluation split. Section 4.1 says only '80/20 random split of the dataset' without stating whether the unit is a line, a module, or a project. OpenABCD is organized by projects (bp_quad has 252 modules, coyote 114, ariane 39, etc.). If the split is at module or line granularity, modules from the same project appear on both sides of the boundary. Because VeriLoC's features include a pooled module embedding plus neighboring-line context, the classifier can exploit project-specific idioms, repeated structural patterns, and even near-duplicate lines across train/test rather than learning transferable RTL semantics. This would inflate the reported F1-scores (0.86 congestion, 0.95 timing) and deflate MAPE (abstract '4%' vs. Table 4's 0.02-0.08 and Table 5's 0.06) relative to a deployment setting where a new design has not been seen during training. The same issue also weakens the module-level timing comparison against MasterRTL and RTL-Timer if those baselines were trained on the same leaked split.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VeriLoC, a method for predicting hardware design quality—specifically routing congestion and timing slack—directly from Verilog source code at both line-of-code and module granularity. VeriLoC uses the penultimate-layer embeddings of CL-Verilog, a Verilog-specialized LLM, to represent individual lines and entire modules; these embeddings are concatenated with neighboring-line context, projected via an autoencoder, and fed into XGBoost, LightGBM, or a small FNN. Line-level congestion and timing are treated as binary classification, while WNS is treated as regression; module-level WNS is obtained by taking the minimum predicted line-level WNS. Experiments on the OpenABCD dataset with Synopsys RTL Architect labels report F1-scores of 0.86 for congestion and 0.95 for timing, and improved module-level timing MAPE compared to MasterRTL, RTL-Timer, VeriDistill, and GNN baselines. The paper also reports runtime speedups over RTL Architect and a qualitative comparison against GPT-4o.","tokens_in":21001,"tokens_out":4183,"duration_ms":44653,"significance":"If the reported results are reliable, VeriLoC is the first method to predict line-level design-quality metrics from RTL code, and the combination of a hardware-specialized LLM with local and global context is a plausible and potentially useful architectural idea. The paper also demonstrates a practical speedup over full synthesis/placement, and it makes an honest attempt to compare against several state-of-the-art baselines. However, the central claims depend on the evaluation protocol: the split unit is unspecified, label-generation details are missing, and the headline MAPE in the abstract is not supported by the reported tables. These issues affect the validity of the generalization claims, so the significance is real but conditional on the evaluation being corrected.","major_comments":[{"comment":"The 80/20 random split is not defined at the line, module, or project level. OpenABCD is organized by projects, and Table 1 shows that a single project (e.g., bp_quad, coyote, ariane) contributes many modules and lines. If the split is at line or module granularity, modules from the same project appear on both sides of the training/test boundary, and the model can exploit project-specific idioms, repeated structural patterns, and near-duplicate lines. This would inflate the reported F1-scores and deflate MAPE relative to deployment on genuinely new designs. The authors should split by project (or perform grouped cross-validation) and report results on held-out projects; this is load-bearing for the paper's central generalization claim.","section":"Section 4.1, Table 1"},{"comment":"The abstract claims a reduction of MAPE to 'only 4%,' but Table 5 reports VeriLoC's aggregate MAPE as 0.06, Table 4 reports per-design module-level MAPE values between 0.02 and 0.08, and Table 3 reports per-design line-level MAPE as high as 0.27. The 4% figure is not derivable from any table, and the paper never defines how the aggregate MAPE is computed. Please either correct the abstract, state the exact metric and aggregation used to obtain 4%, or provide a confidence interval for the aggregate MAPE. As written, the headline number is inconsistent with the empirical section.","section":"Abstract, Section 4.3, Tables 3-5"},{"comment":"The label-generation procedure is not specified in enough detail to assess the validity of the ground truth. The paper states only that RTL Architect was run with a 0.25 ns timing constraint and that congestion/timing labels were back-annotated to lines. It does not explain how individual lines are assigned congestion labels, what threshold defines a 'congestion-causing' line, how line-level WNS is defined for a single line (since WNS is naturally a path- or design-level metric), or how line-level WNS is computed from the tool output. Every F1 and MAPE result depends on these labels, so the paper should include a concrete description of the back-annotation and, ideally, an analysis of label noise.","section":"Section 4.1"},{"comment":"Module-level WNS is computed by taking the minimum predicted line-level WNS across all lines. This is a non-trivial assumption about how line-level slacks compose into module-level WNS, and the paper does not validate it. Moreover, there is an apparent tension between Table 3 and Table 4: for designs like dynamic_node and bp_be_top, the line-level regression is weak (R² = 0.76, MAPE = 0.18 and R² = 0.76, MAPE = 0.27), yet the module-level results derived from those line-level predictions are excellent (R² = 0.99, MAPE = 0.06 and R² = 0.98, MAPE = 0.08). The authors should explain how this can occur, and should compare module-level WNS computed from ground-truth line-level labels against the tool's reported module-level WNS to quantify the error introduced by the min-composition step.","section":"Section 3.5, Tables 3 and 4"},{"comment":"All results are based on a single 80/20 split with no error bars, no multiple seeds, and no statistical significance tests. Given the small number of projects (about a dozen) and the class imbalance in line-level labels, the differences between classifiers and context windows could easily be within run-to-run variation. The paper should report means and standard deviations over multiple random splits, or use repeated cross-validation, to support the quantitative claims in Tables 2-5.","section":"Section 4.1 and all result tables"}],"minor_comments":[{"comment":"The autoencoder for dimensionality reduction is trained on the concatenated line and module embeddings, but the paper does not state whether this training includes test-set embeddings. If the autoencoder is fit on the full dataset before the 80/20 split, this is a form of transductive information leakage that should be disclosed, even if the effect is expected to be small.","section":"Section 3.3"},{"comment":"The heading contains a typo: 'Comaprisons' should be 'Comparisons'.","section":"Section 4.3 heading"},{"comment":"The phrase 'we we first estimate WNS at the line level' contains a duplicated word and should be corrected.","section":"Section 3.5"},{"comment":"The sentence 'Synopsys RTL-A only reports WNS when a timing constraints are actually violated' has a subject-verb agreement error; it should read 'timing constraints are actually violated'.","section":"Section 4.1"},{"comment":"References [13] and [17] are the same Synopsys RTL Architect datasheet and should be merged or cross-referenced.","section":"References"},{"comment":"The table formatting is garbled: the header row 'Design # of Modules # of Lines' appears to be split, and it is not clear whether the second and third columns are module counts and line counts. Please reformat the table for readability.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a relevant and timely problem, and the core idea of using a Verilog-specialized LLM for line-level QoR prediction is interesting. However, the evaluation protocol—particularly the unspecified split unit and the inconsistency between the abstract's 4% MAPE and the reported tables—needs a substantive revision before the claims can be considered reliable. The issues are fixable within the scope of the manuscript, so I do not recommend rejection, but the revision should be treated as more than cosmetic."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: useful paper, real contribution, but evaluation as written can't support the headline claims. I'd send it to review, not desk-reject, because the core idea is good and the issues are fixable.\n\nWhat's new and good: the task matters—designers want to know which Verilog lines will cause timing or congestion problems without running hours of physical design. Prior work mostly did module-level forecasts; line-level congestion prediction from raw RTL text is, as far as I know, genuinely new. The architecture is sensible: per-line CL-Verilog embeddings, pooled module embedding, neighborhood context. The ablations are concrete: comments help, context helps more for congestion than timing, batched embeddings underperform per-line concatenation, and the runtime comparison is honest. Code is shipped.\n\nWhere I'd push back: the biggest problem is the split. Section 4.1 says '80/20 random split of the dataset' without naming the unit. OpenABCD is organized by projects; a split at line or module granularity puts modules from the same project on both sides. Since the features include a module embedding and neighbor context, the model can exploit project-specific idioms and near-duplicate patterns rather than learning transferable RTL semantics. This would inflate F1 and deflate MAPE. Not a flaw in the method, but a flaw in the evidence as presented. Fix: project-level split or at least per-project held-out results.\n\nSecond, the abstract's 'MAPE from 14%–18% down to only 4%' isn't supported by the tables. Table 5 reports 6%; Table 4 has per-design MAPE up to 8%; Table 3 line-level MAPE goes up to 27%. The 4% number needs a definition or should be dropped.\n\nThird, label generation is under-specified: 'RTL Architect back-annotates lines'—how is a per-line congestion label defined? What is the mapping from netlist objects back to lines? This needs to go in the appendix.\n\nFinally, the 'first ever line-level QoR predictor' claim needs care. The DAC'24 reference [1] has 'fine-grained RTL timing evaluation' in its title; the paper should explain how VeriLoC differs from that work rather than dismissing it as module-level.\n\nBottom line: serious work, and a thoughtful reviewer can help sharpen it. The method is likely sound; the evidence currently overstates it. For ML-for-EDA readers this is worth a look, and it's a good reading-group piece for discussing benchmark hygiene. Send it to peer review, but the split question, metric reconciliation, and label-procedure documentation need to be addressed before publication.","headline":"Useful, well-executed first pass at line-level QoR prediction from Verilog, but the split ambiguity and metric inconsistencies mean the headline numbers outrun the evidence.","tokens_in":21602,"tokens_out":3954,"would_cite":false,"duration_ms":45322,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"VeriLoC predicts timing and routing congestion directly from Verilog source code at both the line and module level.","keywords":["Verilog","line-level prediction","timing prediction","congestion prediction","LLM embeddings","RTL quality","hardware design"],"falsifier":"Training on a split where modules from the same project are kept in either train or test, or using labels from a second design-quality tool, would reveal whether the reported F1-scores and error rates persist under stricter generalization.","tokens_in":20554,"feed_emoji":"🔌","tokens_out":10207,"duration_ms":79518,"temperature":0.7,"pith_summary":"This paper introduces a method for predicting hardware design quality directly from Verilog source code at both the individual-line and whole-module levels. Designers would use it to catch timing violations or routing congestion problems in their code before running hours-long synthesis, placement, and routing. The central claim is that embeddings from a Verilog-specialized language model, combined with embeddings of the surrounding lines and of the whole module, carry enough information to identify the exact lines of code that cause physical-design problems. The reported numbers are F1-scores of 0.86 for line-level congestion and 0.95 for line-level timing, with module-level timing error reduced from 14-18% to 4%.","feed_headline":"Flags exact Verilog lines causing chip congestion and timing issues","feed_subtitle":"VeriLoC reads Verilog and flags each line that will cause timing violations or routing congestion","key_machinery":"The central mechanism is the concatenation of local and global context: for each line, the embedding of the line itself is combined with embeddings of up to two neighboring lines on each side and an attention-pooled embedding of the entire module. This preserves the local semantics of the line while providing the context needed to distinguish, for example, a harmless clocked block from a clocked block that causes congestion.","core_discovery":"The central discovery is that a single line of Verilog can carry enough semantic information to determine whether it causes timing or congestion issues, provided its embedding is enriched with embeddings of the surrounding lines and of the whole module. The system uses a Verilog-specialized language model as an encoder, concatenates line-level and module-level embeddings, reduces dimensionality, and then applies classifiers or regressors. The paper claims this approach identifies problem-causing lines with high F1-scores and improves module-level worst-negative-slack prediction over previous RTL-stage methods, and states this is the first line-of-code-level quality prediction result.","pith_inferences":["The approach is likely to be sensitive to the distribution of modules in the training set; if modules from the same project appear on both sides of the split, the reported scores may be inflated and not represent prediction on genuinely new designs.","The labels come from a single EDA tool with a specific aggressive timing constraint, so the findings may transfer differently to designs with different constraints or tool versions.","A natural test is to see whether the predicted line-level hotspots align with the lines a designer would actually modify to relieve congestion in a real design revision, beyond matching the tool's back-annotation."],"forward_implications":["Designers can get early feedback on which lines to rewrite to avoid timing and congestion problems, before running expensive synthesis and physical design.","The same line-level embeddings could be reused for other quality-of-results prediction tasks such as power or area.","Smaller models can be used for faster predictions, with a reported tradeoff of modest accuracy loss but 22x average and 113x median speedups."],"supporting_citations":[{"why":"Supplies the CL-Verilog model whose penultimate-layer activations provide the line and module embeddings.","marker":"[14]"},{"why":"Provides the OpenABCD RTL module dataset used for training and evaluation.","marker":"[18]"},{"why":"Supplies the RTL Architect tool that generates ground-truth congestion and timing labels via back-annotation.","marker":"[13]"},{"why":"MasterRTL, the state-of-art pre-synthesis PPA estimator that VeriLoC compares against and improves upon for module-level timing.","marker":"[4]"},{"why":"RTL-Timer, the state-of-art fine-grained RTL timing estimator that VeriLoC compares against and improves upon.","marker":"[1]"},{"why":"VeriDistill, a multimodal LLM-based PPA predictor used as a baseline in the module-level timing comparison.","marker":"[11]"},{"why":"The GNN-based baseline from synthesized netlists used in the module-level timing comparison.","marker":"[10]"},{"why":"XGBoost, the gradient-boosted tree classifier/regressor used as a downstream head.","marker":"[19]"},{"why":"LightGBM, the alternative gradient-boosted tree classifier that achieves the best congestion and timing F1-scores.","marker":"[20]"}],"fun_headline_variants":["VeriLoC pinpoints exact Verilog lines causing timing and congestion","First line-level method to predict chip quality from Verilog","VeriLoC cuts design quality prediction error to 4%","AI model reads Verilog, flags problematic lines for chip design"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The per-line labels from the automated design tool are correct, and the random split of modules does not leak information across train and test, so the reported scores reflect prediction on genuinely new designs.","fun_headline_variants_meta":{"raw":{"variants":["VeriLoC pinpoints exact Verilog lines causing timing and congestion","First line-level method to predict chip quality from Verilog","VeriLoC cuts design quality prediction error to 4%","AI model reads Verilog, flags problematic lines for chip design"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000855,"raw_usage":{"total_tokens":3686,"prompt_tokens":885,"completion_tokens":2801,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":501,"completion_tokens_details":{"reasoning_tokens":2728}},"tokens_in":501,"tokens_out":2801,"duration_ms":21008,"temperature":1.0,"reasoning_tokens":2728,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:38:28.125417+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Training on a split where modules from the same project are kept in either train or test, or using labels from a second design-quality tool, would reveal whether the reported F1-scores and error rates persist under stricter generalization.","supporting_citations":[{"cited_title":"Openabc-d: A large-scale dataset for machine learning guided integrated circuit synthesis,","cited_arxiv_id":null,"evidence_quote":"Provides the OpenABCD RTL module dataset used for training and evaluation."},{"cited_title":"Masterrtl: A pre-synthesis ppa estimation framework for any rtl design,","cited_arxiv_id":null,"evidence_quote":"MasterRTL, the state-of-art pre-synthesis PPA estimator that VeriLoC compares against and improves upon for module-level timing."},{"cited_title":"Annotating slack directly on your verilog: Fine-grained rtl timing evaluation for early optimization,","cited_arxiv_id":null,"evidence_quote":"RTL-Timer, the state-of-art fine-grained RTL timing estimator that VeriLoC compares against and improves upon."},{"cited_title":"Retrieval-Guided Reinforcement Learning for Boolean Circuit Minimization","cited_arxiv_id":"2401.12205","evidence_quote":"The GNN-based baseline from synthesized netlists used in the module-level timing comparison."},{"cited_title":"Lightgbm: a highly efficient gradient boosting decision tree,","cited_arxiv_id":null,"evidence_quote":"LightGBM, the alternative gradient-boosted tree classifier that achieves the best congestion and timing F1-scores."}],"review_version":1}