{"id":"388f8c27-81cc-47cf-88ee-e5e4c38dbbcf","arxiv_id":"2507.21485","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"HLSDebugger fixes logic bugs in HLS code with 37.6% accuracy, outperforming GPT-4 by over 3x, using an encoder-decoder model trained on a new 300K-sample dataset.","lead":"This paper introduces HLSDebugger, a fine-tuned encoder-decoder language model that locates, classifies, and corrects logic bugs in high-level synthesis (HLS) code, and it releases a new dataset of about 300K buggy code samples. It reports that HLSDebugger corrects bugs more than three times as often as GPT-4 on an existing benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Correction metric tests snippet-text match, not functional repair; HLSDebugger is MLE-trained on those exact snippets, so the 3.6x gain over GPT-4 may be partly a metric artifact.","rationale":"The reader's weakest_assumption is the synthetic bug distribution and its transfer to real-world HLS bugs. That is a genuine external-validity concern, but the more immediately load-bearing issue is internal to the reported benchmark: the evaluation metric for the central '3x better correction' claim is an exact normalized snippet match against labels whose distribution the submitted model was explicitly trained to reproduce. This makes the comparison with GPT-4 unequal even on the benchmark itself, independent of whether the synthetic bugs resemble real-world bugs. The paper has real strengths: it releases a large open-source dataset, a reproducible model, and a clearly described combined-loss training scheme, and its bug-identification results are reported at multiple granularities. No formal verification or semantic equivalence checking is provided, which would be the natural way to validate the correction claim. A functional patch-and-simulate evaluation on a sample of corrections would settle whether the 3x margin is a genuine functional advantage or a label-reproduction artifact. Since the reader already issued a CONDITIONAL verdict, this concern reinforces that conditionality rather than changing the verdict.","tokens_in":14293,"tokens_out":7008,"duration_ms":89400,"concrete_test":"Select a random sample of roughly 150 buggy benchmark samples; for each, take the corresponding outputs of HLSDebugger and GPT-4, patch them into the buggy code, compile with the HLS tool, and simulate with randomized directed testbenches (or have two hardware engineers blind-assess functional equivalence). Compare semantic correction rates. If the semantic gap is materially smaller than the 27.1-point gap in Table III, the headline claim should be re-scoped to text-snippet matching.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The single load-bearing weakness is the bug-correction metric in §V-C. The headline 37.6% vs 10.5% in Table III is scored by 'strict substring match' between the processed generated text and the dataset's canonical correct-snippet label. HLSDebugger's decoder is trained with MLE (Eq. 5) to emit exactly these canonical snippets, so it is directly optimized for the evaluation signal. GPT-4 is a zero-shot baseline with no access to the label distribution; a semantically correct repair that differs lexically from the canonical snippet (different variable naming, reordered statements, or a correct whole-function rewrite) is scored as a miss. Conversely, a snippet-text match does not verify that the repaired HLS code actually compiles and implements the intended behavior. The 3x advantage is therefore not yet evidence of a 3x advantage in functional bug correction.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents HLSDebugger, an encoder-decoder LLM (based on CodeT5-Large) fine-tuned for three HLS logic-debugging subtasks: bug type prediction, bug location identification, and bug correction. The authors construct and release a synthetic supervised dataset of about 300K samples by having GPT-3.5 insert one of eight logic-bug types into correct HLS code, then train a multi-task model with a combined loss. On a 2K-sample subset of the benchmark from [1], HLSDebugger achieves substantially higher bug-identification precision/recall than GPT-4, and 37.6% vs 10.5% strict-substring bug-correction accuracy, which the paper claims as a more than 3x improvement. They also show that providing the bug location improves correction for all models and that HLSDebugger remains best.","tokens_in":14525,"tokens_out":7377,"duration_ms":76283,"significance":"If the reported evaluation is taken at face value, the paper makes a useful contribution: a large open dataset, an open-source fine-tuned model, and a multi-task formulation are potentially valuable for the HLS community. The identification gains are large and internally consistent. However, the central quantitative claim about bug correction rests on an evaluation metric that is biased toward the fine-tuned model, and the training distribution closely mirrors the test benchmark. The actual advance is therefore currently uncertain and requires additional evaluation to be established.","major_comments":[{"comment":"The bug-correction accuracy is measured by 'strict substring match' between the processed generated text and the canonical correct-snippet label. Since the HLSDebugger decoder is trained with MLE (Eq. 5) on exactly these canonical snippets, the model is directly optimized to reproduce the evaluation signal, whereas the GPT-4 baseline is zero-shot and likely to produce semantically correct fixes that differ lexically. A substring match also does not verify that the repaired HLS code compiles or preserves intended behavior. The reported 37.6% vs 10.5% (3.6x) is therefore not yet evidence of a 3.6x functional improvement. Please add an execution-based or semantic-equivalence evaluation (e.g., compile and run the repaired code on test benches, or use a metric tolerant to variable renaming and statement reordering), and report the results for all models.","section":"V-C, Table III"},{"comment":"The synthetic training data are generated by inserting exactly the same eight bug types that appear in the test benchmark, and the label format (buggy snippet, correct snippet) is identical. This means the reported performance reflects the model's ability to reproduce the distribution used to create the benchmark, not necessarily its ability to find and fix naturally occurring HLS logic bugs. The paper should report performance on held-out naturally occurring buggy HLS code, or explicitly frame the benchmark as an in-distribution stress test. Also, Section IV-A states the bug list contains '18 types of logic bugs,' but Section V-A lists only 8; please reconcile this discrepancy.","section":"IV-A and V-A"},{"comment":"The test benchmark from reference [1] lists J. Wang as a co-author, who is also the first author of the current paper. The paper does not disclose this overlap. The ROUGE-L filtering (Section V-B) removes code samples with high similarity to the test benchmark, but it does not address potential overlap in the bug-type and label-generation pipelines. Because both training and test sets use GPT-3.5-style generated snippets, the strict-match evaluation may be influenced by label-generation artifacts. Please disclose the overlap and provide a quantitative analysis of how the bug snippets in the test benchmark differ from those in the training distribution (e.g., lexical overlap, snippet length, edit distance).","section":"V-B and References"},{"comment":"The headline differences (e.g., 37.6% vs 10.5% in Table III, and Table II identification metrics) are reported without confidence intervals or significance tests. The authors state that accuracies converge after 500 samples, but this is not a substitute for uncertainty quantification. Please provide bootstrap confidence intervals or a paired significance test over the 2K benchmark samples, especially for the comparison with GPT-4.","section":"VI"}],"minor_comments":[{"comment":"Line 12 uses 'E<start/>', which is not defined in the notation; it should be the start-token embedding or the start-token sequence, and the decoder input is not clearly specified.","section":"Algorithm 1"},{"comment":"The notation 'log Decoder(<start/>||{c<k i })' omits the conditioning on the encoder context; please make the probability explicit, e.g., log p(c_k | c_<k, E_1, ..., E_N).","section":"Eq. (5)"},{"comment":"GPT-3.5 is cited to the GPT-4 technical report [19]; please provide the correct reference for GPT-3.5.","section":"Section IV-A"},{"comment":"The definition of 'strict substring match' is vague: what exactly is 'processed' and how is a match determined? Please provide a concrete example of an accepted and a rejected output.","section":"Section V-C"},{"comment":"The test set is described as 'randomly selected 2K samples from the benchmark in [1]'; please report the exact split seed and the per-bug-type composition to allow reproducibility.","section":"Section V-A"},{"comment":"The abstract contains a broken sentence: 'The absence of reliable test cases requires multi-tasking solutions, performing both bug identification and correction. complicates the multi-tasking required for effective HLS debugging.' This needs rewriting.","section":"Abstract"},{"comment":"The text says the dataset samples start with 'module' and end with 'end module', but the HLS code examples in Figure 1 are C++ functions; please clarify this mismatch between HLS code and Verilog-style keywords.","section":"Section IV-A"},{"comment":"The conclusion says 'HLSDebugger first generates a large-scale labeled dataset'; this should be 'the authors generate', since the model does not generate the dataset.","section":"Section VIII"}],"recommendation":"major_revision","confidential_remarks":"The overlap between the first author of this paper and the authors of the test benchmark reference [1] is a significant independence concern and should be disclosed to the editor. The editor may want to consider whether the evaluation benchmark is sufficiently independent of the training data generation. The paper's main claim of a >3x improvement over GPT-4 in bug correction depends on a metric that appears to favor the fine-tuned model; an execution-based evaluation would settle this."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on the HLSDebugger paper. It's a solid engineering contribution with a real resource: a 300K labeled HLS bug dataset, open-sourced, plus an encoder-decoder model that jointly localizes, types, and corrects bugs. That's genuinely new for HLS; prior work either had much smaller data or didn't do multitask debugging. The identification results are convincing: high recall and precision, and the ablation with HLSEncoder shows the architecture helps.\n\nThe soft spot is exactly the one flagged: the bug correction metric. Section V-C says they use 'strict substring match' after stripping spaces and special characters. The decoder is trained with MLE to emit the canonical label snippet, so it's directly optimized for that metric. GPT-4 is zero-shot and may produce semantically identical but lexically different fixes, which are scored as failures. That doesn't mean the 3x gain is zero—the model likely does correct real bugs more often—but the magnitude is not established. This is a moderate flaw, not a paper-killer.\n\nThere are two other concerns, both secondary. The synthetic bugs are generated by GPT-3.5 using the same eight-type taxonomy as the test benchmark, so the eval is near-distribution. Real-world HLS bugs may be more varied. The authors acknowledge this, sort of. Also, no error bars or seed runs, so the top-1 gap of 79.5% vs 76.7% might be noise; the correction gap is large enough to survive that.\n\nWhat the paper does well: the dataset is a real gift to the community, the method section is clear, and the authors are honest about the model's practical readiness (they admit 37% is not production-ready). The citation pattern is fine; they build on Wan et al. and RTLCoder.\n\nWho is this for? Researchers working on LLM-assisted hardware debugging, especially HLS. The dataset alone justifies a serious look. I'd send it to peer review, but with the request that the authors add a semantic equivalence metric (e.g., test the repaired code, or use human evaluation on a sample), report variance, and ideally evaluate on a small set of real-world bugs. If they do that, the correction claim will be credible.","headline":"Useful dataset and a sensible multitask model, but the headline 3x correction gain is partly an artifact of a substring-match metric that favors the fine-tuned model.","tokens_in":14985,"tokens_out":2277,"would_cite":true,"duration_ms":25543,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HLSDebugger, a specialized encoder-decoder LLM trained on 300K synthetic HLS bugs, identifies and corrects logic bugs without test cases, outperforming GPT-4 by more than 3x in correction.","keywords":["high-level synthesis","logic bug debugging","encoder-decoder LLM","bug localization","bug correction","HLS dataset","multi-task training","large language models"],"falsifier":"Run HLSDebugger on a held-out set of real HLS bug fixes drawn from version-control commit histories, comparing correction accuracy against GPT-4; if the gap narrows sharply or accuracy falls below 20%, the claim is confined to the synthetic training distribution.","tokens_in":14031,"feed_emoji":"🐛","tokens_out":7468,"duration_ms":76248,"temperature":0.7,"pith_summary":"HLSDebugger claims that HLS logic-bug debugging can be automated by a single fine-tuned encoder-decoder LLM that does not need golden test cases. The paper builds a large synthetic dataset of about 300K buggy HLS code samples, each labeled with the buggy snippet, the bug type, and the correction. The model jointly predicts the bug location and type with the encoder and generates the corrected snippet with the decoder, trained by a combined loss that balances identification and correction. On the eight-type benchmark, it reports 37.6% correction accuracy versus 10.5% for GPT-4, plus substantially higher precision and recall at every localization granularity. If the claim holds, it would mean a purpose-built open-source model can outperform a much larger commercial LLM on this specific debugging task.","feed_headline":"Fine-tuned LLM fixes HLS logic bugs 3x better than GPT-4","feed_subtitle":"Custom encoder-decoder finds the buggy line and generates the fix, trained on a new open 300K-sample dataset.","key_machinery":"The load-bearing object is the encoder-decoder transformer with a combined training loss. The encoder processes the tokenized buggy code prefixed with a special [CLS] token; one MLP head reads the [CLS] embedding to predict the bug type, and another head assigns a bug/non-bug probability to every token. The decoder is a second transformer stack whose cross-attention layers take the encoder's embeddings as keys and values, letting the correction step attend to a soft representation of the buggy region rather than to an explicit, possibly wrong, localization string. The training objective sums a type cross-entropy loss, a token-level binary cross-entropy loss with class-imbalance scaling, and a maximum-likelihood generation loss, with the decoder loss weighted highest; this joint loss is what couples identification and correction in one model.","core_discovery":"On its own terms, the paper's central claim is that HLS logic-bug debugging can be decomposed into bug type prediction, token-level bug location, and bug correction, and solved jointly by one encoder-decoder model fine-tuned on a large synthetic dataset. The encoder reads the buggy code and outputs both a bug-type classification and a per-token bug probability; the decoder generates the corrected snippet while attending to the encoder's context embeddings. The authors report that HLSDebugger corrects 37.6% of benchmark samples without being told where the bug is, versus 10.5% for GPT-4, and identifies the buggy line at 79.5% top-1 code-level accuracy. They also report that the same dataset improves a fine-tuned Gemma-7B from 7.5% to 15.1% correction accuracy, and that giving the ground-truth bug location raises HLSDebugger's correction accuracy to 49.1%.","pith_inferences":["Our inference: the real test of the approach is out-of-distribution generalization, since the benchmark spans only the same eight synthetic bug types used for training; a version of the dataset built from real commit-based bug fixes would show whether the gain survives.","Our inference: the combined-loss idea should transfer to RTL and SystemVerilog debugging, where the same error-accumulation pathology appears when LLMs localize then fix; no HLS-specific mechanism is needed for the core argument.","Our inference: a cheap, testable extension is to vary the loss weighting alpha_decoder and measure the trade-off between identification and correction accuracy, which would reveal how tightly the two tasks are coupled."],"forward_implications":["Applying the same encoder-decoder recipe to other code domains without test cases should let bug localization be treated as a supervised token-classification task, avoiding the error accumulation of separate locate-then-correct prompts.","The released 300K-sample dataset is sufficient to lift an open-source 7B model's correction accuracy from 7.5% to 15.1% by supervised fine-tuning alone, so teams without access to large commercial models can still improve.","Because correction accuracy rises to 49.1% when the true buggy location is given, pairing HLSDebugger with an external locator (static analysis or another model) should push fix rates above the no-hint 37.6%.","The model's 87.6% code-wise top-5 line accuracy means a designer who checks up to five candidate lines will find the bug in most samples, which is a practical interaction pattern for a debugging assistant."],"supporting_citations":[{"why":"Supplies the eight bug types and the 2K-sample testing benchmark used for all comparisons.","marker":"[1]"},{"why":"Provides GPT-4 and GPT-3.5 as baselines and GPT-3.5 as the generator of synthetic bug insertions and analyses.","marker":"[19]"},{"why":"Provides the pre-trained CodeT5-Large encoder-decoder architecture that HLSDebugger is fine-tuned from.","marker":"[22]"},{"why":"Supplies the Gemma-7B open-source baseline used for direct fine-tuning comparisons.","marker":"[21]"},{"why":"Provides the RTL dataset converted into extra HLS code samples for dataset augmentation.","marker":"[11]"},{"why":"Supplies the Rouge-L similarity metric used to remove training samples overlapping the test benchmark.","marker":"[23]"}],"fun_headline_variants":["HLSDebugger triples HLS bug correction over GPT-4","Model finds and fixes HLS bugs 3x better than GPT-4","Encoder-decoder LLM beats GPT-4 on HLS bug repair","300K-sample training powers HLS debugger past GPT-4","HLSDebugger: joint bug location and fix beats GPT-4"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The model is trained and evaluated on bugs artificially inserted by GPT-3.5 from a fixed list of eight bug types, so the reported gains may not transfer to naturally occurring HLS bugs outside that list.","fun_headline_variants_meta":{"raw":{"variants":["HLSDebugger triples HLS bug correction over GPT-4","Model finds and fixes HLS bugs 3x better than GPT-4","Encoder-decoder LLM beats GPT-4 on HLS bug repair","300K-sample training powers HLS debugger past GPT-4","HLSDebugger: joint bug location and fix beats GPT-4"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000275,"raw_usage":{"total_tokens":1679,"prompt_tokens":1014,"completion_tokens":665,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":630,"completion_tokens_details":{"reasoning_tokens":569}},"tokens_in":630,"tokens_out":665,"duration_ms":6536,"temperature":1.0,"reasoning_tokens":569,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T12:40:55.488351+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run HLSDebugger on a held-out set of real HLS bug fixes drawn from version-control commit histories, comparing correction accuracy against GPT-4; if the gap narrows sharply or accuracy falls below 20%, the claim is confined to the synthetic training distribution.","supporting_citations":[{"cited_title":"Invited paper: Software/hardware co-design for llm and its application for design verification,","cited_arxiv_id":null,"evidence_quote":"Supplies the eight bug types and the 2K-sample testing benchmark used for all comparisons."},{"cited_title":"Codet5: Identifier- aware unified pre-trained encoder-decoder models for code understand- ing and generation,","cited_arxiv_id":null,"evidence_quote":"Provides the pre-trained CodeT5-Large encoder-decoder architecture that HLSDebugger is fine-tuned from."},{"cited_title":"Rtlcoder: Outperforming gpt-3.5 in design rtl generation with our open-source dataset and lightweight solution,","cited_arxiv_id":null,"evidence_quote":"Provides the RTL dataset converted into extra HLS code samples for dataset augmentation."},{"cited_title":"ROUGE: A package for automatic evaluation of summaries,","cited_arxiv_id":null,"evidence_quote":"Supplies the Rouge-L similarity metric used to remove training samples overlapping the test benchmark."}],"review_version":1}