{"id":"ec67e933-aa4e-4e75-86c5-6f398fb31a97","arxiv_id":"2504.14641","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"HLSTester combines LLM-based testbench adaptation, backward slicing, feedback-guided mutation, and redundancy filtering to automatically detect behavioral discrepancies between C/C++ programs and HLS-generated FPGA circuits.","lead":"This paper presents HLSTester, a framework that uses large language models to automate testing for mismatches between C/C++ programs and the circuits that high-level synthesis generates for FPGAs. The authors report faster detection of behavioral discrepancies and higher testbench pass rates on ten benchmarks compared with direct LLM use and traditional fuzz testing.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Redundancy-aware filtering (Sec. III-D) assumes inputs with recorded min/max and size cannot trigger new discrepancies; this is false for order- and pipeline-sensitive HLS behavior, so the 15.73% speedup and 2.71x totals are not yet validated.","rationale":"The reader's weakest_assumption correctly identifies the redundancy-aware filtering step as the most load-bearing unvalidated assumption. The paper's main quantitative claims—2.71x total speedup, 15.73% additional acceleration from filtering, and the discrepancy-detection curves in Fig. 9—all depend on skipping hardware simulations for inputs deemed redundant. However, the filter's criterion selects only on global min/max and data size, which does not determine the execution behavior relevant to HLS discrepancies: ordering, arrangement, and intermediate values are not captured. Since the framework's own dynamic mutation includes order mutations, and since the motivating examples involve pipelining and data dependencies, the assumption is not merely unproven but likely false in the targeted regimes. The concern is concrete and testable: disable the filter and check whether skipped inputs ever reveal new discrepancies. It does not require rejecting the framework; the other components (LLM-guided testbench modification, backward-slicing instrumentation, dynamic mutation) may still provide genuine value. Therefore the appropriate verdict remains conditional pending this validation, matching the reader's assessment. No ad hominem or theatrical language is needed: this is a technical soundness issue with a clear experimental resolution.","tokens_in":15509,"tokens_out":2539,"duration_ms":27162,"concrete_test":"Run HLSTester on a pipeline- or dataflow-heavy benchmark (e.g., Gain Control or Edge Detection) with redundancy filtering disabled, and log every input that the filter would have skipped. Execute those skipped inputs anyway. Additionally, inject hand-crafted pairs with identical global range and size but different order or intermediate values (e.g., [1,3,5] vs [5,1,3]) and compare recorded spectra and detected discrepancies. If any skipped input expands the spectra range or reveals a new discrepancy, the monotonicity assumption in Sec. III-D is falsified, and the 15.73% filtering speedup must be recomputed after removing missed discrepancies.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central acceleration claim depends on the redundancy-aware filter in Section III-D, which skips hardware simulation whenever a newly generated test input's value range and data size are contained in the currently recorded bounds. This assumes a monotonicity property: no new behavioral discrepancy can arise unless the global min/max or size of the input changes. That property is not established and is very likely false for exactly the discrepancy classes HLSTester targets. In pipelined or dataflow HLS designs, the execution order and intermediate values matter: the same multiset of values with the same min/max and length can trigger an overflow, a division-by-zero, a feedback-path race, or an out-of-bounds access depending on ordering, alignment, or index-dependent accumulation. The paper itself lists 'Order Mutation' among its eight mutation types, and Table I includes pipeline/dataflow discrepancies where order is semantically relevant. Because the filter key is only (min, max, size), two inputs such as [1,3,5] and [5,1,3] are considered equivalent, although an intermediate accumulator or a feedback variable may behave differently. Consequently, the reported 15.73% acceleration from filtering and the overall 2.71x speedup may be inflated: some skipped inputs could be precisely the ones that reveal new discrepancies, and the comparison curves in Fig. 9 and Fig. 13 may undercount discrepancies and overstate speed. This does not invalidate the other components, but it makes the headline efficiency claims not yet supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HLSTester, an LLM-aided framework for testing behavioral discrepancies between original C/C++ programs and circuits produced by high-level synthesis (HLS). The framework has five stages: modifying C/C++ testbenches into HLS-compatible form using an LLM with retrieval-augmented guidance; instrumenting key variables selected by backward slicing; monitoring runtime spectra in both C and RTL simulation; generating test inputs through dynamically prioritized mutation combined with LLM-guided reasoning; and skipping supposedly redundant hardware simulations via a filter based on recorded input value ranges and data sizes. The evaluation uses ten benchmark tasks with the GPT-4o model and Vitis HLS, repeating each experiment 15 times. The paper reports average improvements of 20.67 percentage points in testbench pass rate, higher instrumentation pass rates, more detected discrepancies within a fixed time budget, and an average 2.71x total speedup, of which 15.73% is attributed to redundancy-aware filtering.","tokens_in":15876,"tokens_out":3843,"duration_ms":37035,"significance":"If the reported results hold, HLSTester would be a useful practical contribution to HLS verification: it addresses a real pain point, namely the human effort and long simulation times involved in checking behavioral equivalence between software and synthesized hardware. The evaluation has genuine strengths: pass-rate results are based on actual HLS compilation and simulation outcomes, the experiments use an external industrial tool (Vitis HLS), the benchmark set spans several hardware-directive categories, and ablation variants are used to isolate components. The paper's central weakness is that the speedup claims rest on the redundancy filter's monotonicity assumption, which is neither proved nor empirically validated, so the headline acceleration figures are not yet trustworthy.","major_comments":[{"comment":"The redundancy-aware filter assumes that a test input whose value range and data size lie within previously recorded bounds 'will not trigger new discrepancies' and can therefore skip hardware simulation. This monotonicity assumption is not established, and the paper itself lists mechanisms that violate it: Table II includes Order Mutation (T4), which rearranges elements while preserving range and size, and Table I includes pipeline and dataflow discrepancies in which execution order, alignment, and intermediate values matter. For example, [1,3,5] and [5,1,3] have the same min, max, and size, but an accumulator or feedback-path variable can behave differently on the two inputs. Because skipped inputs may be exactly the ones that reveal new discrepancies, the reported 15.73% filtering speedup and the overall 2.71x speedup are not yet validated, and the cumulative discrepancy curves in Fig. 9 may be undercounted. The authors should either prove the monotonicity property for the discrepancy classes under test or empirically validate it by running the full simulation on all generated inputs and reporting how many filtered inputs would have triggered a new discrepancy.","section":"Section III-D, Fig. 7, Fig. 9, Fig. 13"},{"comment":"All headline comparisons are reported as point estimates from n=15 runs with no confidence intervals, standard deviations, or significance tests. Several pass-rate improvements are only 13.33 or 20 percentage points, which with n=15 is not convincingly significant: for instance, a 13.33-point improvement could arise from two or three additional successful instances. The total-time speedups in Table III and Fig. 13 are likewise reported without variance, so the 2.71x average has no stated uncertainty. The authors should report per-instance data or at least confidence intervals for pass rates and times, and perform a statistical test (e.g., a paired test across the 15 runs) for the key comparisons. This is required for the 'significantly accelerates' claim to be supported by the evidence.","section":"Section IV, Table III, Figs. 10-13"},{"comment":"The dynamic mutation mechanism depends on the update factor alpha=0.04, the 30% ratio of LLM-guided inputs, and the five LLM correction queries per instance, but no sensitivity analysis is reported. The speedups attributed to dynamic mutation (1.81x and 2.28x) and the comparison curves in Fig. 9 are conditional on these specific values. A small ablation or sensitivity study over alpha and the LLM-input ratio would show whether the reported acceleration is robust or tuned to the chosen parameters.","section":"Section III-C, Eq. (4), Fig. 9"}],"minor_comments":[{"comment":"The text says 'mutation type T7 is activated, producing a new test input (ID3: [1, 4, 7, 8])', but Table II lists ID3 as mutation T1 with input [9,1,4,7,8] and ID2 as T7 with input [1,4,7,8]; the text and table should be aligned.","section":"Table II and Section III-C.1"},{"comment":"The heading 'Base. Fuzz Prop.' is ambiguous; it should clearly state that the three columns are the GPT baseline, the traditional fuzz method, and the proposed method.","section":"Table III, columns 8-10"},{"comment":"The description of the 15 repeated instances is terse: it is not stated whether these are independent reruns with different random seeds, whether the same five LLM queries are reused, or whether the pass rates are aggregated over reruns or over benchmark tasks. Clarifying this would help readers interpret the reported averages.","section":"Section IV, general"},{"comment":"In the filtering example, the text states that the recorded input range (2,5) is updated to (1,5) after seeing [1,4,5], but it does not specify how the data-size bound is updated in the same example; a consistent example including size would make the filter rule easier to follow.","section":"Section III-D, running example"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the venue and the authors' prior work on LLM-based HLS automation is relevant. The main concern is not the pass-rate results, which are externally measured and plausible, but the validity of the speedup claims under the redundancy filter's unvalidated monotonicity assumption. The requested empirical validation of the filter is, in my view, a fixable but necessary condition for accepting the acceleration results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is the integrated pipeline: using existing C/C++ testbenches to guide HLS-compatible testbench generation with RAG over HLS manual rules, backward slicing to select instrumentation targets, spectra feedback to steer mutation, and a redundancy filter to skip hardware simulation. That is a real contribution for a pain point in FPGA verification.\n\nWhat it does well: the pass-rate improvements (20.67% average for testbench modification, 25.33% for HLS instrumentation) are measured by actual compilation and simulation outcomes across 15 trials. That is reproducible evidence, not just LLM vibes. The evaluation on ten benchmarks spanning different directives is decent.\n\nThe soft spot is the redundancy-aware filter in Section III-D. It assumes that if a new input's min/max and size are contained in already-seen bounds, the input cannot trigger a new discrepancy. That is not established, and it is probably false for exactly the classes HLSTester targets: pipelined or dataflow hardware where order and intermediate values matter. The paper itself includes Order Mutation among its eight mutation types, so [1,3,5] and [5,1,3] have the same bounds but can behave differently in an accumulator or feedback path. Because the filter skips simulation for those inputs, the reported 15.73% acceleration and the overall 2.71x speedup could be overstated — skipped inputs might be the ones revealing new bugs. This does not sink the whole paper; the other components stand on their own, but the headline efficiency numbers are not yet supported as stated.\n\nMinor issues: no confidence intervals or significance tests despite n=15, and no artifacts released, which hurts reproducibility. The baseline set is narrow (direct GPT and fuzz); a comparison against a coverage-driven RTL fuzzer would help position the work.\n\nThis is a worthwhile paper for the HLS verification community. The integration is novel enough to warrant a serious referee, and the issues are addressable — prove or bound the filter's assumption, or weaken the claim. I would send it to peer review rather than desk reject, with requests for additional experiments on order-sensitive benchmarks and a justification of the filter.","headline":"A sensible integration of LLM-based HLS testbench adaptation, backward slicing, and mutation for discrepancy testing, but the headline speedups rest on an unproven redundancy-filter assumption.","tokens_in":16334,"tokens_out":1750,"would_cite":false,"duration_ms":15310,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An LLM-guided framework detects HLS behavioral mismatches up to 2.71x faster while raising testbench pass rates by about 20.67%.","keywords":["high-level synthesis","behavioral discrepancy testing","LLM-aided verification","testbench generation","runtime spectra","dynamic mutation","redundancy-aware filtering","FPGA"],"falsifier":"Take a benchmark program with a division or array access, run HLSTester until it records a min/max range and size for some input, then feed it a permutation of that same input that preserves the global min, max, and size but places a large value immediately before the divisor or at a boundary index. If the C and RTL spectra differ for the permuted input while the filter would have skipped its hardware simulation, the redundancy assumption is refuted.","tokens_in":15353,"feed_emoji":"🧪","tokens_out":6374,"duration_ms":53406,"temperature":0.7,"pith_summary":"HLSTester is a testing workflow that uses a large language model to adapt existing C/C++ testbenches into HLS-compatible versions, instruments key variables selected by backward slicing, records their runtime spectra, generates new test inputs through dynamic mutation steered by an LLM reasoning chain, and skips hardware simulations it deems redundant. The paper claims this pipeline detects behavioral discrepancies between original C programs and FPGA circuits produced by high-level synthesis faster and more reliably than both direct LLM use and traditional fuzz testing. Concretely, it reports an average 2.71x total speedup, an average 20.67 percentage-point gain in testbench simulation pass rate, and more detected discrepancies within the same time budget across ten benchmark tasks. A sympathetic reader should care because this is the part of the HLS flow that currently demands manual hardware-software expertise, and the paper is arguing that it can be largely automated.","feed_headline":"LLM framework cuts HLS testing time 2.71x and lifts pass rates 20.67%","feed_subtitle":"Combines testbench rewriting, runtime spectra, and mutation to catch C-to-FPGA behavioral mismatches.","key_machinery":"The load-bearing object is the runtime spectrum of key variables, defined as the tuple of variable value range, array access offsets, loop iteration counts, static stack usage, and FIFO queue sizes recorded from instrumented code. Backward slicing from a mismatched output identifies which variables to instrument; the monitored spectra feed two loops: a dynamic-mutation loop that increases the activation probability of mutation types that produce new spectral extremes, and a redundancy filter that skips hardware simulation when a new input's value range and data size stay within recorded bounds. An LLM progressive reasoning chain splits analysis into code, statement, and directive levels to generate targeted inputs. Together these mechanisms convert a one-shot testbench-writing task into a feedback loop in which each hardware simulation is expected to expand the observed behavior space.","core_discovery":"The central claim is that behavioral discrepancy testing for HLS can be made fast enough for routine use by combining an LLM with program-analysis feedback rather than relying on either alone. On its own terms, HLSTester states that its RAG-guided testbench modification raises the HLS simulation pass rate by an average of 20.67% over asking an LLM directly; that its instrumentation pass rates improve by 25.33% for HLS code and 18.67% for C/C++ code; that LLM-guided reasoning gives a 1.81x speedup in detecting all discrepancies, dynamic mutation raises that to 2.28x, and redundancy-aware filtering adds a further 15.73% acceleration; and that the full pipeline finishes the ten benchmark tasks in 27.88% to 72.23% less total time than baselines, with an average 2.71x speedup.","pith_inferences":["The redundancy filter's skip decisions are testable: permuting a recorded input while preserving its global min, max, and size and checking whether C and RTL spectra still match would reveal whether inputs with identical bounds can still trigger new discrepancies.","The probability update only rewards mutations that expand spectral extremes, so mutations that expose a discrepancy without expanding a range would never be reinforced; the reported speedups may therefore depend on how well discrepancy-triggering inputs correlate with extreme-value expansion.","Because all ten benchmarks share the same HLS tool and one LLM, the 20.67% pass-rate gain and 2.71x speedup are not yet evidence about portability to other synthesis tools, directive sets, or LLMs; re-running the workflow on another toolchain would test that.","The framework's pass rate measures simulation success of generated testbenches and instrumented code, not the rate of finding real bugs; a next step would be to measure precision of the reported discrepancy symptoms against known injected defects."],"forward_implications":["Testbench conversion for HLS can be treated as an LLM task grounded in retrieved synthesis rules, with pass rates of roughly 80-100% on the ten benchmarks versus lower rates for direct LLM generation.","Behavioral discrepancies such as custom-bit-width overflow, out-of-bounds static-array access, and pipeline-induced ordering differences can be detected in a few minutes to about two hours per benchmark instead of several hours.","Mutation types that expand monitored spectra extremes become more likely to fire, so test-input generation adapts to the program under test without manual tuning.","Skipping hardware simulations for inputs whose range and size are already covered is claimed to be safe and to contribute a 15.73% workflow acceleration on top of the mutation and reasoning speedups.","With the same time budget, HLSTester reports detecting roughly 5-9 discrepancies across the ten tasks, compared with 1-2 for traditional fuzz testing."],"supporting_citations":[{"why":"Supplies the HLS compatibility rules and error-log guidance that the LLM uses to modify testbenches.","marker":"[8]"},{"why":"Basis for instrumenting HLS-generated designs and observing runtime behavior, informing the spectra-monitoring stage.","marker":"[10]"},{"why":"The fuzzing survey that defines the traditional mutation baseline HLSTester is compared against.","marker":"[12]"},{"why":"Provides several of the benchmark tasks and the LLM-based C/C++ repair context the framework builds on.","marker":"[31]"},{"why":"Supplies additional benchmark tasks and the refactoring-oriented LLM-HLS workflow the testing pipeline extends.","marker":"[32]"},{"why":"Defines the backward-slicing technique used to pinpoint key variables for instrumentation.","marker":"[36]"},{"why":"Supplies the sentence-embedding model that retrieves relevant HLS rule templates for the RAG prompt enhancement.","marker":"[39]"}],"fun_headline_variants":["LLM-driven HLS testing 2.7x faster, pass rate up 20.7%","HLSTester: LLM aids HLS discrepancy tests, 2.7x speedup","LLM plus slicing speeds HLS testing, lifts pass rate","2.7x faster HLS testing via LLM-guided mutation","LLM-assisted HLS testing: 2.7x speedup, 20.7% better pass"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's time savings depend on the assumption that a test input whose value range and data size stay within previously recorded bounds cannot trigger a new behavioral discrepancy; this is not proven, since inputs with the same global min/max and size can differ in ordering or intermediate values and could still overflow, divide by zero, or access out of bounds.","fun_headline_variants_meta":{"raw":{"variants":["LLM-driven HLS testing 2.7x faster, pass rate up 20.7%","HLSTester: LLM aids HLS discrepancy tests, 2.7x speedup","LLM plus slicing speeds HLS testing, lifts pass rate","2.7x faster HLS testing via LLM-guided mutation","LLM-assisted HLS testing: 2.7x speedup, 20.7% better pass"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000155,"raw_usage":{"total_tokens":1243,"prompt_tokens":999,"completion_tokens":244,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":131}},"tokens_in":615,"tokens_out":244,"duration_ms":2596,"temperature":1.0,"reasoning_tokens":131,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:43:21.984994+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a benchmark program with a division or array access, run HLSTester until it records a min/max range and size for some input, then feed it a permutation of that same input that preserves the global min, max, and size but places a large value immediately before the divisor or at a boundary index. If the C and RTL spectra differ for the permuted input while the filter would have skipped its hardware simulation, the redundancy assumption is refuted.","supporting_citations":[{"cited_title":"Vitis High-Level Synthesis User Guide,","cited_arxiv_id":null,"evidence_quote":"Supplies the HLS compatibility rules and error-log guidance that the LLM uses to modify testbenches."},{"cited_title":"Mu-grind: A Framework for Dynamically Instrumenting HLS- Generated RTL,","cited_arxiv_id":null,"evidence_quote":"Basis for instrumenting HLS-generated designs and observing runtime behavior, informing the spectra-monitoring stage."},{"cited_title":"The Art, Science, and Engi- neering of Fuzzing,","cited_arxiv_id":null,"evidence_quote":"The fuzzing survey that defines the traditional mutation baseline HLSTester is compared against."},{"cited_title":"Automated C/C++ Program Repair for High-Level Syn- thesis via Large Language Models,","cited_arxiv_id":null,"evidence_quote":"Provides several of the benchmark tasks and the LLM-based C/C++ repair context the framework builds on."},{"cited_title":"HLSRewriter: Efficient Refactoring and Optimization of C/C++ Code with LLMs for HLS,","cited_arxiv_id":null,"evidence_quote":"Supplies additional benchmark tasks and the refactoring-oriented LLM-HLS workflow the testing pipeline extends."},{"cited_title":"A Survey on Software Fault Localization,","cited_arxiv_id":null,"evidence_quote":"Defines the backward-slicing technique used to pinpoint key variables for instrumentation."},{"cited_title":"Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks,","cited_arxiv_id":null,"evidence_quote":"Supplies the sentence-embedding model that retrieves relevant HLS rule templates for the RAG prompt enhancement."}],"review_version":1}