{"id":"b2c6040c-2e41-4125-86de-a240390d7184","arxiv_id":"2511.05615","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new open benchmark with 683,176 synthesized hls4ml designs plus GNN/transformer surrogates that predict FPGA resources/latency accurately in-distribution but poorly on out-of-distribution scientific models.","lead":"This paper releases a 683,176-sample open dataset of neural networks synthesized to FPGAs through hls4ml, plus GNN and transformer models that estimate FPGA resources and latency in seconds. It also standardizes a benchmark so future hardware-cost estimators can be compared fairly.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract's 'within several percent' accuracy claim is not supported by Table 4: test-set SMAPE is 10–20% for most metrics, so the 75th-percentile claim needs direct evidence.","rationale":"The reader's weakest_assumption focused on the sufficiency of the 18-dimensional per-layer feature vector and the resulting failure to generalize to exemplar architectures. That is a real limitation, but it is explicitly acknowledged by the authors as a diversity gap and as future work; the headline claim is deliberately confined to the synthetic test set. The more immediately load-bearing issue is whether the stated in-distribution accuracy is actually as advertised. The reader's rationale already noted that the headline '75% percentile within several percent' is less precise in the tables, but did not make this the primary weakest assumption. I think it should be: the practical promise of the surrogates depends on the accuracy claim, and Table 4's SMAPE values are one to two orders of magnitude larger than 'several percent' for most metrics, at least in expectation. A heavy-tailed error distribution could reconcile the two, but the paper does not provide the 75th-percentile RPE values needed to verify that. This is a concrete, checkable concern using the released predictions and labels, and it does not undermine the dataset or benchmark contribution. The verdict should remain CONDITIONAL pending that check, with the accuracy claim clarified or corrected.","tokens_in":20534,"tokens_out":6775,"duration_ms":68875,"concrete_test":"From the released wa-hls4ml test set and the authors' prediction outputs, compute the 75th percentile of the absolute relative percent error (using Eq. 4, RPE = |(y_i - yhat_i)/(y_i + 1)| × 100) for each of the six targets and each surrogate model. Also recompute SMAPE without the '+1' term in the denominator. If the 75th-percentile RPE exceeds roughly 5% for BRAM, DSP, Cycles, or II, the Abstract's 'within several percent' overstates the results; if all six metrics are within a few percent, the mismatch is only due to heavy-tailed outliers and the central claim can stand.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim in the Abstract—'models generally predict latency and resources for the 75% percentile within several percent of the synthesized resources on the synthetic test dataset'—is not supported by the paper's own reported metrics. Table 4 gives SMAPE on the same synthetic test set: for the GNN, 19.5% (BRAM), 15.1% (DSP), 11.6% (FF), 11.4% (LUT), 15.7% (Cycles), 13.4% (II); for the transformer, 14.1% (BRAM), 10.8% (DSP), 2.9% (FF), 2.9% (LUT), 10.1% (Cycles), 14.1% (II). These are mean symmetric errors. Unless the error distribution is extremely heavy-tailed so that the 75th percentile lies far below the mean, 'within several percent' is inconsistent with 10–20% mean SMAPE. Additionally, Eq. (2) inserts '+1' in the denominator; for small resource values (e.g., BRAM can be zero), SMAPE is not a true percentage error, which makes the '75th percentile within several percent' formulation even harder to interpret from the reported numbers. The practical value of the surrogate models—shortening the codesign loop from hours to seconds—depends on this accuracy claim being accurate at the stated level.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces wa-hls4ml, an open benchmark and dataset of 683,176 hls4ml-synthesized neural networks (fully connected, 1D/2D convolutional) targeting Xilinx FPGAs, with train/validation/test splits of 478,220/102,472/102,484 samples plus 887 exemplar samples from scientific applications. It defines evaluation metrics (R^2, SMAPE, RMSE, RPE) and presents three surrogate models: a rule4ml-style MLP baseline, a GATv2-based GNN, and a small transformer, each predicting BRAM, DSP, FF, LUT, clock cycles, and initiation interval. The test-set results show strong in-distribution performance for several targets (e.g., transformer R^2 = 0.95 for cycles and II), while exemplar results show substantial degradation. The abstract claims the models 'generally predict latency and resources for the 75% percentile within several percent' on the synthetic test set.","tokens_in":20900,"tokens_out":11116,"duration_ms":109829,"significance":"The dataset and benchmark, if adopted, are a genuinely useful community resource: the scale (683k fully synthesized samples), the open release of full synthesis projects/logs, and the explicit exemplar-based evaluation of distribution shift are all strengths. The GNN and transformer surrogates are reasonable first attempts and show promising in-distribution results on several targets. However, the paper's headline accuracy claim overstates what the reported metrics support, and several evaluation inconsistencies affect the comparability of the results. The benchmark contribution is likely significant; the surrogate-model evaluation needs to be tightened before the paper can be accepted.","major_comments":[{"comment":"The central claim that the models 'generally predict latency and resources for the 75% percentile within several percent' is not supported by Table 4. Mean SMAPE values on the synthetic test set are 10–20% for most metrics (GNN: BRAM 19.5%, DSP 15.1%, FF 11.6%, LUT 11.4%, Cycles 15.7%, II 13.4%; transformer: BRAM 14.1%, DSP 10.8%, Cycles 10.1%, II 14.1%). Only transformer FF/LUT are near 2.9%. If the claim is based on the 75th percentile of RPE, the paper must report those quantile values directly; the box plots alone do not permit numerical verification. Moreover, Eq. (2) and Eq. (4) add 1 to the denominator, so the reported 'percentages' are not standard relative errors and can be materially smaller than true percentage errors for small resources such as BRAM. The abstract should be revised to match the reported metrics or supplemented with exact quantile evidence.","section":"Abstract; §5.2; Eq. (2)"},{"comment":"The manuscript does not state whether the train/validation/test split is performed at the sample level or at the level of distinct model architectures. Since §2.1.1 explains that each architecture is synthesized multiple times with different reuse factors, a sample-level random split can place near-duplicate architectures in both training and test sets, potentially inflating the in-distribution test metrics in Table 4. Please specify the split procedure explicitly and, ideally, report additional metrics on a split that is disjoint at the architecture level.","section":"§2 (Dataset split)"},{"comment":"The GNN's exemplar evaluation appears to use different data from the other models. The caption of Figure 10 states 'Activation layers are removed from the exemplar set to keep a similar input structure as the GNN was trained on,' while no such caveat accompanies the MLP or transformer figures. If the GNN rows in Table 5 are computed on a modified exemplar set, then the cross-model comparisons in Table 5 are not apples-to-apples. The preprocessing must either be applied consistently to all models or the GNN exemplar results should be clearly separated from the others.","section":"Figure 10 caption; Table 5"},{"comment":"The text states 'For dense layers, all models show improved performance compared to the overall set,' but Table 4 directly contradicts this for the GNN and transformer. In the dense subset, the GNN has R^2 = -0.51 for BRAM and -0.74 for DSP, whereas the overall test set has R^2 = 0.51 and 0.89; the transformer has identical R^2 for BRAM (0.39) and DSP (0.29) in both rows. Since dense networks are the largest category in the dataset, this discrepancy is consequential for the claim that the surrogates perform well in-distribution. Please correct the interpretation or clarify what subset the 'Dense' row actually represents.","section":"§5.2; Table 4 (Dense)"}],"minor_comments":[{"comment":"The description of the '18-dimensional feature vector' is ambiguous and appears inconsistent with the subsequent one-hot encoding and concatenation of global attributes. 'Three input and output dimensions' and the listed categorical features do not obviously sum to 18. Please provide the exact feature list and the dimensionality after preprocessing.","section":"§4.2.1"},{"comment":"Training details are incomplete: no learning-rate schedule, dropout rate, weight decay, or random seeds are reported, and the GNN batch size is missing. Given that the paper proposes a benchmark, reporting these details is important for reproducibility.","section":"§4.2.3; §4.3.3"},{"comment":"Minor typos and wording issues: 'convolutional modes also contain dense, flatten, and pooling layers' has a spacing error; 'an large impact' should be 'a large impact'; 'SMPAPE' appears in §5.2 and should be 'SMAPE'.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The dataset/benchmark contribution is solid and likely valuable to the community, but the surrogate-model evaluation needs substantial revision: the abstract overstates the accuracy, the split methodology is underspecified, and the GNN exemplar evaluation is not clearly comparable. These issues should be resolved before publication. Consider whether the target venue expects tighter reporting of training configurations."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Nick,\n\nQuick take: this is a useful community resource — the 683k-sample open synthesis dataset is the real contribution — but the paper oversells the surrogates in the abstract, and Table 4 has at least one number that can't be right. We should engage with it, but the authors have some fixing to do.\n\nWhat's new: wa-hls4ml is the largest open dataset of hls4ml-synthesized networks I've seen, with full projects, logs, and both HLS and logic-synthesis reports. That's a solid foundation for future work. The benchmark metrics and exemplar set are a good start, and they're honest about the dataset's limited diversity. The GNN and transformer surrogates are a reasonable step up from the rule4ml MLP — representing the model as a per-layer graph/sequence is more flexible than the collapsed feature vector.\n\nThe in-distribution results are decent: the transformer gets R² of 0.95 on cycles and II, and SMAPE around 3% for FF/LUT. But the abstract's claim that models 'generally predict latency and resources for the 75% percentile within several percent' is not what Table 4 shows. SMAPE on the same test set is 10-20% for BRAM, DSP, cycles, and II for both models. I suspect they're looking at the 75th percentile of signed RPE, which can be small even when large absolute errors occur in the tails. That's a misleading way to state accuracy, and it needs to be clarified or softened.\n\nTwo other flags. First, there's an internal inconsistency in Table 4: the GNN's RMSE for DSP on the dense subset is 18,950.8, while the overall test-set RMSE is 580.0. Since dense models dominate the test set, that's implausible; probably a typo, but the authors need to fix it. Second, the SMAPE formula adds +1 in the denominator, so for small resource counts (e.g., BRAM can be zero) the 'percent' error is not a real percentage; they should say that explicitly.\n\nThe exemplar results are poor — negative R² in many cases — but they report it openly and attribute it to distribution shift. That's not a fatal flaw; it's a clear limitation. I'd also like to see some uncertainty estimate (e.g., multiple runs or bootstraps) for the model comparison, but that's minor.\n\nBottom line: The dataset is worth having and the benchmark is a useful step. The paper should be revised before acceptance: rewrite the abstract, fix the table, and show the actual percentile-of-absolute-error if they want to keep the claim. I'd send it to review.\n\nCheers.","headline":"Large, open hls4ml synthesis dataset and a reasonable benchmark package; the abstract overstates the surrogate accuracy and Table 4 has at least one implausible value.","tokens_in":21470,"tokens_out":9886,"would_cite":true,"duration_ms":83690,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces the wa-hls4ml dataset of 683,176 fully synthesized neural networks and shows that GNN- and transformer-based surrogate models predict FPGA resource usage and latency within a few percent on the synthetic test set, redu","keywords":["surrogate model","FPGA","hls4ml","resource estimation","latency estimation","high-level synthesis","graph neural network","transformer"],"falsifier":"Synthesize a neural network with a skip connection or other cross-layer interaction that is absent from the training set, run the surrogate, and compare predictions against actual logic synthesis; if the error substantially exceeds the few-percent accuracy reported on the synthetic test set, the claim that the per-layer feature representation suffices is falsified.","tokens_in":20438,"feed_emoji":"⚙️","tokens_out":4342,"duration_ms":44188,"temperature":0.7,"pith_summary":"The authors are trying to establish that a large open dataset of fully synthesized neural network designs, together with graph- and attention-based surrogate models, can make FPGA resource and latency estimation nearly instantaneous and accurate for in-distribution architectures. They build wa-hls4ml, a benchmark containing over 680,000 hls4ml-synthesized fully-connected and convolutional networks targeting Xilinx FPGAs, and train GNN and transformer surrogates on per-layer features. On the synthetic test set, these models generally predict resources and latency within a few percent of the post-logic-synthesis ground truth, for example the transformer achieves 2.9 percent SMAPE for flip-flops and lookup tables and an R² of 0.95 for clock cycles. The authors also show, through seven exemplar scientific models, that accuracy degrades markedly outside the training distribution, often worse than predicting the mean, which they attribute to limited architectural diversity in the dataset. A sympathetic reader would care because this is a concrete path to compressing the FPGA codesign loop from hours to seconds, with a public resource for further work.","feed_headline":"680,000 synthesized networks train fast FPGA cost predictors","feed_subtitle":"GNN- and transformer-based surrogates estimate latency and resources within a few percent for in-distribution designs.","key_machinery":"The key machinery is the per-layer 18-dimensional feature vector plus the network-as-graph encoding, processed by a five-layer GATv2 graph attention network or a two-block transformer with a [CLS] token. The GNN's attention weights dynamically determine which layer relationships matter, and learnable pooling—additive, mean, and max—aggregates node embeddings into a graph-level prediction of six targets: LUTs, FFs, DSPs, BRAM, latency cycles, and initiation interval. The transformer treats each layer as a token with positional encoding and uses the [CLS] output to regress the same six targets.","core_discovery":"The central claim is that resource and latency outcomes of hls4ml logic synthesis can be predicted from a per-layer description of the neural network architecture without running synthesis. The authors represent each layer as a node with an 18-dimensional feature vector—input/output dimensions, precision, reuse factor, strategy, layer/activation type, filters, kernel size, stride, padding, batch normalization, and I/O type—and connect nodes along the sequential dataflow. A GATv2-based graph neural network and a transformer trained on 683,176 synthesized samples achieve the stated few-percent accuracy on the synthetic test set, while the included exemplar models reveal that this representatio","pith_inferences":["The sharp exemplar drop suggests the 18-dimensional local feature vector omits cross-layer interactions such as skip connections and layer fusion that materially change synthesis outcomes; adding explicit graph-level or pairwise features would be a testable fix.","Because the dataset stores the full hls4ml project for every sample, the same prediction problem can be re-targeted to other high-level synthesis tools or FPGA families by re-synthesizing a subset, making the benchmark portable.","While generalization improves, a practical workaround is to use the surrogate only for early design-space pruning and verify the final candidate with a real synthesis run.","The negative R² values on several exemplar metrics, such as the Quarks model, imply that current surrogates are worse than predicting the dataset mean for those cases, so users should not trust them outside the synthetic distribution without additional validation."],"forward_implications":["If a designer's target architecture is in-distribution, surrogate predictions within a few percent let them evaluate design alternatives in seconds rather than hours, without running synthesis.","The open benchmark and standardized metrics (R², SMAPE, RMSE, RPE box plots) give the community a common ground for comparing future resource and latency estimators.","The dataset's full synthesis artifacts—HLS code, intermediate representations, reports, and logs—support downstream tasks beyond regression, such as code- and IR-driven learning and LLM-based HLS assistants.","The authors' own exemplar results imply that broader architectural diversity, not just more samples, is needed for surrogates to generalize to realistic scientific models.","Extending the dataset with skip connections, larger reuse ranges, and more target boards is the stated direction for closing the exemplar gap."],"fun_headline_variants":["Surrogate models predict FPGA costs without synthesis","GNN and transformer estimate hls4ml latency and resources","680k synthesized networks train FPGA cost predictors","Replacing synthesis: ML predicts FPGA resource needs","From netlist to cost: surrogate models skip hardware synthesis"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"That each layer's local attributes and the sequential order of layers fully determine the post-synthesis resource and latency counts, so a model trained on those features can predict unseen architectures; the exemplar results show this premise weakens outside the training distribution.","fun_headline_variants_meta":{"raw":{"variants":["Surrogate models predict FPGA costs without synthesis","GNN and transformer estimate hls4ml latency and resources","680k synthesized networks train FPGA cost predictors","Replacing synthesis: ML predicts FPGA resource needs","From netlist to cost: surrogate models skip hardware synthesis"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000292,"raw_usage":{"total_tokens":1547,"prompt_tokens":757,"completion_tokens":790,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":501,"completion_tokens_details":{"reasoning_tokens":716}},"tokens_in":501,"tokens_out":790,"duration_ms":7647,"temperature":1.0,"reasoning_tokens":716,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T23:38:05.079803+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Synthesize a neural network with a skip connection or other cross-layer interaction that is absent from the training set, run the surrogate, and compare predictions against actual logic synthesis; if the error substantially exceeds the few-percent accuracy reported on the synthetic test set, the claim that the per-layer feature representation suffices is falsified.","supporting_citations":[],"review_version":1}