{"id":"cdb8090b-1de3-443b-8f87-2b634fb5fb81","arxiv_id":"2508.17826","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LLMulator uses an LLM with digit-wise numeric output, DPO-based dynamic calibration, and synthetic data augmentation to predict dataflow accelerator performance, reporting 12.2% mean absolute percentage error.","lead":"This paper introduces LLMulator, a framework that predicts power, area, flip-flop count, and cycle count of dataflow accelerators using pre-trained large language models. It predicts digits one at a time and adds an online calibration step that refines predictions by profiling real runs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline accuracy is inflated by DPO calibration on test workloads' ground-truth profiles; a held-out calibration evaluation is needed before the generalization claim is credible.","rationale":"The reader's verdict is REJECT, and my concern supports that rejection, so I recommend no change to the reader's verdict. I partially agree with the reader's weakest-assumption identification: the reader focused on the unspecified inference-time protocol for reasoning features, while I see the stronger and more immediate threat as the use of test-workload ground-truth labels to drive DPO calibration before evaluation. Both are load-bearing: the first concerns what the model actually receives at deployment; the second concerns whether the reported accuracy numbers are predictive at all. The paper's central claim—a generalizable, input-adaptive cost model with 12.2% MAPE—requires that cycles predictions improve without having seen the test inputs' ground-truth profiles. Section 5.1 and Section 7.2 do not establish this; they describe an online procedure that uses the profiler's actual result for the exact input being predicted. In that setting the model is being updated with the answer key, so the reported post-calibration error is not a fair estimate of predictive accuracy on unseen inputs or unseen control-flow variations.\n\nAn honest re-evaluation with a disjoint calibration set is straightforward and could plausibly rescue the claim if the method generalizes. Similarly, the reasoning-feature issue is testable by withholding the <think> features at inference and measuring the drop in static-metric accuracy. In both cases the required evidence is absent, and for a conference paper whose abstract promises a generalizable, first-of-its-kind cost model, that absence is decisive. I credit the paper for its clear descriptions of the calibration loop and data-generation pipeline, but those descriptions make the methodological problem explicit rather than hidden.","tokens_in":813,"tokens_out":1411,"duration_ms":62344,"concrete_test":"Re-run Section 7.2's dynamic-cycle evaluation with a held-out calibration protocol: select a disjoint set of workloads for DPO calibration, recalibrate exactly as in Section 5.1, then compute MAPE on the original 14 Table 2 workloads. If the dynamic-cycle MAPE (currently 16.4%) rises materially, or the aggregate 12.2% MAPE exceeds a threshold such as 15%, the headline generalization claim is unsupported. Additionally, for the static-metric results in Table 3, report whether the <think>-tag RTL features were extracted from the test workloads themselves; if so, re-evaluate with those features withheld or predicted by the model.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—12.2% MAPE, 11.2% cycle-error convergence—is not supported as a clean prediction result. Section 5.1 (Figure 4) describes DPO updating on preference triplets built from the profiler's ground-truth y_w for the same {x, data}. Section 7.2 evaluates dynamic cycles for the Table 2 workloads after 5 DPO iterations, and Section 7.4 states LLMulator is \"dynamically calibrated using input profiles collected during TPU runs\" for those same workloads. No held-out calibration set is described. Thus the dynamic-cycle improvements in Table 3 (28.9%→16.4%) and the aggregate 12.2% MAPE are test-label-influenced numbers rather than measurements of generalization to unseen inputs. This undermines the paper's headline contribution.\n\nA secondary ambiguity compounds this: Section 6.2's reasoning format inserts SiliconCompiler-extracted RTL features (module counts, MUX numbers) into the model input. The paper never states whether these features are present at inference or must be generated/predicted by the model. If they require a SiliconCompiler run, the static-metric results are not end-to-end cost-model predictions, and the speed advantage claimed over synthesis collapses. Section 2 also contains an explicit placeholder (\"specific experimental numbers or benchmark results to be added\"), indicating missing support for part of the motivation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"LLMulator proposes an LLM-based cost model for dataflow accelerators that (1) tokenizes numeric values in programs and predicts performance as digit-wise classification with beam search, (2) calibrates cycle predictions online via DPO using profiler feedback, and (3) augments training data with progressively synthesized programs and RTL-level \"thinking\" features. The paper reports a 12.2% average MAPE across static and dynamic metrics, a 9.7% improvement from dynamic calibration, and convergence of cycle-prediction error to 11.2% after several iterations.","tokens_in":33271,"tokens_out":4829,"duration_ms":61720,"significance":"The problem is important and the three proposed components are individually plausible. Treating numeric outputs as categorical digit sequences is a sensible way to avoid regression saturation, and using DPO with execution feedback for input-adaptive calibration is a novel direction. The progressive data synthesizer also addresses a real limitation of existing datasets. However, the current evaluation does not support the central generalization claims: the dynamic-calibration numbers are computed on the same workloads whose ground truth drives the DPO updates, and the static predictions appear to require RTL features extracted by the same SiliconCompiler flow the model is meant to replace. If these issues are fixed with a held-out calibration protocol and a clearly specified inference-time feature pipeline, the contribution could be significant; as it stands, the headline accuracy is not an out-of-sample prediction result.","major_comments":[{"comment":"The dynamic-calibration evaluation leaks test labels. The DPO procedure in Eq. (2) uses the profiler's ground-truth y_w for a state {x, data} to construct preference pairs, and Section 7.4 states that LLMulator is \"dynamically calibrated using input profiles collected during TPU runs\" for the same workloads. Section 7.2 then reports the Dynamic-Cycles MAPE (16.4% after 5 iterations) on those same workloads. This is in-sample fitting, not prediction, and it is not a like-for-like comparison with GNNHLS, TLP, or Tenset-MLP, which are not updated with the corresponding ground-truth profiles. The abstract's \"9.7% improvement\" and \"11.2% convergence\" are therefore not supported as generalization results. The authors need a held-out calibration set: DPO updates on a subset of inputs, evaluation on disjoint inputs, and reporting of both before- and after-calibration MAPE on that held-out set.","section":"§5.1, Fig. 4, Eq. (2); §7.2, Table 3; §7.4, Table 11"},{"comment":"The inference-time availability of the \"thinking\" features is unspecified. The reasoning format inserts SiliconCompiler-extracted RTL-level features (module counts, MUX counts, estimated areas) into the model input, and Table 3's static power/area/FF results are obtained with this format. If these features require running HLS/synthesis at inference time, then the static results are not end-to-end cost-model predictions, and the runtime advantage over synthesis claimed in Table 4 is not demonstrated because the feature-extraction cost is omitted. If the model is expected to predict or bypass these features, that variant must be evaluated explicitly. Please state the inference protocol and report end-to-end latency including any required EDA steps.","section":"§6.2, Figs. 8–10; §7.2, Table 3; §7.1, Table 4"},{"comment":"The confidence-calibration claim is based on a Pearson correlation of -0.44 between the final-layer logit and MSE on 12 randomly sampled workloads. This sample size is too small to establish a calibration property, and a correlation with MSE is not a calibration curve. A proper evaluation would report expected calibration error (ECE) or a reliability diagram across confidence bins, with confidence intervals, on a larger held-out sample.","section":"§7.3, Table 6"},{"comment":"The reported aggregate 12.2% MAPE mixes static metrics (trained and evaluated on the synthesized dataset) and dynamic-cycle metrics (obtained after DPO calibration on the test workloads). The headline comparison against TLP and GNNHLS therefore conflates two different evaluation protocols. The paper should separate the static and dynamic claims, and for the dynamic claim it should compare every method under the same calibration/update protocol, or explicitly state that the baselines are not dynamically calibrated.","section":"§7.2, Table 3"}],"minor_comments":[{"comment":"The second bullet under \"Numerical range compression distortion\" contains an explicit placeholder: \"(specific experimental numbers or benchmark results to be added)\". A submitted manuscript should not contain such placeholders, and the surrounding claims about threefold edge errors and >40% relative error are not yet supported by the text.","section":"§2"},{"comment":"The table formatting is corrupted: rows for Polybench kernels appear merged with values, e.g., \"correlation1.0% 1.6% ...\" and \"covariance28.7% ...\". This makes the per-benchmark results difficult or impossible to verify.","section":"Table 3"},{"comment":"The column label \"No-A\" is never defined. Also, the cycles column shows several workloads where the full pipeline is worse than the No-A configuration (e.g., Tab. 2-6: 51.4% → 67.3%; Tab. 2-8: 12.9% → 31.6%). This contradicts the blanket statement that the dataset synthesizer delivers consistent improvement and needs explanation.","section":"Table 7"},{"comment":"The Timeloop comparison figure is largely unreadable: the text is tiny, axis labels are missing, and the legend is ambiguous. Please redraw it with legible fonts and explicit per-workload labels.","section":"Figure 11"},{"comment":"The evaluation reports pass@5 sampling but gives no standard deviation or confidence intervals across sampling runs. Given the small benchmark set and the stochasticity of LLM decoding, this makes it hard to assess whether the reported differences are meaningful.","section":"§7.1"}],"recommendation":"major_revision","confidential_remarks":"The core idea has merit, but the evaluation section needs a fundamental redesign rather than a local edit. The most serious issue is that the DPO calibration is evaluated on the same test workloads used to obtain its ground-truth labels, so the headline 12.2% MAPE and the 9.7%/11.2% dynamic-calibration numbers are not clean prediction results. The second issue is the unstated inference-time dependence on SiliconCompiler-extracted RTL features. Both are fixable in principle with a held-out calibration protocol and a clarified feature pipeline, so I do not recommend outright rejection, but I would require the revised version to report out-of-sample numbers before the generalization claim is accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper has two genuinely useful pieces—the digit-wise categorical output for numeric prediction and the progressive data synthesizer. The static-metric results (power, area, FF) look credible and the ablations show these components earn their keep. I would not be surprised if the static model alone is a solid step beyond TLP/GNNHLS on out-of-range values.\n\nThe problem is the headline. The 12.2% MAPE and the 28.9%->16.4% dynamic-cycle improvement are not clean prediction numbers. Section 5.1 describes DPO updates built from preference triplets where y_w is the profiler's ground-truth for the same {x, data} that is later reported in Table 3 and Section 7.4. The paper says in 7.4 that LLMulator is \"dynamically calibrated using input profiles collected during TPU runs\" for those same workloads. No held-out calibration set is described. That makes the dynamic-cycle numbers test-label-influenced, closer to fitting than prediction. This is not a minor caveat; it directly undermines the central generalization claim.\n\nA second, more fixable ambiguity: the reasoning format in Section 6.2 inserts SiliconCompiler-extracted RTL features (module counts, MUX numbers) into the input. The paper never says whether these features are available at inference or must be predicted by the model. If they require a synthesis run, the end-to-end speed advantage collapses and the static results are not what they seem. That needs to be stated and tested.\n\nAlso worth flagging: Section 2 contains an explicit placeholder (\"specific experimental numbers or benchmark results to be added\"), which means part of the motivation is unsupported as written. And the confidence-calibration claim rests on r = -0.44 with n = 12, which is suggestive but not strong.\n\nWhat the paper does well: the numerical encoding idea is explained clearly, the dataset synthesizer seems to generalize across memory delays and loop mappings, and the ablation study is reasonably thorough. The authors are transparent about their calibration procedure, which is more sloppy evaluation design than deliberate deception.\n\nWho is this for? People building learned cost models for dataflow accelerators and the LLM-for-EDA crowd. It deserves a serious referee, but the referee should demand a held-out calibration evaluation and a clear statement of the inference-time feature protocol. As submitted, the dynamic calibration claim should not be taken at face value.","headline":"Static modeling and data synthesis are solid contributions, but the dynamic-calibration headline numbers are inflated by DPO updates on the test workloads' own ground-truth profiles, so the generalization claim needs a clean held-out evaluation before it can be believed.","tokens_in":33672,"tokens_out":2058,"would_cite":true,"duration_ms":28021,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLMulator claims a pre-trained LLM can predict dataflow-accelerator performance at 12.2% average error across unseen applications, inputs, and hardware.","keywords":["dataflow accelerators","cost modeling","performance prediction","large language models","numeric modeling","dynamic control flow","design space exploration","high-level synthesis"],"falsifier":"Evaluate LLMulator on the same workloads in direct mode, where the program and parameters are the only inputs and no synthesis-extracted module or multiplexer counts are supplied; if average MAPE rises to the baselines' level, the end-to-end advantage for users who skip synthesis would disappear.","tokens_in":32808,"feed_emoji":"⚙️","tokens_out":6006,"duration_ms":69392,"temperature":0.7,"pith_summary":"The paper claims that a pre-trained large language model can serve as a general-purpose performance predictor for dataflow accelerators—hardware that executes programs as networks of operators moving data through memory and compute units. LLMulator's central bet is that the program-semantic knowledge an LLM already has can be repurposed to estimate cycles, power, area, and flip-flop count for designs it has never seen, across different applications, input sizes, and hardware mappings. To make numbers learnable beyond the training range, the paper converts performance targets into digit-by-digit classification problems rather than regression; to handle programs whose control flow changes with the input, it adds an online calibration loop that refines predictions from execution feedback; and to cover new hardware, it synthesizes training data spanning memory delays, loop mappings, and dataflow structures. If correct, this would mean a designer can get early-cycle estimates for new accelerator designs in about a second per query without hand-built simulator rules, with accuracy competitive with—and often better than—existing learned cost models. The reported headline is an average absolute percentage error of 12.2% across static and dynamic metrics, with dynamic cycle error converging to 11.2% after a few calibration iterations.","feed_headline":"Cost model hits 12.2% error on dataflow accelerators","feed_subtitle":"A pre-trained LLM predicts cycles, power, and area for unseen apps and hardware, adapting when control flow depends on input.","key_machinery":"Numeric modeling is the mechanism that carries the argument: instead of regressing a single number, the model predicts each digit of the performance value as a separate classification decision, ordered from most to least significant, with beam search to correct high-order errors and logits that read as confidence. Supporting it are the progressive tokenizer, which isolates numeric tokens so their length scales with digit count, and the dynamic calibration loop, which uses direct-preference optimization on preference pairs (predicted vs. profiled performance) to adapt to input-dependent control flow. The dataset synthesizer supplies the breadth of (program, hardware, performance) triples that","core_discovery":"LLMulator treats dataflow performance prediction as a language-model task: input is the dataflow graph program, operator implementations, hardware parameters, and runtime input scalars; output is a vector of power, area, flip-flops, and cycles. Its core discovery is that three design choices together make this generalization work. First, progressive numeric modeling: numbers inside the program are isolated and tokenized digit-wise, and the predicted performance value is decoded digit-by-digit from most-significant to least-significant digit using categorical classification with beam search, yielding confidence estimates at each position and reducing extreme-value errors. Second, dynamic pred","pith_inferences":["The paper's accuracy numbers are measured with intermediate RTL-level features supplied by a synthesis tool. A user who wants the speed advantage without running synthesis would need the model to generate those features itself; that path is not evaluated, so the end-to-end speed claim is an open question.","The digit-wise categorical decoding is a general trick: any learned predictor with a bounded target metric could adopt it, though the paper only demonstrates it for power, area, flip-flops, and cycles.","Attention masking in dynamic prediction hints at a compositional extension: predict per-operator costs independently and cache them across design-space iterations, making latency scale with the number of changed operators rather than the whole graph.","A testable extension is to run LLMulator on a held-out accelerator whose control flow depends on input in an unseen way, such as variable-length sequence models, and measure whether the same few DPO iterations still converge."],"forward_implications":["Design-space exploration for dataflow accelerators could use LLMulator as a fast filter before expensive synthesis, since a prediction costs about a second versus minutes to hours for HLS and physical synthesis.","Cost models that output confidence at each digit position would let designers know when an early estimate is unreliable and needs a synthesis check.","The dynamic calibration recipe means a deployed cost model can keep improving as new input profiles arrive, without retraining from scratch.","The progressive data synthesizer's approach could be reused as a data-generation recipe for other learned hardware models, since adding those synthesized examples also lowers error for the baselines tested.","If the reasoning-data variant is used, the model exposes intermediate RTL-level features as a chain of thought, making predictions more interpretable than a black-box regression."],"supporting_citations":[{"why":"Supplies the language-model regression baseline whose fixed output range LLMulator's numeric decoding is designed to improve upon.","marker":"[89]"},{"why":"Supplies the rule-based simulator baseline and the motivation that hand-built rules fail on dynamic control flow.","marker":"[64]"},{"why":"Supplies the graph-based baselines and their HLS benchmark dataset used for comparative evaluation.","marker":"[15, 80]"},{"why":"Supplies the direct-preference optimization algorithm that the dynamic calibration loop is built on.","marker":"[66]"},{"why":"Supplies the open-source synthesis-and-profiling pipeline that produces ground-truth labels and intermediate RTL features.","marker":"[63]"},{"why":"Supplies a dataset and an MLP baseline that reads only coarse input features, motivating the input-adaptive design.","marker":"[92]"}],"fun_headline_variants":["LLM cost model hits 12.2% error on dataflow accelerators","RL calibration cuts cycle error by 9.7% in LLM cost model","Digit-wise tokenization helps LLM predict accelerator costs","LLM predicts cycles, power, area for unseen hardware","Input-adaptive control flow? LLM cost model handles it"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The accuracy claims assume the intermediate hardware features the model reasons over are available at prediction time; if a user needs a fast prediction without running synthesis, the model must generate those features itself, and the paper does not test whether that preserves accuracy.","fun_headline_variants_meta":{"raw":{"variants":["LLM cost model hits 12.2% error on dataflow accelerators","RL calibration cuts cycle error by 9.7% in LLM cost model","Digit-wise tokenization helps LLM predict accelerator costs","LLM predicts cycles, power, area for unseen hardware","Input-adaptive control flow? LLM cost model handles it"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001359,"raw_usage":{"total_tokens":5320,"prompt_tokens":682,"completion_tokens":4638,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":426,"completion_tokens_details":{"reasoning_tokens":4547}},"tokens_in":426,"tokens_out":4638,"duration_ms":35563,"temperature":1.0,"reasoning_tokens":4547,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T16:44:32.400046+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate LLMulator on the same workloads in direct mode, where the program and parameters are the only inputs and no synthesis-extracted module or multiplexer counts are supplied; if average MAPE rises to the baselines' level, the end-to-end advantage for users who skip synthesis would disappear.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the language-model regression baseline whose fixed output range LLMulator's numeric decoding is designed to improve upon."},{"cited_title":"Ying, Anurag Mukkara, Rangharajan Venkatesan, Brucek Khailany, Stephen W","cited_arxiv_id":null,"evidence_quote":"Supplies the rule-based simulator baseline and the motivation that hand-built rules fail on dynamic control flow."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the open-source synthesis-and-profiling pipeline that produces ground-truth labels and intermediate RTL features."},{"cited_title":"Gonzalez, Ion Stoica, and Ameer Haj Ali","cited_arxiv_id":null,"evidence_quote":"Supplies a dataset and an MLP baseline that reads only coarse input features, motivating the input-adaptive design."}],"review_version":1}