{"id":"402a32e4-7990-4e97-be34-55a747396ede","arxiv_id":"2412.15548","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A new performance-modeling and search tool, Polaris, uses transfer learning and online RTL simulation to find deep learning accelerator designs with 2.7x lower energy-delay product than the prior DOSA tool.","lead":"The paper presents a tool that uses machine learning to quickly compare many possible designs for deep learning accelerator chips. It claims to find better designs in minutes than existing tools find in hours.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section IV-B reports 212 FireSim samples, but Figure 10 plots training-set sizes up to 1,600; the central accuracy and sample-efficiency claims rest on an unresolved dataset-size inconsistency.","rationale":"The reader's stated weakest assumption was the delay-only validation (Footnote 2): that the Timeloop/FireSim hybrid EDP label shares a Timeloop energy term, so the transfer-learning benefit could be inflated by label leakage, and no delay-only results are shown. That is a legitimate and well-targeted concern about attribution of the improvement to transfer learning. However, the more load-bearing issue is the internal inconsistency in the reported dataset size, which undermines the quantitative evidence for the central claim before attribution is even considered. Section IV-B says 212 FireSim samples; Figure 10 shows training-set sizes up to 1,600; an 80/20 split of 212 samples cannot produce a training set of 1,600. This inconsistency directly affects the headline ρ=0.99 and the 61% sample-savings claim, both of which are the paper's primary contributions. If the dataset is truly 212, Figure 10 is inexplicable; if it is 1,600, the paper's stated sample count and the derived savings are wrong. Either way, the manuscript as written does not permit a reader to verify the central results. This is a stronger and more fundamental problem than the delay-only concern, so I disagree with the reader's choice of weakest assumption, though I note the reader did flag the dataset issue in their rationale. The appropriate verdict is UNVERDICTED: the paper cannot be judged until the authors correct the dataset-size statement and re-run the affected experiments, or release the data and code so the inconsistency can be resolved externally. A conditional acceptance could be justified if the authors convincingly explain the discrepancy, but given that no code or data is provided and the inconsistency is in the core evidence, UNVERDICTED is the honest assessment for now.","tokens_in":19376,"tokens_out":4393,"duration_ms":37286,"concrete_test":"Obtain the authors' FireSim dataset and Starlight training scripts. Count the unique HW/SW configurations in the FireSim dataset, then re-run the Figure 10 ablation and the VII-A2 comparisons with that exact count. If the dataset has 212 samples, the training-set-size axis must be capped near 170 and all correlation values and error bars recomputed; if it has 1,600 samples, correct Section IV-B and recompute the 61% sample-savings claim against DOSA's actual RTL simulation budget. Report the corrected dataset size, the resulting ρ, and the revised sample-efficiency number.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The central claim is that Starlight achieves ρ=0.99 against RTL simulation while requiring 61% fewer high-fidelity samples than DOSA. This claim depends entirely on the size of the FireSim training set. Section IV-B states: 'We collect a total of 216 samples from Timeloop and 212 samples from FireSim.' Yet Section VII-A2 and Figure 10 present an ablation over 'Training Set Size' with x-axis ticks at 400, 700, 1000, 1300, and 1600, and the text refers to 'the full training set.' If the FireSim dataset contains only 212 samples, then an 80/20 split yields at most ~170 training samples, making Figure 10's x-axis impossible. If the dataset actually contains at least 1,600 samples, then Section IV-B understates the dataset by roughly 7.5×, which would materially change the reported 61% sample-savings figure and the characterization of transfer learning. This is not a cosmetic discrepancy: the paper's headline quantitative results—Starlight's accuracy and its training efficiency—cannot be evaluated until this is resolved. The absence of code/data release means the inconsistency cannot be checked externally, so the current manuscript leaves the central claim unverifiable.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Starlight, a transfer-learned deep kernel learning (DKL) model that predicts the energy-delay product (EDP) of the Gemmini DLA, and Polaris, a Bayesian optimization tool that uses Starlight as a surrogate with RTL simulation in the optimization loop. Starlight is first trained as a variational autoencoder with a predictor on Timeloop analytical-model data, and then its encoder is transferred and fine-tuned with FireSim RTL data. The paper claims that Starlight reaches Spearman rank correlation 0.99 against FireSim, trains with 61% fewer high-fidelity samples than the DOSA baseline, and supports Polaris in finding designs that reduce EDP by 2.7x over DOSA within 3.3 hours. The evaluation compares Polaris against Offline Random, DOSA, and Spotlight on four workloads, and includes wall-clock time comparisons. The central methodology is promising, but the current manuscript leaves key quantitative claims ambiguous because of an unresolved dataset-size inconsistency and because the delay-only validation that rules out label contamination appears only in a footnote without supporting results.","tokens_in":19719,"tokens_out":7236,"duration_ms":64058,"significance":"If the claims are correct, this is a significant contribution to accelerator design-space exploration: it is the first work to transfer a performance model trained on a low-fidelity analytical model to predict high-fidelity RTL outcomes, and it provides evidence that online RTL-in-the-loop optimization can outperform offline proxy optimization. The paper also gives creditworthy concrete predictions, including 6,500 predictions per second, Spearman rho values, and wall-clock times in Table II, and it compares against two strong baselines (DOSA and Spotlight). The principal weakness is verifiability: the reported FireSim dataset size (212 samples) is inconsistent with the training-set-size axis in Figure 10, and the claim that delay-only experiments behave identically is asserted but never shown. These issues affect the headline accuracy and sample-efficiency claims, so the paper cannot be accepted in its current form.","major_comments":[{"comment":"Section IV-B states that the authors collected 216 Timeloop samples and 212 FireSim samples, but Figure 10 plots training-set sizes up to 1,600 and Section VII-A2 refers to 'the full training set.' With an 80/20 split of 212 samples, only about 170 training samples would be available, making the x-axis in Figure 10 impossible. If the dataset actually contains at least 1,600 samples, then Section IV-B undercounts it by roughly 7.5x, which would materially change the 61% sample-savings claim and the transfer-learning comparison. Please reconcile the dataset size, report the exact number of training samples used for each curve in Figure 10, and recompute the headline sample-efficiency numbers accordingly.","section":"Section IV-B, Section VII-A2, Figure 10"},{"comment":"The transferability justification in Section IV-C is based on KL divergence between Timeloop-EDP and the hybrid EDP label (Timeloop energy x FireSim delay). Because both distributions share the Timeloop energy term, a low KL divergence does not demonstrate that the delay signal transfers. Footnote 2 claims that all experiments were reproduced using delay-only measurements and that the behavior is identical, but no delay-only results are presented. Please provide the delay-only versions of the accuracy and training-set-size experiments, report the delay-only KL divergences, and clarify in the abstract and Section I that the EDP label is a hybrid measure rather than one measured entirely by RTL simulation.","section":"Section IV-B footnote 2, Section IV-C"},{"comment":"The 61% sample-savings claim is repeated in the Abstract, Section I, and Section VIII, but the manuscript never defines the comparison precisely: it does not state DOSA's training-set size, the number of FireSim samples used to train Starlight in the final configuration, or the formula used to compute 61%. Given the dataset-size ambiguity in Figure 10, this central claim is not evaluable as written. Please add a table or explicit sentence that states the exact sample counts for Starlight and the DOSA baseline and shows how 61% is derived.","section":"Abstract, Section I, Section VIII"}],"minor_comments":[{"comment":"The sentence 'The datasets are collected by performing Sobol sampling [59] cut for space: —a sampling method...' contains a broken phrase 'cut for space:' and should be reworded.","section":"Section IV-B"},{"comment":"Footnote 3 contains the typo 'coorelation coefficient' and should read 'correlation coefficient.'","section":"Section VI-B, footnote 3"},{"comment":"The text says 'Starlight achieves rho >= 0.98 after just 100 trials,' but the context is about epochs of training; this should be '100 epochs' to avoid confusion with the independent trials used for variance reporting.","section":"Section VII-A1"},{"comment":"Section V-B says the hardware design space has '8x32x32 designs,' but Table I lists four spatial-array choices, 32 accumulator sizes, and 32 scratchpad sizes, which is 4x32x32 = 4,096 designs, not 8x32x32. Please correct the count.","section":"Section V-B, Table I"},{"comment":"Table II uses dashes for Spotlight in the software-DSE rows for ResNet-50 and BERT without an explanation; please state why these entries are missing.","section":"Table II"}],"recommendation":"major_revision","confidential_remarks":"The paper is a good fit for the venue and the core idea is publishable, but the central quantitative claims need to be made verifiable. I strongly encourage the authors to release code and data; without them, the dataset-size inconsistency and the absent delay-only validation are difficult to resolve externally. My recommendation is major_revision rather than reject because the issues appear fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nPunchline: Polaris combines two ideas that haven't been put together for DLA design space exploration: transfer learning from a Timeloop-trained model to a FireSim-tuned deep kernel learner, and a Bayesian optimizer that runs RTL simulation inside the loop. The offline-versus-online comparison is clean, and the ablation against DKL-from-scratch, transferred-encoder-plus-NN, and fine-tuned Starlight-Low is exactly the right control set. On top of that, the 10-trial robustness analysis shows real care. The literature review is engaged, and the comparison points to DOSA and Spotlight are the ones that matter. So there is substance here.\n\nBut I can't take the headline numbers at face value in this version. Section IV-B says the FireSim dataset contains 212 samples. Figure 10 then plots training-set sizes out to 1600 and calls that 'the full training set.' Both statements can't be true. The 61% sample-savings claim appears to be read off those curves, so it is unverifiable until this is resolved. That is a factor of eight, not a rounding error.\n\nSecond, footnote 2 says all experiments were repeated using delay-only measurements and behavior was 'identical,' but no delay-only results are shown. Since the EDP label shares Timeloop's energy term in both source and target distributions, the KL divergence of 0.04 used to justify transfer learning could be inflated by that shared component. The delay-only check the authors describe is the right experiment; they just need to show it.\n\nThe 2.7x EDP improvement over DOSA also comes with a caveat the authors partially acknowledge: in HW/SW co-design, Polaris can choose a 32x32 array while DOSA's design space omits spatial array size. In their software-DSE control, where that advantage is removed, Polaris's edge shrinks to around 1.5x. The control is honest, but the abstract's wording overstates the algorithmic gain. And '99% accuracy' is Spearman rank correlation, not absolute prediction error; a small precision issue in the abstract.\n\nNo code or data release accompanies the paper, which makes the dataset inconsistency uncheckable externally. For a systems paper with quantitative efficiency claims, that is a meaningful gap.\n\nBottom line: this deserves a serious referee. The idea is new, the experimental design is mostly sound, and the flaws are fixable. The referee should require the dataset provenance, the delay-only plots, a derivation of the 61% number, and ideally a code/data release. With those, this becomes a strong paper.","headline":"Useful integration of transfer learning and online RTL-in-the-loop BO for DLA design, but the headline accuracy and sample-efficiency claims are unverifiable as written due to a 212-vs-1600 dataset inconsistency and a missing delay-only ablation.","tokens_in":20170,"tokens_out":5966,"would_cite":false,"duration_ms":51601,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A transferred surrogate predicts RTL-level energy-delay with 99% rank accuracy using 61% fewer simulations, and keeping RTL simulation in the optimization loop yields designs 2.7x better in energy-delay product.","keywords":["deep learning accelerators","design space exploration","transfer learning","deep kernel learning","Bayesian optimization","RTL simulation","performance modeling","hardware/software co-design"],"falsifier":"Repeat the Starlight training and the Polaris search using delay-only labels, comparing FireSim delay against Timeloop delay with no shared energy term; if the rank correlation or the 61% sample savings drop materially, the central transfer-learning claim is inflated by the shared energy model, and if they hold, the claim is robust.","tokens_in":19172,"feed_emoji":"⚡","tokens_out":6418,"duration_ms":51655,"temperature":0.7,"pith_summary":"This paper claims that transfer learning can break the accuracy-versus-speed tradeoff in evaluating deep learning accelerator designs. It introduces Starlight, a performance model trained on cheap analytical estimates and then fine-tuned on a small set of slow RTL simulations, reaching 99% rank correlation with RTL-measured energy-delay product using 61% fewer high-fidelity samples than the prior state of the art. The paper also builds Polaris, a Bayesian optimizer that keeps RTL simulation inside the optimization loop; in under 3.3 hours it finds accelerator designs and software mappings that beat the best designs of the offline baseline DOSA by 2.7x in energy-delay product. If true, this means accurate design evaluation no longer requires huge one-time simulation budgets, and online evaluation produces hardware-faithful designs.","feed_headline":"Transfer learning makes AI-chip design search 61% cheaper","feed_subtitle":"A transferred model reaches RTL-grade rank accuracy and guides a fast search that beats prior tools.","key_machinery":"Starlight is built from deep kernel learning (DKL): a variational autoencoder's encoder network (trained with a predictor head that imposes a smooth EDP gradient on the latent space) is transferred by hard weight sharing from Starlight-Low, then attached to a Gaussian process with a Matérn kernel that supplies the uncertainty estimate needed for Bayesian optimization. Polaris wraps this surrogate in an outer hardware loop (enumerating discrete array/scratchpad/accumulator choices) and an inner per-layer software loop (sampling 10,000 Sobol candidates per iteration), selects candidates with an Upper Confidence Bound acquisition function, and evaluates them on the RTL simulator, feeding each result back into Starlight.","core_discovery":"The paper's central claim is that the encoder of a variational autoencoder trained on cheap analytical-model evaluations of a DLA (Starlight-Low) can be transplanted into a deep-kernel-learning model and fine-tuned on a small set of RTL-simulation evaluations to produce Starlight, a surrogate that predicts RTL-measured energy-delay product with Spearman rank correlation 0.99 while using 61% fewer high-fidelity samples than the DOSA baseline. On top of that, the paper claims Polaris—a Bayesian optimizer that keeps an RTL simulator inside the optimization loop—consistently finds designs with lower EDP than offline optimizers, beating DOSA's best designs by an average of 2.7x in under 3.3 hours. The authors argue this is the first demonstration that RTL simulation in the loop, rather than only as a final check, materially improves the quality of the produced hardware/software co-designs.","pith_inferences":["The same transfer-learning recipe could be reused whenever a cheap low-fidelity estimator and a slow high-fidelity validator share structure, so Starlight's architecture is a template for other accelerator families beyond the one evaluated here.","The online-versus-offline result implies that even a 0.99-rank-accurate surrogate leaves a systematic fidelity gap; designers should therefore choose in-loop high-fidelity evaluation whenever the RTL time per candidate is affordable.","If the delay-only experiments promised in the footnote were published, they would either confirm that the method's gains are robust (no energy-term artifact) or bound how much of the 61% saving is due to the shared Timeloop energy model."],"forward_implications":["Training a high-fidelity DLA performance model can be done with 61% fewer RTL simulations, reducing the one-time data-collection bottleneck.","Design space exploration can use RTL simulation as the evaluator without giving up search breadth, because Starlight evaluates ~6,500 configurations per second while Polaris spends RTL time only on chosen candidates.","Online evaluation beats offline evaluation: keeping the high-fidelity simulator in the optimization loop yields designs that are faithful when translated to real hardware, not just optimal under the proxy.","Polaris reaches parity with a 6-hour DOSA run in under 35 minutes and surpasses it by 2.7x in EDP within 3.3 hours.","Because Starlight's initial accuracy is already high, transfer learning itself provides the head start that makes sample-efficient Bayesian optimization possible."],"supporting_citations":[{"why":"Supplies the low-fidelity analytical-model labels used to train Starlight-Low.","marker":"[46]"},{"why":"Supplies the high-fidelity RTL-simulation labels for Starlight and the in-loop evaluations for Polaris.","marker":"[27]"},{"why":"Serves as the state-of-the-art data-driven predictor and offline DSE baseline that Starlight and Polaris are compared against.","marker":"[22]"},{"why":"Provides the deep kernel learning architecture that combines a transferred encoder with a Gaussian process for uncertainty-aware prediction.","marker":"[64]"},{"why":"Supplies the online Bayesian-optimization baseline (Spotlight) and the feature-transform methodology adapted for comparison.","marker":"[52]"},{"why":"Defines the Gemmini accelerator target and the hardware/software design space used in all experiments.","marker":"[19]"},{"why":"Motivates the variational autoencoder architecture (encoder, decoder, latent space) that Starlight-Low builds on.","marker":"[30]"}],"fun_headline_variants":["Transfer learning makes AI chip design search 61% cheaper","Polaris finds AI chip designs 2.7x better in 3.3 hours","AI chip design search: 61% fewer samples, 99% accuracy","RTL-aware optimizer beats prior AI chip design tools by 2.7x"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the knowledge transferred from the analytical model to the RTL simulator comes from genuinely shared performance structure, but because the hybrid energy-delay label uses the same analytical-model energy term in both source and target, the transfer gain could be partly an artifact of that shared measurement.","fun_headline_variants_meta":{"raw":{"variants":["Transfer learning makes AI chip design search 61% cheaper","Polaris finds AI chip designs 2.7x better in 3.3 hours","AI chip design search: 61% fewer samples, 99% accuracy","RTL-aware optimizer beats prior AI chip design tools by 2.7x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000534,"raw_usage":{"total_tokens":2577,"prompt_tokens":963,"completion_tokens":1614,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":1530}},"tokens_in":579,"tokens_out":1614,"duration_ms":10468,"temperature":1.0,"reasoning_tokens":1530,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:19:38.527012+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the Starlight training and the Polaris search using delay-only labels, comparing FireSim delay against Timeloop delay with no shared energy term; if the rank correlation or the 61% sample savings drop materially, the central transfer-learning claim is inflated by the shared energy model, and if they hold, the claim is robust.","supporting_citations":[{"cited_title":"Timeloop: A Systematic Approach to DNN Accelerator Evaluation,","cited_arxiv_id":null,"evidence_quote":"Supplies the low-fidelity analytical-model labels used to train Starlight-Low."},{"cited_title":"Firesim: FPGA- Accelerated Cycle-Exact Scale-Out System Simulation in the Public Cloud,","cited_arxiv_id":null,"evidence_quote":"Supplies the high-fidelity RTL-simulation labels for Starlight and the in-loop evaluations for Polaris."},{"cited_title":"DOSA: Differentiable Model-Based One-Loop Search for DNN Accelerators,","cited_arxiv_id":null,"evidence_quote":"Serves as the state-of-the-art data-driven predictor and offline DSE baseline that Starlight and Polaris are compared against."},{"cited_title":"Deep Kernel Learning,","cited_arxiv_id":null,"evidence_quote":"Provides the deep kernel learning architecture that combines a transferred encoder with a Gaussian process for uncertainty-aware prediction."},{"cited_title":"Leveraging Domain Information for the Efficient Automated Design of Deep Learning Accelerators,","cited_arxiv_id":null,"evidence_quote":"Supplies the online Bayesian-optimization baseline (Spotlight) and the feature-transform methodology adapted for comparison."},{"cited_title":"Gemmini: Enabling Systematic Deep-Learning Architecture Evaluation via Full-Stack Integration,","cited_arxiv_id":null,"evidence_quote":"Defines the Gemmini accelerator target and the hardware/software design space used in all experiments."},{"cited_title":"Auto-Encoding Variational Bayes,","cited_arxiv_id":null,"evidence_quote":"Motivates the variational autoencoder architecture (encoder, decoder, latent space) that Starlight-Low builds on."}],"review_version":1}