{"id":"86b975b5-117e-4999-a396-ceed03158ca0","arxiv_id":"2411.16342","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A data-driven framework predicts GNN inference latency for different accelerator dataflows and uses those predictions to schedule jobs, reporting large speedups in simulation.","lead":"Researchers trained small machine-learning models on simulated data to predict how fast graph neural network workloads run under different accelerator dataflow designs, then used those predictions to schedule inference jobs. The method reports up to 91% accuracy in picking the best dataflow and large scheduling speedups, but all results come from simulation and no code or data was released.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim rests on unvalidated simulator fidelity: all latency labels, optimal-dataflow ranks, and scheduling speedups are STONNE-Omega-internal; a wrong cycle model would invalidate the 91.28% top-1 and speedup figures.","rationale":"I agree with the reader's weakest assumption. The single most load-bearing premise is not 'machine learning can fit a simulator'; it is 'the simulator's relative ordering of dataflows is correct.' The paper is internally consistent: the ablation study (Table 4) shows the added features and log-transform help, and the OOD evaluations are honestly reported. But internal consistency does not establish external validity. I considered the alternative concern that the scheduling comparison is unfair because baselines get random tiling (Figure 6) while the proposed method gets predictor-chosen tiling; however, the authors partially answer this in Figure 7 by giving baselines optimal tiling, at which point the mean completion/execution gaps narrow but turnaround-time gains remain. That issue is secondary to the fact that every label and evaluation metric comes from the same untested simulator. A wrong simulator affects all three pillars: the 91.28% top-1 accuracy, the 3.78% MAPE, and the scheduling speedups. The concrete test above directly probes the ranking fidelity of STONNE-Omega; if it passes, the paper's claims are substantially supported, and if it fails, the headline numbers are simulator artifacts. Therefore the reader's CONDITIONAL verdict should stand unchanged.","tokens_in":12346,"tokens_out":5330,"duration_ms":52467,"concrete_test":"Obtain or build an RTL or FPGA implementation of the STONNE-Omega target architecture (512 PEs, 64B register files, global buffer, NoC) and measure cycle-accurate latencies for all 24 dataflows on a stratified sample of 100 graphs from Graphlaxy.Medium and the real-world test sets. Compare STONNE-Omega's latency predictions with the RTL measurements using Kendall's tau on the ranking of configurations and the top-1 optimal-config agreement. If Kendall's tau is below 0.85 or top-1 agreement is below 75%, the simulator does not faithfully reproduce dataflow ranking, and both the dataflow-selection accuracy and the scheduling speedups should be re-estimated with hardware-measured labels.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's empirical claim—that the learned regressors can pick the best of 24 dataflows and that SJF-Predicted achieves up to 3.17x/6.26x speedups—is only as strong as the simulator that supplies every training label and every evaluation number. Section 3.1 states that STONNE-Omega is 'a framework for accurate simulation of the latency' but no comparison to real silicon, FPGA, or an independently validated cycle-accurate model is reported. The target accelerator is a 512-PE spatial array with 64B register files taken from [5], and the simulator's correctness is inherited by citation. The 24 dataflows differ mostly in tiling/unrolling and inter-phase pipelining; small modeling errors in NoC traffic, global-buffer banking, or degree-dependent PE under-utilization can reorder configs that are close in latency. Since top-1 selection is a ranking problem, even unbiased per-config errors can flip the argmax. The scheduler inherits the same problem: predicted job lengths are simulator cycles, and the predictor-overhead comparison (2,312 cycles vs 18,756 average waiting time) is computed inside the simulation. Thus the 91.28% top-1 accuracy and the 3.17x speedup are currently statements about a modeled system, not about a physical accelerator.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a data-driven framework for predicting GNN inference latency across 24 dataflow configurations on spatial accelerators, using lightGBM regressors trained on STONNE-Omega simulations of synthetic Graphlaxy graphs. It reports top-1 dataflow-selection accuracy up to 91.28% and a MAPE of 3.78% on in-distribution graphs, with substantially worse performance on real-world out-of-distribution datasets. The predictors are then embedded in an SJF-based online scheduler for a heterogeneous multi-accelerator setup, with claimed speedups up to 3.17x in mean completion time and 6.26x in mean execution time over the best feasible baseline.","tokens_in":82,"tokens_out":5938,"duration_ms":134358,"significance":"If STONNE-Omega's cycle counts faithfully represent hardware behavior, the method is a practical, low-overhead way to choose dataflows and to guide online scheduling for GNN inference. The paper has real strengths: it builds a large synthetic training set, includes an ablation study in Table 4, evaluates on held-out real-world graph datasets, and measures predictor cost with a compiled LightGBM runtime. The main limitation is that all labels, all evaluation numbers, and the scheduling results are internal to a simulator that is not validated against silicon, FPGA, or an independently cycle-accurate model in this manuscript. As a result, the headline accuracy and speedup figures are currently claims about a modeled system. The scheduling comparison also uses favorable baseline settings that conflate dataflow selection with scheduling quality. These issues are fixable and do not undermine the value of the proposed framework as a simulation-based design tool, but the hardware-level conclusions need either additional validation or more careful framing.","major_comments":[{"comment":"The entire training and evaluation pipeline uses STONNE-Omega cycle counts as ground truth, but the manuscript provides no validation of the simulator against real hardware or an independently validated cycle-accurate model. Because the task is to rank 24 dataflows, even unbiased per-configuration cycle errors can reorder configurations that are close in latency, and the scheduling results inherit the same issue. Please add a validation study on at least a subset of the 24 configurations (e.g., against FPGA/RTL measurements or a published cycle-accurate model) and report per-configuration error distributions and rank-flip rates, or substantially temper the claims so that they are explicitly about the simulator's model.","section":"Section 3.1"},{"comment":"Features S1--S6 are hand-derived closed-form cycle estimates from the same loop-level analytical model family that underlies STONNE-Omega, and the ablation in Table 4 shows that adding them reduces Graphlaxy.Medium MAPE from 35.77% to 10.76% (and further to 3.78% with log transformation). This means the learned regressor is to a large extent fitting the simulator's own analytical expressions, which weakens the 'data-driven' interpretation and limits expectations for hardware whose behavior deviates from those expressions. Please discuss this explicitly and, if feasible, report the scheduling results with a model trained without the S1--S6 features to separate the contribution of the analytical priors from the learned part.","section":"Table 1 and Table 4"},{"comment":"In the first scheduling scenario, all feasible baselines (Random, FCFS, LCFS, SJF-Nodes, SJF-Edges) use random tiling selection, while the proposed SJF-Predicted uses predicted tiling. The reported speedups 'against the best feasible baseline' therefore conflate dataflow-selection quality with job-ordering quality. A fairer feasible baseline would combine SJF-Nodes or SJF-Edges with the best fixed tiling per dataset (which can be chosen offline from Table 3). Without such a baseline, the 3.17x and 6.26x figures may overstate the scheduling-only benefit.","section":"Section 4.2, Figure 6"},{"comment":"The abstract and conclusion claim 'more than 1000× speedup in turnaround time', but Section 4.2 reports no raw numeric value for this metric; Figures 6 and 7 are normalized separately by the largest value, so the reader cannot verify the claim. Please provide the absolute values or the computed speedup, or remove the claim.","section":"Abstract and Section 4.2"}],"minor_comments":[{"comment":"The formula for S5 uses T_F without a definition; it should be T_Fa (or defined explicitly). The expression for S6, S3 + S5/T_Va, is also dimensionally unclear because S5 already appears to sum over all nodes; please clarify.","section":"Table 1"},{"comment":"The paper says scheduling experiments are run 'over five runs with a fixed seed for all algorithms'; if the same seed is used for all runs, the runs are deterministic repetitions. Please clarify whether multiple seeds were used and report variance across seeds.","section":"Section 4.2"},{"comment":"Normalizing each metric separately by its largest value makes it difficult to compare absolute differences across metrics and algorithms; consider reporting a table of raw mean values alongside the normalized figures.","section":"Figures 6 and 7"},{"comment":"The Pareto inter-arrival distribution is not fully specified; reproducibility would benefit from reporting the shape/scale parameters or the exact procedure used to obtain a mean utilization of 85%.","section":"Section 4.2"},{"comment":"Reference [25] appears twice in the citation list '[19, 25, 25]'; this looks like a typo and should be corrected.","section":"Section 2.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the journal's scope and the idea is timely, but the main reservations are the missing simulator validation and the favorable baseline setup in the scheduler comparison. The work is not, in my view, fatally flawed; a revision that adds a validation study (or clearly reframes the claims as simulation-based) and includes a feasible dataflow-aware baseline in the scheduling experiments could make the contribution solid. The 'more than 1000x turnaround speedup' claim also needs support."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper does something new and useful. It shows that a cheap gradient-boosted regressor trained on simulator labels can pick a good dataflow for a given GNN graph, and that feeding those estimates into an SJF scheduler helps in a heterogeneous multi-accelerator setup. The ablation in Table 4 is well done, the held-out evaluation across several real-world graph datasets is a strong point, and the scheduling experiments separate tiling selection from job-length estimation, which is more thought than most papers in this space put in. The writing is clear and the feature design is sensible.\n\nThe soft spot is the one the stress-test note flags: the entire empirical chain is anchored to STONNE-Omega. There is no comparison to silicon, FPGA, or an independently validated cycle model; the simulator's accuracy is inherited by citation from [5]. For a ranking problem like dataflow selection, even small per-config errors can flip the argmax, and the paper's own out-of-distribution numbers show the regressors degrade sharply (13–40% MAPE, top-1 as low as 19% on some datasets). That does not kill the contribution, but it means the headline 91.28% and 3.17×/6.26× speedups are claims about a modeled system, not physical hardware. The authors should say that clearly and, ideally, validate the simulator or at least report sensitivity.\n\nOther issues are secondary. No code or data is released, which hurts reproducibility. The scheduling numbers come from five fixed-seed runs with no error bars. The abstract leads with the best-case in-distribution result while the out-of-distribution performance is much weaker. And the Figure 6 comparison gives baselines random tiling but the proposed method learned tiling, which conflates two effects—though Figure 7 partially corrects for this by giving baselines optimal tiling.\n\nNet: this is a solid, clearly presented piece of systems research for the GNN-accelerator crowd. It deserves a serious referee. I would recommend conditional acceptance: require hardware validation or a clear statement of the simulator-dependent nature of the results, code/data release, and error bars. If the authors address those, this would be a useful reference for anyone building dataflow-aware GNN schedulers.","headline":"A well-integrated simulation study of learned dataflow selection for GNN accelerators, but the missing hardware validation makes the headline numbers proof-of-concept rather than findings.","tokens_in":13187,"tokens_out":3626,"would_cite":false,"duration_ms":32994,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Learned latency models pick the fastest GNN dataflow 91% of the time","keywords":["graph neural network inference","spatial accelerators","dataflow selection","latency prediction","gradient boosting","online scheduling","shortest job first","simulation-based training"],"falsifier":"Run the trained 24 regressors against latency measurements on a real GNN accelerator (FPGA or ASIC) for graphs drawn from the same test sets; if the top-1 dataflow selection accuracy drops far below the reported 91% (or below random choice for some datasets) or the MAPE climbs far above 3.78%, the simulator-to-hardware transferability premise fails. A cheaper version: compare STONNE-$\\Omega$'s cycle counts to cycle-accurate RTL simulation on a handful of representative graphs.","tokens_in":12130,"feed_emoji":"⚡","tokens_out":4877,"duration_ms":40393,"temperature":0.7,"pith_summary":"This paper claims that the latency of GNN inference on a spatial accelerator depends strongly on which dataflow configuration is used, and that this dependence can be learned: lightweight gradient-boosting regressors, trained only on simulated synthetic graphs, predict execution latency across 24 dataflow configurations accurately enough to select the fastest one for a given graph most of the time. If true, it would let GNN accelerators adapt dataflow per input graph without expensive per-graph simulation, and would enable a practical online scheduler for heterogeneous accelerator systems. The authors report up to 91.28% top-1 accuracy in choosing the optimal configuration, a mean absolute percentage error of 3.78% on in-distribution graphs, and scheduling speedups of up to 3.17x in mean completion time and 6.26x in mean execution time against the best feasible baseline. The same predictor is then used as the job-length estimator in a shortest-job-first scheduler, matching the performance of an oracle that knows true execution times.","feed_headline":"Learned latency models pick the fastest GNN dataflow 91% of the time","feed_subtitle":"Trained on synthetic graphs, the same regressors drive an SJF scheduler that cuts completion time up to 3.17x.","key_machinery":"The central object is a latency predictor mapping a graph, an intra-phase dataflow, and an inter-phase dataflow to an execution latency estimate, implemented as a set of 24 LightGBM gradient-boosted regression models, one per configuration, trained on cycles simulated by STONNE-$\\Omega$. The input features combine ordinary graph statistics (node count, edge count, density, clustering coefficient, degree quantiles) with custom cycle-estimate features for the aggregation and combination phases, and the target latency is log-transformed before training. The scheduling machinery is a shortest-job-first dispatcher that uses the predicted latency as the job-length estimate, deciding both which accelerator in a heterogeneous multi-accelerator system receives a graph and which tiling configuration that accelerator should use.","core_discovery":"On the paper's own terms: GNN inference latency for a given graph and dataflow is a learnable function of cheap graph statistics. Training one gradient-boosted tree per dataflow configuration on latency labels produced by the STONNE-$\\Omega$ simulator over synthetic graphs from Graphlaxy yields regressors whose relative error stays under 5 percent for medium and large in-distribution graphs. Ranking the 24 configurations by predicted latency selects the true optimal dataflow with 91.28 percent top-1 accuracy on the in-distribution medium dataset, and using the same predictions inside a shortest-job-first online scheduler yields up to a 3.17x speedup in mean completion time and 6.26x speedup in mean execution time versus the strongest feasible baseline, closely matching an oracle that knows true job lengths.","pith_inferences":["Our inference: if the simulator-to-hardware gap is small, this result suggests dataflow selection for GNNs could become a per-graph runtime decision in production systems, shifting accelerator design toward more flexible dataflow support instead of fixed, design-time configuration choices.","Our inference: the latency predictors could be reused for other resource-management decisions beyond scheduling, such as choosing among heterogeneous accelerators in a data center, partitioning graphs across devices, or guiding compiler-style mapping searches for GNN layers.","Our inference: a testable extension is to replace or augment the degree-quantile features with spectral graph features (for example, eigenvalues of the normalized Laplacian) and measure whether top-1 accuracy improves on the hardest out-of-distribution datasets, Pattern and Enzymes, where the paper reports the lowest selection accuracy.","Our inference: the reported 1000x turnaround-time speedup is tied to the heavy-load, high-utilization setting built into the experiments, so on lighter workloads the advantage of SJF-Predicted over simpler baselines would shrink; stress-testing across utilization levels would map where the method retains its edge."],"forward_implications":["A single flexible accelerator can choose its dataflow per input graph instead of using a fixed configuration, achieving up to 93.63% latency improvement over a random choice and within 0.56% of the oracle on the medium in-distribution dataset.","The learned dataflow selection transfers to out-of-distribution real-world graphs with limited degradation, staying under 15% from optimal across all tested datasets and under 5% for half of them, suggesting the mapping is not overfit to synthetic graphs.","The same predictors enable an online scheduling policy that approximates the performance of an oracle shortest-job-first scheduler with true job lengths, and does so at negligible runtime cost: roughly 2,312 accelerator cycles per prediction versus an average job waiting time of 18,756 cycles.","Because the predictor's inference cost is constant regardless of graph size, the relative overhead of this approach shrinks as graphs grow, making it progressively more attractive for large GNN workloads."],"supporting_citations":[{"why":"Supplies the sparse/dense multiphase dataflow taxonomy, the STONNE-Omega simulator that produces all latency labels, and the 512-PE accelerator architecture used in simulation.","marker":"[5]"},{"why":"Provides the Graphlaxy synthetic graph generator whose output forms the entire training distribution for the latency regressors.","marker":"[24]"},{"why":"Supplies the LightGBM gradient-boosting implementation used to train the 24 per-configuration latency predictors.","marker":"[9]"},{"why":"Provides the speed-oblivious online scheduling theory that justifies using predicted job lengths in place of exact execution times for the SJF-based scheduler.","marker":"[14]"},{"why":"Supplies the TUDataset real-world graphs (ENZYMES, PROTEINS, REDDIT-BINARY) used as out-of-distribution test sets for both prediction and scheduling experiments.","marker":"[16]"},{"why":"Supplies the SuiteSparse Matrix Collection graphs used as additional real-world test cases for evaluating transfer of the trained predictors.","marker":"[12]"},{"why":"Provides the Pattern and TSP semi-synthetic benchmark graphs used to test prediction and scheduling performance on structured, non-synthetic graph distributions.","marker":"[4]"}],"fun_headline_variants":["Data-driven scheduler speeds GNN inference by up to 6.26x","Predicting optimal GNN dataflow with 91% accuracy using graph stats","Online scheduling with learned latency cuts GNN completion time 3.17x","Learned regressors guide GNN scheduling to match oracle speedups"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central premise is that STONNE-$\\Omega$'s simulated cycle counts accurately represent real GNN accelerator latency; the paper validates predictions only against the simulator, not against silicon, so if the simulator diverges from real hardware, the reported accuracies and speedups describe a modeled system rather than physical accelerators.","fun_headline_variants_meta":{"raw":{"variants":["Data-driven scheduler speeds GNN inference by up to 6.26x","Predicting optimal GNN dataflow with 91% accuracy using graph stats","Online scheduling with learned latency cuts GNN completion time 3.17x","Learned regressors guide GNN scheduling to match oracle speedups"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000259,"raw_usage":{"total_tokens":1588,"prompt_tokens":953,"completion_tokens":635,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":554}},"tokens_in":569,"tokens_out":635,"duration_ms":6622,"temperature":1.0,"reasoning_tokens":554,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:14:45.144886+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the trained 24 regressors against latency measurements on a real GNN accelerator (FPGA or ASIC) for graphs drawn from the same test sets; if the top-1 dataflow selection accuracy drops far below the reported 91% (or below random choice for some datasets) or the MAPE climbs far above 3.78%, the simulator-to-hardware transferability premise fails. A cheaper version: compare STONNE-$\\Omega$'s cycle counts to cycle-accurate RTL simulation on a handful of representative graphs.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the sparse/dense multiphase dataflow taxonomy, the STONNE-Omega simulator that produces all latency labels, and the 512-PE accelerator architecture used in simulation."},{"cited_title":"Bias Reduction via Cooperative Bargaining in Synthetic Graph Dataset Generation","cited_arxiv_id":"2205.13901","evidence_quote":"Provides the Graphlaxy synthetic graph generator whose output forms the entire training distribution for the latency regressors."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the LightGBM gradient-boosting implementation used to train the 24 per-configuration latency predictors."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the speed-oblivious online scheduling theory that justifies using predicted job lengths in place of exact execution times for the SJF-based scheduler."}],"review_version":1}