{"id":"78c384ed-45a2-4715-9846-8e90cb56e2b0","arxiv_id":"2411.16954","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A Random Forest regression model predicts GEMM runtime (R²=0.98) and power (R²=0.78) on an RTX 4070, but the reported best model is an unspecified stacking ensemble and the dataset counts are contradictory.","lead":"This paper trains a machine learning model to predict how fast matrix multiplication runs and how much power it uses on an NVIDIA RTX 4070 GPU. It also measures many CUTLASS configurations and concludes that a 16x16 tile size gives the best balance of speed and energy.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's headline R² values are attributed by the paper's own Table VI to a Stacking Ensemble, while the described and released model is a Random Forest (Algorithm 2); the central claim as stated has no supporting result.","rationale":"I read the paper in good faith and find a real internal inconsistency at the center of the strongest claim. The abstract and Section III-C describe a Random Forest-based model, yet the reported headline metrics match the Stacking Ensemble row in Table VI, not the Random Forest row. Algorithm 2 gives the only concrete model construction and it is a Random Forest with no stacking components. No ensemble details are provided anywhere, so the reader cannot check the model, its hyperparameters, or its training procedure. This is more load-bearing than the power-measurement concern identified by the reader, because even perfectly clean nvidia-smi readings would not resolve the model-attribution problem. The contradictory dataset split (16,128 samples versus 2,076 training plus 519 test) further prevents any independent evaluation. I am not disputing the tiled-kernel analysis, the occupancy table, or the CUTLASS data collection infrastructure; those parts are plausible and potentially useful. The issue is that the paper's own evidence assigns the abstract's numbers to a model that is never specified, so the central claim cannot be verified as stated. The reader's verdict of REJECT remains appropriate because the headline result is unsupported by the manuscript text, but my reason differs from the reader's weakest assumption.","tokens_in":9831,"tokens_out":4762,"duration_ms":47551,"concrete_test":"Obtain the GPPerf training script and dataset. Run the exact Algorithm 2 pipeline (StandardScaler + MultiOutputRegressor(RandomForestRegressor(n_estimators=100, max_depth=6))) on the collected CUTLASS data with a properly documented split, and record runtime/power R². If the Random Forest reproduces 0.9808/0.7783, Table VI is mislabeled. If it produces approximately 0.9456/0.7234, then the abstract's numbers belong to the undisclosed Stacking Ensemble; the paper must either specify that ensemble completely or the central claim must be revised to the Random Forest's actual metrics.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, stated in the abstract and Section III-C, is that a Random Forest multi-output regression model achieves R²=0.98 for runtime and 0.78 for power. The paper's own evidence does not support that attribution. Table IV reports runtime R²=0.9808 and power R²=0.7783, and Table VI assigns exactly these values to the 'Stacking Ensemble' row, while the Random Forest row is 0.9456 and 0.7234. Algorithm 2, the only model construction given, builds a MultiOutputRegressor wrapping RandomForestRegressor; no stacking ensemble, base learners, meta-learner, or weights are specified anywhere. Section IV-C4 says 'stacked model approach' but does not reconcile this with Algorithm 2. Additionally, Section V-B states the dataset has 16,128 samples split into 2,076 training and 519 test samples; 2,076+519=2,595, not 16,128, with no explanation. Consequently the headline numbers are not traceable to the model the paper proposes, and the evaluation set is undefined. This internal inconsistency is independent of any debate about measurement methodology: as written, the abstract's strongest claim cannot be reproduced or even attributed.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an analytical/machine-learning framework for predicting GEMM kernel runtime, power consumption, and energy efficiency on an NVIDIA RTX 4070 (Ada Lovelace). It combines a custom tiled matrix multiplication kernel, CUTLASS-based profiling of 16,128 configurations, and a multi-output Random Forest regressor. The authors report R² scores of 0.9808 for runtime and 0.7783 for power, and claim that tile size 16×16 provides the best balance, with up to 3.2× performance improvement and 22% power reduction. The paper also analyzes SM occupancy and correlation patterns between matrix dimensions and performance metrics.","tokens_in":10114,"tokens_out":5184,"duration_ms":45079,"significance":"The paper's practical contribution includes a large GEMM profiling dataset, an open-source implementation (GPPerf), and a systematic analysis of tile-size effects on performance and power on a modern consumer GPU. If the reported accuracy could be attributed to the described model, the result would be a useful empirical predictor for GEMM configuration selection. However, in the current manuscript, the central claim is not reproducible: the headline R² values are assigned to a stacking ensemble that is never specified, while the only described model (Random Forest, Algorithm 2) shows lower values (0.9456 runtime, 0.7234 power). The dataset split arithmetic is inconsistent, and the power measurements are acknowledged to be noisy due to shared-machine use. These problems must be resolved before the paper's findings can be considered reliable.","major_comments":[{"comment":"The paper's headline accuracy numbers (R²=0.98 for runtime, 0.78 for power) are attributed in Table VI to a 'Stacking Ensemble,' but the only model construction described (Algorithm 2) is a MultiOutputRegressor wrapping RandomForestRegressor. Section IV-C4 mentions a 'stacked model approach' but provides no base learners, meta-learner, or weights, and it is not reconciled with Algorithm 2. As a result, the abstract's central claim is not traceable to the model specified in the paper, and the reported results cannot be reproduced or even attributed. The authors must either describe the stacking ensemble in full (including algorithms, hyperparameters, and code) or revise the abstract and tables to report the Random Forest results actually supported by the methodology.","section":"Abstract / §III-C / §IV-C4 / Table VI / Algorithm 2"},{"comment":"Section V-B states that the dataset comprised 16,128 samples, split into 2,076 training samples and 519 test samples. Since 2,076 + 519 = 2,595, these numbers are inconsistent. The discrepancy is load-bearing because the evaluation set is undefined; the R² values in Table IV cannot be interpreted without knowing the exact split and whether the 16,128 figure includes pre-processing steps such as outlier removal.","section":"§V-B"},{"comment":"The power measurements used to train and evaluate the model were collected via nvidia-smi on a shared machine, and Section V-A2 states that power usage is 'more stable to noise like other users using the same device.' This admission indicates that the power labels may include contributions from other processes, which directly affects the validity of the R²=0.7783 power prediction claim. The authors should either isolate the GPU or clearly quantify the noise contribution and report error bars on the power predictions.","section":"§V-A2"},{"comment":"Equations (1)-(3) present the relationships as Runtime_pred = α·Runtime_actual + β + ε, Power_pred = f(...) ± ε_power, and Energy_pred = Runtime_pred · Power_pred · γ, but these are not analytical models derived from first principles; they are empirical fits to the collected data. The abstract and Section III-C call the framework 'analytical' and present the coefficients α, β, and γ as if they were derived, when in fact they are fitted parameters of a regression. This framing overstates the theoretical contribution. Please revise the terminology to accurately describe the method as a machine-learning-based empirical model.","section":"§III-C and §V-B3"}],"minor_comments":[{"comment":"In Section V-B3(a), the runtime prediction plot is referred to as 'Figure 6' but the actual figure is Figure 7; similarly, in Section V-B3(b), the power prediction graph is called 'Figure 7' but it is Figure 8. Please correct the cross-references.","section":"§V-B3(a) and §V-B3(b)"},{"comment":"In Section V-A1, the phrase 'after tile size researches 16' should be 'reaches 16.'","section":"§V-A1"},{"comment":"In Equation (3), the energy prediction is written as Energy_pred = Runtime_pred · Power_pred · γ, but if the units are compatible, multiplying runtime and power already gives energy; the role of γ as an empirical correction factor should be explained.","section":"§V-B3(c)"}],"recommendation":"major_revision","confidential_remarks":"This paper touches on a timely topic, but the internal inconsistencies are severe enough that I would not consider acceptance in the current form. The authors need to provide a complete description of the stacking ensemble (or drop it and align all reported numbers with the Random Forest model), correct the dataset split inconsistency, and address the power measurement contamination. Given that these are fixable in principle, I recommend major revision rather than rejection, but the revision must be thorough."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Rough but not worthless. The main finding is internally inconsistent: the abstract and Algorithm 2 describe a Random Forest with multi-output regression, yet Table IV reports R² = 0.98 runtime and 0.78 power, and Table VI assigns those exact numbers to a \"Stacking Ensemble\" that is never defined. The paper's central claim is therefore not traceable to any actual model. The dataset split adds to the problem—16,128 samples are said to split into 2,076 training and 519 test, which does not add up.\n\nWhat the paper does well: the CUTLASS profiling sweep over 16,000+ configurations on the RTX 4070 is a useful dataset, and the tile-size analysis is plausible. The finding that 16×16 tiles give the best balance, supported by the occupancy table (dropping from 24 to 6 blocks per SM past tile 16), is concrete and likely correct. The correlation analysis between matrix dimensions and runtime/power is straightforward but reasonable. Releasing code is a plus.\n\nThe soft spots are serious. Beyond the model mix-up, power measurements come from nvidia-smi on a shared machine, and the paper itself notes that other users' load can affect readings. That alone makes the R² 0.78 power claim shaky. There is no error analysis, no confidence intervals, and no discussion of run-to-run variance. The word \"analytical\" is doing too much work for a fitted regression.\n\nThe underlying dataset could help people doing GEMM autotuning or energy-aware scheduling on consumer Ada GPUs, and the tile-size observations are real empirical work. But as written, the headline numbers are not reproducible. The paper needs a clear description of the actual model, a corrected dataset split, and careful re-evaluation before I would trust the numbers.\n\nRecommendation: this deserves a serious referee only if the authors get a chance to fix the model description and evaluation. I would not desk-reject it outright because the data and observations have value, but the current version should not be accepted. Send it out with a strong request for overhaul.","headline":"A useful profiling dataset and a plausible 16x16 tile-size result, but the headline R² numbers belong to an undefined stacking ensemble, not the Random Forest the paper describes.","tokens_in":10650,"tokens_out":2795,"would_cite":false,"duration_ms":26729,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a single Random Forest multi-output model can predict GEMM runtime and power on an NVIDIA RTX 4070 from configuration features alone, reporting $R^2 = 0.98$ for runtime and $R^2 = 0.78$ for power on CUTLASS SGEMM…","keywords":["GEMM performance prediction","GPU energy efficiency","Random Forest multi-output regression","CUTLASS profiling","NVIDIA Ada Lovelace","tiled matrix multiplication","power consumption modeling","arithmetic intensity"],"falsifier":"Run the same 16,128 CUTLASS SGEMM configurations on an idle, dedicated RTX 4070 with kernel-scoped power sampling (for example, NCU or an external power meter) and compare measured per-kernel power against the model's predictions; if the power $R^2$ does not stay near 0.78 or the runtime mean error exceeds the reported 15.57%, the central accuracy claim is refuted.","tokens_in":9637,"feed_emoji":"⚡","tokens_out":9937,"duration_ms":79542,"temperature":0.7,"pith_summary":"The paper's central claim is that machine learning can replace hand-built analytical formulas for predicting GEMM performance and energy on a modern GPU, because interactions among matrix shape, tile size, and memory access are too complex for linear models. To support this, the authors profile 16,128 CUTLASS SGEMM operations on an RTX 4070 and train a Random Forest multi-output regression pipeline that predicts runtime, power, energy, and TFLOPS simultaneously, reporting $R^2 = 0.98$ for runtime (mean error 15.57%) and $R^2 = 0.78$ for power (median error 5.42%). A parallel hand-written tiled kernel isolates tile-size effects and leads to a $16 \\times 16$ tile recommendation that the paper says improves performance up to 3.2× while cutting power by 22% versus baseline. A sympathetic reader would care because, if these accuracies hold, predicting kernel behavior no longer requires exhaustive re-profiling on every new matrix shape.","feed_headline":"Machine learning predicts GEMM speed and power on Ada GPUs","feed_subtitle":"A multi-output model trained on 16,128 CUTLASS runs hits 0.98 R² for runtime and 0.78 for power on NVIDIA's RTX 4070.","key_machinery":"The namable machinery is the stacked multi-output regression pipeline: numerical features are standardized and fed to a Random Forest regressor with 100 trees and maximum depth 6, wrapped to predict runtime, power, energy, and TFLOPS together from a single feature set. The second object is the hand-written tiled GEMM kernel, run with tile sizes 1 through 32, which isolates the shared-memory and occupancy effects that the CUTLASS sweep cannot vary independently. The features include matrix dimensions, derived arithmetic intensity, block sizes, layouts, and alpha-beta scalars, so the model sees the configuration knobs a developer can actually tune.","core_discovery":"On its own terms, the paper establishes that a multi-output Random Forest regressor trained on CUTLASS profiler output can capture GEMM behavior on Ada Lovelace hardware: runtime tracks total computational volume ($r=0.98$ with $M \\times N \\times K$), power tracks the output matrix size more than the contraction dimension ($r=0.80$ with $M \\times N$ versus $r=0.02$ with $K$), and a tile size of $16 \\times 16$ gives the best shared-memory and occupancy trade-off. The measured model beats individual Random Forest and XGBoost baselines and a linear regression baseline on all targets, and the paper reports fitted relationships including $Runtime_{pred} = 0.97 \\cdot Runtime_{actual} + 1.23$. The account also identifies the resource ceiling: moving from tile 16 to tile 32 drops active blocks per SM from 6 to 1 because shared-memory demand scales with tile area.","pith_inferences":["Because the dataset comes from a single RTX 4070, the model's transfer to other Ada or Hopper GPUs is untested; if the same features generalize without retraining, it would become a cheap cross-architecture predictor.","The power labels come from nvidia-smi on a shared machine, so the 5.42% median power error likely understates the error for kernel-only power; an isolated measurement would be a stronger test of the power claim.","The $16 \\times 16$ tile conclusion is drawn from a SIMT SGEMM kernel and may not carry over to tensor-core or split-K GEMM variants, where shared-memory and register pressure behave differently.","A natural next step would be to train the same pipeline on tensor-core GEMMs and on multiple GPU generations and report per-architecture error, separating hardware-specific effects from configuration effects."],"forward_implications":["On the paper's reported results, a developer on RTX 4070-class hardware can query the trained model for unseen GEMM shapes and get runtime predictions with mean error around 15.6% without running the kernel.","Power-aware GEMM scheduling should weight output-matrix dimensions more heavily than contraction depth, since power correlates with $M \\times N$ at 0.80 and with $K$ at only 0.02.","A tile size of $16 \\times 16$ is a practical default for SIMT SGEMM kernels on this architecture: it gives the best runtime and power balance before shared-memory occupancy collapses from 24 to 6 active blocks per SM.","Ensemble stacking over Random Forest and XGBoost is worth the overhead, improving runtime $R^2$ from 0.9623 (XGBoost) to 0.9808.","Energy predictions inherit error from both runtime and power targets, so energy-aware use of the model should not treat its 43% mean energy error as precise."],"supporting_citations":[{"why":"Provides the DNN-based GPU frequency selection baseline for machine-learning performance and energy prediction.","marker":"[1]"},{"why":"Supplies the DSO energy-efficiency optimizer that motivates combining static and dynamic information.","marker":"[2]"},{"why":"Gives the CUDA platform and RTX 4070 hardware context used for kernel implementation and profiling.","marker":"[5]"},{"why":"Establishes the need for machine-learning performance prediction of GPU applications, the problem this paper extends.","marker":"[6]"},{"why":"Presents the analytical GPU power and performance model that the paper positions its approach against.","marker":"[9]"},{"why":"Offers a simpler portable prediction model used as a comparison point for prediction accuracy.","marker":"[10]"},{"why":"Grounds the power-prediction methodology in statistical modeling from performance counters.","marker":"[11]"},{"why":"Provides a simulation-based energy prediction alternative for CUDA kernels.","marker":"[12]"}],"fun_headline_variants":["Random Forest nails GEMM runtime and power on Ada GPUs","GEMM tile 16 gives 3.2x speedup and 22% power cut","Ada GEMM: Random Forest beats baselines with 0.98 R2","ML predicts GEMM runtime and power on RTX 4070 with R2 up to 0.98","0.98 R2 runtime prediction for GEMM on Ada GPUs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The power labels are treated as accurate per-kernel readings even though they come from nvidia-smi on a shared machine, so activity from other users can contaminate the power measurements that the $R^2 = 0.78$ claim rests on.","fun_headline_variants_meta":{"raw":{"variants":["Random Forest nails GEMM runtime and power on Ada GPUs","GEMM tile 16 gives 3.2x speedup and 22% power cut","Ada GEMM: Random Forest beats baselines with 0.98 R2","ML predicts GEMM runtime and power on RTX 4070 with R2 up to 0.98","0.98 R2 runtime prediction for GEMM on Ada GPUs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001316,"raw_usage":{"total_tokens":5412,"prompt_tokens":1048,"completion_tokens":4364,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":664,"completion_tokens_details":{"reasoning_tokens":4255}},"tokens_in":664,"tokens_out":4364,"duration_ms":27430,"temperature":1.0,"reasoning_tokens":4255,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:41:50.267453+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 16,128 CUTLASS SGEMM configurations on an idle, dedicated RTX 4070 with kernel-scoped power sampling (for example, NCU or an external power meter) and compare measured per-kernel power against the model's predictions; if the power $R^2$ does not stay near 0.78 or the runtime mean error exceeds the reported 15.57%, the central accuracy claim is refuted.","supporting_citations":[{"cited_title":"Performance-aware energy-efficient GPU frequency selection using DNN-based models,","cited_arxiv_id":null,"evidence_quote":"Provides the DNN-based GPU frequency selection baseline for machine-learning performance and energy prediction."},{"cited_title":"DSO: A GPU energy efficiency optimizer by fusing dynamic and static information,","cited_arxiv_id":null,"evidence_quote":"Supplies the DSO energy-efficiency optimizer that motivates combining static and dynamic information."},{"cited_title":"CUDA C++ programming guide,","cited_arxiv_id":null,"evidence_quote":"Gives the CUDA platform and RTX 4070 hardware context used for kernel implementation and profiling."},{"cited_title":"Performance prediction of GPU-based deep learning applications,","cited_arxiv_id":null,"evidence_quote":"Establishes the need for machine-learning performance prediction of GPU applications, the problem this paper extends."},{"cited_title":"An analytical model for a GPU architecture with memory-level and thread-level parallelism awareness,","cited_arxiv_id":null,"evidence_quote":"Presents the analytical GPU power and performance model that the paper positions its approach against."},{"cited_title":"A simple model for portable and fast prediction of execution time and power consumption of GPU kernels,","cited_arxiv_id":null,"evidence_quote":"Offers a simpler portable prediction model used as a comparison point for prediction accuracy."},{"cited_title":"Sta- tistical power modeling of GPU kernels using performance counters,","cited_arxiv_id":null,"evidence_quote":"Grounds the power-prediction methodology in statistical modeling from performance counters."},{"cited_title":"Predicting the energy consumption of CUDA kernels using SimGrid,","cited_arxiv_id":null,"evidence_quote":"Provides a simulation-based energy prediction alternative for CUDA kernels."}],"review_version":1}