{"id":"aa8cb093-3f64-4946-b03d-cbd42ae32952","arxiv_id":"2412.15519","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"PreNeT predicts per-epoch training time for deep learning layers, including attention and embedding, using computational operations, memory, and GPU peak FLOPs features, and reports up to 72% improvement over a 2018 baseline.","lead":"This paper introduces PreNeT, a machine learning framework that predicts how long it takes to train a neural network for one epoch by combining layer-level details such as arithmetic operations, memory use, and GPU speed. It reports accuracy improvements up to 72% over an older predictor, mainly by adding attention and embedding layers and richer computational features.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central full-model claim depends on Eq. (13) summing isolated per-layer runtimes; without in-situ per-layer profiling, the additive assumption is unvalidated and could bias epoch predictions.","rationale":"The reader's weakest_assumption correctly identifies Eq. (13)'s additivity and isolated-layer representativeness. I agree this is the load-bearing issue for the full-model training-time claim. The per-layer RMSE improvements over the baseline are plausible and self-contained, but the paper's broader claims about predicting DNN training time depend on how those per-layer predictions compose. The missing validation is an addressable experimental gap, not a fundamental flaw, so the CONDITIONAL verdict is appropriate. No new concern beyond the reader's is identified; the recommended verdict remains UNCHANGED.","tokens_in":14863,"tokens_out":4620,"duration_ms":41383,"concrete_test":"Run PyTorch Profiler on VGG-16 and BERT training for one epoch (e.g., batch 64 on the same GPUs used in Section 5.3) to collect per-layer forward/backward times. Feed the same layer configurations through PreNeT's isolated per-layer predictors and compare the summed isolated predictions to the in-situ per-layer sum. If the ratio differs from 1 by more than the reported epoch-level error (e.g., >5%), Eq. (13)'s additivity is falsified; if it is ~1, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline result (up to 72% improvement) is computed per layer, but the paper's stated purpose is predicting DNN training time per epoch. Section 3.4 (Eq. 13) assumes epoch time is the sum of independently predicted per-layer, per-batch runtimes. The benchmark data (Section 3.1) measures isolated layers with random configurations, 20,000 per GPU, with no optimizer updates, data loading, memory contention, or kernel launch overhead. These overheads do not necessarily factorize across layers. For real models (VGG-16, BERT), Section 5.3 reports only aggregate epoch-level prediction errors; it never checks whether the sum of isolated per-layer times matches the sum of in-situ per-layer times. If isolated runs are faster (or slower) than in-model runs by a constant or architecture-dependent factor, the summed prediction is biased even when every per-layer model is accurate on its own test set. The 27.65% error for VGG-16/P100/batch 64 (vs. 4.50% on V100) suggests such a systematic effect rather than layer-level feature performance.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"PreNeT is a framework for predicting the per-epoch training time of deep neural networks by training machine-learning models on isolated layer measurements across seven Nvidia GPUs. The model uses hand-crafted computational features—layer-specific parameters, computational operations (CO), computational memory (CM), and computational time (CT)—to predict the runtime of each layer, then sums these predictions over layers and batches (Eq. 13) to estimate an epoch's total time. The paper reports that PreNeT achieves up to 72% lower RMSE than a re-implemented 2018 baseline (Justus et al.) for attention, convolutional, embedding, recurrent, and fully connected layers, and that it generalizes to two unseen Nvidia GPUs. Full-model experiments on VGG-16 and BERT report epoch-level prediction errors, with the best case around 4.5% and worst around 27.65%. The central claim is that the proposed feature set materially improves layer-wise training-time prediction and that summing these predictions yields accurate full-model epoch times.","tokens_in":15086,"tokens_out":5600,"duration_ms":47608,"significance":"If the central claim were rigorously supported, PreNeT would be a practical tool for hardware selection and configuration planning, and its extension to transformer-based layers would address a gap in older performance models. The release of code and data on GitHub is a positive step for reproducibility. However, the current evidence is insufficient in several load-bearing respects: the additivity assumption in Eq. (13) is unvalidated, the reported improvements lack statistical error bars, and the comparison is only against a re-implemented 2018 baseline rather than contemporary frameworks. The idea is plausible and the per-layer results are suggestive, but the full-model claims need substantial additional validation.","major_comments":[{"comment":"Eq. (13) assumes that the per-epoch training time equals the sum of independently predicted per-layer, per-batch runtimes. These per-layer rimes were measured in isolation (Section 3.1) on randomly generated configurations, without optimizer updates, data loading, or memory contention. The paper never validates this additivity against in-situ per-layer timings inside VGG-16 or BERT; Section 5.3 reports only aggregate epoch-level errors. The unexplained large gap between VGG-16 errors on V100 (4.50%) and P100 (27.65%) suggests a systematic bias that may arise from the additivity assumption breaking down. The authors should profile at least one full model to compare predicted versus actual layer-wise times, or otherwise bound the effects of kernel launch, data loading, optimizer, and memory contention.","section":"Section 3.4 / Section 5.3"},{"comment":"Section 3.1 states that the median runtime was taken over multiple runs, while Section 4.2 says the mean of five repetitions was used to calculate the runtime. These are different statistics, and the paper must specify which one was used for the reported results. In addition, no variance or confidence intervals are given for any RMSE in Tables 4 and 5, so the improvement percentages (e.g., 72.60% in the Attention/GDBT row) cannot be assessed for statistical significance. The authors should provide error bars, confidence intervals, or significance tests for the key comparisons.","section":"Section 3.1 vs. Section 4.2"},{"comment":"The abstract claims up to 72% improvement over 'contemporary state-of-the-art frameworks,' but the only quantitative comparison is to a re-implemented version of Justus et al. (2018) as described in Section 5.1. Section 6 discusses DNNPerf and Habitat, yet no comparison to these or any other recent framework is made. For Attention, Embedding, and Normalization layers, whose baseline did not exist in the original Justus et al. work, the 'baseline' is the authors' own layer-feature-only model. The claim should either be qualified as 'improvement over a re-implemented 2018 baseline' or be backed by experiments against genuinely contemporary predictors.","section":"Abstract / Section 5.1"},{"comment":"The unseen-hardware experiments cover only two Nvidia GPUs (L4 and RTX A4000), which does not support the abstract's claim of predicting training time on 'novel accelerator architectures' such as TPU, IPU, or Gaudi. Additionally, GPU Model is listed as a one-hot feature in Table 1; for an unseen GPU, the one-hot category is new, and the paper does not explain how this is handled at test time. The evidence therefore supports at most 'unseen Nvidia GPU models' using the continuous hardware features, and the authors should either adjust the claim or add experiments on genuinely novel accelerator types.","section":"Section 5.2 / Table 1"}],"minor_comments":[{"comment":"Equation (13) is garbled in the manuscript (OCR artifacts), making the summation notation illegible; it should be rewritten in standard notation. Also, the variables O and L are used for numbers of layers and batches, while L is used earlier for batch size; notation should be unified.","section":"Section 3.4"},{"comment":"The text contains numerous OCR artifacts (e.g., '!gure', 'con!guration', 'e\"ciency') that obscure meaning. A thorough proofreading and typesetting pass is needed before publication.","section":"Throughout"},{"comment":"The GitHub repository link is described as 'anonymous' but the URL is https://github.com/pacslab/PreNeT, which reveals the lab name. If double-blind review is intended, the link and repository should be anonymized.","section":"Section 4"},{"comment":"The formulas for computational operations for RNN, LSTM, GRU, and Attention are stated without derivation or citation; a brief justification or a note that these are established approximations (with reference) would help readers assess their validity.","section":"Section 3.2.1"},{"comment":"DNNPerf and Habitat are discussed in Related Work but are not included in the experimental comparison. Even a short discussion of why they are not directly comparable would strengthen the evaluation.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"The central idea—adding computational operation, memory, and time features to layer-wise runtime prediction—is plausible, and the per-layer results are encouraging. The main weaknesses are the unvalidated additivity assumption in Eq. (13), the absence of statistical error bars, and the overclaim relative to 'contemporary state-of-the-art frameworks.' I believe the paper can be made publishable with additional experiments (in-situ layer profiling, comparison to recent baselines, and repeated-run statistics), but as it stands the full-model prediction claim is not adequately supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"PreNeT is a modest but useful feature-engineering extension of Justus et al. The genuinely new pieces are the CO/CM/CT features and the benchmark data for attention, embedding, and layer-norm layers across seven GPUs. The per-layer RMSE improvements over the 2018 baseline are plausible, and the authors are honest that CT does not always help—memory-bound layers like attention and embedding do better without it. Shipping code and data is real evidence, and the paper is easy to reproduce in spirit.\n\nThe soft spots are real but addressable. First, the baseline is only the 2018 Justus work. The abstract says \"contemporary state-of-the-art frameworks,\" but there is no comparison to DNNPerf, Habitat, or any recent method; that is an overclaim. Second, there are no variance estimates or significance tests, and Section 3.1 says median while Section 4.2 says mean of five repetitions—an inconsistency that should be fixed. Third, the unseen-hardware section references Figures 2 and 3, but those figures are missing from the text I have; that experiment cannot be checked. Fourth, and most important, the full-model prediction uses Eq. (13), summing isolated per-layer runtimes. The stress-test note is right that additivity is unvalidated. The VGG-16 result on P100 (27.65% error) versus V100 (4.50%) at the same batch size suggests a systematic, hardware-dependent bias, not just layer-level feature performance. An in-situ profiling check or an error decomposition would settle this.\n\nThe LLM claim is loose: BERT is a transformer but not an LLM, and there is no test on a large language model. Similarly, \"novel accelerator architectures\" is not demonstrated; all benchmarks are Nvidia GPUs.\n\nNone of this is fatal. The core method is sound, the benchmark data is a contribution on its own, and the paper's own caveats show clear thinking. The paper is for people doing cost-aware hardware selection and capacity planning, and for researchers in performance prediction who want a layer-level dataset. It deserves a serious referee, but the referee should ask for major revisions: real baselines, error bars, visible figures, an additivity check, and toned-down claims.","headline":"Modest feature-engineering extension with useful new layer benchmarks; the paper overclaims SOTA status and the full-model additivity assumption needs validation before the headline numbers can be trusted.","tokens_in":15617,"tokens_out":2676,"would_cite":false,"duration_ms":24555,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding computational features to layer-wise predictors lowers training-time prediction error by up to 72%.","keywords":["training time prediction","deep neural networks","large language models","computational complexity","memory footprint","GPU benchmarking","layer-wise prediction","resource allocation"],"falsifier":"Run a real VGG-16 or BERT training step while profiling each layer's forward and backward time with standard profiler hooks, then compare the sum of the individually measured layer times with the whole-model step time across batch sizes. If the gap grows systematically with batch size or with the number of memory-heavy layers, the per-layer summation that PreNeT's epoch prediction relies on is wrong.","tokens_in":14668,"feed_emoji":"⏱️","tokens_out":7604,"duration_ms":60054,"temperature":0.7,"pith_summary":"PreNeT is a framework for predicting how long a single training epoch will take before the training run starts. Its central claim is that per-layer prediction accuracy improves sharply when each layer is described not only by its shape and the GPU it runs on, but also by three computational features: the number of arithmetic operations it performs, the memory its weights and tensors occupy, and a hardware-normalized computational time equal to those operations divided by the GPU's theoretical peak FLOP rate. In experiments across attention, convolutional, embedding, recurrent, dense, and normalization layers, the added features lowered RMSE by up to 72% compared with a layer-wise baseline, with the largest gains on attention and recurrent layers, and the trained models predicted per-epoch times on two GPUs they had never seen. If the claim holds, practitioners can compare model configurations and GPU choices by cost and runtime before spending compute.","feed_headline":"Training-time predictor cuts error up to 72% with compute features","feed_subtitle":"PreNeT adds per-layer compute and memory features, then sums predictions to forecast epoch cost on unseen GPUs.","key_machinery":"The load-bearing machinery is the per-layer feature vector $F_{\\text{layer}}$ that combines layer-specific shapes with three computational feature families: CO, the arithmetic operation count (e.g., $CO_{\\text{Dense}} = S \\cdot R_{\\text{in}} \\cdot R_{\\text{out}}$ and $CO_{\\text{Attention}} = P \\cdot X^2 \\cdot R$); CM, the memory required for weights, input, and output tensors; and CT, the computational time $t(O,M) = CO(L)/Z_{\\text{FLOPs}}(M)$, where $Z_{\\text{FLOPs}}(M)$ is the GPU's theoretical peak FLOP rate. A regressor per layer type maps these vectors to median measured runtimes, and the predicted epoch time is the sum over all layers and all batches of the per-layer predictions. These features let the regressor separate how much work a layer demands from how fast a given GPU can supply that work, which is why the framework claims to generalize to GPUs it has not been benchmarked on.","core_discovery":"On the paper's own terms, the discovery is that training-time prediction is materially improved by making the feature set carry the layer's computational load explicitly, rather than leaving the regressor to infer it from shapes alone. PreNeT benchmarks each layer type in isolation on seven GPUs, records median runtimes over 20,000 random configurations, trains per-layer regressors on layer-specific and computational features, and then predicts a full epoch as the sum of the per-layer, per-batch predictions. Across all layer types the computational features beat the baseline, with the best per-layer models reaching RMSE values such as 0.312 ms for attention layers and 7.909 ms for convolutional layers, and with a 72.60% RMSE reduction for attention layers under gradient-boosted trees. The framework also holds out two GPU models during training and reports that the same feature set predicts their per-epoch times with 19.4% to 72.5% lower error than the baseline, and it validates the end-to-end sum on VGG-16 and BERT. The paper takes this as evidence that the computational features, not just the layer shapes or hardware identifiers, carry the signal that determines training duration.","pith_inferences":["If the additivity assumption holds under real training conditions, the same feature design should extend to distributed training by summing per-device segment times, but it would need an explicit communication and synchronization overhead term; the paper does not test that case.","A natural testable extension is a memory-bandwidth-normalized time feature alongside the peak-FLOP time; the paper's results suggest it would improve the already memory-sensitive attention and embedding layers further.","The epoch-level predictor becomes a total-cost predictor only when chained to a convergence-step estimator; the paper names this as future work, but the epoch feature set here is the component that would make such a chain practical."],"forward_implications":["Transformer and LLM configurations can be costed before training, because attention and embedding layers, absent from the earlier baseline, are benchmarked and predicted explicitly.","A new GPU can be scored with no benchmark runs, since the regressors take only hardware specifications and layer features as input.","Layer-type-specific model selection matters: dense, attention, and recurrent layers favor MLP regressors, while convolutional and fully connected layers favor random forests, so a production predictor should keep a per-layer model ensemble.","The feature set can be trimmed by layer type: computational time mainly helps compute-bound layers such as CNN and dense layers, while memory features drive accuracy on memory-bound layers such as attention, embedding, and RNN layers."],"supporting_citations":[{"why":"Supplies the per-layer, per-epoch prediction baseline that PreNeT extends and compares against.","marker":"[18]"},{"why":"Provides the peak-FLOP hardware performance model that motivates the Computational Time feature.","marker":"[35]"},{"why":"Defines multi-head self-attention, whose complexity formula the attention CO feature is built on.","marker":"[44]"},{"why":"Supplies the convolutional-layer operations formula used for the CNN CO feature.","marker":"[20]"},{"why":"Defines the LSTM gating structure behind the recurrent-layer CO formulas.","marker":"[6]"},{"why":"Defines the GRU gating structure behind the recurrent-layer CO formulas.","marker":"[4]"},{"why":"Used as a full-model validation target for end-to-end epoch-time prediction.","marker":"[8]"},{"why":"Used as a full-model validation target for end-to-end epoch-time prediction.","marker":"[39]"}],"fun_headline_variants":["PreNeT predicts training time on unseen GPUs with up to 72% better accuracy","Computational features boost training-time prediction by up to 72%","Per-layer compute metrics cut training time prediction error up to 72%","Training time forecast improves up to 72% using layer compute features","PreNeT: up to 72% more accurate training time on new hardware"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result depends on assuming that the time for one training epoch is exactly the sum of independently predicted per-layer, per-batch runtimes, and that runtimes measured on isolated layers with random configurations behave the same inside a real model; memory contention, kernel launch overhead, optimizer updates, and data loading could all break that additivity.","fun_headline_variants_meta":{"raw":{"variants":["PreNeT predicts training time on unseen GPUs with up to 72% better accuracy","Computational features boost training-time prediction by up to 72%","Per-layer compute metrics cut training time prediction error up to 72%","Training time forecast improves up to 72% using layer compute features","PreNeT: up to 72% more accurate training time on new hardware"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000963,"raw_usage":{"total_tokens":4107,"prompt_tokens":961,"completion_tokens":3146,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":3046}},"tokens_in":577,"tokens_out":3146,"duration_ms":17429,"temperature":1.0,"reasoning_tokens":3046,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:20:58.967457+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a real VGG-16 or BERT training step while profiling each layer's forward and backward time with standard profiler hooks, then compare the sum of the individually measured layer times with the whole-model step time across batch sizes. If the gap grows systematically with batch size or with the number of memory-heavy layers, the per-layer summation that PreNeT's epoch prediction relies on is wrong.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the per-layer, per-epoch prediction baseline that PreNeT extends and compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the peak-FLOP hardware performance model that motivates the Computational Time feature."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the convolutional-layer operations formula used for the CNN CO feature."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the LSTM gating structure behind the recurrent-layer CO formulas."}],"review_version":1}