{"id":"5c59290a-5023-4614-ae0f-737ac82433ca","arxiv_id":"2608.10506","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A characterization of 13,419 CNN configurations on two GPUs reveals target-dependent transferability, and the CARB ensemble predicts energy, latency, and memory with R2 around 0.99 while screening out over 90% of candidates.","lead":"This paper measures energy, latency, and peak memory for 13,419 CNN configurations on two GPUs, and finds that energy and latency scale differently while memory transfers across GPUs better than the other two targets. It then builds CARB, an ensemble predictor with R2 around 0.99, plus a screening workflow that trims large configuration spaces to a small Pareto shortlist before hardware profiling.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The deployed screening pipeline scores candidates with median-imputed telemetry, but Section 7.3 validates only after replacing those features with measured values; the 3/3 hardware check does not certify the workflow as run.","rationale":"The R2≈0.99 claim is internally supported: the held-out split is stratified, the leave-one-batch-tier-out check addresses interpolation, and the ablation shows telemetry is not necessary for accuracy. The load-bearing weakness is therefore not the training-mode SM utilization proxy that the reader highlighted, though that proxy does weaken the characterization narrative. The more decisive gap is that the screening pipeline as deployed differs from the pipeline as validated. In Section 7.1 candidates are scored with per-tier median telemetry; in Section 7.3 the validation replaces those imputed values with measured feature vectors. The reported 3/3 correct budget decisions and the 95.8% classification rate are for the actual-feature predictor, not for the median-imputed screen that produced the Pareto shortlist. Section 6.3 shows an architecture-only retrained model is nearly as accurate, but no experiment tests the full telemetry model under median imputation, nor whether the top-100 or Pareto set is stable under plausible telemetry perturbations. With only three hardware-validated configurations, all in a low-energy corner, the end-to-end screening claim remains unproven. That is an addressable validation gap, so the appropriate verdict is unchanged: accept only after pipeline-level validation is supplied.","tokens_in":14911,"tokens_out":7973,"duration_ms":74782,"concrete_test":"Re-run the exact Section 7.2 workflow with median-imputed telemetry to regenerate the top-100 and the 7-config Pareto shortlist, then hardware-profile all 7 (or a random 50 of the top-100) on the RTX 3080. Also rerun Stage 2 with each of 10 bootstrap samples of per-tier telemetry medians and with the M3 architecture-only model; if the Pareto shortlist membership changes or any re-profiled candidate crosses the 75 J / 500 MB budget, the screening claim is not robust to imputation. Report the false-accept rate on this pipeline-level set, not just on actual-feature test vectors.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing gap is in the screening claim, not the R2. Section 7.1 says all candidates are scored with runtime telemetry fixed to training-set medians stratified by batch-size tier. Section 7.3 then validates CARB after re-running on actual feature vectors—not the synthetic architectural approximations used during screening. The 3/3 correct budget decisions therefore certify a predictor that receives real telemetry, while the deployed two-stage workflow that actually produces the 7-config Pareto shortlist runs on imputed medians. Section 6.3's ablation is not a substitute: it compares a retrained architecture-only model to a full-telemetry model; it does not test the sensitivity of the exact deployed model to median imputation. If imputed telemetry shifts predicted energies at the 75 J boundary or reorders candidates near the Pareto front, the claimed 99.8% reduction and 100% feasible shortlist may not describe the pipeline as run. The hardware check is also too narrow: all three validated configurations are basic depth-14, bs=1, fp32, so they exercise only one corner of the screened space.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a workload characterization of 13,419 ResNet-style CNN configurations on RTX 5090 and RTX 3080 GPUs under controlled clock settings, reporting that energy and latency diverge substantially under high computational demand (35.4× vs 11.2× as batch size grows) and that cross-GPU transferability is target-dependent (energy slope 1.61, latency slope 2.09, memory slope ≈0.92). It then proposes CARB, a cascade-blended ensemble of XGBoost, LightGBM, and ExtraTrees with regime-specific residual correctors, which jointly predicts peak memory, energy, and latency with held-out R² ≈ 0.99 across the three targets. Finally, it demonstrates a two-stage deployment screening workflow that reduces a 3,072-configuration grid to a seven-configuration Pareto shortlist for an RTX 3080 deployment budget, with three of three matched configurations receiving correct budget classifications on real hardware.","tokens_in":15171,"tokens_out":6367,"duration_ms":55413,"significance":"If the prediction and screening claims hold, CARB would be a practically useful tool for pre-deployment, energy-aware CNN selection, and the characterization results would strengthen the case against using FLOPs or latency as proxies for energy. The paper has several genuine strengths: a large and carefully collected dataset with clock locking, cooling intervals, and averaged measurements; a stratified split plus a leave-one-batch-tier-out evaluation (R² 0.956–0.991) showing generalization beyond interpolation; an ablation demonstrating that a telemetry-free mode is within 0.0012 R² of the full model; and an honest acknowledgment that the screening validation is re-run on measured feature vectors. However, the screening workflow is validated with a protocol mismatch—imputed telemetry in deployment versus measured telemetry in validation—and the hardware check covers only a narrow architectural corner. These gaps must be closed before the deployment-screening claims are fully credible.","major_comments":[{"comment":"The screening validation does not validate the screening protocol as run. Section 7.1 Step 1 fixes runtime telemetry features to training-set medians stratified by batch-size tier, so the 3,072 candidates are scored without GPU measurements. Section 7.3 then states that CARB is \"re-run on actual feature vectors—not the synthetic architectural approximations used during screening\" before comparing against the 75 J budget. The 3/3 correct classifications and 2.49 J mean absolute error therefore certify a model that receives real telemetry, not the deployed model that receives imputed medians. The Section 6.3 ablation compares retrained architecture-only and full-telemetry models and does not quantify how median imputation shifts predicted energies near the 75 J boundary or reorders the Pareto front. Please run the actual screening pipeline with imputed telemetry on the held-out RTX 3080 configurations and report budget-classification accuracy and the resulting shortlist, or provide an explicit sensitivity analysis of the 75 J decisions and Pareto ranking to the imputation.","section":"§7.1, §7.3, Table 9"},{"comment":"The hardware validation covers only one corner of the screened space. All three matched configurations in Table 9 are basic-block depth-14, batch size 1, FP32, and differ only in width multiplier. The candidate grid in Section 7.2 includes four block types, three precision formats, and four batch sizes, and the Pareto shortlist spans three unique architecture signatures with input-resolution and early-downsampling variants. The 3/3 success therefore does not support the claim that the seven-configuration shortlist is feasible, nor the broader statement that \"screening decisions are reliable.\" Please validate at least one representative configuration from each architecture signature in the shortlist, or narrow the screening claim to the subspace actually validated.","section":"§7.2, §7.3, Table 9"},{"comment":"The SM utilization used to characterize energy–latency divergence is measured in a different execution mode from the targets. Section 3.3 records GPU utilization over a 2-second window during a training-mode forward-backward pass, while energy and latency are measured in evaluation mode. Figure 2 bins configurations by this training-mode SM utilization and uses the resulting high/low separation to support the 35.4× vs 11.2× divergence claim, and Section 5.1 builds interaction features (batch_x_sm, flops_x_sm) from the same quantity. If training-mode utilization does not represent inference hardware load, the characterization finding and the physical-grounding narrative of the cascade are unsupported. Please report eval-mode SM utilization for the same configurations and re-run the Figure 2 analysis; if the divergence changes materially, revise the characterization claims accordingly.","section":"§3.3, §4.3, Fig. 2"},{"comment":"The cross-GPU transferability asymmetry is reported without uncertainty quantification. The energy fit (y = 1.61x + 52.8), latency fit (y = 2.09x − 2.7), and memory fit (y = 0.92x + 6.7) are presented as point estimates, with the claim that memory transfers \"well\" because its slope is ≈1. Since the design of Section 7 (shared memory model across GPUs, per-GPU models for energy and latency) rests on the distinction between slope > 1 and slope ≈ 1, please report confidence intervals for the slopes and intercepts, the number of configurations in each fit, and a rank-correlation or residual diagnostic to support the rank-preservation claim.","section":"§4.5, Fig. 4"}],"minor_comments":[{"comment":"The statement that feature importances \"independently reproduce\" the Section 4 findings is overstated, because the interaction features and cascade order in Section 5.1 were designed from those same findings; this is a confirmatory consistency check, not an independent replication. Please rephrase to reflect that.","section":"§6.1"},{"comment":"The residual standard deviation analysis that identifies the low-batch regime as the shared axis of elevated error is described only verbally; please include the underlying analysis (e.g., residual SD by batch tier or a figure) that justifies restricting the corrector to batch size ≤ 8.","section":"§5.5, §6.5"},{"comment":"The notation M_param and M_activation is used without explicit definition; Section 3.4 defines activation memory but not the parameter-memory symbol. Please define both quantities in the text or in the table caption.","section":"Table 2, §3.4"},{"comment":"The secondary axis of Figure 1 is labeled \"Latency / Memory,\" but the two quantities have different units (milliseconds versus megabytes); please use separate axes or explicit units to avoid conflating them.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The central prediction-accuracy result is credible and well supported by the held-out evaluations and the leave-one-batch-tier-out test. The main risk is the gap between the deployed screening protocol (median-imputed telemetry) and the validation protocol (measured telemetry), which the manuscript itself openly acknowledges. This is fixable within the paper's scope by validating the actual screening pipeline or adding a sensitivity analysis, and by broadening the hardware check beyond the single architecture corner. I recommend major revision rather than rejection because the core R² claim does not appear circular or unsound."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know: this paper builds a large characterization dataset (13,419 CNN configs on two GPUs) and a cascade ensemble that predicts energy, latency, and peak memory with held-out R2 around 0.99. That part is credible. The more interesting novel claims are empirical: energy and latency diverge sharply as batch size grows, and cross-GPU transferability is target-dependent—memory transfers with slope near 1, energy and latency do not. These are useful results if they hold up.\n\nThe measurement methodology is careful: clock locking, cooling intervals, averaging over 100 runs, and a leave-one-batch-tier-out evaluation that addresses interpolation concerns. The ablation showing that removing telemetry costs at most 0.0012 R2 is a nice result—it means predictions are robust even without live GPU instrumentation.\n\nThe soft spots are real but not catastrophic. The biggest one is a mismatch in the screening section. Section 7.1 says candidates are scored with runtime telemetry fixed to training-set medians; Section 7.3 then validates by re-running CARB on measured telemetry. So the 3/3 budget-classification check certifies a model that receives real telemetry, not the deployed pipeline that runs on imputed medians. The ablation in 6.3 is not a substitute because it compares full-telemetry to no-telemetry, not the exact imputed-telemetry model used in screening. That said, the ablation suggests telemetry is nearly irrelevant, so this may be minor; but the paper should test imputed versus measured telemetry directly. Also, the three validated configurations are all basic depth-14, bs=1, fp32—one corner of the space.\n\nSecond, the SM utilization used in characterization comes from a training-mode forward-backward pass, while energy and latency are measured in eval mode. This proxy drives part of the energy-latency divergence argument. The raw batch-size medians in Table 3 already show divergence without SM utilization, so the finding is probably robust, but the paper should either measure utilization in eval mode or state the limitation.\n\nThird, the baselines are too weak. Comparing to FLOPs-only linear regression and a latency-as-energy proxy does not tell us whether CARB beats existing predictors like NeuralPower or nn-Meter. The novelty claim needs that comparison.\n\nAlso: no code or data release, and cross-GPU slopes are reported without confidence intervals, so the \"non-unit\" claim is plausible but not quantified.\n\nOverall, this deserves a serious referee. It's a good empirical study with a real screening claim that needs one more validation pass. The dataset, if released, would be valuable. I'd bring it to a reading group and would engage with a revision.","headline":"A credible R2~0.99 prediction study with a real, but addressable, validation gap in the screening workflow and baselines too weak to prove superiority.","tokens_in":15668,"tokens_out":5200,"would_cite":true,"duration_ms":41350,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single model predicts CNN energy, latency, and memory at R2 ≈ 0.99","keywords":["CNN inference cost prediction","energy-latency divergence","GPU telemetry","cascade ensemble","deployment screening","Pareto optimization","peak memory prediction","cross-GPU transferability"],"falsifier":"Measure the same configurations on a third GPU (or on the same GPUs) with SM utilization sampled during evaluation-mode inference only, and retrain CARB; if the $R^2 \\approx 0.99$ accuracy, the 3× energy-latency divergence, or the cross-GPU slope pattern (energy 1.61×, latency 2.09×, memory ≈1) does not reproduce, the characterization-guided design loses its empirical foundation.","tokens_in":14717,"feed_emoji":"⚡","tokens_out":5262,"duration_ms":44118,"temperature":0.7,"pith_summary":"This paper tries to establish that the three practical costs of running a CNN on a GPU—energy, latency, and peak memory—cannot be safely inferred from FLOPs or from each other, and that a data-driven ensemble can predict all three before deployment. From 13,419 configurations measured on two GPU platforms, it reports that energy and latency diverge by a factor of about 3 under heavy computational demand, and that energy and latency need platform-specific models while peak memory transfers across the tested GPUs. The proposed framework, CARB, cascades memory, energy, and latency predictions through a blended tree ensemble and corrects residuals in the low-batch regime, reaching $R^2 \\approx 0.99$ for all three targets. If correct, this lets practitioners score thousands of design candidates in seconds and reserve hardware profiling for a Pareto-prioritized shortlist.","feed_headline":"One model predicts CNN energy, latency, and memory at R2 ≈ 0.99","feed_subtitle":"CARB scores thousands of GPU deployment candidates in seconds and cuts the profiling shortlist by over 90%.","key_machinery":"The central object is CARB, a cascade-blended ensemble with three structurally diverse tree learners blended per target and chained so the memory prediction feeds the energy model, and the energy prediction feeds the latency model. Multiplicative interaction features (e.g., batch size times SM utilization) encode compounding hardware-load effects, and per-target residual correctors are trained only in the low-batch regime—where residual analysis shows structured error—with a 0.9 damping factor to avoid overcorrection. All targets are learned in log space. The cascade is the load-bearing mechanism: the paper reports that the upstream energy prediction ranks first among latency features, acting as a compressed summary of execution-time determinants.","core_discovery":"On its own terms, the paper claims that energy is not a shadow of latency or FLOPs: across batch-size tiers under high SM utilization, energy scales 35.4× while latency scales only 11.2×, so a latency-only proxy underestimates energy severalfold in deployment-relevant regimes. Cross-GPU analysis shows energy and latency follow non-unit transfer slopes (1.61× and 2.09×), whereas peak memory follows slope ≈1, meaning memory models transfer between the two tested platforms but energy and latency models must be per-GPU. CARB operationalizes this characterization: a cascade-blended ensemble predicts peak memory first, then energy using memory, then latency using both, achieving $R^2$ of 0.997, 0.993, and 0.992 on held-out test configurations. A two-stage screening workflow exploiting the near-monotonic ranking across GPUs reduces a 3,072-configuration space to seven Pareto-prioritized candidates, with 95.8% budget-classification accuracy and 2.1% false accepts at a 75 J threshold.","pith_inferences":["If the rank-preservation result (Spearman ρ = 0.95 between GPU rankings) holds beyond the two tested GPUs, CARB-style screening could order candidates for multi-device deployment from a single ranking model—an extension the paper does not test.","The telemetry-free mode's near-parity with full telemetry suggests static architectural features may sustain accurate cost prediction on unseen devices, but this is only shown within the same architecture family and two GPUs; extending to new block types or vendors is untested.","The low-batch residual asymmetry implies kernel-launch and driver overheads dominate small-batch energy; adding explicit launch-overhead or occupancy features could shrink the remaining error—a testable modification of the feature set.","The 3× energy-latency divergence under high SM utilization predicts that energy-aware NAS rankings will differ materially from latency-aware NAS in high-batch regimes; comparing the two Pareto fronts on a real deployment would test the practical consequence."],"forward_implications":["A practitioner who profiles only latency will mis-budget energy by roughly 3× in high-load, large-batch regimes; energy must be predicted or measured directly.","Cross-GPU energy and latency estimates need a per-GPU model, while a single peak-memory model can be shared across the tested GPUs.","FLOPs alone cannot drive reliable cost screening: a FLOPs-only linear baseline achieves $R^2 < 0.38$ for all three targets.","Design spaces of thousands of CNN configurations can be reduced to a handful of Pareto candidates in under a second of scoring, before any hardware profiling.","Budget classification is reliable enough for pre-screening (95.8% accuracy, 2.1% false accepts), with borderline candidates flagged for measurement."],"supporting_citations":[{"why":"Supplies the gradient-boosting tree algorithm used as one of the three specialist learners in the ensemble.","marker":"[6]"},{"why":"Supplies the leaf-wise boosting learner whose predictions carry the largest weight in the energy blend.","marker":"[11]"},{"why":"Supplies the extra-randomized tree learner that dominates the latency blend.","marker":"[8]"},{"why":"Defines the ResNet-style architecture family and block types that generate the configuration search space.","marker":"[10]"},{"why":"Computes the FLOPs feature used both as a baseline proxy and as an input to the interaction features.","marker":"[14]"},{"why":"Provides the Pareto-dominance criterion used to extract the priority shortlist from the screened candidates.","marker":"[7]"}],"fun_headline_variants":["CARB predicts CNN cost: R² 0.99, cuts design space by 90%","Energy ≠ latency: CARB predicts all three at R² 0.99","Cascade ensemble predicts GPU cost, cuts candidates 90% in seconds","FLOPs can't model energy: CARB nails all three costs at R²~0.99","Cross-GPU memory transfers; CARB predicts all three at R²~0.99"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole case rests on the assumption that SM utilization recorded during a training-mode forward-backward pass faithfully represents hardware load during evaluation-mode inference, because the interaction features and regime correctors are built on that proxy.","fun_headline_variants_meta":{"raw":{"variants":["CARB predicts CNN cost: R² 0.99, cuts design space by 90%","Energy ≠ latency: CARB predicts all three at R² 0.99","Cascade ensemble predicts GPU cost, cuts candidates 90% in seconds","FLOPs can't model energy: CARB nails all three costs at R²~0.99","Cross-GPU memory transfers; CARB predicts all three at R²~0.99"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000861,"raw_usage":{"total_tokens":3749,"prompt_tokens":970,"completion_tokens":2779,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":586,"completion_tokens_details":{"reasoning_tokens":2664}},"tokens_in":586,"tokens_out":2779,"duration_ms":16637,"temperature":1.0,"reasoning_tokens":2664,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:18:39.165077+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the same configurations on a third GPU (or on the same GPUs) with SM utilization sampled during evaluation-mode inference only, and retrain CARB; if the $R^2 \\approx 0.99$ accuracy, the 3× energy-latency divergence, or the cross-GPU slope pattern (energy 1.61×, latency 2.09×, memory ≈1) does not reproduce, the characterization-guided design loses its empirical foundation.","supporting_citations":[{"cited_title":"Extremely randomized trees.Machine Learning, 63(1):3–42, Mar 2006","cited_arxiv_id":null,"evidence_quote":"Supplies the extra-randomized tree learner that dominates the latency blend."},{"cited_title":"fvcore: Facebook’s core library for computer vision research, 2020","cited_arxiv_id":null,"evidence_quote":"Computes the FLOPs feature used both as a baseline proxy and as an input to the interaction features."}],"review_version":1}