{"id":"d4e1907b-240c-4767-95cc-47440ace6f7f","arxiv_id":"2412.15441","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Serving configuration choice, especially runtime engine and execution provider, significantly changes energy, time, and resource use of code SLMs, with TORCH+CUDA most efficient on the tested hardware.","lead":"This paper measured how different software setups for running small code-writing AI models change energy use, speed, and computer load. It found that the PyTorch-plus-GPU setup used the least energy, while the ONNX engine was best for CPU-only machines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline TORCH+CUDA energy-savings range is internally inconsistent (37.99–89.16% in §1 vs 10.84–62.01% in §8) and the larger bound is partly an admitted software artifact; the quantitative claim as stated is not yet supported.","rationale":"The reader's CONDITIONAL verdict is appropriate and I do not move it. My stress-test agrees with the reader that the snapshot/library-version dependency is a serious limitation, but I would place the most load-bearing weight on a more concrete defect: the paper itself reports two incompatible ranges for the same headline comparison. Section 1 and the abstract give 37.99–89.16% energy savings for TORCH+CUDA versus ONNX+CUDA, while Section 8 gives 10.84–62.01% for the same comparison. Since both cannot be correct, at least one of the paper's central numeric claims is wrong as written, and the abstract repeats the unverified range. This is not a disagreement with the field's consensus; it is an internal inconsistency in the reported evidence. The second concern, also flagged by the reader, is that the authors explicitly attribute the ONNX+CUDA deficit to missing kernels, CPU fallback, and memcpy nodes in the evaluated release, and Section 8 concedes the ranking could invert with kernel parity and IOBinding. That means the headline 'greatest energy efficiency' claim is stronger than the evidence supports for other software versions, even though it may be true for the exact tested snapshot. The qualitative central claim, that serving configuration choice significantly affects energy, time, and resource use, is well supported by the experimental design, the replication package, and the candid limitation statements. The paper needs a revision of the reported savings ranges and a hedge of the abstract's unqualified 'greatest energy efficiency' statement, but the underlying study is not fatally flawed. Hence CONDITIONAL remains the right verdict, and my read does not change it.","tokens_in":28860,"tokens_out":3699,"duration_ms":32242,"concrete_test":"Using the Zenodo replication package (doi:10.5281/zenodo.15258679), recompute per-model global energy for TORCH+CUDA and ONNX+CUDA from the raw energibridge/nvidia-smi logs using the aggregation described in §5.1, and calculate the percentage savings for each of the 12 models as (E_ONNX - E_TORCH)/E_ONNX. Report the min–max across models and compare with both claimed ranges, 37.99–89.16% (§1, abstract) and 10.84–62.01% (§8). Whichever range fails to reproduce identifies the erroneous headline number; if neither reproduces, the paper's central quantitative claim is not yet supported by the provided data.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim is that TORCH+CUDA is the most energy-efficient serving configuration, with energy savings from 37.99% up to 89.16% compared with other configurations. For this claim to hold, the reported savings must be a well-defined, reproducible quantity. Two concrete problems undermine it. (1) Internal inconsistency: Section 1 states that the 37.99–89.16% range is 'compared to the worst-performing runtime engine (ONNX Runtime) in CUDA execution provider configurations,' yet Section 8 gives 'from 10.84% up to 62.01%' for the same comparison. These ranges cannot both describe the same computed savings; the abstract repeats the Section 1 range, so as currently written the headline number is not trustworthy. (2) The advantage is partly a software artifact, as the authors concede: Section 6 attributes the ONNX+CUDA deficit to missing decoder CUDA kernels, CPU fallback, and host–device memcpy nodes in the evaluated ONNX Runtime release; Section 7 states this 'can overstate resource utilization advantages' for TORCH+CUDA; Section 8 calls the result a 'best-case snapshot' in which the ranking 'could invert' with kernel parity and IOBinding. The qualitative finding that configuration choice matters is well supported; the quantitative headline is not robust to the very software-version sensitivity the paper documents.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a controlled experiment measuring energy consumption, execution time, and computing-resource utilization for twelve code-generation small language models served under five configurations defined by the runtime engine (TORCH, ONNX Runtime, OpenVINO) and execution provider (CPU, CUDA). Using 164 short prompts derived from HumanEval and ten repetitions per configuration, the authors find that CUDA configurations generally outperform CPU configurations in both energy and time, with TORCH+CUDA being the most energy-efficient and ONNX+CPU the best among CPU-only configurations. They provide practitioner guidelines and a public replication package.","tokens_in":28909,"tokens_out":10368,"duration_ms":79271,"significance":"If the quantitative claims were robust, this would be a practically valuable study for software engineers making serving-infrastructure decisions without changing models. The methodological strengths are substantial: energy readings cross-validated against a wattmeter, ten repetitions per configuration, formal normality/variance checks with suitable parametric and non-parametric tests, documented exclusion of JIT configurations, and a fully open replication package. However, the headline energy-savings ranges are internally inconsistent and, by the authors' own admission, partly a software artifact of the specific ONNX Runtime release and hardware snapshot. The qualitative conclusion that configuration choice significantly affects resource utilization is well supported, but the quantitative headline as currently stated is not yet trustworthy.","major_comments":[{"comment":"The headline energy-savings range for TORCH+CUDA is stated as 'from 37.99% up to 89.16%' in the Abstract and Section 1, where it is specified as relative to the worst-performing runtime engine (ONNX Runtime) in CUDA execution provider configurations, but Section 8 reports 'from 10.84% up to 62.01%' for the same comparison. These two ranges cannot both be correct descriptions of the same measured quantity; the authors must reconcile them, state the exact reference configuration used for each bound, and specify whether the extrema are taken over models, model-size categories, or configurations.","section":"Abstract; §1; §8"},{"comment":"The paper itself documents that the TORCH+CUDA advantage over ONNX+CUDA is partly a software artifact: Section 6 attributes it to missing CUDA kernels, CPU fallback, and host-device memcpy nodes in the evaluated ONNX Runtime release; Section 7 says this 'can overstate resource utilization advantages' for TORCH+CUDA; and Section 8 characterizes the study as a 'best-case snapshot' in which the ranking 'could invert' once kernel parity and IOBinding are enabled. Because these concessions directly qualify the magnitude and even the ordering of the central result, the Abstract and Section 1 should carry the same caveat, or the quantitative claim should be reformulated as version- and hardware-specific.","section":"§6; §7; §8"},{"comment":"The claimed savings percentages (37.99%–89.16% for TORCH+CUDA, 8.98%–72.04% for ONNX+CPU, and the Section 8 alternatives) are not derivable from the aggregated values in Tables 4 and 5, and the paper does not define the calculation (e.g., per-model (E_reference − E_TORCH,CUDA)/E_reference, with extrema over models or categories). To make the headline result independently verifiable, the authors should report per-model energy savings or provide the exact formula and the underlying data in the paper or an appendix.","section":"§5; Tables 4–5"}],"minor_comments":[{"comment":"The sentence 'The only exceptions were that the execution times of the ⟨ONNX, CPU⟩ and ⟨TORCH, CPU⟩ configurations did not differ significantly for three out of the twelve models (...), resulting in the rejection of H.2.0 for only these models' is logically inconsistent: non-significant pairwise differences for those three models do not lead to rejection of the global null hypothesis H.2.0 for those models; the phrasing should be corrected to state that the pairwise comparison was not significant or that H.2.0 was retained for those cases.","section":"§5.2"},{"comment":"The claim that 'we manually verified the model metadata to ensure that the selected models are base pretrained models, not subjected to quantization or fine-tuning' conflicts with the inclusion of TinyLlama-1.1B-Chat-v1.0 and stablecode-instruct-alpha-3b, which are chat or instruct models and therefore likely fine-tuned; the authors should either correct the selection description or justify the inclusion of these models.","section":"§4.2; Table 1"},{"comment":"The Abstract reports '8.98% up to 72.04%' energy savings for ONNX+CPU within CPU-based configurations, while Section 8 reports '27.96% up to 91.02%' for 'OV or ONNX' compared to the baseline TORCH engine; the relation between these ranges (different reference configurations or model sets) should be stated explicitly to avoid apparent contradiction.","section":"Abstract; §8"},{"comment":"The subsection title contains a typo 'W attmeter' (extra space), and 'energibridge' is capitalized inconsistently (e.g., 'EnergiBridge' vs 'energibridge'); both should be standardized.","section":"§4.6"},{"comment":"The caption for the last section of Table 4 says 'Last section presents mean values,' but the rows labeled '1,2,3' are described elsewhere as mean values for configurations while the table title says 'aggregated median energy consumption'; the distinction should be clarified.","section":"Tables 4–5"},{"comment":"The random selection of 12 models from 29 is not described with a reproducible seed or explicit criteria beyond diversity; adding the seed or a detailed selection procedure would strengthen reproducibility.","section":"§4.2"}],"recommendation":"major_revision","confidential_remarks":"The empirical protocol is careful and the replication package is a real strength. The main barrier is that the headline numbers are internally inconsistent and the authors' own caveats undermine the unqualified abstract claims. These are fixable within the manuscript's scope by reconciling the ranges, defining the savings metric precisely, and carrying the version- and hardware-specific caveats into the abstract and introduction. The model-selection inconsistency in Section 4.2 should also be corrected. I do not see a need for full rejection, but the revised version must be checked carefully."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuinely useful measurement study, but the headline number doesn't survive contact with the paper's own caveats. Section 1 and the abstract report TORCH+CUDA energy savings of 37.99–89.16% over ONNX+CUDA; Section 8 gives 10.84–62.01% for the same comparison. Both can't be right, and the authors admit the larger bound is inflated by missing decoder CUDA kernels, CPU fallback, and memcpy nodes in the ONNX Runtime release they used. They even call the result a 'best-case snapshot' and say the ranking could invert with kernel parity and IOBinding. So the quantitative headline as written is not trustworthy, but the qualitative finding is solid: serving configuration clearly matters for energy, time, and resource use.\n\nWhat's new: first study I've seen that systematically varies runtime engine (TORCH, ONNX, OpenVINO, JIT) and execution provider (CPU, CUDA) for code SLMs in short-prompt autocomplete, across 12 models. The protocol is careful: ten repetitions, 164 prompts per model, two profilers cross-validated against a wattmeter, normality and variance checks with appropriate post-hoc tests, documented exclusions, open Zenodo package. The statistical work is honest and the replication package is a real asset.\n\nSoft spots, in order of seriousness. First, the internal inconsistency above. It's a straightforward fix—recompute and state one range with its definition—but as submitted the abstract repeats a number the authors' own Section 8 contradicts. Second, the unhedged 'greatest energy efficiency' claim in the abstract conflicts with the paper's own concession that the TORCH lead over ONNX+CUDA is partly a software artifact. Third, JIT was dropped after producing erroneous outputs, narrowing the designed seven-configuration comparison to five; the exclusion is justified, but readers should know the headline comparisons are a post-hoc subset. Fourth, the single-hardware snapshot limits generalization; the authors say so themselves, so this is a limitation rather than a hidden flaw.\n\nWho should read it: software engineers choosing a serving stack for code completion, and green-AI researchers needing empirical ground truth for inference energy. It deserves a serious referee: the design is sound, the data is openly available, and the qualitative conclusion will likely hold even if the specific percentages change with newer library versions. The revision just has to reconcile the ranges and align the abstract with the caveats.","headline":"A careful, reproducible measurement study whose qualitative finding about serving configuration is solid, but whose headline energy-savings range is internally inconsistent and partly an admitted software artifact.","tokens_in":29667,"tokens_out":2522,"would_cite":true,"duration_ms":22123,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Serving configuration alone can cut code-SLM inference energy by up to 89%.","keywords":["small language models","code generation","model serving","runtime engines","execution providers","energy consumption","resource utilization","inference optimization"],"falsifier":"Repeat the same 12-model, 164-prompt experiment with a newer ONNX Runtime release that includes CUDA kernels for decoder operators and with I/O binding enabled, on the same GPU and driver; if ONNX+CUDA then matches or beats TORCH+CUDA on energy or execution time, the paper's headline recommendation would fail.","tokens_in":28454,"feed_emoji":"⚡","tokens_out":6292,"duration_ms":47428,"temperature":0.7,"pith_summary":"This paper tries to establish that software engineers can substantially improve the energy, speed, and resource efficiency of code-completion small language models simply by choosing which runtime engine and execution provider serve the model, without retraining or compressing the model. It reports a controlled experiment with twelve models and 164 short-prompt requests per model, repeated ten times across five serving configurations. The headline result is that TORCH paired with CUDA was the most energy-efficient configuration, with energy savings from 37.99% up to 89.16% compared with other configurations, and that ONNX Runtime paired with CPU achieved 8.98% to 72.04% energy savings within CPU-based setups. If correct, this gives practitioners an immediately actionable lever for greener and faster code-AI inference, and it highlights that execution time alone is not a reliable proxy for energy use.","feed_headline":"Right runtime can cut code-model energy up to 89 percent","feed_subtitle":"A 12-model test shows TORCH plus CUDA beats other serving setups on energy, speed, and resource use.","key_machinery":"The load-bearing object is the serving configuration, defined as a duplet of a runtime engine and an execution provider. The runtime engine loads the model and applies optional optimizations such as graph rewriting or just-in-time compilation; the execution provider is the backend library that selects and tunes low-level kernels for the available hardware. The experiment treats these duplets as the independent variable and measures energy consumption, execution time, and five resource-utilization metrics through a reproducible multi-stage pipeline using profiling tools and a wattmeter for cross-validation.","core_discovery":"The paper's central claim is that the choice of serving configuration—the pairing of a runtime engine (TORCH, ONNX Runtime, OpenVINO, or Torch JIT) with an execution provider (CPU or CUDA)—substantially changes how much energy, time, and computing resources code-generation small language models consume during inference. Across twelve models and short-prompt code-completion requests, CUDA-based configurations consistently beat CPU-based ones, with TORCH paired with CUDA the most energy-efficient overall: energy savings of 37.99% to 89.16% and execution-time reductions of 47.84% to 89.74% compared with other serving configurations. Within CPU-only setups, ONNX paired with CPU delivered 8.98% to 72.04% energy savings over other CPU configurations. The paper argues these gains come from the serving stack alone, requiring no model changes, and it frames the TORCH+CUDA result as a best-case snapshot for the tested hardware and library versions, since the ONNX+CUDA gap is partly explained by missing decoder CUDA kernels and host-device memory copies in the evaluated release.","pith_inferences":["The ranking may be specific to the library versions tested; a newer ONNX Runtime with full CUDA decoder kernels and IOBinding could close or invert the gap with TORCH+CUDA, so the savings should be re-measured on every deployment stack.","Because the workload uses only 10-15 token prompts from HumanEval, the energy and time advantages may not carry over to longer completions, multi-file contexts, or conversational code generation; extending the benchmark would test that.","The reported energy savings, if realized at scale in IDE auto-completion, would also reduce the carbon footprint and electricity cost of code AI, giving sustainability-motivated teams a concrete deployment lever.","The methodology—profiling a fixed set of models and prompts across serving duplets—could serve as a lightweight internal benchmark for teams choosing an inference stack."],"forward_implications":["Software teams can cut inference energy and latency for code SLMs by switching serving configuration alone, with TORCH+CUDA recommended when a GPU is available.","CPU-only deployments should consider ONNX or OpenVINO runtimes, which can reduce energy and time relative to the plain framework, though at higher RAM usage.","In latency-critical settings like IDE auto-completion, CUDA configurations are consistently faster, with TORCH+CUDA fastest in this study.","Execution time should not be used as a proxy for energy: the paper found OV+CPU faster than ONNX+CPU yet more energy-hungry.","Engineers should inspect execution-provider placement logs and enable I/O binding before concluding which runtime is best on their hardware."],"supporting_citations":[{"why":"Defines deep learning serving infrastructure types and motivates runtime engine and execution provider as architectural decisions this study evaluates.","marker":"[16]"},{"why":"Supplies the HumanEval benchmark and Codex context used to build the short-prompt code-completion dataset and filter candidate SLMs.","marker":"[25]"},{"why":"The model-loading library through which all serving configurations were implemented and exported.","marker":"[74]"},{"why":"Provides the cross-platform energy and resource profiler used for CPU measurements.","marker":"[77]"},{"why":"Documents ONNX Runtime deployment and execution-provider behavior used to interpret the ONNX+CUDA energy gap.","marker":"[79]"},{"why":"Cited as the basis for the paper's warning that the TORCH+CUDA lead could shrink or invert once ONNX gains kernel parity and IOBinding.","marker":"[80]"},{"why":"Explains I/O binding and host-device copy overheads that inflate ONNX+CUDA energy and time in the evaluated version.","marker":"[81]"},{"why":"Supports the claim that the used ONNX Runtime release lacked CUDA kernels for some operators and fell back to CPU.","marker":"[96]"}],"fun_headline_variants":["TORCH+CUDA slashes code-model energy up to 89%","Serving config choice can cut code-SLM energy by 89%","Runtime+provider pair drives code-model energy down 89%","Right serving setup: TORCH+CUDA saves 38–89% energy","Code SLM energy drops up to 89% with TORCH+CUDA serving"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking reflects the specific runtime-engine versions and the single NVIDIA GPU used in the study, so a different library release or different hardware could reorder the configurations.","fun_headline_variants_meta":{"raw":{"variants":["TORCH+CUDA slashes code-model energy up to 89%","Serving config choice can cut code-SLM energy by 89%","Runtime+provider pair drives code-model energy down 89%","Right serving setup: TORCH+CUDA saves 38–89% energy","Code SLM energy drops up to 89% with TORCH+CUDA serving"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000701,"raw_usage":{"total_tokens":3218,"prompt_tokens":1049,"completion_tokens":2169,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":665,"completion_tokens_details":{"reasoning_tokens":2067}},"tokens_in":665,"tokens_out":2169,"duration_ms":14093,"temperature":1.0,"reasoning_tokens":2067,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:26:41.498882+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the same 12-model, 164-prompt experiment with a newer ONNX Runtime release that includes CUDA kernels for decoder operators and with I/O binding enabled, on the same GPU and driver; if ONNX+CUDA then matches or beats TORCH+CUDA on energy or execution time, the paper's headline recommendation would fail.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The model-loading library through which all serving configurations were implemented and exported."},{"cited_title":"URL: https://www.nvidia.com/en-us/ on-demand/session/gtc24-s62336/ , last accessed April 12th, 2025","cited_arxiv_id":null,"evidence_quote":"Documents ONNX Runtime deployment and execution-provider behavior used to interpret the ONNX+CUDA energy gap."},{"cited_title":"Stiles, Efficient Segment Anything on the Edge, Ph.D","cited_arxiv_id":null,"evidence_quote":"Cited as the basis for the paper's warning that the TORCH+CUDA lead could shrink or invert once ONNX gains kernel parity and IOBinding."},{"cited_title":"URL: https: //onnxruntime.ai/docs/performance/tune-performance/ iobinding.html, last accessed April 12th, 2025","cited_arxiv_id":null,"evidence_quote":"Explains I/O binding and host-device copy overheads that inflate ONNX+CUDA energy and time in the evaluated version."},{"cited_title":"URL: https:// onnxruntime.ai/docs/api/python/api_summary.html, last ac- cessed April 12th, 2025","cited_arxiv_id":null,"evidence_quote":"Supports the claim that the used ONNX Runtime release lacked CUDA kernels for some operators and fell back to CPU."}],"review_version":1}