Pith. sign in

REVIEW 3 major objections 6 minor 98 references

Insights into resource utilization of code small language models serving with runtime engines and execution providers

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Serving configuration alone can cut code-SLM inference energy by up to 89%.

desk verdict A careful, reproducible measurement study whose qualitative finding about serving configuration is solid, but whose headline energy-savings range is internally inconsistent and partly an admitted software artifact. read the letter →

arxiv 2412.15441 v2 pith:XXVPULAU submitted 2024-12-19 cs.SE cs.AIcs.LG

Francisco Durán , Matias Martinez , Patricia Lago , Silverio Martínez-Fernández This is my paper · ORCID
classification cs.SEcs.AIcs.LG
keywords smalllanguagemodelscodegenerationmodelservingruntimeenginesexecutionprovidersenergyconsumptionresourceutilizationinferenceoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that software engineers can substantially improve the energy, speed, and resource efficiency of code-completion small language models simply by choosing which runtime engine and execution provider serve the model, without retraining or compressing the model. It reports a controlled experiment with twelve models and 164 short-prompt requests per model, repeated ten times across five serving configurations. The headline result is that TORCH paired with CUDA was the most energy-efficient configuration, with energy savings from 37.99% up to 89.16% compared with other configurations, and that ONNX Runtime paired with CPU achieved 8.98% to 72.04% energy savings within CPU-based setups. If correct, this gives practitioners an immediately actionable lever for greener and faster code-AI inference, and it highlights that execution time alone is not a reliable proxy for energy use.

What carries the argument

The load-bearing object is the serving configuration, defined as a duplet of a runtime engine and an execution provider. The runtime engine loads the model and applies optional optimizations such as graph rewriting or just-in-time compilation; the execution provider is the backend library that selects and tunes low-level kernels for the available hardware. The experiment treats these duplets as the independent variable and measures energy consumption, execution time, and five resource-utilization metrics through a reproducible multi-stage pipeline using profiling tools and a wattmeter for cross-validation.

What would settle it

Repeat the same 12-model, 164-prompt experiment with a newer ONNX Runtime release that includes CUDA kernels for decoder operators and with I/O binding enabled, on the same GPU and driver; if ONNX+CUDA then matches or beats TORCH+CUDA on energy or execution time, the paper's headline recommendation would fail.

Watch

Extended reading notes

Core claim

The paper's central claim is that the choice of serving configuration—the pairing of a runtime engine (TORCH, ONNX Runtime, OpenVINO, or Torch JIT) with an execution provider (CPU or CUDA)—substantially changes how much energy, time, and computing resources code-generation small language models consume during inference. Across twelve models and short-prompt code-completion requests, CUDA-based configurations consistently beat CPU-based ones, with TORCH paired with CUDA the most energy-efficient overall: energy savings of 37.99% to 89.16% and execution-time reductions of 47.84% to 89.74% compared with other serving configurations. Within CPU-only setups, ONNX paired with CPU delivered 8.98% to 72.04% energy savings over other CPU configurations. The paper argues these gains come from the serving stack alone, requiring no model changes, and it frames the TORCH+CUDA result as a best-case snapshot for the tested hardware and library versions, since the ONNX+CUDA gap is partly explained by missing decoder CUDA kernels and host-device memory copies in the evaluated release.

Load-bearing premise

The ranking reflects the specific runtime-engine versions and the single NVIDIA GPU used in the study, so a different library release or different hardware could reorder the configurations.

Editorial extensions

If this is right

  • Software teams can cut inference energy and latency for code SLMs by switching serving configuration alone, with TORCH+CUDA recommended when a GPU is available.
  • CPU-only deployments should consider ONNX or OpenVINO runtimes, which can reduce energy and time relative to the plain framework, though at higher RAM usage.
  • In latency-critical settings like IDE auto-completion, CUDA configurations are consistently faster, with TORCH+CUDA fastest in this study.
  • Execution time should not be used as a proxy for energy: the paper found OV+CPU faster than ONNX+CPU yet more energy-hungry.
  • Engineers should inspect execution-provider placement logs and enable I/O binding before concluding which runtime is best on their hardware.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The ranking may be specific to the library versions tested; a newer ONNX Runtime with full CUDA decoder kernels and IOBinding could close or invert the gap with TORCH+CUDA, so the savings should be re-measured on every deployment stack.
  • Because the workload uses only 10-15 token prompts from HumanEval, the energy and time advantages may not carry over to longer completions, multi-file contexts, or conversational code generation; extending the benchmark would test that.
  • The reported energy savings, if realized at scale in IDE auto-completion, would also reduce the carbon footprint and electricity cost of code AI, giving sustainability-motivated teams a concrete deployment lever.
  • The methodology—profiling a fixed set of models and prompts across serving duplets—could serve as a lightweight internal benchmark for teams choosing an inference stack.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper reports a controlled experiment measuring energy consumption, execution time, and computing-resource utilization for twelve code-generation small language models served under five configurations defined by the runtime engine (TORCH, ONNX Runtime, OpenVINO) and execution provider (CPU, CUDA). Using 164 short prompts derived from HumanEval and ten repetitions per configuration, the authors find that CUDA configurations generally outperform CPU configurations in both energy and time, with TORCH+CUDA being the most energy-efficient and ONNX+CPU the best among CPU-only configurations. They provide practitioner guidelines and a public replication package.

Significance. If the quantitative claims were robust, this would be a practically valuable study for software engineers making serving-infrastructure decisions without changing models. The methodological strengths are substantial: energy readings cross-validated against a wattmeter, ten repetitions per configuration, formal normality/variance checks with suitable parametric and non-parametric tests, documented exclusion of JIT configurations, and a fully open replication package. However, the headline energy-savings ranges are internally inconsistent and, by the authors' own admission, partly a software artifact of the specific ONNX Runtime release and hardware snapshot. The qualitative conclusion that configuration choice significantly affects resource utilization is well supported, but the quantitative headline as currently stated is not yet trustworthy.

major comments (3)
  1. [Abstract; §1; §8] The headline energy-savings range for TORCH+CUDA is stated as 'from 37.99% up to 89.16%' in the Abstract and Section 1, where it is specified as relative to the worst-performing runtime engine (ONNX Runtime) in CUDA execution provider configurations, but Section 8 reports 'from 10.84% up to 62.01%' for the same comparison. These two ranges cannot both be correct descriptions of the same measured quantity; the authors must reconcile them, state the exact reference configuration used for each bound, and specify whether the extrema are taken over models, model-size categories, or configurations.
  2. [§6; §7; §8] The paper itself documents that the TORCH+CUDA advantage over ONNX+CUDA is partly a software artifact: Section 6 attributes it to missing CUDA kernels, CPU fallback, and host-device memcpy nodes in the evaluated ONNX Runtime release; Section 7 says this 'can overstate resource utilization advantages' for TORCH+CUDA; and Section 8 characterizes the study as a 'best-case snapshot' in which the ranking 'could invert' once kernel parity and IOBinding are enabled. Because these concessions directly qualify the magnitude and even the ordering of the central result, the Abstract and Section 1 should carry the same caveat, or the quantitative claim should be reformulated as version- and hardware-specific.
  3. [§5; Tables 4–5] The claimed savings percentages (37.99%–89.16% for TORCH+CUDA, 8.98%–72.04% for ONNX+CPU, and the Section 8 alternatives) are not derivable from the aggregated values in Tables 4 and 5, and the paper does not define the calculation (e.g., per-model (E_reference − E_TORCH,CUDA)/E_reference, with extrema over models or categories). To make the headline result independently verifiable, the authors should report per-model energy savings or provide the exact formula and the underlying data in the paper or an appendix.
minor comments (6)
  1. [§5.2] The sentence 'The only exceptions were that the execution times of the ⟨ONNX, CPU⟩ and ⟨TORCH, CPU⟩ configurations did not differ significantly for three out of the twelve models (...), resulting in the rejection of H.2.0 for only these models' is logically inconsistent: non-significant pairwise differences for those three models do not lead to rejection of the global null hypothesis H.2.0 for those models; the phrasing should be corrected to state that the pairwise comparison was not significant or that H.2.0 was retained for those cases.
  2. [§4.2; Table 1] The claim that 'we manually verified the model metadata to ensure that the selected models are base pretrained models, not subjected to quantization or fine-tuning' conflicts with the inclusion of TinyLlama-1.1B-Chat-v1.0 and stablecode-instruct-alpha-3b, which are chat or instruct models and therefore likely fine-tuned; the authors should either correct the selection description or justify the inclusion of these models.
  3. [Abstract; §8] The Abstract reports '8.98% up to 72.04%' energy savings for ONNX+CPU within CPU-based configurations, while Section 8 reports '27.96% up to 91.02%' for 'OV or ONNX' compared to the baseline TORCH engine; the relation between these ranges (different reference configurations or model sets) should be stated explicitly to avoid apparent contradiction.
  4. [§4.6] The subsection title contains a typo 'W attmeter' (extra space), and 'energibridge' is capitalized inconsistently (e.g., 'EnergiBridge' vs 'energibridge'); both should be standardized.
  5. [Tables 4–5] The caption for the last section of Table 4 says 'Last section presents mean values,' but the rows labeled '1,2,3' are described elsewhere as mean values for configurations while the table title says 'aggregated median energy consumption'; the distinction should be clarified.
  6. [§4.2] The random selection of 12 models from 29 is not described with a reproducible seed or explicit criteria beyond diversity; adding the seed or a detailed selection procedure would strengthen reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the study's claims are arithmetic summaries of its own measurements, not derivations from fitted or self-cited premises.

full rationale

The paper is an empirical benchmarking study, not a derivation. Its central claims (e.g., that TORCH paired with CUDA shows the lowest energy consumption among the tested configurations) are descriptive summaries of measured energy, time, and resource-utilization data collected against an external benchmark (HumanEval) and an independently documented experimental pipeline. The savings percentages are computed directly from the reported median/mean values in Tables 4 and 5, with no fitted parameters, no equations that reduce to earlier results, and no prediction that is defined in terms of the input it claims to explain. The only self-citation of note is reference [16], used for taxonomical background on deep learning serving infrastructures; it does not supply any measured values, uniqueness theorem, or assumed ranking, and the empirical results stand on the paper's own profiling data. The paper's own limitations sections (e.g., Section 7 noting that ONNX's missing CUDA kernels 'can overstate resource utilization advantages' for TORCH+CUDA, and Section 8 describing the study as a 'best-case snapshot') weaken the generalizability and robustness of the quantitative claims, but these are internal-validity and version-sensitivity concerns, not circularity. The internal inconsistency between the 37.99%-89.16% savings range in Sections 1/Contributions and the 10.84%-62.01% range in Section 8 is a correctness/reporting issue, not a definitional or self-referential reduction. No step in the paper's reasoning assumes what it purports to show, so the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

No free parameters: the paper reports direct measurements of energy, time, and resource usage, and the savings percentages are arithmetic summaries of those measurements, not fitted constants. The experimental design contains judgment calls (twelve-model selection, 10-15 token prompts, ten repetitions), but none is a fitted value. No invented entities: the serving configuration duplet is a classification of pre-existing software components. Six domain assumptions carry the measurements; the most fragile is that the rankings are properties of the engines rather than of the specific library versions and hardware snapshot, a premise the paper itself qualifies.

assumptions (6)
  • domain assumption EnergiBridge, nvidia-smi, and the NETIO wattmeter correctly attribute energy and resource usage to the serving process.
    Load-bearing measurement premise for all three RQs, relied on in Sections 4.6 and 4.7; the wattmeter serves only as a cross-check, not as per-process attribution.
  • domain assumption A 10-15 token prompt sampled from each HumanEval problem approximates real IDE auto-complete serving workloads.
    Section 4.3; the design trades away multi-file, long-context, and conversational code generation, as the paper acknowledges in Sections 4.3 and 7.
  • domain assumption The 12 selected models are representative of code-generation SLMs.
    Section 4.2: candidates were filtered to models whose metadata explicitly mentions HumanEval results, then 12 were randomly drawn from 29 for practical runtime reasons; this introduces a mild selection constraint on the model pool.
  • domain assumption Ten repetitions with five-minute idle windows control thermal drift and background activity.
    Section 4.7 and internal validity discussion in Section 7; the authors note fixed-order execution is a residual risk.
  • domain assumption Results observed with one ONNX Runtime version and one GPU characterize the runtime engines generally.
    Section 6 concedes the ONNX-CUDA deficit is largely caused by missing decoder CUDA kernels and memcpy overhead in the used release, and Section 8 calls the study a 'best-case snapshot'; the headline recommendation depends on this generalizability premise.
  • standard math The statistical tests are validly applied and the reported p-values are correct.
    Section 4.8 specifies Welch's ANOVA/Games-Howell and Kruskal-Wallis/Dunn after normality and variance checks, but the p-value tables live only in the replication package, so the stated 'all configurations significantly different' cannot be audited from the text.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Insights into resource utilization of code small language models serving with runtime engines and execution providers." pith.science (2026). https://pith.science/paper/XXVPULAU

@misc{pith2026241215441,
  author       = {Pith},
  title        = {Pith review of: Insights into resource utilization of code small language models serving with runtime engines and execution providers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XXVPULAU}},
  note         = {Machine review of arXiv:2412.15441}
}
read the original abstract

The rapid growth of language models, particularly in code generation, requires substantial computational resources, raising concerns about energy consumption and environmental impact. Optimizing language models inference resource utilization is crucial, and Small Language Models (SLMs) offer a promising solution to reduce resource demands. Our goal is to analyze the impact of deep learning serving configurations, defined as combinations of runtime engines and execution providers, on resource utilization, in terms of energy consumption, execution time, and computing-resource utilization from the point of view of software engineers conducting inference in the context of code generation SLMs. We conducted a technology-oriented, multi-stage experimental pipeline using twelve code generation SLMs to investigate energy consumption, execution time, and computing-resource utilization across the configurations. Significant differences emerged across configurations. CUDA execution provider configurations outperformed CPU execution provider configurations in both energy consumption and execution time. Among the configurations, TORCH paired with CUDA demonstrated the greatest energy efficiency, achieving energy savings from 37.99% up to 89.16% compared to other serving configurations. Similarly, optimized runtime engines like ONNX with the CPU execution provider achieved from 8.98% up to 72.04% energy savings within CPU-based configurations. Also, TORCH paired with CUDA exhibited efficient computing-resource utilization. Serving configuration choice significantly impacts resource utilization. While further research is needed, we recommend the above configurations best suited to software engineers' requirements for enhancing serving resource utilization efficiency.

Figures

Figures reproduced from arXiv: 2412.15441 by the authors.

Figure 1
Figure 1. Architectural design decisions for deep learning serving: runtime engine as a serving infrastructure and execution provider options [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Schema of our experiment The complete experiment and data analysis is imple￾mented in Python 3.10.12, using HuggingFace Transform￾ers 4.40.2 [74], and Optimum 1.19.27 . To implement the serving configuration, we followed the implementations and model exportation provided by HuggingFace. To accurately measure the variables outlined in our study, we use the energibridge profiler, nvidia-smi profiler, and a wattmeter. … view at source ↗
Figure 3
Figure 3. Aggregated energy consumption and execution time by model size category: CUDA execution provider (left) and CPU execution [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Aggregated computing-resource utilization by model size category: CUDA execution provider (left) and CPU execution provider [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Aggregated computing-resource utilization by model size category: CUDA execution provider configurations measured by Nvidia [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Example of energy consumption and execution time for model [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Example of computing-resource utilization for model [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Example of computing-resource utilization for model [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

98 extracted references · 58 canonical work pages

  1. [1]

    Shoeybi, M

    M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, B. Catanzaro, Megatron-lm: Training multi-billion parame- ter language models using model parallelism, arXiv preprint arXiv:1909.08053 (2019)

    arXiv 2019

  2. [2]

    Strubell, A

    E. Strubell, A. Ganesh, A. McCallum, Energy and policy con- siderations for deep learning in NLP, in: Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, Association for Computational Linguistics, Florence, Italy, 2019, pp. 3645–3650. URL: https://aclanthology.org/ P19-1355. doi:10.18653/v1/P19-1355

    doi:10.18653/v1/p19-1355 2019

  3. [3]

    Cowls, A

    J. Cowls, A. Tsamados, M. Taddeo, L. Floridi, The ai gambit: leveraging artificial intelligence to combat climate change—opportunities, challenges, and recommendations, Ai & Society (2023) 1–25

    2023

  4. [4]

    Perrault, J

    R. Perrault, J. Clark, Artificial intelligence index report 2024 (2024)

    2024

  5. [5]

    Schwartz, J

    R. Schwartz, J. Dodge, N. A. Smith, O. Etzioni, Green AI, Commun. ACM 63 (2020) 54–63. doi: 10.1145/3381831

    doi:10.1145/3381831 2020

  6. [6]

    Dodge, T

    J. Dodge, T. Prewitt, R. Tachet des Combes, E. Odmark, R. Schwartz, E. Strubell, A. S. Luccioni, N. A. Smith, N. De- Cario, W. Buchanan, Measuring the carbon intensity of ai in cloud instances, in: 2022 ACM Conference on Fairness, Ac- countability, and Transparency, 2022, pp. 1877–1894

    2022

  7. [7]

    S. Ajel, F. Ribeiro, R. Ejbali, J. Saraiva, Energy efficiency of python machine learning frameworks, in: International Confer- ence on Intelligent Systems Design and Applications, Springer, 2022, pp. 586–595

    2022

  8. [8]

    Mart ´ ınez-Fern´ andez, X

    S. Mart ´ ınez-Fern´ andez, X. Franch, F. Dur´ an, Towards green ai-based software systems: an architecture-centric approach (gaissa), in: 2023 49th Euromicro Conference on Software En- gineering and Advanced Applications (SEAA), IEEE, 2023, pp. 432–439

    2023

Show all 98 references
  1. [9]

    Bhatt, S

    H. Bhatt, S. Arun, A. Kakran, K. Vaidhyanathan, Towards ar- chitecting sustainable mlops: A self-adaptation approach, arXiv preprint arXiv:2404.04572 (2024)

    2024 arXiv

  2. [10]

    Verdecchia, J

    R. Verdecchia, J. Sallou, L. Cruz, A systematic review of green ai, Wiley Interdisciplinary Reviews: Data Mining and Knowl- edge Discovery (2023) e1507

    2023

  3. [11]

    D. Zhao, N. C. Frey, J. McDonald, M. Hubbell, D. Bestor, M. Jones, A. Prout, V. Gadepally, S. Samsi, A green (er) world for ai, in: 2022 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), IEEE, 2022, pp. 742–750

    2022

  4. [12]

    Zhang, Y

    H. Zhang, Y. Huang, Y. Wen, J. Yin, K. Guan, Inferbench: Un- derstanding deep learning inference serving with an automatic benchmarking system, arXiv preprint arXiv:2011.02327 (2020)

    2020 arXiv

  5. [13]

    Kumara, R

    I. Kumara, R. Arts, D. Di Nucci, W. J. Van Den Heuvel, D. A. Tamburri, Requirements and reference architecture for mlops: insights from industry, Authorea Preprints (2023)

    2023

  6. [14]

    Face, Model deployment considerations, 2024

    H. Face, Model deployment considerations, 2024. URL: https://huggingface.co/learn/computer-vision-course/ en/unit9/model_deployment, last accessed October 28th, 2024

    2024

  7. [15]

    Face, Model optimization tools and frameworks, 2024

    H. Face, Model optimization tools and frameworks, 2024. URL: https://huggingface.co/learn/computer-vision-course/ en/unit9/tools_and_frameworks, last accessed October 28th, 2024

    2024

  8. [16]

    Dur´ an, S

    F. Dur´ an, S. Mart ´ ınez-Fern´ andez, M. Martinez, P. Lago, Identi- fying architectural design decisions for achieving green ml serv- ing, in: Proceedings of the IEEE/ACM 3rd International Con- ference on AI Engineering-Software Engineering for AI, 2024, pp. 18–23

    2024

  9. [17]

    Last accessed April 12th, 2024

    ONNX Runtime, Onnx runtime execution providers, https://onnxruntime.ai/docs/execution-providers/, 2024. Last accessed April 12th, 2024

    2024

  10. [18]

    Alizadeh, F

    N. Alizadeh, F. Castor, Green ai: A preliminary empirical study on energy consumption in dl models across different runtime infrastructures, arXiv preprint arXiv:2402.13640 (2024)

    2024 arXiv

  11. [19]

    ISO/IEC 25010:2012

    ISO/IEC 25010:2012 system and software quality requirements and evaluation (square) – system and software quality models, Standard, 2012. ISO/IEC 25010:2012

    2012

  12. [20]

    Ziegler, E

    A. Ziegler, E. Kalliamvakou, X. A. Li, A. Rice, D. Rifkin, S. Simister, G. Sittampalam, E. Aftandilian, Productivity as- sessment of neural code completion, in: Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Pro- gramming, 2022, pp. 21–29

    2022

  13. [21]

    X. Gu, M. Chen, Y. Lin, Y. Hu, H. Zhang, C. Wan, Z. Wei, Y. Xu, J. Wang, On the effectiveness of large language mod- els in domain-specific code generation, ACM Transactions on Software Engineering and Methodology 34 (2025) 1–22

    2025

  14. [22]

    F. F. Xu, U. Alon, G. Neubig, V. J. Hellendoorn, A systematic evaluation of large language models of code, in: Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, 2022, pp. 1–10

    2022

  15. [23]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, I. Polosukhin, Attention is all you need, Advances in neural information processing systems 30 (2017)

    2017

  16. [24]

    Jiang, K

    N. Jiang, K. Liu, T. Lutellier, L. Tan, Impact of code lan- guage models on automated program repair, arXiv preprint arXiv:2302.05020 (2023)

    2023 arXiv

  17. [25]

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al., Evaluating large language models trained on code, arXiv preprint arXiv:2107.03374 (2021)

    2021 arXiv

  18. [26]

    D. Zan, B. Chen, F. Zhang, D. Lu, B. Wu, B. Guan, W. Yongji, J.-G. Lou, Large language models meet nl2code: A survey, in: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023, pp. 7443–7464

    2023

  19. [27]

    F. Wang, Z. Zhang, X. Zhang, Z. Wu, T. Mo, Q. Lu, W. Wang, R. Li, J. Xu, X. Tang, et al., A comprehensive survey of small language models in the era of large language models: Tech- niques, enhancements, applications, collaboration with llms, and trustworthiness, arXiv preprint...

    2024 arXiv

  20. [28]

    Y. Zhu, M. Zhu, N. Liu, Z. Xu, Y. Peng, Llava-phi: Efficient multi-modal assistant with small language model, in: Proceed- ings of the 1st International Workshop on Efficient Multimedia Computing under Limited, 2024, pp. 18–22

    2024

  21. [29]

    S. Hu, Y. Tu, X. Han, C. He, G. Cui, X. Long, Z. Zheng, Y. Fang, Y. Huang, W. Zhao, et al., Minicpm: Unveiling the potential of small language models with scalable training strate- gies, arXiv preprint arXiv:2404.06395 (2024)

    2024 arXiv

  22. [30]

    Van Nguyen, X

    C. Van Nguyen, X. Shen, R. Aponte, Y. Xia, S. Basu, Z. Hu, J. Chen, M. Parmar, S. Kunapuli, J. Barrow, et al., A survey of small language models, arXiv preprint arXiv:2410.20011 (2024)

    2024 arXiv

  23. [31]

    Weber, M

    T. Weber, M. Brandmaier, A. Schmidt, S. Mayer, Significant productivity gains through programming with large language models, Proceedings of the ACM on Human-Computer Inter- action 8 (2024) 1–29

    2024

  24. [32]

    Jiang, F

    J. Jiang, F. Wang, J. Shen, S. Kim, S. Kim, A survey on large language models for code generation, arXiv preprint arXiv:2406.00515 (2024)

    2024 arXiv

  25. [33]

    Wohlin, P

    C. Wohlin, P. Runeson, M. H¨ ost, M. C. Ohlsson, B. Regnell, A. Wessl´ en, et al., Experimentation in software engineering, vol- ume 236, Springer, 2012

    2012

  26. [34]

    Goldfarb, Evaluating the pytorch compiler in the vision domain (2024)

    A. Goldfarb, Evaluating the pytorch compiler in the vision domain (2024)

    2024

  27. [35]

    NVIDIA Corporation, End-to-end ai for nvidia- based pcs: Onnx runtime and optimiza- tion, https://developer.nvidia.com/blog/ end-to-end-ai-for-pcs-onnx-runtime-and-optimization/ ,

  28. [36]

    Runtime, Onnx runtime, https://onnxruntime.ai/, 2024

    O. Runtime, Onnx runtime, https://onnxruntime.ai/, 2024. Last accessed July 17th, 2024

    2024

  29. [37]

    URL: https://onnxruntime.ai/docs/execution-providers/ CUDA-ExecutionProvider.html, last accessed April 12th, 2025

    ONNX Runtime, Cuda execution provider, 2024. URL: https://onnxruntime.ai/docs/execution-providers/ CUDA-ExecutionProvider.html, last accessed April 12th, 2025

    2024

  30. [38]

    Georgiou, M

    S. Georgiou, M. Kechagia, T. Sharma, F. Sarro, Y. Zou, Green ai: Do deep learning frameworks have different costs?, in: Pro- ceedings of the 44th International Conference on Software En- 23 gineering, 2022, pp. 1082–1094

    2022

  31. [39]

    Last accessed April 12th, 2024

    Intel, Openvino toolkit, https://software.intel.com/openvino- toolkit, 2019. Last accessed April 12th, 2024

    2019

  32. [40]

    Lenherr, R

    N. Lenherr, R. Pawlitzek, B. Michel, New universal sustainabil- ity metrics to assess edge intelligence, Sustainable Computing: Informatics and Systems 31 (2021) 100580

    2021

  33. [41]

    URL: https://www.nvidia.com/en-us/on-demand/session/ gtcspring21-s32240/, last accessed April 12th, 2025

    NVIDIA Corporation, Onnx runtime: Accelerating py- torch and tensorflow inferencing on cloud and edge, 2021. URL: https://www.nvidia.com/en-us/on-demand/session/ gtcspring21-s32240/, last accessed April 12th, 2025

    2021

  34. [42]

    URL: https://developer.nvidia.com/blog/ end-to-end-ai-for-nvidia-based-pcs-cuda-and-tensorrt-execution-providers-in-onnx-runtime/ , last accessed April 12th, 2025

    NVIDIA Corporation, End-to-end ai for nvidia-based pcs: Cuda and tensorrt execution providers in onnx run- time, 2022. URL: https://developer.nvidia.com/blog/ end-to-end-ai-for-nvidia-based-pcs-cuda-and-tensorrt-execution-providers-in-onnx-runtime/ , last accessed April 12th, 2025

    2022

  35. [43]

    C. Meng, M. Zhang, Z. Luan, Y. He, Dnabert: Efficient and differentiable kernel fusion for deep learning, in: Pro- ceedings of the Twenty-Fifth International Conference on Ar- chitectural Support for Programming Languages and Operat- ing Systems (ASPLOS), 2020. URL: https://dl...

    2020 doi

  36. [44]

    Last accessed April 12th, 2024

    NVIDIA Corporation, Cuda, https://developer.nvidia.com/cuda- zone, 2024. Last accessed April 12th, 2024

    2024

  37. [45]

    Hsieh, C.-L

    C.-Y. Hsieh, C.-L. Li, C.-K. Yeh, H. Nakhost, Y. Fujii, A. Rat- ner, R. Krishna, C.-Y. Lee, T. Pfister, Distilling step-by- step! outperforming larger language models with less training data and smaller model sizes, arXiv preprint arXiv:2305.02301 (2023)

    2023 arXiv

  38. [46]

    Gunasekar, Y

    S. Gunasekar, Y. Zhang, J. Aneja, C. C. T. Mendes, A. Del Giorno, S. Gopi, M. Javaheripi, P. Kauffmann, G. de Rosa, O. Saarikivi, et al., Textbooks are all you need, arXiv preprint arXiv:2306.11644 (2023)

    2023 arXiv

  39. [47]

    Schick, H

    T. Schick, H. Sch¨ utze, It’s not just size that matters: Small language models are also few-shot learners, arXiv preprint arXiv:2009.07118 (2020)

    2020 arXiv

  40. [48]

    J. Shi, Z. Yang, H. J. Kang, B. Xu, J. He, D. Lo, Greening large language models of code, in: Proceedings of the 46th In- ternational Conference on Software Engineering: Software En- gineering in Society, 2024, pp. 142–153

    2024

  41. [49]

    Svyatkovskiy, S

    A. Svyatkovskiy, S. Lee, A. Hadjitofi, M. Riechert, J. V. Franco, M. Allamanis, Fast and memory-efficient neural code comple- tion, in: 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR), IEEE, 2021, pp. 329–340

    2021

  42. [50]

    Kalliamvakou, Research: quantifying github copilot’s impact on developer productivity and happiness, The GitHub Blog (2022)

    E. Kalliamvakou, Research: quantifying github copilot’s impact on developer productivity and happiness, The GitHub Blog (2022)

    2022

  43. [51]

    S. A. Khowaja, P. Khuwaja, K. Dev, W. Wang, L. Nkenyereye, Chatgpt needs spade (sustainability, privacy, digital divide, and ethics) evaluation: A review, Cognitive Computation (2024) 1– 23

    2024

  44. [52]

    Y. Zhou, X. Lin, X. Zhang, M. Wang, G. Jiang, H. Lu, Y. Wu, K. Zhang, Z. Yang, K. Wang, et al., On the opportunities of green computing: A survey, arXiv preprint arXiv:2311.00447 (2023)

    2023 arXiv

  45. [53]

    Desislavov, F

    R. Desislavov, F. Mart ´ ınez-Plumed, J. Hern´ andez-Orallo, Trends in ai inference energy consumption: Beyond the performance-vs-parameter laws of deep learning, Sustainable Computing: Informatics and Systems 38 (2023) 100857

    2023

  46. [54]

    Zhang, Y

    X. Zhang, Y. Wang, W. Shi, {pCAMP}: Performance compar- ison of machine learning packages on the edges, in: USENIX workshop on hot topics in edge computing (HotEdge 18), 2018

    2018

  47. [55]

    Koubaa, A

    A. Koubaa, A. Ammar, A. Kanhouch, Y. AlHabashi, Cloud versus edge deployment strategies of real-time face recognition inference, IEEE Transactions on Network Science and Engi- neering 9 (2021) 143–160

    2021

  48. [56]

    R. M. Hampau, M. Kaptein, R. Van Emden, T. Rost, I. Mala- volta, An empirical study on the performance and energy con- sumption of ai containerization strategies for computer-vision tasks on the edge, in: Proceedings of the International Con- ference on Evaluation and Assessme...

    2022

  49. [57]

    Klimiato, Utilizing deep learning models for image analysis at scale: comparison of deployment solutions, Ph.D

    R. Klimiato, Utilizing deep learning models for image analysis at scale: comparison of deployment solutions, Ph.D. thesis, Kauno technologijos universitetas, 2022

    2022

  50. [58]

    Escribano Perez, Energy consumption of machine learn- ing deployment in cloud providers, B.S

    D. Escribano Perez, Energy consumption of machine learn- ing deployment in cloud providers, B.S. thesis, Universitat Polit` ecnica de Catalunya, 2023

    2023

  51. [59]

    Z. Yang, J. Shi, P. Devanbu, D. Lo, Ecosystem of large language models for code, arXiv preprint arXiv:2405.16746 (2024)

    2024 arXiv

  52. [60]

    Ilager, L

    S. Ilager, L. F. Briem, I. Brandic, Green-code: Optimizing energy efficiency in large language models for code generation, arXiv preprint arXiv:2501.11006 (2025)

    2025 arXiv

  53. [61]

    Z. Sun, X. Du, F. Song, S. Wang, L. Li, When neural code completion models size up the situation: Attaining cheaper and faster completion through dynamic model inference, in: Pro- ceedings of the IEEE/ACM 46th International Conference on Software Engineering, 2024, pp. 1–12

    2024

  54. [62]

    S. J. Warnett, U. Zdun, Architectural design decisions for ma- chine learning deployment, in: 2022 IEEE 19th International Conference on Software Architecture (ICSA), IEEE, 2022, pp. 90–100

    2022

  55. [63]

    S. C. Jepsen, A. Ø. Nielsen, M. B. Kjærgaard, Architecting ma- chine learning systems: Which parts are the architect’s pain?, in: 2024 IEEE 21st International Conference on Software Ar- chitecture Companion (ICSA-C), IEEE, 2024, pp. 303–306

    2024

  56. [64]

    V. R. Basili, G. Caldiera, H. D. Rombach, The goal question metric approach. encyclopedia of software engineering-2 volume set, Copyright by John Wiley & Sons, Inc (1994) 528–532

    1994

  57. [65]

    Casta˜ no, S

    J. Casta˜ no, S. Mart ´ ınez-Fern´ andez, X. Franch, Lessons learned from mining the hugging face repository, in: Proceedings of the 1st IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering, 2024, pp. 1–6

    2024

  58. [66]

    Fried, A

    D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, W.-t. Yih, L. Zettlemoyer, M. Lewis, Incoder: A generative model for code infilling and synthesis, arXiv preprint arXiv:2204.05999 (2022)

    2022 arXiv

  59. [67]

    Bavarian, H

    M. Bavarian, H. Jun, N. Tezak, J. Schulman, C. McLeavey, J. Tworek, M. Chen, Efficient training of language models to fill in the middle, arXiv preprint arXiv:2207.14255 (2022)

    2022 arXiv

  60. [68]

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, K. Narasimhan, Swe-bench: Can language models resolve real- world github issues?, arXiv preprint arXiv:2310.06770 (2023)

    2023 arXiv

  61. [69]

    Jedlitschka, M

    A. Jedlitschka, M. Ciolkowski, D. Pfahl, Reporting experiments in software engineering, Guide to advanced empirical software engineering (2008) 201–228

    2008

  62. [70]

    L. E. Lwakatare, I. Crnkovic, J. Bosch, Devops for ai–challenges in development of ai-enabled applications, in: 2020 interna- tional conference on software, telecommunications and com- puter networks (SoftCOM), IEEE, 2020, pp. 1–6

    2020

  63. [71]

    R. C. Castanyer, S. Mart ´ ınez-Fern´ andez, X. Franch, Which design decisions in ai-enabled mobile applications contribute to greener ai?, Empirical Software Engineering 29 (2024) 2

    2024

  64. [72]

    URL: https://docs.openvino.ai/2024/openvino-workflow/ running-inference.html, last accessed July 22nd, 2024

    Intel, Running inference with openvino, 2024. URL: https://docs.openvino.ai/2024/openvino-workflow/ running-inference.html, last accessed July 22nd, 2024

    2024

  65. [73]

    Guldner, R

    A. Guldner, R. Bender, C. Calero, G. S. Fernando, M. Funke, J. Gr¨ oger, L. M. Hilty, J. H¨ ornschemeyer, G.-D. Hoffmann, D. Junger, et al., Development and evaluation of a reference measurement model for assessing the resource and energy ef- ficiency of software products and ...

    2024

  66. [74]

    T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, et al., Transformers: State-of-the-art natural language processing, in: Proceedings of the 2020 conference on empirical methods in natural language processing: system demo...

    2020

  67. [75]

    Y. Xu, S. J. Mart ´ ınez Fern´ andez, M.-S. Mart ´ ınez Mart ´ ınez, J. Franch Guti´ errez, Energy efficiency of training neural net- work architectures: an empirical study, in: Proceedings of the 56th Annual Hawaii International Conference on System Sci- 24 ences: January 3-6...

    2023

  68. [76]

    Martinez, S

    M. Martinez, S. Mart ´ ınez-Fern´ andez, X. Franch, Energy con- sumption of automated program repair, in: Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings, 2024, pp. 358–359

    2024

  69. [77]

    Sallou, L

    J. Sallou, L. Cruz, T. Durieux, Energibridge: Empowering software sustainability through cross-platform energy measure- ment, arXiv preprint arXiv:2312.13897 (2023)

    2023 arXiv

  70. [78]

    Bekhelifi, N.-E

    O. Bekhelifi, N.-E. Berrached, On optimizing deep neural net- works inference on cpus for brain-computer interfaces using in- ference engines, in: 2024 IEEE International Symposium on Circuits and Systems (ISCAS), IEEE, 2024, pp. 1–5

    2024

  71. [79]

    URL: https://www.nvidia.com/en-us/ on-demand/session/gtc24-s62336/ , last accessed April 12th, 2025

    NVIDIA Corporation, Onnx runtime: Accelerated ai deploy- ment for pc apps, 2024. URL: https://www.nvidia.com/en-us/ on-demand/session/gtc24-s62336/ , last accessed April 12th, 2025

    2024

  72. [80]

    Stiles, Efficient Segment Anything on the Edge, Ph.D

    N. Stiles, Efficient Segment Anything on the Edge, Ph.D. thesis, Massachusetts Institute of Technology, 2024

    2024

  73. [81]

    URL: https: //onnxruntime.ai/docs/performance/tune-performance/ iobinding.html, last accessed April 12th, 2025

    ONNX Runtime, I/o binding, 2024. URL: https: //onnxruntime.ai/docs/performance/tune-performance/ iobinding.html, last accessed April 12th, 2025

    2024

  74. [82]

    J. Shi, Z. Yang, B. Xu, H. J. Kang, D. Lo, Compressing pre- trained models of code into 3 mb, in: Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, 2022, pp. 1–12

    2022

  75. [83]

    Singh, B

    J. Singh, B. Adams, A. E. Hassan, On the impact of black- box deployment strategies for edge ai on latency and model performance, arXiv preprint arXiv:2403.17154 (2024)

    2024 arXiv

  76. [84]

    Sever, S

    M. Sever, S. ¨O˘ g¨ ut, A performance study depending on execution times of various frameworks in machine learning inference, in: 2021 15th Turkish National Software Engineering Symposium (UYMS), IEEE, 2021, pp. 1–5

    2021

  77. [85]

    Pochelu, O

    P. Pochelu, O. Castro Lopez, Mastering computer vision infer- ence frameworks, in: Companion of the 15th ACM/SPEC In- ternational Conference on Performance Engineering, 2024, pp. 28–33

    2024

  78. [86]

    N. P. Lopes, Torchy: A tracing jit compiler for pytorch, in: Pro- ceedings of the 32nd ACM SIGPLAN International Conference on Compiler Construction, 2023, pp. 98–109

    2023

  79. [87]

    Ansel, E

    J. Ansel, E. Yang, H. He, N. Gimelshein, A. Jain, M. Vozne- sensky, B. Bao, P. Bell, D. Berard, E. Burovski, et al., Pytorch 2: Faster machine learning through dynamic python bytecode transformation and graph compilation, in: Proceedings of the 29th ACM International Conferenc...

    2024

  80. [88]

    URL: https://pytorch.org/docs/ stable/jit.html, last accessed July 30th, 2024

    PyTorch, Torchscript, 2023. URL: https://pytorch.org/docs/ stable/jit.html, last accessed July 30th, 2024

    2023

  81. [89]

    URL: https://discuss.pytorch.org/t/ why-torchscript-module-does-not-take-less-gpu-memory-than-pytorch-model/ 157560/4, last accessed July 30th, 2024

    PyTorch Community, Why torchscript module does not take less gpu memory than pytorch model?, 2022. URL: https://discuss.pytorch.org/t/ why-torchscript-module-does-not-take-less-gpu-memory-than-pytorch-model/ 157560/4, last accessed July 30th, 2024

    2022

  82. [90]

    Z. Chen, A. Kerr, R. Cai, J. Kosaian, H. Wu, Y. Ding, Y. Xie, Evt: Accelerating deep learning training with epilogue visitor tree, in: Proceedings of the 29th ACM International Confer- ence on Architectural Support for Programming Languages and Operating Systems, Volume 3, 202...

    2024

  83. [91]

    Kaplan, S

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, D. Amodei, Scaling laws for neural language models, arXiv preprint arXiv:2001.08361 (2020)

    2020 arXiv

  84. [92]

    Z. Zhou, X. Ning, K. Hong, T. Fu, J. Xu, S. Li, Y. Lou, L. Wang, Z. Yuan, X. Li, et al., A survey on efficient inference for large language models, arXiv preprint arXiv:2404.14294 (2024)

    2024 arXiv

  85. [93]

    S. Mart ´ ınez-Fern´ andez, Environmental sustainability of ma- chine learning systems: Reducing the carbon impact of their lifecycle process, in: International Conference on Product- Focused Software Process Improvement, Springer, 2024, pp. 3– 7

    2024

  86. [94]

    Y. Han, G. Huang, S. Song, L. Yang, H. Wang, Y. Wang, Dy- namic neural networks: A survey, IEEE Transactions on Pat- tern Analysis and Machine Intelligence 44 (2021) 7436–7456

    2021

  87. [95]

    W. Wang, W. Chen, Y. Luo, Y. Long, Z. Lin, L. Zhang, B. Lin, D. Cai, X. He, Model compression and efficient in- ference for large language models: A survey, arXiv preprint arXiv:2402.09748 (2024)

    2024 arXiv

  88. [96]

    URL: https:// onnxruntime.ai/docs/api/python/api_summary.html, last ac- cessed April 12th, 2025

    ONNX Runtime, Api overview, 2024. URL: https:// onnxruntime.ai/docs/api/python/api_summary.html, last ac- cessed April 12th, 2025

    2024

  89. [97]

    URL: https://eur-lex.europa.eu/eli/reg/2024/1689/oj, last ac- cessed November, 2024

    European Union, Artificial intelligence act., 2024. URL: https://eur-lex.europa.eu/eli/reg/2024/1689/oj, last ac- cessed November, 2024. 25

    2024

  90. [2022]

    Last accessed April 12th, 2024

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.