Pith. sign in

REVIEW 3 major objections 5 minor 4 cited by

Energy-Aware Code Generation with LLMs: Benchmarking Small vs. Large Language Models for Sustainable AI Programming

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that on 52.6% of 150 LeetCode-style problems, at least one small open-source code model produced correct code that used as much or less execution energy than large commercial LLMs, while large models remained more accurate

desk verdict A carefully documented benchmark that is undercut by its own ChatGPT-cleaning step; the headline energy result is not attributable to the small models. read the letter →

arxiv 2508.08332 v1 pith:JCZMKLKI submitted 2025-08-10 cs.SE cs.AI

Humza Ashraf , Syed Muhammad Danish , Aris Leivadeas , Yazan Otoum , Zeeshan Sattar This is my paper
classification cs.SEcs.AI
keywords codegenerationsmalllanguagemodelslargeenergyefficiencysustainabilityperformanceevaluationLeetbenchmarkPython
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that small open-source code models can serve as energy-efficient alternatives to large commercial models for a meaningful share of routine programming problems. On a benchmark of 150 Python problems evenly split across easy, medium, and hard difficulty, large models GPT-4.0 and DeepSeek-Reasoner remain the most accurate, but in 79 of 150 problems (52.6%) at least one of three 3-billion-parameter models produced a correct solution whose execution energy was equal to or lower than the large models' and human-written solutions. The result matters because energy use from code assistants is a growing environmental cost, and small models are cheaper to run locally. The comparison is about the energy consumed by the generated code when executed, not the energy of training or serving the models. The paper also shows that SLMs are not interchangeable: Qwen2.5-Coder-3B-Instruct clearly outperforms the other two small models.

What carries the argument

The central object is a standardized execution-energy benchmark. Each model receives the same LeetCode prompt with instructions to emit only runnable Python (a class Solution plus an __main__ block); generated code is executed ten times per sample in an isolated Linux VM with a five-second cooling period; correctness is checked against LeetCode test cases plus manual review; energy is tracked with CodeCarbon, peak memory with tracemalloc, and runtime with Python's time module. The comparison is defined over the energy of the generated code when run, not over the energy of training or serving the models.

What would settle it

Re-run the 150-problem protocol on raw SLM outputs before the ChatGPT cleanup step, or measure correctness and energy both before and after cleanup on a sample. If raw outputs are substantially less correct or have a different energy profile, the 52.6% claim is an artifact of the cleanup stage rather than a property of the small models.

Watch

Extended reading notes

Core claim

Stated on the paper's own terms, the central discovery is conditional: if a small model's output is functionally correct, that output is often as energy-efficient as a large model's output. Concretely, across 150 LeetCode problems, the three SLMs together produced correct and energy-competitive solutions in 79 problems (52.6% of the total; 66% of easy, 42% of medium, 50% of hard). Large models maintain the highest correctness at every difficulty level, with DeepSeek-Reasoner reaching 88%, 86%, and 74% success by difficulty and GPT-4.0 reaching 80%, 82%, and 74%, versus the best SLM Qwen2.5-Coder-3B-Instruct at 74%, 72%, and 66%. The energy differences are small in absolute terms (averages ar

Load-bearing premise

The load-bearing premise is that cleaning generated code with the ChatGPT API, described in Section III-E, does not materially change the algorithm being tested; if the cleanup rewrites the code, the correctness and energy numbers attributed to the small models are actually attributable to ChatGPT.

Editorial extensions

If this is right

  • Energy-constrained deployments can consider small models for easy and medium LeetCode-class tasks without sacrificing per-task energy, provided correctness is verified.
  • Model choice matters: the energy-efficiency result is driven mostly by Qwen2.5-Coder-3B-Instruct; StarCoderBase-3B contributes little in correct, energy-competitive outputs.
  • Because the 52.6% figure is a union over three small models, a practical system may need to generate with several SLMs and select a correct output, which changes the total energy accounting.
  • Improving SLM correctness would convert a conditional energy advantage into a broader one, since energy parity already holds when outputs are correct.
  • Human-written LeetCode solutions remain close to both model families in energy, so the sustainability gain of SLMs over LLMs is modest unless combined with correctness screening.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that the absolute energy differences (roughly 0.01 to 0.02 mWh per run) are small, so the practical case for SLMs is cumulative fleet-level savings plus local-deployment benefits such as privacy, latency, and avoiding API costs rather than large per-task savings.
  • A testable extension is to instrument the ChatGPT cleanup step itself, measuring correctness and energy on raw versus cleaned SLM outputs; without that measurement, the SLM-specific credit for the 52.6% result is uncertain.
  • A natural follow-up is a correctness-screening cascade: run a small model first and escalate only failed or low-confidence cases to a large model; the paper's per-model correctness and energy data supply the inputs such a policy would need.
  • The protocol could be extended to end-to-end energy by including the local SLM inference on commodity hardware and the cloud API inference for LLMs, which would likely favor SLMs further but is outside the paper's execution-code-only scope.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper evaluates three 3-billion-parameter open-source small language models (StableCode-3B, StarCoderBase-3B, Qwen2.5-Coder-3B-Instruct) against two commercial large language models (GPT-4.0, DeepSeek-Reasoner) and human-written LeetCode solutions. For 150 Python problems (50 each of easy, medium, hard), the authors generate code, clean it, and measure correctness, runtime, memory, and energy consumption. Their central claim is that, while LLMs achieve higher correctness, SLMs are often more energy-efficient when their outputs are correct, with 52.6% of problems having at least one correct SLM solution that consumes equal or less energy than LLM/human baselines. The paper does not measure inference energy; it measures only the energy of the final generated code.

Significance. If the central claim were established, the paper would provide a useful, practically relevant benchmark for choosing smaller models in energy-constrained or privacy-sensitive deployment scenarios. The topic is timely, and the comparison across difficulty levels, model families, and human baselines is appropriate. The authors also state their scope clearly: they do not measure training or inference energy, only the energy of the generated code. The paper's related-work section gives a reasonable overview of prior energy-efficiency benchmarks. However, the empirical load-bearing steps have significant gaps: the ChatGPT-based cleaning step contaminates attribution, the reported energy differences are extremely small with no uncertainty quantification, and the derivation of the 52.6% result is ambiguous. These issues are fixable in principle but currently prevent the headline result from being interpretable.

major comments (3)
  1. [Section III-E, Section V] The central attribution of correctness and energy to the SLMs is contaminated by the ChatGPT API cleaning step. The text states: 'In cases where code required minor modifications to ensure executability, we used the ChatGPT API to automate the code-cleaning process' and, for small models that produced multiple solutions, 'we used the paid ChatGPT API to clean the code and keep only the first valid solution.' The final code that is measured is therefore not necessarily the SLM's output: GPT-4.0 may have fixed syntax by rewriting logic, or selected among multiple SLM outputs. The manuscript does not report how often cleaning occurred, what changes were made, whether cleaning was applied only to SLM outputs, or whether any algorithmic structure was altered. Section V lists limitations but does not mention this confound. Because the headline 52.6% result and Table III / Fig. 4 counts are com
  2. [Section IV-A, Table II] The energy data underlying the main claim are not statistically supported. In Table II, all average energy values lie between 1.442 and 1.458 mWh, a range of 0.016 mWh, and no standard deviations, confidence intervals, or significance tests are reported. The runtimes are also very small (about 20 ms), where Python interpreter startup and CodeCarbon sampling resolution can dominate. With only 10 runs per sample, the paper needs to show that the per-problem energy differences used to build Table III are not measurement noise. In addition, Table II reports averages over all generated outputs, not only correct outputs, while the energy-efficiency claim is about correct outputs. The authors should report per-problem paired comparisons, standard deviations, effect sizes (e.g., a paired bootstrap or Cliff's delta), and separate statistics for the correct-output subset.
  3. [Section IV-C, Table III, Fig. 4] The derivation of 79 out of 150 problems (52.6%) is ambiguous. The text says 'at least one SLM produced a correct and energy-efficient solution comparable to LLMs,' implying a union over the three SLMs. However, Table III lists per-model counts of 33 (Qwen2.5-Coder), 29 (StableCode), and 17 (StarCoderBase), which sum to exactly 79. If the 79 is a true union over models, then the per-model counts cannot be simply summed without overlap information; if it is a sum of model-level counts, the 'at least one SLM' statement is unsupported. Overlaps are likely, since the same problem may be solved correctly and efficiently by multiple SLMs. The paper should report the union explicitly (e.g., a Venn-style breakdown). In addition, the comparison baseline is unclear: the text says 'same amount of energy or less than LLMs ... along with human-written solutions,' while the Fig. 4 caption says 'equal
minor comments (5)
  1. [Section IV-D, Table III] The Medium column for StarCoderBase-3B reads '4 (%)' in the text; the percentage '8%' appears to be missing.
  2. [Section IV-B, Fig. 3] The text says success rates are 'out of 150 attempts,' but the figure appears to break down by difficulty, so each category has 50 attempts. Clarify whether the denominator is 50 per difficulty or 150 overall.
  3. [Section III-E] The prompt includes 'two test cases with their outputs,' but the paper does not report the exact test cases or how many random trials per model/prompt were run. Reporting the sampling temperature, seed, and number of generations would improve reproducibility.
  4. [Section III-G] The correctness evaluation is described as automated testing followed by a manual review, but the manual review procedure is not specified (e.g., who did it, how disagreements were resolved). A short protocol description would strengthen confidence in the correctness labels.
  5. [Section VI] The conclusion repeats the 52.6% result without the 'at least one SLM' and 'when correct' qualifications used in Section IV-C. Please ensure the conclusion matches the qualified finding.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central comparisons are external measurements against fixed benchmarks and human baselines.

full rationale

The paper's central claim (Section IV-C: 'in 52.6% of the total problems, at least one SLM produced a correct and energy-efficient solution comparable to LLMs') is a summary statistic over directly measured quantities: code correctness (checked against LeetCode test cases), runtime (Python time module), memory (tracemalloc), and energy (CodeCarbon), all executed in a controlled environment. There is no fitted parameter, no model whose definition embeds the target result, and no derivation in which an output quantity is equal by construction to an input quantity. The SLM/LLM outputs are generated from fixed prompts and measured externally; human-written LeetCode solutions provide an independent baseline. The only notable methodological concern is Section III-E's code-cleaning step ('we used the paid ChatGPT API to clean the code and keep only the first valid solution'), which is a potential attribution confound because the final measured code may reflect ChatGPT post-processing rather than pure SLM output. However, this is a validity threat, not circular reasoning: no result is defined in terms of a quantity the paper itself produced, and the cleaning step is not a fitted parameter that is later renamed as a prediction. The Limitations section (Section V) omits this confound, and the omission weakens the paper's internal validity, but it does not make the claimed comparison circular. There is no load-bearing self-citation, no imported uniqueness theorem, and no renaming of a known result as a new derivation. Hence the circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters or invented entities are introduced. The benchmark rests on standard measurement assumptions and on the integrity of external APIs and the human baseline. The most fragile assumption is that ChatGPT-assisted cleaning preserves the identity of the small-model-generated code.

assumptions (4)
  • domain assumption CodeCarbon CPU-based energy estimates accurately capture relative energy differences for short Python scripts on a Google Cloud VM.
    Invoked in Section III-G.1. The central energy metric depends entirely on CodeCarbon's hardware model and CPU readings, and no calibration or cross-check is reported.
  • domain assumption English-language LeetCode problem prompts plus two test cases are sufficient and fair inputs for all five models.
    Section III-E. If the commercial APIs saw LeetCode problems in training, memorization could inflate LLM correctness, an issue the paper itself acknowledges in Section V.
  • domain assumption The most-upvoted LeetCode community solution is an appropriate efficiency baseline.
    Section III-C. The paper claims these solutions are high quality, but also acknowledges in Section V that upvotes may not reflect optimal efficiency.
  • domain assumption Ten repeated runs with a five-second cooldown provide statistically stable estimates at the observed runtimes around 20 ms.
    Section III-F. No variance or confidence intervals are reported, and differences of 0.01 mWh are near measurement resolution.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Energy-Aware Code Generation with LLMs: Benchmarking Small vs. Large Language Models for Sustainable AI Programming." pith.science (2026). https://pith.science/paper/JCZMKLKI

@misc{pith2026250808332,
  author       = {Pith},
  title        = {Pith review of: Energy-Aware Code Generation with LLMs: Benchmarking Small vs. Large Language Models for Sustainable AI Programming},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JCZMKLKI}},
  note         = {Machine review of arXiv:2508.08332}
}
read the original abstract

Large Language Models (LLMs) are widely used for code generation. However, commercial models like ChatGPT require significant computing power, which leads to high energy use and carbon emissions. This has raised concerns about their environmental impact. In this study, we evaluate open-source Small Language Models (SLMs) trained explicitly for code generation and compare their performance and energy efficiency against large LLMs and efficient human-written Python code. The goal is to investigate whether SLMs can match the performance of LLMs on certain types of programming problems while producing more energy-efficient code. We evaluate 150 coding problems from LeetCode, evenly distributed across three difficulty levels: easy, medium, and hard. Our comparison includes three small open-source models, StableCode-3B, StarCoderBase-3B, and Qwen2.5-Coder-3B-Instruct, and two large commercial models, GPT-4.0 and DeepSeek-Reasoner. The generated code is evaluated using four key metrics: run-time, memory usage, energy consumption, and correctness. We use human-written solutions as a baseline to assess the quality and efficiency of the model-generated code. Results indicate that LLMs achieve the highest correctness across all difficulty levels, but SLMs are often more energy-efficient when their outputs are correct. In over 52% of the evaluated problems, SLMs consumed the same or less energy than LLMs.

Figures

Figures reproduced from arXiv: 2508.08332 by the authors.

Figure 1
Figure 1. CO2 Emissions: LLM Training vs. 83 Years of Home Power [3] total daily energy consumption would be approximately 0.2 gigawatt-hours (GWh) [4]. As a result of this growing energy demand, LLM-based applications have faced critical questions regarding their sustainability. In light of this, a key question arises: are large LLMs always necessary, especially for routine or simpler coding tasks? In this context, Small Lan… view at source ↗
Figure 2
Figure 2. Methodology for Code Generation and Evaluation Using LLM Models [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Energy-efficient correctness comparison across difficulty levels, [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Evaluating the Environmental Impact of using SLMs and Prompt Engineering for Code Generation

    cs.SE 2026-04 unverdicted novelty 7.0 of 10

    Chain-of-Thought prompting balances high accuracy with low energy use in small language models for code generation, while multi-sampling strategies add high energy costs for small accuracy gains.

  2. Beyond the Need for Speed: Energy-Aware Code Generation via Simulation-Guided Reinforcement Learning

    cs.LG 2026-07 conditional novelty 6.5 of 10

    Simulation-in-the-loop GRPO on energy-labeled C++ pairs yields 12.63% CARET and beats human energy efficiency on 58.4% of valid held-out outputs, while IPC misranks energy on 67.8% of problems.

  3. EcoAssist: Embedding Sustainability into AI-Assisted Frontend Development

    cs.HC 2026-04 unverdicted novelty 5.0 of 10

    EcoAssist embeds energy estimation and optimization into AI-assisted frontend coding, reducing website energy use by 13-16% in benchmarks while preserving developer productivity.

  4. Sustainable Code Generation Using Large Language Models: A Systematic Literature Review

    cs.SE 2026-03 unverdicted novelty 3.0 of 10

    A systematic review finds research on the sustainability of LLM-generated code to be limited, fragmented, and without accepted frameworks for measurement or benchmarking.

Reference graph

Works this paper leans on

25 extracted references · 13 canonical work pages · cited by 4 Pith papers

  1. [18]

    Evaluating the energy-efficiency of the code generated by llms,

    M. A. Islam, D. V . Jonnala, R. Rekhi, P. Pokharel, S. Cilamkoti, A. Imran, T. Kosar, and B. Turkkan, “Evaluating the energy-efficiency of the code generated by llms,” arXiv preprint arXiv:2505.20324, 2025

    arXiv 2025

  2. [19]

    Can llms generate green code-a comprehensive study through leetcode,

    J. F. Tuttle, D. Chen, A. Nasrin, N. Soto, and Z. Zong, “Can llms generate green code-a comprehensive study through leetcode,” in 2024 IEEE 15th International Green and Sustainable Computing Conference (IGSC). IEEE, 2024, pp. 39–44

    work page 2024

  3. [1]

    Herrington, Code generation in action

    J. Herrington, Code generation in action . Manning Publications Co., 2003

    work page 2003

  4. [2]

    Trends in ai inference energy consumption: Beyond the performance-vs-parameter laws of deep learning,

    R. Desislavov, F. Mart ´ınez-Plumed, and J. Hern ´andez-Orallo, “Trends in ai inference energy consumption: Beyond the performance-vs-parameter laws of deep learning,” Sustainable Computing: Informatics and Sys- tems, vol. 38, p. 100857, 2023

    work page 2023

  5. [3]

    Holistically evaluating the environmental impact of creating language models. arxiv 2025,

    J. Morrison, C. Na, J. Fernandez, T. Dettmers, E. Strubell, and J. Dodge, “Holistically evaluating the environmental impact of creating language models. arxiv 2025,” arXiv preprint arXiv:2503.05804 , 2025

    arXiv 2025

  6. [4]

    Learn to code sustainably: An empirical study on llm-based green code generation,

    T. Vartziotis, I. Dellatolas, G. Dasoulas, M. Schmidt, F. Schneider, T. Hoffmann, S. Kotsopoulos, and M. Keckeisen, “Learn to code sustainably: An empirical study on llm-based green code generation,” arXiv preprint arXiv:2403.03344 , 2024

    arXiv 2024

  7. [5]

    What is the role of small models in the llm era: A survey,

    L. Chen and G. Varoquaux, “What is the role of small models in the llm era: A survey,” arXiv preprint arXiv:2409.06857 , 2024

    arXiv 2024

  8. [6]

    Mercury: A code efficiency benchmark for code large language models,

    M. Du, A. T. Luu, B. Ji, Q. Liu, and S.-K. Ng, “Mercury: A code efficiency benchmark for code large language models,” Advances in Neural Information Processing Systems , vol. 37, pp. 16 601–16 622, 2024

    work page 2024

Show all 25 references
  1. [7]

    Effibench: Benchmarking the efficiency of automatically generated code,

    D. Huang, Y . Qing, W. Shang, H. Cui, and J. M. Zhang, “Effibench: Benchmarking the efficiency of automatically generated code,”Advances in Neural Information Processing Systems , vol. 37, pp. 11 506–11 544, 2024

    2024

  2. [8]

    Codereval: A benchmark of pragmatic code generation with generative pre-trained models,

    H. Yu, B. Shen, D. Ran, J. Zhang, Q. Zhang, Y . Ma, G. Liang, Y . Li, Q. Wang, and T. Xie, “Codereval: A benchmark of pragmatic code generation with generative pre-trained models,” in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering , 2024, pp. 1–12

    2024

  3. [9]

    Measuring coding challenge competence with apps,

    D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Songet al., “Measuring coding challenge competence with apps,” arXiv preprint arXiv:2105.09938 , 2021

    2021 arXiv

  4. [10]

    Recode: Robustness evaluation of code generation models,

    S. Wang, Z. Li, H. Qian, C. Yang, Z. Wang, M. Shang, V . Kumar, S. Tan, B. Ray, P. Bhatia et al., “Recode: Robustness evaluation of code generation models,” arXiv preprint arXiv:2212.10264 , 2022

    2022 arXiv

  5. [11]

    Effi-code: Unleashing code efficiency in language models,

    D. Huang, G. Zeng, J. Dai, M. Luo, H. Weng, Y . Qing, H. Cui, Z. Guo, and J. Zhang, “Effi-code: Unleashing code efficiency in language models,” arXiv preprint arXiv:2410.10209v1 , 2024

    2024 arXiv

  6. [12]

    An exploration of prompting llms to generate energy-efficient code,

    T. Cappendijk, P. de Reus, and A. Oprescu, “An exploration of prompting llms to generate energy-efficient code,” in 2025 IEEE/ACM 9th Interna- tional Workshop on Green and Sustainable Software (GREENS) . IEEE Computer Society, 2025, pp. 31–38

    2025

  7. [13]

    Large language models for energy-efficient code: Emerging results and future directions,

    H. Peng, A. Gupte, N. J. Eliopoulos, C. C. Ho, R. Mantri, L. Deng, W. Jiang, Y .-H. Lu, K. L ¨aufer, G. K. Thiruvathukal et al. , “Large language models for energy-efficient code: Emerging results and future directions,” arXiv preprint arXiv:2410.09241 , 2024

    2024 arXiv

  8. [14]

    Can we make code green? understanding trade-offs in llms vs. human code optimizations,

    P. Rani, J.-A. Bard, J. Sallou, A. Boll, T. Kehrer, and A. Bacchelli, “Can we make code green? understanding trade-offs in llms vs. human code optimizations,” arXiv preprint arXiv:2503.20126 , 2025

    2025 arXiv

  9. [15]

    Comparative analysis of carbon footprint in manual vs. llm-assisted code development,

    K. S. Cheung, M. Kaul, G. Jahangirova, M. R. Mousavi, and E. Zie, “Comparative analysis of carbon footprint in manual vs. llm-assisted code development,” in Proceedings of the 1st International Workshop on Responsible Software Engineering , 2025, pp. 13–20

    2025

  10. [16]

    Green my llm: Studying the key factors affecting the energy consumption of code assistants,

    T. Coignion, C. Quinton, and R. Rouvoy, “Green my llm: Studying the key factors affecting the energy consumption of code assistants,” arXiv preprint arXiv:2411.11892, 2024

    2024 arXiv

  11. [17]

    Sustainability via llm right-sizing,

    J. Haase, F. Klessascheck, J. Mendling, and S. Pokutta, “Sustainability via llm right-sizing,” arXiv preprint arXiv:2504.13217 , 2025

    2025 arXiv

  12. [20]

    A controlled experiment on the energy efficiency of the source code generated by code llama,

    V .-A. Cursaru, L. Duits, J. Milligan, D. Ural, B. R. Sanchez, V . Stoico, and I. Malavolta, “A controlled experiment on the energy efficiency of the source code generated by code llama,” in International Confer- ence on the Quality of Information and Communications Technology...

    2024

  13. [21]

    How efficient is llm-generated code? a rigorous & high-standard benchmark,

    R. Qiu, W. W. Zeng, J. Ezick, C. Lott, and H. Tong, “How efficient is llm-generated code? a rigorous & high-standard benchmark,” arXiv preprint arXiv:2406.06647, 2024

    2024 arXiv

  14. [22]

    Carbon footprint evaluation of code generation through llm as a service,

    T. Vartziotis, M. Schmidt, G. Dasoulas, I. Dellatolas, S. Attademo, V . D. Le, A. Wiechmann, T. Hoffmann, M. Keckeisen, and S. Kotsopoulos, “Carbon footprint evaluation of code generation through llm as a service,” in International Stuttgart Symposium . Springer, 2024, pp. 230–241

    2024

  15. [23]

    Ai-powered, but power- hungry? energy efficiency of llm-generated code,

    L. Solovyeva, S. Weidmann, and F. Castor, “Ai-powered, but power- hungry? energy efficiency of llm-generated code,” in 2025 IEEE/ACM Second International Conference on AI Foundation Models and Software Engineering (Forge). IEEE, 2025, pp. 49–60

    2025

  16. [24]

    The goal question metric approach,

    V . R. B. G. Caldiera and H. D. Rombach, “The goal question metric approach,” Encyclopedia of software engineering , pp. 528–532, 1994

    1994

  17. [25]

    On evaluating the efficiency of source code generated by llms,

    C. Niu, T. Zhang, C. Li, B. Luo, and V . Ng, “On evaluating the efficiency of source code generated by llms,” in Proceedings of the 2024 IEEE/ACM First International Conference on AI Foundation Models and Software Engineering, 2024, pp. 103–107

    2024

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.