{"id":"c3d1441e-2a75-4eec-9e9c-653d49c07ef4","arxiv_id":"2508.08332","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"On 150 LeetCode problems, GPT-4.0 and DeepSeek-Reasoner beat three 3B-parameter models on correctness and speed; the 52% energy-efficiency claim counts any of three SLMs on correct outputs, not a per-model advantage.","lead":"A study benchmarked three small open-source code models and two large commercial models on 150 LeetCode problems, measuring correctness, runtime, memory, and energy of generated Python. Large models were more often correct, while small models only sometimes matched energy use in correct outputs, a result the abstract presents more strongly than the data support.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ChatGPT cleaning of SLM outputs confounds attribution: measured code may not reflect SLM capability.","rationale":"I read the paper carefully. The experimental setup is clearly described, and the idea of comparing SLMs and LLMs on energy-aware code generation is reasonable. However, the most load-bearing assumption is that the code attributed to SLMs is actually produced by SLMs. The paper's own methodology (Section III-E) admits to using ChatGPT to clean SLM outputs. This is not a minor detail: it directly affects the validity of the independent variable. If ChatGPT modifies or selects the code, then the energy and correctness results are confounded. The reader's weakest_assumption correctly identifies this. I considered other possible concerns, such as the lack of variance reporting and the union-over-models nature of the 52.6% claim, but these are secondary: even if they were addressed, the cleaning issue would still undermine the attribution. The paper's Limitations section omits this confound, making it more concerning. I propose a concrete test: compare raw SLM outputs to executed code and re-run measurements without cleaning. If the claim survives, the paper could be revised to emphasize the need for post-processing, which would change the interpretation but not necessarily reject it. However, given the current state, REJECT is justified.","tokens_in":11213,"tokens_out":5320,"duration_ms":51256,"concrete_test":"For all 150 problems, store the raw SLM output before cleaning. Have independent reviewers (or an automated diff tool) compare the raw output to the final code that was executed. Count how many problems had non-trivial changes (e.g., changes beyond whitespace/imports, or changes that could affect time complexity). Then re-run the energy and correctness measurements on the raw SLM outputs for the subset that is directly executable, and recompute the 'at least one SLM correct and energy-efficient' percentage. If the percentage drops materially, or if a large fraction of final solutions were changed by ChatGPT, the central claim is contaminated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on attributing energy and correctness to SLM-generated code. Section III-E states that 'in cases where code required minor modifications to ensure executability, we used the ChatGPT API to automate the code-cleaning process' and, for small models that produced more than one solution, 'we used the paid ChatGPT API to clean the code and keep only the first valid solution.' This means the final code that is tested for correctness and energy may have been substantially rewritten or selected by ChatGPT, not by the SLM. If ChatGPT alters the algorithmic structure (e.g., fixing a syntax error by changing logic, or choosing an efficient variant among multiple outputs), the measured performance reflects ChatGPT's contribution, not the SLM's. The paper does not quantify how often cleaning occurred, what changes were made, or whether cleaning is applied only to SLMs (as the phrasing suggests). Without this, the 52.6% claim is uninterpretable: it may indicate that ChatGPT can post-process SLM outputs into efficient code, rather than that SLMs themselves are energy-efficient. The Limitations section (Section V) does not mention this confound, underscoring its omission.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates three 3-billion-parameter open-source small language models (StableCode-3B, StarCoderBase-3B, Qwen2.5-Coder-3B-Instruct) against two commercial large language models (GPT-4.0, DeepSeek-Reasoner) and human-written LeetCode solutions. For 150 Python problems (50 each of easy, medium, hard), the authors generate code, clean it, and measure correctness, runtime, memory, and energy consumption. Their central claim is that, while LLMs achieve higher correctness, SLMs are often more energy-efficient when their outputs are correct, with 52.6% of problems having at least one correct SLM solution that consumes equal or less energy than LLM/human baselines. The paper does not measure inference energy; it measures only the energy of the final generated code.","tokens_in":11446,"tokens_out":6884,"duration_ms":72101,"significance":"If the central claim were established, the paper would provide a useful, practically relevant benchmark for choosing smaller models in energy-constrained or privacy-sensitive deployment scenarios. The topic is timely, and the comparison across difficulty levels, model families, and human baselines is appropriate. The authors also state their scope clearly: they do not measure training or inference energy, only the energy of the generated code. The paper's related-work section gives a reasonable overview of prior energy-efficiency benchmarks. However, the empirical load-bearing steps have significant gaps: the ChatGPT-based cleaning step contaminates attribution, the reported energy differences are extremely small with no uncertainty quantification, and the derivation of the 52.6% result is ambiguous. These issues are fixable in principle but currently prevent the headline result from being interpretable.","major_comments":[{"comment":"The central attribution of correctness and energy to the SLMs is contaminated by the ChatGPT API cleaning step. The text states: 'In cases where code required minor modifications to ensure executability, we used the ChatGPT API to automate the code-cleaning process' and, for small models that produced multiple solutions, 'we used the paid ChatGPT API to clean the code and keep only the first valid solution.' The final code that is measured is therefore not necessarily the SLM's output: GPT-4.0 may have fixed syntax by rewriting logic, or selected among multiple SLM outputs. The manuscript does not report how often cleaning occurred, what changes were made, whether cleaning was applied only to SLM outputs, or whether any algorithmic structure was altered. Section V lists limitations but does not mention this confound. Because the headline 52.6% result and Table III / Fig. 4 counts are com","section":"Section III-E, Section V"},{"comment":"The energy data underlying the main claim are not statistically supported. In Table II, all average energy values lie between 1.442 and 1.458 mWh, a range of 0.016 mWh, and no standard deviations, confidence intervals, or significance tests are reported. The runtimes are also very small (about 20 ms), where Python interpreter startup and CodeCarbon sampling resolution can dominate. With only 10 runs per sample, the paper needs to show that the per-problem energy differences used to build Table III are not measurement noise. In addition, Table II reports averages over all generated outputs, not only correct outputs, while the energy-efficiency claim is about correct outputs. The authors should report per-problem paired comparisons, standard deviations, effect sizes (e.g., a paired bootstrap or Cliff's delta), and separate statistics for the correct-output subset.","section":"Section IV-A, Table II"},{"comment":"The derivation of 79 out of 150 problems (52.6%) is ambiguous. The text says 'at least one SLM produced a correct and energy-efficient solution comparable to LLMs,' implying a union over the three SLMs. However, Table III lists per-model counts of 33 (Qwen2.5-Coder), 29 (StableCode), and 17 (StarCoderBase), which sum to exactly 79. If the 79 is a true union over models, then the per-model counts cannot be simply summed without overlap information; if it is a sum of model-level counts, the 'at least one SLM' statement is unsupported. Overlaps are likely, since the same problem may be solved correctly and efficiently by multiple SLMs. The paper should report the union explicitly (e.g., a Venn-style breakdown). In addition, the comparison baseline is unclear: the text says 'same amount of energy or less than LLMs ... along with human-written solutions,' while the Fig. 4 caption says 'equal","section":"Section IV-C, Table III, Fig. 4"}],"minor_comments":[{"comment":"The Medium column for StarCoderBase-3B reads '4 (%)' in the text; the percentage '8%' appears to be missing.","section":"Section IV-D, Table III"},{"comment":"The text says success rates are 'out of 150 attempts,' but the figure appears to break down by difficulty, so each category has 50 attempts. Clarify whether the denominator is 50 per difficulty or 150 overall.","section":"Section IV-B, Fig. 3"},{"comment":"The prompt includes 'two test cases with their outputs,' but the paper does not report the exact test cases or how many random trials per model/prompt were run. Reporting the sampling temperature, seed, and number of generations would improve reproducibility.","section":"Section III-E"},{"comment":"The correctness evaluation is described as automated testing followed by a manual review, but the manual review procedure is not specified (e.g., who did it, how disagreements were resolved). A short protocol description would strengthen confidence in the correctness labels.","section":"Section III-G"},{"comment":"The conclusion repeats the 52.6% result without the 'at least one SLM' and 'when correct' qualifications used in Section IV-C. Please ensure the conclusion matches the qualified finding.","section":"Section VI"}],"recommendation":"major_revision","confidential_remarks":"I see the reviewer's rejection take, but I disagree with outright rejection: the central design is not circular, and the confound and statistical issues are, in principle, fixable with additional experiments and per-problem data. The main risk is that the authors may not be able to redo the experiments without ChatGPT cleaning, in which case the headline claim should be withdrawn or reframed as a benchmark of ChatGPT-assisted SLM output. I recommend major revision with a strong request for the missing audit and uncertainty quantification."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a reasonable benchmark effort with a serious attribution problem. The paper measures correctness, runtime, memory, and energy for three 3B-parameter code models (StableCode-3B, StarCoderBase-3B, Qwen2.5-Coder-3B-Instruct) and two commercial APIs (GPT-4.0, DeepSeek-Reasoner) on 150 LeetCode problems, 50 per difficulty, using CodeCarbon on an isolated VM, ten runs per sample, with cooling periods. That is real work, and the setup is described clearly enough to reproduce. The manual correctness review is a nice touch. Qwen2.5-Coder-3B-Instruct genuinely looks like the best SLM and closer to LLM correctness than the other two.\n\nThe problems are not minor. Section III-E says that when SLM outputs had minor issues or multiple solutions, the authors used the paid ChatGPT API to clean the code and keep the first valid solution. That means the code being measured for energy and correctness may have been rewritten or selected by ChatGPT. The paper never quantifies how often this happened or what changed. So the 52.6% claim (79/150 problems where at least one SLM was both correct and as energy-efficient as the LLMs) is uninterpretable: it could be ChatGPT post-processing, not SLM capability. The Limitations section mentions nondeterminism and memorization but not this confound. That is a load-bearing omission.\n\nSecond, the energy differences are immeasurably small. Table II shows all model means lie between 1.442 and 1.458 mWh, a span of 0.016 mWh, and no variances are reported. With ten runs, that range is plausibly measurement noise. The authors even say 'the differences are minor,' yet the abstract says SLMs are 'often more energy-efficient' — that overstates the evidence. The 52.6% is a union over three SLMs, not a per-model rate, and the abstract phrasing makes it sound like a blanket SLM advantage. It isn't.\n\nWhat is genuinely new here is narrow: prior energy benchmarks on LeetCode code exist ([18], [19]), but this specific model set and difficulty-stratified comparison is a legitimate extension. The paper is clearly written and the experimental protocol is transparent. I also noticed no data or scripts are made available, which is a fixable omission.\n\nWho this is for: someone looking for a template for model-selection energy benchmarks, or a reading group on measurement confounds in LLM evaluation. As a citable result, I wouldn't rely on it without a cleaned re-run that quantifies the cleaning step.\n\nRecommendation: send it to peer review, but expect a critical referee. The core experiment is salvageable: rerun without the ChatGPT cleaning (or at minimum quantify how often cleaning occurred and show the comparison holds on untouched outputs), report variance, and soften the abstract to match the union claim. As-is, the central attribution claim fails.","headline":"A carefully documented benchmark that is undercut by its own ChatGPT-cleaning step; the headline energy result is not attributable to the small models.","tokens_in":805,"tokens_out":837,"would_cite":false,"duration_ms":35714,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that on 52.6% of 150 LeetCode-style problems, at least one small open-source code model produced correct code that used as much or less execution energy than large commercial LLMs, while large models remained more accurate","keywords":["code generation","small language models","large language models","energy efficiency","sustainability","performance evaluation","LeetCode benchmark","Python"],"falsifier":"Re-run the 150-problem protocol on raw SLM outputs before the ChatGPT cleanup step, or measure correctness and energy both before and after cleanup on a sample. If raw outputs are substantially less correct or have a different energy profile, the 52.6% claim is an artifact of the cleanup stage rather than a property of the small models.","tokens_in":11127,"feed_emoji":"⚡","tokens_out":7437,"duration_ms":66733,"temperature":0.7,"pith_summary":"The paper tries to establish that small open-source code models can serve as energy-efficient alternatives to large commercial models for a meaningful share of routine programming problems. On a benchmark of 150 Python problems evenly split across easy, medium, and hard difficulty, large models GPT-4.0 and DeepSeek-Reasoner remain the most accurate, but in 79 of 150 problems (52.6%) at least one of three 3-billion-parameter models produced a correct solution whose execution energy was equal to or lower than the large models' and human-written solutions. The result matters because energy use from code assistants is a growing environmental cost, and small models are cheaper to run locally. The comparison is about the energy consumed by the generated code when executed, not the energy of training or serving the models. The paper also shows that SLMs are not interchangeable: Qwen2.5-Coder-3B-Instruct clearly outperforms the other two small models.","feed_headline":"Small models match large models' energy on 52.6% of code tasks","feed_subtitle":"Correct code from 3-billion-parameter models ran as energy-efficiently as GPT-4.0 and DeepSeek-Reasoner on a 150-problem LeetCode benchmark.","key_machinery":"The central object is a standardized execution-energy benchmark. Each model receives the same LeetCode prompt with instructions to emit only runnable Python (a class Solution plus an __main__ block); generated code is executed ten times per sample in an isolated Linux VM with a five-second cooling period; correctness is checked against LeetCode test cases plus manual review; energy is tracked with CodeCarbon, peak memory with tracemalloc, and runtime with Python's time module. The comparison is defined over the energy of the generated code when run, not over the energy of training or serving the models.","core_discovery":"Stated on the paper's own terms, the central discovery is conditional: if a small model's output is functionally correct, that output is often as energy-efficient as a large model's output. Concretely, across 150 LeetCode problems, the three SLMs together produced correct and energy-competitive solutions in 79 problems (52.6% of the total; 66% of easy, 42% of medium, 50% of hard). Large models maintain the highest correctness at every difficulty level, with DeepSeek-Reasoner reaching 88%, 86%, and 74% success by difficulty and GPT-4.0 reaching 80%, 82%, and 74%, versus the best SLM Qwen2.5-Coder-3B-Instruct at 74%, 72%, and 66%. The energy differences are small in absolute terms (averages ar","pith_inferences":["The paper leaves implicit that the absolute energy differences (roughly 0.01 to 0.02 mWh per run) are small, so the practical case for SLMs is cumulative fleet-level savings plus local-deployment benefits such as privacy, latency, and avoiding API costs rather than large per-task savings.","A testable extension is to instrument the ChatGPT cleanup step itself, measuring correctness and energy on raw versus cleaned SLM outputs; without that measurement, the SLM-specific credit for the 52.6% result is uncertain.","A natural follow-up is a correctness-screening cascade: run a small model first and escalate only failed or low-confidence cases to a large model; the paper's per-model correctness and energy data supply the inputs such a policy would need.","The protocol could be extended to end-to-end energy by including the local SLM inference on commodity hardware and the cloud API inference for LLMs, which would likely favor SLMs further but is outside the paper's execution-code-only scope."],"forward_implications":["Energy-constrained deployments can consider small models for easy and medium LeetCode-class tasks without sacrificing per-task energy, provided correctness is verified.","Model choice matters: the energy-efficiency result is driven mostly by Qwen2.5-Coder-3B-Instruct; StarCoderBase-3B contributes little in correct, energy-competitive outputs.","Because the 52.6% figure is a union over three small models, a practical system may need to generate with several SLMs and select a correct output, which changes the total energy accounting.","Improving SLM correctness would convert a conditional energy advantage into a broader one, since energy parity already holds when outputs are correct.","Human-written LeetCode solutions remain close to both model families in energy, so the sustainability gain of SLMs over LLMs is modest unless combined with correctness screening."],"supporting_citations":[{"why":"Large-scale prior benchmark of 20 LLMs on 878 LeetCode problems that established the energy-efficiency comparison this study extends.","marker":"[18]"},{"why":"Introduced RuntimeRatio and EnergyRatio metrics for comparing LLM-generated code against human-written code, the measurement approach adapted here.","marker":"[19]"},{"why":"Recent evaluation of LLM-generated code energy across Python, Java, and C++ on hard LeetCode problems, the direct predecessor this work broadens to small models.","marker":"[23]"},{"why":"Supplies the Goal-Question-Metric formulation used to structure the research questions and methodology.","marker":"[24]"},{"why":"Establishes LeetCode and community-voted solutions as a benchmark for evaluating the efficiency of LLM-generated code.","marker":"[25]"},{"why":"Shows LLMs can generate greener code when prompted and provides the green-code-generation baseline this study contrasts with SLM outputs.","marker":"[4]"},{"why":"Proposes the eff@k efficiency metric that extends pass@k, informing the correctness-plus-efficiency evaluation design.","marker":"[21]"}],"fun_headline_variants":["Small models match big-LLM energy on correct code","SLMs energy-efficient but only when output is correct","52.6% of coding tasks: Small models energy-match GPT-4","When correct, 3B-parameter models rival GPT-4's energy","Correct small-model code uses as little energy as GPT-4"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The load-bearing premise is that cleaning generated code with the ChatGPT API, described in Section III-E, does not materially change the algorithm being tested; if the cleanup rewrites the code, the correctness and energy numbers attributed to the small models are actually attributable to ChatGPT.","fun_headline_variants_meta":{"raw":{"variants":["Small models match big-LLM energy on correct code","SLMs energy-efficient but only when output is correct","52.6% of coding tasks: Small models energy-match GPT-4","When correct, 3B-parameter models rival GPT-4's energy","Correct small-model code uses as little energy as GPT-4"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000295,"raw_usage":{"total_tokens":1603,"prompt_tokens":850,"completion_tokens":753,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":594,"completion_tokens_details":{"reasoning_tokens":664}},"tokens_in":594,"tokens_out":753,"duration_ms":7245,"temperature":1.0,"reasoning_tokens":664,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T22:09:25.635449+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 150-problem protocol on raw SLM outputs before the ChatGPT cleanup step, or measure correctness and energy both before and after cleanup on a sample. If raw outputs are substantially less correct or have a different energy profile, the 52.6% claim is an artifact of the cleanup stage rather than a property of the small models.","supporting_citations":[{"cited_title":"Can llms generate green code-a comprehensive study through leetcode,","cited_arxiv_id":null,"evidence_quote":"Introduced RuntimeRatio and EnergyRatio metrics for comparing LLM-generated code against human-written code, the measurement approach adapted here."},{"cited_title":"Ai-powered, but power- hungry? energy efficiency of llm-generated code,","cited_arxiv_id":null,"evidence_quote":"Recent evaluation of LLM-generated code energy across Python, Java, and C++ on hard LeetCode problems, the direct predecessor this work broadens to small models."},{"cited_title":"The goal question metric approach,","cited_arxiv_id":null,"evidence_quote":"Supplies the Goal-Question-Metric formulation used to structure the research questions and methodology."},{"cited_title":"On evaluating the efficiency of source code generated by llms,","cited_arxiv_id":null,"evidence_quote":"Establishes LeetCode and community-voted solutions as a benchmark for evaluating the efficiency of LLM-generated code."}],"review_version":1}