{"id":"7df2be91-fe8f-47da-9a3b-b74555d0c272","arxiv_id":"2505.20324","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLM-generated Python solutions typically consume more energy than canonical human-written solutions, with DeepSeek-v3 and GPT-4o the most efficient LLMs and worst-case gaps near 450 times on certain problems.","lead":"This paper measured how much energy code written by 20 large language models consumes when run, and compared it with efficient human-written solutions to 878 programming problems. It found LLM code usually uses more energy, sometimes far more, and identified which models write the most energy-efficient code.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Paper's own case study contradicts the central claim: the canonical human solution for LeetCode 740 consumes 5x more energy than the best LLM solution, so the baseline is not a reliable 'human-efficiency' reference.","rationale":"The reader's weakest_assumption was that EffiBench canonical solutions are genuinely the most efficient human solutions; I agree this is the load-bearing point, but the paper itself provides a concrete counterexample in Appendix F.2. This is more serious than a generic external-data concern because it is internally inconsistent: the authors chose Problem 740 as a 'comparable difference' case and then report the canonical solution consuming 5x more energy than the best LLM solution, without noting that this undercuts their abstract and conclusion. The central claim is an average over problems, so one counterexample does not by itself falsify the average; however, it does show the baseline is not a validated efficient-human reference. The right response is not to reject the paper—the tables and code are valuable—but to require the authors to report the per-problem distribution and the fraction of LLM-beats-canonical cases, and to soften the claim from 'LLMs are less efficient than human-written solutions' to 'LLMs are less efficient than the particular canonical solutions in EffiBench on average.' Since the reader's verdict is already CONDITIONAL and my concern reinforces that conditionality without changing its direction, I recommend UNCHANGED. This is a technical, checkable inconsistency, not a question of author intent or consensus.","tokens_in":51,"tokens_out":4959,"duration_ms":66999,"concrete_test":"For the 298- and 576-problem sets, recompute the relative-efficiency metric of Eq. 13 using, as baseline, the per-problem minimum total energy across all 20/11 LLMs and the canonical solution; then report the fraction of problems where the canonical solution is not the minimum and recompute the average ratios. Also report how the abstract's 1.17x/1.21x figures change when Problem 740 (and any other problems where canonical is not the minimum) are excluded or when the best-available-solution baseline is used. If the canonical solution is beaten by any LLM on a substantial fraction of problems, or if the average ratios shrink materially, the central human-efficiency claim is an artifact of the chosen baseline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that human-written canonical solutions are more energy-efficient than LLM-generated code depends entirely on EffiBench's canonical solutions being a fair, near-optimal human baseline. The paper's own analysis contradicts this. In Appendix F.2 (Problem 740, 'Delete and Earn'), the authors report that the canonical solution uses an O(max(nums)) list-based algorithm and consumes 5 times more energy than the best LLM-generated solutions (GPT-4o, Grok2, LLaMA 3.1 70B, Gemini 2.0 Flash-Lite), which use O(len(nums)) iterative DP. This is a direct counterexample to the claim that canonical solutions are the 'most efficient' human-written solutions, and it shows the baseline is not a consistent lower-energy reference. The paper never reconciles this case with the abstract's assertion that canonical solutions are ~1.17–2x more efficient on average. If the canonical solution is merely one arbitrary human solution per problem rather than an efficient one, the measured ratios in Tables 8–9 may reflect the particular LeetCode forum solutions chosen, not a general human-vs-LLM gap. The paper should at minimum report the per-problem distribution of relative efficiency and the fraction of problems where an LLM beats the canonical; without that, the headline conclusion is unsubstantiated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper benchmarks the energy efficiency of code generated by 20 LLMs on 878 LeetCode problems drawn from EffiBench, comparing the LLM outputs against EffiBench's canonical human-written solutions. The authors measure package and RAM energy with perf, runtime, and memory, and also report pass rates, token counts, and API costs. Their headline claim is that human canonical solutions are on average about 1.17x more energy-efficient than DeepSeek-v3, 1.21x more than GPT-4o, and over 2x more than Grok-2 and Gemini-1.5-Pro, with much larger gaps for dynamic programming, backtracking, and bit-manipulation problems. The paper also identifies DeepSeek-v3 and GPT-4o as the most energy-efficient LLMs and provides two case studies explaining why some LLM solutions consume far more energy than the canonical baseline.","tokens_in":34411,"tokens_out":4659,"duration_ms":41304,"significance":"If the central result holds, this is a useful and timely empirical contribution to green software engineering and LLM code generation. The study covers a broad set of models (20) and problems (878), uses a public benchmark, and provides a reproducible workflow (the anonymous repository is a concrete asset). The energy measurement methodology, including idle-power baseline subtraction, cooldown periods, and five repeated runs, is more careful than many prior token-based efficiency studies. The breakdown by difficulty and algorithmic category is valuable for practitioners. However, the significance is currently limited by an unreliable baseline claim: the paper's own case study shows that a canonical 'most efficient human' solution can be 5x less efficient than the best LLM solution. The headline ratios are also drawn from a 576-problem/11-model subset rather than the full 878-problem/20-model set, and several aggregate statements are contradicted by the paper's own per-category tables. These issues need to be fixed before the quantitative conclusions can be taken as a general human-versus-LLM efficiency gap.","major_comments":[{"comment":"The paper's own case study contradicts the assumption that EffiBench canonical solutions are the 'most efficient' human-written solutions. For LeetCode Problem 740 (Delete and Earn), the canonical solution uses an O(max(nums)) list-based algorithm and consumes 5x more energy than the best LLM solutions, which use O(len(nums)) iterative DP. Since §3 states that EffiBench pairs problems with 'the most efficient solutions from the LeetCode discussion forum,' this counterexample shows the baseline is not a reliable lower-energy reference. The authors should report the per-problem distribution of the relative energy ratio (Eq. 13) and the fraction of problems where each LLM beats the canonical solution, rather than only aggregate ratios in Tables 8 and 9. Without this, the headline conclusion that human-written solutions are more energy-efficient than LLM-generated code may be an artifact of the particular canonical solutions chosen.","section":"Appendix F.2, §3"},{"comment":"The abstract and introduction present the 1.17x, 1.21x, and over-2x ratios as if they apply to the full 878-problem, 20-model study, but these numbers come from Benchmark Set-II, which contains only 576 problems successfully solved by 11 LLMs. The all-20-model comparison in Benchmark Set-I is restricted to 298 common problems, with only 30 hard problems (Table 4). The authors should state these subset sizes explicitly in the abstract and introduction, and qualify the generalization of the headline ratios accordingly.","section":"Abstract, §4.2.2, Tables 7 and 9"},{"comment":"The text overclaims that canonical solutions 'consistently outperform all evaluated LLMs' and that LLM energy efficiency is 'always poorer,' but the paper's own tables contain numerous counterexamples. For example, in Table 13 (Benchmark Set-II, easy problems), Claude 3.5 Haiku has average total energy 5.1176 J versus canonical 5.1220 J; in Table 16 (Greedy, Benchmark Set-I), at least nine LLM solutions have lower average total energy than canonical; in Table 25 (Bit Manipulation, Benchmark Set-I), GPT-4o, DeepSeek-v3, and several others are below canonical; and in Table 27 (Greedy, Benchmark Set-II), Grok 2 and DeepSeek-v3 are below canonical. The central claim should be reframed as 'on average across the benchmark' and 'for certain categories,' not as a universal property. The authors should also provide confidence intervals or significance tests for the aggregate differences, since some reported gaps (e.g., 5.77 J vs 5.91 J) may be within measurement noise.","section":"§4.2.1, §5, Tables 13, 16, 25, 27"},{"comment":"The statement that LLM-generated code can consume 'up to 450 times more energy' for dynamic programming, backtracking, and bit manipulation is based on a single problem (LeetCode 2305, Fair Distribution of Cookies) discussed in Appendix F.1. The abstract and introduction present this as a general pattern for entire algorithmic groups. The authors should either provide the per-problem maximum across all problems in each category or explicitly label the 450x figure as a single-case example. A distribution of per-problem relative energies is needed to support any claim about the maximum or about category-level behavior.","section":"Abstract, Introduction, Appendix F.1"}],"minor_comments":[{"comment":"The text says GPT-4o has a Pass@10 score of '89.1%', but Table 3 reports 89.7%; the values should be reconciled.","section":"§4.1, Table 3"},{"comment":"The paper refers to '11 LLMs' in Benchmark Set-II but never lists which 11 models are included; the reader must infer the set from Table 7. Please state the model list explicitly.","section":"§4.2.2"},{"comment":"Bullet 4 of the Problem 740 case study says the worst solution is produced by 'Claude-Sonnet and LLaMA 3.1 70B,' but bullet 1 lists LLaMA 3.1 70B among the best solutions. This appears to be a typo (likely LLaMA 3.3 70B) and should be corrected.","section":"Appendix F.2"},{"comment":"The table caption says 'TOTAL' but one problem can belong to multiple algorithmic categories, so the TOTAL column is not the sum of the category columns; this should be clarified in the caption or text.","section":"§3.1, Table 1"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is a serious empirical benchmark of 20 LLMs' code energy efficiency. It's worth reading for the rankings alone. But the headline that human canonical solutions are consistently more efficient than LLM-generated code is overstated: the numbers come from subsets (576 or 298 problems, not the full 878), and the paper's own case study in Appendix F.2 shows a canonical solution that uses 5x more energy than the best LLM solution. That undercuts the 'human most efficient' framing.\n\nWhat's new and good: this is the broadest comparison I know of across 20 models, with energy, memory, runtime, and token cost measured on the same problems. The methodology is careful — perf with idle-power subtraction, five runs averaged, a common prompt, separate memory profiling. The relative rankings in Tables 8 and 9 are internally consistent and useful: DeepSeek-v3, GPT-4o, and Claude 3.5 Sonnet come out best among LLMs; Grok-2 and Gemini-1.5-Pro worst. If you are choosing an LLM for code generation and care about efficiency, this is a real data point.\n\nWhere it's soft — and these are addressable, not fatal:\n\n1. The baseline. EffiBench's 'canonical' solutions are one human solution from LeetCode forums, not demonstrated optimal. The paper itself, in Appendix F.2 (LeetCode 740), reports the canonical solution is 5x worse than the best LLM solution because it uses an O(max(nums)) list where the LLM uses O(n) DP. On average the canonical might still do better, but the paper needs to show the per-problem distribution and report how often an LLM beats the canonical. Without that, the central claim is not substantiated.\n\n2. The abstract says '878 problems' but the energy comparisons use 576 (11 models) or 298 (all 20 models), with only 30 hard problems in the all-20 set. The 1.17x and 1.21x figures in the abstract come from the 576-problem subset. That should be disclosed up front.\n\n3. No error bars or significance tests. Energy is noisy; five runs averaged helps but doesn't show variance. The differences between close models (e.g., DeepSeek at 1.02x vs Gemini 2.0 Flash at 1.06x) could easily be noise.\n\n4. Algorithm-level breakdowns are based on small n in some categories (e.g., 4 Divide & Conquer problems in Set I), so those conclusions are tentative. The anonymous repo could not be inspected.\n\nWho it's for: practitioners choosing between LLMs, and researchers in green software. It deserves a serious referee, but I'd send it back with the request to fix the baseline reporting and either add variance or soften the aggregate claims. The underlying data is useful enough that I'd engage with it.","headline":"Solid benchmark data on LLM code energy efficiency, but the headline claim overstates a baseline that even the paper's own case study contradicts.","tokens_in":35043,"tokens_out":5607,"would_cite":true,"duration_ms":44741,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that LLM-generated code, even when functionally correct, is less energy-efficient than canonical human-written solutions across 878 coding problems: the best models are 17–21% behind on average, the worst more than twice…","keywords":["energy efficiency","LLM code generation","green software","sustainable AI","code optimization","energy measurement","software benchmarking","runtime analysis"],"falsifier":"Gather alternative human-written solutions for a meaningful sample of the 878 problems and measure whether a substantial fraction beat the canonical solutions on energy; if they do, the reported human-vs-LLM ratios are an artifact of baseline choice rather than a general human advantage.","tokens_in":33980,"feed_emoji":"⚡","tokens_out":5896,"duration_ms":48812,"temperature":0.7,"pith_summary":"This paper tries to establish that code produced by large language models, although often functionally correct, is on average less energy-efficient than canonical human-written solutions for the same programming problems. The authors measured processor-package and RAM energy for solutions generated by 20 LLMs across 878 interview-style coding problems, using human-written canonical solutions as the baseline. They report that human solutions are about 1.17 times more energy-efficient than DeepSeek-v3, 1.21 times more efficient than GPT-4o, and over 2 times more efficient than Grok-2 and Gemini-1.5-Pro. If this is right, current correctness-focused evaluations of AI code generators miss a real environmental and economic cost, and model choice plus algorithmic optimization become first-order concerns for sustainable software development.","feed_headline":"LLM code trails human code on energy in 878 tests","feed_subtitle":"Even the best model lags reference human solutions by 17%; worst cases consume 450x.","key_machinery":"The central mechanism is a relative-efficiency comparison: each LLM solution's total energy (processor package plus RAM, with measured idle power subtracted) and runtime are divided by the same metrics for the problem's canonical human solution, producing a ratio in which values above 1 mean the LLM used more energy. This ratio, averaged over two common-problem sets derived after filtering out broken problems, is what carries the conclusion that human solutions are more energy-efficient.","core_discovery":"The paper's central claim is that LLM-generated code is measurably less energy-efficient than canonical human-written solutions once correctness is held fixed. On the 298 problems that all 20 LLMs solved correctly, canonical solutions averaged 5.77 J of total energy while the most efficient LLM, DeepSeek-v3, averaged 5.91 J, and the least efficient, GPT-4 Turbo, averaged 12.00 J. On the broader set of 576 problems solved by 11 LLMs, the gap widens: canonical solutions averaged 5.46 J whereas DeepSeek-v3 averaged 6.37 J. The paper also identifies algorithmic categories where the gap is extreme: for dynamic programming, backtracking, and bit manipulation, some LLM solutions consumed up to 450 times more energy than the canonical solution, often because they omit pruning, duplicate logic, or choose worse algorithms.","pith_inferences":["If the dominant cause of the gap is missing algorithmic optimizations like pruning, then energy-aware repair loops that feed measured energy back into the model could close much of the gap; the paper does not test this.","The measured energy covers only execution of the generated code, not the energy spent by the LLM during generation; including generation-side energy would make the total environmental cost of LLM-generated code larger, not smaller.","Because the results are for Python only, the ranking may differ in compiled languages where constant-factor overheads that inflate LLM energy use could shrink.","Since the benchmark problems are publicly available and may have appeared in model training data, the gap could shift on problems created after the models' training cutoffs."],"forward_implications":["Energy efficiency should become a reported metric in LLM code-generation benchmarks, alongside functional correctness.","Model selection has a direct energy consequence: switching from Gemini-1.5-Pro or Grok-2 to DeepSeek-v3 roughly halves the execution energy of generated code.","Optimization pressure on dynamic programming, backtracking, and bit manipulation problems would yield the largest sustainability gains.","The models that are cheapest to query are not the ones whose output is cheapest to run, so cost analyses that ignore execution energy can mislead.","Even after allowing up to 25 regeneration attempts to obtain correct code, no LLM matched the average energy of human canonical solutions."],"supporting_citations":[{"why":"Supplies the benchmark dataset of coding problems, canonical human-written solutions, and the 100 test cases per problem that the paper filters down to 878.","marker":"[8]"},{"why":"Source of the interview-style problems whose discussion-forum solutions serve as the canonical human baselines.","marker":"[29]"},{"why":"Documents the power/energy-pkg and power/energy-ram counters used to measure package and RAM energy for each executed solution.","marker":"[30]"},{"why":"Provides the standardized cloud testbed hardware on which all energy, runtime, and memory measurements were collected.","marker":"[31]"}],"fun_headline_variants":["LLM code trails human energy efficiency by up to 450x","Human-written code uses less energy than LLM output in tests","Grok-2 and Gemini-1.5-Pro most energy-hungry LLM coders","DeepSeek-v3, GPT-4o top LLMs for code energy efficiency","LLMs' code consumes over 2x energy on average for some models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the canonical human solutions used as baselines are genuinely near-optimal in energy for each problem, and that each problem's 100 test cases capture correctness fully; the paper itself removed 122 problems whose canonical solutions or tests were broken, so the remaining baselines carry the entire comparison.","fun_headline_variants_meta":{"raw":{"variants":["LLM code trails human energy efficiency by up to 450x","Human-written code uses less energy than LLM output in tests","Grok-2 and Gemini-1.5-Pro most energy-hungry LLM coders","DeepSeek-v3, GPT-4o top LLMs for code energy efficiency","LLMs' code consumes over 2x energy on average for some models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000494,"raw_usage":{"total_tokens":2436,"prompt_tokens":964,"completion_tokens":1472,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":1369}},"tokens_in":580,"tokens_out":1472,"duration_ms":13594,"temperature":1.0,"reasoning_tokens":1369,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:33:50.496386+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Gather alternative human-written solutions for a meaningful sample of the 878 problems and measure whether a substantial fraction beat the canonical solutions on energy; if they do, the reported human-vs-LLM ratios are an artifact of baseline choice rather than a general human advantage.","supporting_citations":[{"cited_title":"Effibench: Benchmarking the efficiency of automatically generated code","cited_arxiv_id":null,"evidence_quote":"Supplies the benchmark dataset of coding problems, canonical human-written solutions, and the 100 test cases per problem that the paper filters down to 878."},{"cited_title":"Problems, 2025","cited_arxiv_id":null,"evidence_quote":"Source of the interview-style problems whose discussion-forum solutions serve as the canonical human baselines."},{"cited_title":"perf-stat— Linux manual page , 2023","cited_arxiv_id":null,"evidence_quote":"Documents the power/energy-pkg and power/energy-ram counters used to measure package and RAM energy for each executed solution."},{"cited_title":"Gunawi, Cody Hammock, Joe Mambretti, Alexander Barnes, François Halbach, Alex Rocha, and Joe Stubbs","cited_arxiv_id":null,"evidence_quote":"Provides the standardized cloud testbed hardware on which all energy, runtime, and memory measurements were collected."}],"review_version":1}