{"id":"f1874a62-7a73-4717-9072-740fea9d5f07","arxiv_id":"2505.04521","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"LLM-assisted coding on 12 Codeforces tasks was estimated to emit 19.9 to 43.8 times more CO2 than manual coding, with the gap growing with task complexity.","lead":"This study estimates that using GPT-4 to write Codeforces solutions produces, on average, 32.72 times more carbon emissions than writing the same solutions by hand. The estimate rests on assumed per-query energy costs and a very low laptop power value, not on direct energy measurements.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 32.72x ratio is an arithmetic artifact of two unvalidated constants, not a measured result; without a sensitivity analysis the headline claim is unsupported.","rationale":"The reader's weakest assumption is the same as mine: the 32.72x mean ratio is almost entirely set by two constants. Section 6.1 reports that 98.68% of LLM CF is query energy, and Section 5.3 fixes each query at 0.011 kWh. Because this constant already implies an order-of-magnitude per-query energy compared with the manual per-task laptop energy, the ratio is largely a consequence of the chosen inputs. The 4.075 W laptop power is also far below typical active-coding power; a realistic 20–40 W value raises the manual denominator by roughly 5–10×, reducing the mean ratio to single digits. The paper candidly acknowledges that no power meters were used and that the GPT-4 lifetime query count is unknown, so the constants are not independently established. A simple sensitivity analysis over these two constants would settle whether the ranking and the 32.72x figure are robust; none is provided. The correlation may survive some perturbations because query counts and manual times both scale with task complexity, but the quantitative headline cannot be accepted as stated. This concern supports the reader's rejection verdict; I would keep it unchanged.","tokens_in":11456,"tokens_out":5326,"duration_ms":54083,"concrete_test":"Recompute the mean ratio in Table 3 after substituting p_laptop = 30 W and per-query energy = 0.0022 kWh (inference only) in the Section 5 formulas; if the mean ratio falls below about 5 or changes sign, the 32.72x headline is an artifact of the two assumed constants.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline ratio is effectively determined by two assumed constants. LLM CF is 98.68% query energy (Section 6.1), and each query is charged 0.011 kWh = 0.0022 kWh inference (Ludvigsen blog estimate) + 0.0088 kWh training amortization computed from 50 GWh divided by 5.68 Giga queries (Section 5.3). Manual CF is dominated by p_laptop × MTS, with p_laptop fixed at 4.075 W, the average of two laptop LCA values (Section 5.1). The paper's own limitations (Sections 5.3 and 7) state that no power meters were used and the GPT-4 lifetime query count is unknown. Because the per-query charge is heavily influenced by an unvalidated training amortization and the laptop power assumption is roughly 5–10× lower than typical active-coding power, plausible alternative values shift the mean ratio from 32.72 to single digits or below. No sensitivity analysis is provided, so the central quantitative claim is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper compares the carbon footprint of manual software development (using Codeforces contest data as a proxy for developer effort) to an LLM-assisted process in which GPT-4 generates and iteratively fixes code for the same tasks. The authors estimate the end-to-end energy of both processes and report that the LLM-assisted approach has on average 32.72 times the carbon footprint of the manual approach, and that the difference correlates with task complexity (Pearson r=0.890). The paper also proposes best practices for reducing LLM-related emissions.","tokens_in":11753,"tokens_out":8615,"duration_ms":76727,"significance":"If the result held, it would be a useful contribution to the growing body of work on the environmental cost of AI-assisted software development, and the replication package would support follow-up studies. The paper is transparent about its formulas, data tables, and assumptions, which makes the identified weaknesses tractable. However, the headline ratio is not robust to the two most influential constants, and the comparison is not outcome-matched for half the tasks, so the significance as a measured result is currently low. The study does provide a structured, reproducible methodology that could be adapted if the underlying estimates are improved.","major_comments":[{"comment":"The per-query energy constant of 0.011 kWh (0.0022 kWh inference plus 0.0088 kWh training amortization) is the dominant driver of the LLM result, since query energy constitutes 98.68% of LLM CF (§6.1). The training amortization is computed by dividing 50 GWh of training energy by 5.68 Giga queries, but Section 7 acknowledges that GPT-4's lifetime query count is unknown and that no power meters or energy profiling tools were used. A single error in this constant scales the headline ratio almost linearly, yet no sensitivity analysis is provided. Plausible alternative per-query energy values from the literature (e.g., 0.001-0.005 kWh) would reduce the ratio from 32.72 to single digits or below. The paper should present the 32.72x result as conditional on this unvalidated constant and include a sensitivity analysis across published estimates.","section":"§5.3"},{"comment":"The manual arm's power draw is fixed at p_laptop = 4.075W, the average of two laptop life-cycle assessment values (Asus 3.59W, Dell 4.62W). These LCA figures represent average power over the product lifetime, including off and idle states, and are not a valid proxy for the power drawn while a developer is actively coding. Realistic active coding power on a standard laptop is on the order of 20-40W. Replacing 4.075W with 30W alone would increase the manual CF by a factor of about 7.4, reducing the mean ratio from 32.72 to roughly 4.4 even before any change to the query energy constant. The authors need to justify p_laptop with an active-coding power source or demonstrate that the conclusion is invariant to a realistic range of values.","section":"§5.1"},{"comment":"For 6 of the 12 tasks, TPAH (percentage of test cases passed after human insight) is below 100%, and for two tasks it is 0%. This means the LLM-assisted process did not produce a fully working solution for half the tasks, yet these tasks are included in the aggregate ratio of 32.72. The comparison is therefore not between two processes that achieve the same outcome. The research question RQ1 requires both approaches to be evaluated on equivalent outcomes; otherwise one is comparing successful manual development to a partially failing LLM-assisted process. The authors should either restrict the ratio computation to tasks where both arms reach 100% test pass, or report a separate analysis for task success rates and explicitly discuss the implications for the comparison.","section":"§6.1 / Table 2"},{"comment":"The reported correlation of 0.890 between task complexity and the CF difference is structurally dominated by the LLM term. Since query energy is 98.68% of LLM CF (§6.1) and the manual term is negligible in comparison, the 'difference' is approximately equal to the LLM CF. Thus the Pearson and Spearman tests largely measure whether more complex tasks lead to more LLM queries, which is also constrained by the experimental cap of 1-8 queries (§4). The paper should clarify that the correlation does not provide independent evidence about the 'gap' between approaches; it is essentially a property of the query budget.","section":"§6.2"}],"minor_comments":[{"comment":"\"32.72 higher carbon footprint\" should read \"32.72 times higher carbon footprint\" or \"a factor of 32.72.\"","section":"Abstract"},{"comment":"The unit 'kwh' appears several times (e.g., '0.011 kwh'); use the standard 'kWh' for consistency.","section":"§5.3"},{"comment":"The sentence \"the LLM-based approach has at least 19.92 times and at most 43.81 times more carbon footprint (resp. energy consumption)\" is awkward; consider phrasing as \"by factors of 19.92 to 43.81.\"","section":"§6.1"},{"comment":"The claim of \"up to 30% based on our observed data\" is not substantiated; specify which metric is reduced by 30% and where in the data this appears.","section":"§6.3"},{"comment":"The use of Codeforces relative submission time as a proxy for time spent on a task assumes that participants work continuously from contest start; this is not discussed in the threats to validity.","section":"§5.2"},{"comment":"The inference energy value is sourced from a blog post [19]; the authors should seek peer-reviewed sources for per-query inference energy, especially since this constant is so influential.","section":"References"}],"recommendation":"reject","confidential_remarks":"The manuscript addresses a timely question and provides a replicable calculation pipeline, but the abstract's numeric claim is likely to be quoted and used in policy discussions. In its current form, the headline ratio is not supported by the evidence due to unvalidated constants and non-equivalent outcome arms. Even with a sensitivity analysis added, the paper would need to substantially retract or qualify the 32.72x figure. I recommend reject; if the editors are inclined to offer a revision, the authors should be asked to provide: (1) a sensitivity analysis over per-query energy and laptop power, (2) a re-analysis restricted to tasks where both arms fully pass tests, and (3) a revised abstract that removes the unqualified ratio."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: this paper deserves a serious referee because it is the first in its cited literature to directly compare the carbon footprint of manual versus LLM-assisted development effort, and it ships a complete replication package with code and full data. The equations are internally consistent, the tables are complete, and the authors are explicit about what they assumed. That is real work and the right kind of scaffolding for a hard measurement problem.\n\nThat said, the reader's rejection is fair. The headline ratio is not measured; it is computed from two assumed constants. The LLM arm's footprint is 98.68% query energy, and each query is charged a fixed 0.011 kWh (0.0022 inference from a blog post plus 0.0088 training amortization from 50 GWh divided by an estimated 5.68 Giga lifetime queries for GPT-4, based on GPT-3's lifetime). The manual arm is dominated by a laptop power of 4.075 W, an average of two lifecycle-assessment values. As the stress-test note correctly says, plausible variations in either constant drop the 32.72x to single digits or reverse it. No sensitivity analysis is provided. The paper itself states no power meters were used and the GPT-4 lifetime query count is unknown, so the authors know the weak points; they just didn't quantify them.\n\nThe correlation result (Pearson 0.890) is also weaker than it appears. Since query energy linearly scales with query count and the manual term contributes only a few percent, the correlation mostly restates that harder tasks trigger more queries. That is a trivial insight, not an independent empirical discovery.\n\nThere is also a modeling wrinkle that the reader's report under-emphasizes: for the LLM arm, the estimated human insight and add-functionalities times are derived from the same average manual task time t, so the two arms are not independent. For half the tasks the LLM-assisted process never reached 100% test pass, so the comparison is not outcome-matched. The authors acknowledge this only indirectly.\n\nThe paper's own limitations section is candid, which is a credit, but candor about a load-bearing assumption does not make the assumption robust. This is a pilot study that illustrates a methodology; the abstract's quantitative claim overstates it.\n\nWho should read it: researchers working on sustainable AI-assisted software engineering, especially those designing benchmarks for query budgeting. The paper is a valuable template for what a transparent energy-accounting study should look like. I would send it to peer review, but the revision has to add sensitivity analysis, replace the blog-based per-query energy with something more credible, use realistic laptop power ranges, and clearly reframe the 32.72x as an example calculation under stated assumptions rather than a finding. The methodology is worth developing; the current headline is not defensible.","headline":"An honest, fully transparent first attempt at comparing manual vs. LLM-assisted coding carbon footprints, but the 32.72x headline is an arithmetic product of two unvalidated constants, not a measured result.","tokens_in":12215,"tokens_out":1553,"would_cite":false,"duration_ms":17068,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM-assisted coding is estimated to emit about 33 times the carbon of manual coding per task.","keywords":["carbon footprint","LLM-assisted software development","code generation","energy consumption","GPT-4","sustainability","competitive programming","empirical software engineering"],"falsifier":"Measure actual wall-plug energy for both the laptop and the LLM server while solving the same twelve tasks, and recompute the ratio with the measured per-query energy and a realistic laptop power draw; if the measured query energy is well below 0.011 kWh or laptop power is above a few watts, the reported ratio will not reproduce.","tokens_in":11296,"feed_emoji":"🌍","tokens_out":7750,"duration_ms":70390,"temperature":0.7,"pith_summary":"This paper tries to establish that using a large language model as a coding assistant is not carbon-cheaper than writing code by hand. On twelve competitive-programming tasks from the Codeforces platform, the authors compare the estimated end-to-end energy of a human writing Python with that of a GPT-4-based loop of generate, test, and repair. They report that the LLM-assisted approach uses on average 32.72 times more energy and carbon per task, with the ratio ranging from about 20 to 44. They also find a strong correlation between task difficulty and the size of the gap. If true, the result matters because AI-assisted coding is often promoted as a productivity gain, and the environmental cost of that gain would need to be priced in.","feed_headline":"LLM-assisted coding emits 33x the carbon of manual coding","feed_subtitle":"The gap widens with task difficulty, and query energy drives 98.7% of the AI footprint.","key_machinery":"The argument is carried by the ratio of two energy estimates: $CF_{\\text{LLM}} / CF_{\\text{manual}}$. The manual side multiplies laptop power ($p_{\\text{laptop}}=4.075$ W in the base case) by Codeforces submission times, with debugging and testing added through fixed fractions from prior studies. The LLM side is dominated by a fixed per-query energy of $0.011$ kWh, composed of $0.0022$ kWh for inference and $0.0088$ kWh for amortized training, multiplied by the number of queries the repair loop uses. Because query energy is nearly constant per query while manual energy scales with time, the reported ratio is set largely by these two constants rather than by measured execution.","core_discovery":"The central claim is that the end-to-end carbon footprint of LLM-assisted code development exceeds manual development for every task in the sample, by a mean factor of 32.72 with standard deviation 8.41, and that the gap grows with task complexity. The authors model the manual footprint as coding plus debugging plus testing energy, using Codeforces submission times as a proxy for effort. They model the LLM footprint as query energy plus the energy of human insight and repair, with query energy contributing 98.68% of the total. They report a Pearson correlation of 0.890 with a p-value of 0.00011 between task difficulty and the carbon-footprint difference, and interpret this as evidence that LLM assistance is especially costly on harder tasks.","pith_inferences":["The headline ratio is extremely sensitive to the 0.011 kWh per-query constant and the 4.075 W laptop power; using a realistic laptop draw of 20 to 40 W, or dropping the amortized training component from the per-query cost, would shrink the ratio to single digits or reverse it, and the paper itself notes that no power meters were used.","The task sample is competitive-programming problems solved in Python, not typical business software, so the ratio for longer, interactive, or more ambiguous real-world tasks could differ because human effort and query counts scale differently.","A direct extension would be to run the same twelve tasks with wall-plug energy meters and several different LLMs, varying the per-query energy cost, to map where the crossover between manual and assisted coding lies."],"forward_implications":["If the result holds, AI-assisted coding on this kind of task is roughly 20 to 44 times more carbon-intensive per completed task than manual coding, before any productivity savings are counted.","Because query energy makes up 98.68% of the LLM footprint, reducing the number of queries per task, or using a cheaper model for easy tasks, would cut the footprint almost linearly.","The correlation with task complexity implies that as tasks get harder, the environmental penalty of LLM assistance grows; the authors recommend decomposing complex tasks into smaller subtasks, estimating that this could reduce the gap by up to 30%.","For organizations tracking software carbon budgets, the result implies that LLM-based coding should be used selectively rather than as a default for all tasks."],"supporting_citations":[{"why":"Supplies the per-query inference energy constant of 0.0022 kWh used to compute query energy consumption.","marker":"[19]"},{"why":"Provides the energy-estimation tool used to measure CPU and RAM runtime power for the testing and debugging components.","marker":"[23]"},{"why":"Supplies the 42% share of development time spent on debugging, used in both the manual and LLM-side estimates.","marker":"[24]"},{"why":"Supplies the 38% task-understanding share used to estimate the human-insight time in the LLM-assisted approach.","marker":"[22]"},{"why":"Provides the reading and editing fractions of debugging time used to estimate the time to add missing functionalities.","marker":"[17]"},{"why":"Supplies the 217 gCO2/kWh carbon intensity used to convert every energy estimate into a carbon footprint.","marker":"[2]"},{"why":"Provides the platform data: tasks, submission times, runtimes, and memory usage that ground the manual-side estimates.","marker":"[1]"}],"fun_headline_variants":["LLM-assisted coding: 33x more carbon than manual","AI coding's carbon cost: 33x manual, worse on hard tasks","Coding with LLMs emits 32.7x more CO2 than hands-on","LLM code help: carbon footprint 33x higher, task-complexity linked","Manual coding beats AI on emissions by 33x, study finds"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire ratio rests on the fixed assumptions that one LLM query costs 0.011 kWh and that coding on a laptop draws only 4.075 W; if either constant is off, especially if real laptop power is 20 to 40 W, the 32.72 times result shrinks or flips.","fun_headline_variants_meta":{"raw":{"variants":["LLM-assisted coding: 33x more carbon than manual","AI coding's carbon cost: 33x manual, worse on hard tasks","Coding with LLMs emits 32.7x more CO2 than hands-on","LLM code help: carbon footprint 33x higher, task-complexity linked","Manual coding beats AI on emissions by 33x, study finds"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000122,"raw_usage":{"total_tokens":1039,"prompt_tokens":830,"completion_tokens":209,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":446,"completion_tokens_details":{"reasoning_tokens":109}},"tokens_in":446,"tokens_out":209,"duration_ms":2867,"temperature":1.0,"reasoning_tokens":109,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:26:31.298461+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure actual wall-plug energy for both the laptop and the LLM server while solving the same twelve tasks, and recompute the ratio with the measured per-query energy and a realistic laptop power draw; if the measured query energy is well below 0.011 kWh or laptop power is above a few watts, the reported ratio will not reproduce.","supporting_citations":[{"cited_title":"ChatGPT’s Energy Use per Query","cited_arxiv_id":null,"evidence_quote":"Supplies the per-query inference energy constant of 0.0022 kWh used to compute query energy consumption."},{"cited_title":"The developer coefficient","cited_arxiv_id":null,"evidence_quote":"Supplies the 42% share of development time spent on debugging, used in both the manual and LLM-side estimates."},{"cited_title":"I know what you did last summer - an investigation of how developers spend their time","cited_arxiv_id":null,"evidence_quote":"Supplies the 38% task-understanding share used to estimate the human-insight time in the LLM-assisted approach."},{"cited_title":"Ko, Brad A","cited_arxiv_id":null,"evidence_quote":"Provides the reading and editing fractions of debugging time used to estimate the time to add missing functionalities."},{"cited_title":"https://www.nowtricity.com/country/ united-kingdom/","cited_arxiv_id":null,"evidence_quote":"Supplies the 217 gCO2/kWh carbon intensity used to convert every energy estimate into a carbon footprint."},{"cited_title":"https://codeforces.com/","cited_arxiv_id":null,"evidence_quote":"Provides the platform data: tasks, submission times, runtimes, and memory usage that ground the manual-side estimates."}],"review_version":1}