REVIEW 3 major objections 5 minor 12 references
The Generative Energy Arena (GEA): Incorporating Energy Awareness in Large Language Model (LLM) Human Evaluations
T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Displaying relative energy consumption to LLM evaluators shifts their votes substantially, giving smaller models energy-aware win rates above 75 percent.
desk verdict Useful raw data on energy-aware switching, but the 75% headline rests on an unsupported tie-transfer assumption. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is the two-stage GEA protocol plus the reweighting identity. Stage one is a blind quality vote; stage two asks only users who preferred the more energy-hungry model whether they would back down on seeing that the other answer consumed less energy. The fraction who switch, called $E_c$, is inserted into $W_S(E) = W_S + T + W_L E_c$ and $W_L(E) = W_L(1 - E_c)$, which moves all ties and every back-down from the large model to the small one. This makes relative energy information the tested intervention and gives the paper a single number to measure its effect on rankings.
What would settle it
Run a randomized arena experiment in which the energy prompt is neutral rather than leading, such as 'This answer used less energy; you may change your vote,' and keep ties as ties; if the switch rate drops to near zero or the small-model win rate falls below 50 percent, the reported 75 percent result is an artifact.
Extended reading notes
Core claim
GEA compares pairs of models from the same family that differ mainly in scale, so relative energy use is simple and trustworthy. Users vote on which answer is better before any energy information appears; only when they prefer the larger model are they asked whether they would change their choice knowing the other response consumes less energy, assuming a loss in quality. With a switch fraction of roughly 46 percent, the energy-aware win rates computed from $W_S(E) = W_S + T + W_L E_c$ and $W_L(E) = W_L(1 - E_c)$ reverse the initial ranking: smaller models win more than 75 percent of the time, compared with a difference of under 2 percent before energy was shown. The paper concludes that energy awareness is a critical factor in human evaluation and that larger models are only worth their extra cost for specific questions.
Load-bearing premise
The load-bearing premise is that a user's 'yes' to the energy question reflects a genuine preference change, and that all ties would shift to the smaller model in the energy-aware tally.
Editorial extensions
If this is right
- Arena-style human evaluations that omit energy information can systematically overstate the effective quality of large models relative to user preferences.
- For common conversational and generative tasks, same-family smaller models are sufficient, so providers can serve many queries with lower energy and cost without losing perceived quality.
- Energy-aware win rates, not raw quality votes, should be the reported metric when the goal is to reflect what users would actually choose.
- LLM evaluation and development should treat energy as a design dimension: improving efficiency is a way to improve the user-facing product, not just an environmental side goal.
Reading between the lines
- The switch question's phrasing, 'assuming a loss in quality,' may overstate real willingness to change because it invites agreement; actual adoption of energy-aware routing could be lower than 75 percent.
- The MOOC student sample and Spanish-language questions mean the effect may not transfer to expert users, high-stakes tasks, or other languages; a replication with diverse users would test that.
- If the result holds at scale, it points to a practical routing policy: automatically send routine queries to small models and reserve large models for requests where users demonstrably value the extra quality.
- Counting every tie as a small-model win in Eq. (1) is a modeling choice; a conservative analysis that keeps ties unresolved would show a smaller but still non-negligible shift.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces GEA, a two-step LLM comparison arena in which a user first selects the better response between two models of the same family and, if the user chose the larger model, is then asked whether they would switch to the smaller model knowing that it consumes less energy, assuming a loss in quality. The paper defines a switch fraction Ec and derives post-energy win rates via Eqs. (1)–(2), reporting that about 41–52% of responses changed and that smaller models are chosen more than 75% of the time once energy information is accounted for. It concludes that energy awareness should be included in human LLM evaluations and that larger models are not worth their extra cost for most user queries.
Significance. If the headline result were robust, the paper would make a valuable contribution to LLM evaluation by showing that energy information materially changes human preferences and by providing a public arena artifact (code and deployment on Hugging Face) for further study. The underlying question—whether energy-aware users still prefer larger models—is timely and practically important, and the raw switch fractions (41–52%) do indicate that a sizable share of users state willingness to switch. However, the quantitative headline (>75% smaller-model win rate) is not directly measured; it is produced by a deterministic formula that transfers all tie votes to the smaller model even though ties were never queried about energy, and it relies on a one-sided, hypothetical switch question asked only to large-model voters. These issues currently undermine the central claim as stated, though the data gathering approach and raw measurements are a reasonable starting point that could support a more carefully qualified conclusion.
major comments (3)
- [Section 2.3, Eqs. (1)–(2)] The definition of Ec is ambiguous and the equations are only consistent under one reading. The text says Ec is 'the fraction of the responses for which the users changed their vote', but Eq. (2), WL(E) = WL · (1 − Ec), is correct only if Ec is the conditional fraction of initial large-model voters who switched, because the number of switches is WL · N · Ec. If Ec were instead the unconditional fraction of all responses that changed, the correct transformation would be WL(E) = WL − Ec, not WL · (1 − Ec). Please clarify which quantity Ec denotes and correct the text or the equations accordingly. Additionally, regardless of the intended reading, Eq. (1) transfers the entire tie rate T to the smaller model even though Section 2.2 shows that the energy question is posed only to users who selected the larger model; tied users are never asked how energy information would affect their choice, so T is not measured. This unverified assumption is precisely what pushes the pooled WS(E) above the 75% threshold reported in Section 4, so the paper must either measure tie responses, remove the T transfer, or explicitly flag it as a modeling assumption with sensitivity analysis.
- [Section 4, Figures 3–4 and the >75% claim] The protocol is one-sided: only users who initially chose the larger model are offered the switch to the smaller model, while users who chose the smaller model are never asked whether energy information could make them switch to the larger one. Combined with the leading wording of the switch question ('would you change your choice assuming a loss in quality?'), the measured responses cannot support the summary statement that 'for most user interactions, the extra cost and energy incurred by the more complex and top-performing models do not provide an increase in the perceived quality of the responses that justifies their use.' The >75% figure is not an observed preference share; it is a deterministic transformation of a one-sided hypothetical response through Eq. (1). Please report the directly measured conditional switch rates and the win rates computed without the unmeasured tie transfer, and rephrase the conclusions to match what the data actually show.
- [Section 4, statistics and sampling] No confidence intervals, significance tests, or user-level clustering are reported. With 694 questions, with each MOOC student answering ten questions, and with initial family-level preferences that differ by up to roughly 20 percentage points (e.g., GPT families initially favoring the small model), the claims that initial preferences are 'comparable' (within 2%) and that smaller models win 'more than 75% of the time' need bootstrap or hierarchical estimates to establish that these differences are not sampling noise. The paper should also report the number of unique users, not just the number of questions, since votes from the same user are not independent.
minor comments (5)
- [Section 1] There is a typo in the first paragraph: 'becnhmarks' should be 'benchmarks'.
- [Section 4] The estimate that 'at least 83% of the questions were made by the MOOC students' is not derived anywhere. If 295 fixed questions account for 42.5% of 694 total questions, then the remaining 399 questions were user-created, which would imply the MOOC students authored all of the remaining questions (assuming only students created questions), not 83%. Please provide the computation or correct the estimate.
- [Section 3] The paper states that the arena code is publicly available, but no repository link or URL is given. Please provide the exact location of the code and data to support reproducibility.
- [Section 2.1] The paper assumes that larger models in each family consume more energy than smaller ones, but this assumption is not validated with any measured energy data. A sentence acknowledging that the relative ordering is based on the model-size heuristic would make the presentation more precise.
- [References] Some entries are formatted inconsistently (e.g., Chung et al. 2025 with many authors is abbreviated while others are not). Please standardize the reference list to the journal's style.
Circularity Check
Partial circularity: the >75% small-model win rate is built into Eq. (1) via an unmeasured tie transfer, and the one-sided protocol guarantees the direction.
-
self definitional
[Section 2.2-2.3 (Eqs. (1)-(2)) and Section 4 (reported >75%)]
"WS(E) = WS + T + WL · Ec (1) ... If the user selected as best answer that from the model with the largest energy consumption, then the user is asked “Knowing that the other response consumes less energy, would you change your choice assuming a loss in quality?” ... the winning rates after accounting for the back-down rate significantly favor the smaller models (blue bars), since the users choose them more than 75% of the time."
Eq. (1) defines the post-energy small-model win rate as WS + T + WL·Ec. The protocol in Sec. 2.2 asks the energy question only to users who initially selected the larger model, so tied users are never asked how energy information would affect them; adding the entire tie rate T to S is an unverified assumption embedded in the metric. The Section 4 sentence “users choose them more than 75% of the time” then treats this constructed quantity as an observed choice. Because the equations can only move votes from L to S, and always move all ties to S, the direction of the headline result is guaranteed by construction; only the magnitude of the switch fraction Ec is genuinely measured.
full rationale
The paper's genuine empirical input is Ec, the fraction of large-model choosers who state they would switch to the smaller model when told it uses less energy (41–52%). Under the conditional reading, Eqs. (1)–(2) are accounting identities: WS(E) = WS + T + WL·Ec and WL(E) = WL(1−Ec). The circularity lies in the headline interpretation: “users choose them more than 75% of the time” is not a directly measured outcome but a deterministic transform of WS, T, WL, and Ec that assigns all unmeasured ties to the smaller model. Since only voters for the larger model were asked the energy question, the direction of the result is fixed by design, and the magnitude is inflated by the T transfer. This makes the central conclusion partly self-definitional rather than an independent result. The measured switch rate itself is real empirical content, so the circularity is partial rather than total. There is no load-bearing self-citation chain; the Conde et al. citation is background support for energy variability. The Limitations section honestly notes sample size, language, and model-family scope, but it does not acknowledge the tie-transfer assumption or the one-sided question, so the identified issue is not mitigated.
Assumptions & free parameters
assumptions (4)
- domain assumption Larger models in the same family consume more energy than smaller counterparts.
- domain assumption A hypothetical answer to the switch question predicts actual voting behavior.
- domain assumption The first vote is unaffected by the knowledge that energy information will be revealed afterwards.
- domain assumption MOOC students are representative of general LLM arena users.
Cite this review
Pith. "Pith review of The Generative Energy Arena (GEA): Incorporating Energy Awareness in Large Language Model (LLM) Human Evaluations." pith.science (2026). https://pith.science/paper/DALASW7S
@misc{pith2026250713302,
author = {Pith},
title = {Pith review of: The Generative Energy Arena (GEA): Incorporating Energy Awareness in Large Language Model (LLM) Human Evaluations},
year = {2026},
howpublished = {\url{https://pith.science/paper/DALASW7S}},
note = {Machine review of arXiv:2507.13302}
}
read the original abstract
The evaluation of large language models is a complex task, in which several approaches have been proposed. The most common is the use of automated benchmarks in which LLMs have to answer multiple-choice questions of different topics. However, this method has certain limitations, being the most concerning, the poor correlation with the humans. An alternative approach, is to have humans evaluate the LLMs. This poses scalability issues as there is a large and growing number of models to evaluate making it impractical (and costly) to run traditional studies based on recruiting a number of evaluators and having them rank the responses of the models. An alternative approach is the use of public arenas, such as the popular LM arena, on which any user can freely evaluate models on any question and rank the responses of two models. The results are then elaborated into a model ranking. An increasingly important aspect of LLMs is their energy consumption and, therefore, evaluating how energy awareness influences the decisions of humans in selecting a model is of interest. In this paper, we present GEA, the Generative Energy Arena, an arena that incorporates information on the energy consumption of the model in the evaluation process. Preliminary results obtained with GEA are also presented, showing that for most questions, when users are aware of the energy consumption, they favor smaller and more energy efficient models. This suggests that for most user interactions, the extra cost and energy incurred by the more complex and top-performing models do not provide an increase in the perceived quality of the responses that justifies their use.
Figures
Reference graph
Works this paper leans on
-
[4]
Speed and conversational large language models: Not all is about tokens per second. Computer 57, 74–80. doi:10.1109/MC.2024.3399384 . Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., Stein - hardt, J.,
-
[6]
arXiv preprint arXiv:2501.08219
Investigating energy efficiency and performance trade -offs in llm inference across tasks and dvfs settings. arXiv preprint arXiv:2501.08219 . Min, R., Pang, T., Du, C., Liu, Q., Cheng, M., Lin, M.,
-
[7]
arXiv preprint arXiv:2501.17858
Improving your model ranking on chatbot arena by vote rigging. arXiv preprint arXiv:2501.17858 . 12 Morrison, J., Na, C., Fernandez, J., Dettmers, T., Strubell, E., Dodge, J.,
-
[8]
arXiv preprint arXiv:2503.05804
Holistically evaluating the environmental impact of creating language models. arXiv preprint arXiv:2503.05804 . Panickssery, A., Bowman, S.R., Feng, S.,
-
[9]
arXiv preprint arXiv:2404.13076
Llm evaluators recognize and favor their own generations. arXiv preprint arXiv:2404.13076 . Phan, L., Gatti, A., Han, Z., Li, N., Hu, J., Zhang, H., Zhang, C.B.C., Shaaban, M., Ling, J., Shi, S., et al.,
-
[10]
arXiv preprint arXiv:2501.14249
Humanity’s last exam. arXiv preprint arXiv:2501.14249 . Wu, Y., Hua, I., Ding, Y.,
-
[11]
arXiv preprint arXiv:2502.11256
Unveiling environmental impacts of large language model serving: A functional unit view. arXiv preprint arXiv:2502.11256 . Xu, C., Guan, S., Greene, D., Kechadi, M., et al.,
-
[12]
arXiv preprint arXiv:2406.04244
Benchmark data contamination of large language models: A survey. arXiv preprint arXiv:2406.04244 . Zheng, C., Zhou, H., Meng, F., Zhou, J., Huang, M.,
Show all 12 references
-
[15]
Chiang, W.L., Zheng, L., Sheng, Y., Angelopoulos, A.N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonzalez, J.E., et al.,
URL: https://doi.org/10.1145/ 3641289, doi:10.1145/3641289. Chiang, W.L., Zheng, L., Sheng, Y., Angelopoulos, A.N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonzalez, J.E., et al.,
-
[2020]
arXiv preprint arXiv:2009.03300
Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 . Maliakel, P.J., Ilager, S., Brandic, I.,
2009 arXiv
-
[2024]
arXiv preprint arXiv:2403.04132
Chatbot arena: An open platform for evaluating llms by human preference. arXiv preprint arXiv:2403.04132 . Chung, J.W., Liu, J., Ma, J.J., Wu, R., Kweon, O.J., Xia, Y., Wu, Z., Chowdhury, M.,
-
[2025]
arXiv preprint arXiv:2505.06371
The ML.ENERGY Benchmark: Toward auto - mated inference energy measurement and optimization. arXiv preprint arXiv:2505.06371 . Conde, J., González, M., Reviriego, P., Gao, Z., Liu, S., Lombardi, F.,
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.