Pith. sign in

REVIEW 4 major objections 5 minor 17 references

Breaking the ICE: Exploring promises and challenges of benchmarks for Inference Carbon & Energy estimation for LLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Benchmark-trained regressions estimate prompt-level LLM inference energy within about 15 percent average error, enabling non-intrusive carbon accounting.

desk verdict The benchmark-based estimation idea is practical and honestly presented, but the external validation rescales away the input-token granularity that is the whole point, so the headline accuracy numbers overstate what is demonstrated. read the letter →

arxiv 2506.08727 v1 pith:JAJOILQX submitted 2025-06-10 cs.LG cs.AIcs.CYcs.SE

Samarth Sikand , Rohit Mehra , Priyavanshi Pathania , Nikhil Bamby , Vibhu Saujanya Sharma , Vikrant Kaulgud , Sanjay Podder , Adam P. Burden This is my paper
classification cs.LGcs.AIcs.CYcs.SE
keywords LLMinferencecarbonemissionsestimationenergybenchmark-basedmodelingprompt-levelgranularityregressionsustainableAInon-intrusive
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that public LLM efficiency benchmarks, which already report latency and energy for many models, can be repurposed as training data for lightweight regression models that estimate the carbon footprint of a single inference prompt. It presents R-ICE, a framework that takes only a model's parameter count and the input and output token counts, then predicts end-to-end latency and energy use before the prompt is sent. On an unseen external leaderboard, energy estimates land within about 15 percent of reported values on average and latency within about 23 percent. The authors claim this makes prompt-level carbon accounting practical without intrusive monitoring or access to proprietary model internals.

What carries the argument

The mechanism is a latency decomposition: end-to-end runtime equals prompt-encoding latency plus the product of per-output-token latency and the number of generated tokens. Three regression models feed this decomposition — parameter count to minimum GPU count, parameter count and input tokens to prompt-encoding latency, and parameter count to per-output-token latency — and the resulting latency is multiplied by hardware count, thermal design power, utilization, data-center power usage effectiveness, and regional carbon intensity to get grams of CO2-equivalent. The framework deliberately replaces proprietary architecture details with two public numeric features, parameter count and token counts, which is what makes closed models addressable.

What would settle it

Measure actual energy with a power meter on a production model served at batch size 8 with known utilization, run the same prompts through R-ICE, and compare: if energy error substantially exceeds 15 percent, the fixed utilization and singleton-batch assumptions are the weak point.

Watch

Extended reading notes

Core claim

The central claim is that prompt-level inference energy and carbon emissions can be estimated accurately from benchmark data alone, using regression models that map model size and token counts to latency, then convert latency to energy through a standard operational-carbon formula. The framework trains one linear model for minimum GPU deployment, a random-forest model for prompt-encoding latency, and a polynomial model for per-output-token latency, with total latency as the sum of encoding time and per-token time times the number of output tokens. Validation on 46 filtered data points from an independent efficiency leaderboard gives an average mean absolute percentage error of 22.9 percent for end-to-end latency and 15.3 percent for energy consumption. The authors read this as evidence that benchmark-based estimation is accurate enough for pre-inference use cases such as dynamic LLM routing and carbon accounting.

Load-bearing premise

The estimates assume every model runs on a minimum number of a single GPU model at 26 percent utilization with a power usage effectiveness of 1.1 and a batch size of one, so any deployment with different utilization, batching, or hardware scales the estimates away from truth.

Editorial extensions

If this is right

  • Organizations can obtain prompt-level carbon estimates for closed or restrictive-license LLMs without vendor architecture disclosures.
  • Carbon-aware routing becomes feasible at design time, letting a router compare estimated emissions across models before sending a prompt.
  • A green ranking of LLMs can be derived from benchmark-based estimates and plugged into orchestration engines.
  • The approach works pre-inference, unlike monitoring tools that require access to the deployment environment after inference.
  • With only a few input features, the framework can be adopted at enterprise scale with low integration effort.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the 15 percent energy error holds across broader model families and hardware, benchmark-based estimation could complement or replace fine-grained monitoring for carbon reporting, since the same method scales to any LLM covered by a public benchmark.
  • The fixed utilization and PUE assumptions make estimates proportional to the true value when deployments differ only by these factors; a straightforward extension would calibrate utilization as a function of batch size, making the framework valid for production batching.
  • The latency decomposition may transfer to other autoregressive sequence models, including multimodal models whose text decoding dominates runtime, a testable extension.
  • A public repository of benchmark-derived carbon estimates would let auditors compare vendor claims against independent numbers, which the paper does not itself develop.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes R-ICE, a regression-based framework for estimating prompt-level inference energy and carbon emissions of LLMs without intrusive monitoring. It trains three regression models on HELM efficiency data: minimum GPU count versus parameter count, prompt-encoding latency versus input tokens and parameters, and per-output-token latency versus parameter count; these are combined with TDP, utilization, PUE, and regional carbon intensity. The framework is validated on 46 filtered points from the LLM-Perf leaderboard, with reported average MAPEs of 22.9% for E2E latency and 15.3% for energy, from which the paper concludes that prompt-level energy/carbon estimation is accurate.

Significance. The core idea—using public benchmark efficiency data to build a non-intrusive, low-input estimator for inference emissions—is timely and potentially useful for pre-inference LLM routing and sustainability accounting. The paper makes its assumptions explicit, attempts external validation on an unseen leaderboard, and candidly lists limitations such as small training sets and singleton-batch assumptions. However, the current validation does not actually test prompt-level granularity, so the headline accuracy numbers are not yet evidence for the paper's central claim. If the validation is reworked, the approach could become a useful, practical baseline in this space.

major comments (4)
  1. [Section V, Eq. (2)] The validation ground truth is constructed to be a function of output-token count only: Scaled(E2E Lat.) rescales the leaderboard E2E by output tokens/256, and Scaled(Energy) is output tokens divided by tok/kWh. Neither expression contains any input-token or prefill information, even though Eq. (1) and Regression Model 2 in the framework explicitly model prompt-encoding latency as a function of input tokens. Because all 46 data points are evaluated at a single prompt length (192 input tokens, 250 output tokens), the reported 22.9%/15.3% MAPE can only speak to output-token scaling and model-specific constant factors; it does not substantiate the Section V statement that estimates are 'accurate at a prompt level.' The authors should compare against leaderboard measurements with distinct, reported input and output lengths, or otherwise preserve the prefill contribution in the ground truth, and should also compare against a trivial output-token-only baseline to show that the prompt-aware model adds value.
  2. [Section V] The validation is performed at one fixed operating point: 192 input tokens and 250 output tokens for every one of the 46 models. This yields a single MAPE per model at one prompt, not a distribution over prompts, so it cannot establish that the method tracks variation in input prompt length, which is precisely the paper's stated granularity contribution. At minimum, the authors should report errors over a range of input and output token counts and provide confidence intervals for the aggregate MAPE.
  3. [Section IV-B] Regression Model 3 (per-output-token latency) is fit to 10 data points and Regression Model 1 (minimum devices) to fewer than 15 data points, with reported R² values computed in-sample and no held-out split for these components. Because the per-token latency enters Eq. (1) multiplicatively and the device count enters the carbon formula as a factor, overfitting in either component propagates directly into every energy estimate. The Limitations section acknowledges the small datasets, but Section V does not report how sensitive the validation MAPE is to these two regressions; leave-one-out or bootstrap assessment is needed before the accuracy claim can be regarded as robust.
  4. [Section IV-A1 and Fig. 1] The energy and carbon estimates inherit three unvalidated multiplicative constants: fixed hardware utilization 0.26, PUE 1.1, and the minimum-A100 deployment assumption with batch size 1. These constants are not calibrated or varied in Section V, and the leaderboard's hardware and workload conditions are not reported. Since any error in these constants scales the final estimate linearly, the 15.3% energy MAPE should be stated as conditional on these assumptions; without a sensitivity analysis (for example, utilization in [0.1, 0.9] or batch size > 1), the headline accuracy number is not informative about real-world deployments.
minor comments (5)
  1. [Eq. (1)] The symbol p is used both for the input token count and for 'prompt encoding time(p)'; this should be written as a function, e.g., T(p,o) = f(p) + (o-1)β, to avoid ambiguity.
  2. [Table III] The XGBoost MAPE is reported as 0.60, which is an order of magnitude worse than Gradient Boosting despite having a higher R²; this is likely a typo for 0.060 and should be corrected.
  3. [Eq. (2) and Table V] The units are inconsistent: Scaled(Energy) = Output tokens / (tok/kWh) yields kWh, while Table V labels Energy in J. The conversion factor should be stated explicitly.
  4. [Section V, Eq. (2)] The origin of the denominator 256 is not explained; the authors should state the leaderboard's native output-token length and input-token length so that the scaling in Eq. (2) can be reproduced and assessed.
  5. [Throughout] The acronym is given as RICE in the Section IV title but R-ICE elsewhere; use one consistent form.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: regression models are fit to HELM data and validated against unseen LLM-Perf leaderboard targets, so the energy/carbon predictions are not forced by construction.

full rationale

The paper's derivation chain is: (i) fit three regression models to HELM-Efficiency data (minimum deployment hardware vs. parameter count; prompt-encoding latency vs. parameters/input tokens; per-output token latency vs. parameters), (ii) combine predicted latency and device counts with fixed external constants (TDP, utilization 0.26, PUE 1.1, RCI) in a standard carbon formula, and (iii) validate the resulting latency/energy estimates against 46 unseen points from the HuggingFace LLM-Perf leaderboard. No model parameter is tuned to the validation targets; the only transformation applied to the targets is the linear rescaling in Eq. (2), which is a preprocessing step rather than a fitted prediction. That rescaling does discard input-token information and thus weakens the prompt-level validation claim as a correctness/interpretability matter, but it does not make the predicted values equal to the ground truth by construction. The load-bearing references ([11], [12], [14]) are external to the authors, so there is no self-citation chain. Section VI openly lists the framework's assumptions (small training sets, singleton batch, 10k-token context limit, fixed utilization/PUE, vendor-specific stacks); these are stated limitations, not circular inputs. No quoted equation or fitted parameter reduces to the reported outcome, so no circular step can be identified under the specified patterns.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The framework introduces no new physical entities. It relies on fitted regression models, fixed operational constants, and several domain assumptions inherited from prior work or chosen for convenience. The load-bearing constants are utilization and PUE, which directly multiply the final carbon estimate.

free parameters (5)
  • Hardware utilization default = 0.26
    Chosen from a BLOOM study, not from the target validation data; directly scales the carbon estimate in Fig. 1.
  • PUE default = 1.1
    Assumed default; directly scales carbon estimate.
  • Minimum-device linear regression coefficients = Not disclosed
    Fit to fewer than 15 HELM data points (R^2=0.99) to predict number of A100 GPUs from model parameters.
  • Prompt-encoding latency Random Forest model = Black-box model
    Trained on about 400 HELM data points; R^2=0.986 on training set, MAPE 5.0%.
  • Per-output token latency polynomial regression coefficients = Not disclosed
    Fit to 10 HELM data points (R^2=0.94); smallest training set in the framework.
assumptions (5)
  • domain assumption End-to-end runtime decomposes as prompt encoding time plus (output tokens - 1) times per-token latency (Eq. 1)
    Inherited from Narayanan et al.'s HELM cost model; assumes idealized deployment on standardized hardware.
  • domain assumption Power draw of a GPU is approximated by TDP times utilization
    Used in the carbon formula in Fig. 1; no validation of this approximation in the paper.
  • domain assumption Inference is run with batch size 1 and context length under 10k tokens
    Stated in Section IV-A1; real services often batch and use longer contexts.
  • ad hoc to paper The number of GPU devices needed scales linearly with model parameter count
    Observed from fewer than 15 data points; used as Regression Model 1.
  • ad hoc to paper Validation ground truth is rescaled linearly by token count (Eq. 2)
    Used to compare against LLM-Perf data measured at different token lengths; the linearity is not justified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Breaking the ICE: Exploring promises and challenges of benchmarks for Inference Carbon & Energy estimation for LLMs." pith.science (2026). https://pith.science/paper/JAJOILQX

@misc{pith2026250608727,
  author       = {Pith},
  title        = {Pith review of: Breaking the ICE: Exploring promises and challenges of benchmarks for Inference Carbon & Energy estimation for LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JAJOILQX}},
  note         = {Machine review of arXiv:2506.08727}
}
read the original abstract

While Generative AI stands to be one of the fastest adopted technologies ever, studies have made evident that the usage of Large Language Models (LLMs) puts significant burden on energy grids and our environment. It may prove a hindrance to the Sustainability goals of any organization. A crucial step in any Sustainability strategy is monitoring or estimating the energy consumption of various components. While there exist multiple tools for monitoring energy consumption, there is a dearth of tools/frameworks for estimating the consumption or carbon emissions. Current drawbacks of both monitoring and estimation tools include high input data points, intrusive nature, high error margin, etc. We posit that leveraging emerging LLM benchmarks and related data points can help overcome aforementioned challenges while balancing accuracy of the emission estimations. To that extent, we discuss the challenges of current approaches and present our evolving framework, R-ICE, which estimates prompt level inference carbon emissions by leveraging existing state-of-the-art(SOTA) benchmark. This direction provides a more practical and non-intrusive way to enable emerging use-cases like dynamic LLM routing, carbon accounting, etc. Our promising validation results suggest that benchmark-based modelling holds great potential for inference emission estimation and warrants further exploration from the scientific community.

Figures

Figures reproduced from arXiv: 2506.08727 by the authors.

Figure 1
Figure 1. Overall approach to RICE. TABLE I HIGH-LEVEL DESCRIPTION OF COMPONENTS IN FIG.1 Component Description # of Devices Min. number of hardware units to deploy a LLM Thermal Design Power (TDP) Power consumption of hardware under maximum theoretical load Utilization Average Hardware utilization/efficiency End-to-end(E2E) latency Total time taken to process the input prompt and generate the final response(till the last out… view at source ↗
Figure 2
Figure 2. Correlation Per-output Token Latency and Model Size(# Parameters) [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

17 extracted references · 10 canonical work pages

  1. [1]

    Reducing the carbon impact of generative ai inference (today and in 2035),

    A. A. Chien, L. Lin, H. Nguyen, V . Rao, T. Sharma, and R. Wijayawar- dana, “Reducing the carbon impact of generative ai inference (today and in 2035),” inProceedings of the 2nd workshop on sustainable computer systems, 2023, pp. 1–7

    work page 2023

  2. [2]

    Github, Jul. 2024. [Online]. Available: https://github.com/features/copilot

    work page 2024

  3. [3]

    The carbon footprint of machine learning training will plateau, then shrink,

    D. Patterson, J. Gonzalez, U. H ¨olzle, Q. Le, C. Liang, L.-M. Munguia, D. Rothchild, D. R. So, M. Texier, and J. Dean, “The carbon footprint of machine learning training will plateau, then shrink,”Computer, vol. 55, no. 7, pp. 18–28, 2022

    2022

  4. [4]

    Green ai,

    R. Schwartz, J. Dodge, N. A. Smith, and O. Etzioni, “Green ai,” Communications of the ACM, vol. 63, no. 12, pp. 54–63, 2020

    2020

  5. [5]

    Google, Jul. 2024. [Online]. Available: https://blog.google/outreach- initiatives/sustainability/2024-environmental-report/

    work page 2024

  6. [6]

    mlco2/codecarbon: v2.4.1,

    B. Courty, “mlco2/codecarbon: v2.4.1,” May 2024. [Online]. Available: https://doi.org/10.5281/zenodo.11171501

    doi:10.5281/zenodo.11171501 2024

  7. [7]

    Eco2ai: carbon emissions tracking of machine learning models as the first step towards sustainable ai,

    S. A. Budennyy, V . D. Lazarev, N. N. Zakharenko, A. N. Korovin, O. Plosskaya, D. V . Dimitrov, V . Akhripkin, I. Pavlov, I. V . Oseledets, I. S. Barsolaet al., “Eco2ai: carbon emissions tracking of machine learning models as the first step towards sustainable ai,” inDoklady Mathematics, vol. 106, no. Suppl 1. Springer, 2022, pp. S118–S128

    work page 2022

  8. [8]

    LLMCarbon: Modeling the end-to-end carbon footprint of large language models,

    A. Faiz, S. Kaneda, R. Wang, R. C. Osi, P. Sharma, F. Chen, and L. Jiang, “LLMCarbon: Modeling the end-to-end carbon footprint of large language models,” inThe Twelfth International Conference on Learning Representations, 2024

    work page 2024

Show all 17 references
  1. [9]

    Quanti- fying the carbon emissions of machine learning,

    A. Lacoste, A. Luccioni, V . Schmidt, and T. Dandres, “Quanti- fying the carbon emissions of machine learning,”arXiv preprint arXiv:1910.09700, 2019

    1910 arXiv

  2. [10]

    AWS, Jul. 2024. [Online]. Available: https://aws.amazon.com/aws-cost- management/aws-customer-carbon-footprint-tool/

    2024

  3. [11]

    Holistic evaluation of language models,

    P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y . Zhang, D. Narayanan, Y . Wu, A. Kumaret al., “Holistic evaluation of language models,”arXiv preprint arXiv:2211.09110, 2022

    2022 arXiv

  4. [12]

    Cheaply estimating inference efficiency metrics for autoregressive transformer models,

    D. Narayanan, K. Santhanam, P. Henderson, R. Bommasani, T. Lee, and P. S. Liang, “Cheaply estimating inference efficiency metrics for autoregressive transformer models,”Advances in Neural Information Processing Systems, vol. 36, pp. 66 518–66 538, 2023

    2023

  5. [13]

    Beyond efficiency: Scaling ai sustainably,

    C.-J. Wu, B. Acun, R. Raghavendra, and K. Hazelwood, “Beyond efficiency: Scaling ai sustainably,”IEEE Micro, 2024

    2024

  6. [14]

    Estimating the carbon footprint of bloom, a 176b parameter language model,

    A. S. Luccioni, S. Viguier, and A.-L. Ligozat, “Estimating the carbon footprint of bloom, a 176b parameter language model,”Journal of Machine Learning Research, vol. 24, no. 253, pp. 1–15, 2023

    2023

  7. [15]

    Llm-perf leaderboard,

    R. P. Ilyas Moutawwakil, “Llm-perf leaderboard,” https://huggingface.co/spaces/optimum/llm-perf-leaderboard, 2023

    2023

  8. [16]

    Gptq: Accurate post-training quantization for generative pre-trained transformers,

    E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “Gptq: Accurate post-training quantization for generative pre-trained transformers,”arXiv preprint arXiv:2210.17323, 2022

    2022 arXiv

  9. [17]

    Flashattention: Fast and memory-efficient exact attention with io-awareness,

    T. Dao, D. Fu, S. Ermon, A. Rudra, and C. R ´e, “Flashattention: Fast and memory-efficient exact attention with io-awareness,”Advances in Neural Information Processing Systems, vol. 35, pp. 16 344–16 359, 2022

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.