Pith. sign in

REVIEW 3 major objections 5 minor 12 references

Batched LLM energy cannot be fairly split by token counts alone.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 01:39 UTC pith:NVHIGTYC

load-bearing objection Measured Shapley ground truth for request-level LLM energy is real and carefully done, but the reference is protocol-bound, not yet production-validated; still, it deserves serious review. the 3 major comments →

arxiv 2608.00026 v1 pith:NVHIGTYC submitted 2026-07-11 cs.AI cs.DC

Request-Level Energy Attribution for Batched LLM Serving

Qi Luo , Kunlin Li , Ziwen Wang , Dongsheng Wang , Yun Chen This is my paper
classification cs.AI cs.DC
keywords LLM energy attributionShapley valuetoken-proportional accountingbatched inferenceGPU power telemetrycalibration modelcontinuous batchingsustainability accounting
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that the common practice of charging each request in a batched LLM run in proportion to its token count systematically misattributes a large share of GPU energy. It builds an offline harness that replays every subset of requests under a fixed protocol, measures active GPU power, and computes the exact Shapley energy share of each request as a fairness reference. Across 16 model/workload runs, token-based attribution deviates from this reference by 0.44–0.46 normalized L1, meaning roughly a quarter of each batch's energy ends up charged to the wrong requests. A lightweight ridge-regression calibrator trained on the measured Shapley shares, using separated prefill and decode token counts, cuts the error to 0.12–0.18 while adding about 0.003 ms per request. If correct, this gives serving systems a deployable path to fairness-like energy accounting without isolated runs.

Core claim

The central discovery is empirical: under both static and continuous batching, token-proportional attribution overcharges long-prompt requests and undercharges decode-heavy ones, because prefill and decode contribute asymmetrically to energy and batching amortizes shared costs in a way token counts do not capture. The paper establishes this by measuring the energy of every subset of 8-request groups on real GPUs and evaluating the exact Shapley value of each request, then uses those measurements as supervision for JCalib, a calibration model that predicts Shapley shares from cheap token-count features. The result is a measured, reproducible ground truth for request-level energy attribution a

What carries the argument

The Shapley value of a request under the coalition game whose characteristic function E(S) is the measured active GPU energy of serving subset S. The exact value requires every subset energy; the paper obtains these by exhaustive replay under a fixed protocol (static batches or continuous batching with 0.5s interleaved arrivals) with greedy decoding and power telemetry. JCalib then fits a ridge-regression mapping from per-request features — separated prefill and decode token counts, logs, group shares, and ratios — to those measured Shapley shares, with clip-and-renormalize to guarantee charges sum to the measured batch energy.

Load-bearing premise

The whole reference stands on the counterfactual replay protocol: if a production scheduler's actual interleaving, preemption, or memory sharing makes the energy of request subsets differ from the fixed 0.5s-gap replays used here, then the Shapley values that audit token attribution and train JCalib are no longer the right target.

What would settle it

Run the same audit on a production serving system with organic arrival traces, capturing actual subset and interleaving energies; if token-proportional attribution shows normalized L1 near zero against the production-based Shapley reference, or if JCalib no longer beats token attribution, the central claim is falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Token-normalized energy benchmarks, when applied to shared batches, should be read as approximate: their per-request charges can be wrong by roughly 25% of batch energy in heterogeneous mixes.
  • A single offline calibration trained on n=8 groups transfers to unseen models, workloads, GPUs, and to group sizes up to n=16, so the expensive subset-replay stage does not need to be repeated per deployment.
  • Because the calibrator uses only token counts the serving system already has and costs about 0.003 ms per request, it can be applied online per accounting window while preserving exact batch-energy efficiency.
  • The token–Shapley gap does not shrink with group size; it plateaus near 0.44–0.46 through n=16, so larger batches do not make token attribution fairer.
  • Shapley supervision, not the feature set, drives the improvement: fitting the same two token features against batch energies instead of Shapley shares yields only about 0.37 L1.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If production schedulers interleave, preempt, or share KV-cache in ways that differ from the fixed replay protocol, the measured Shapley labels could shift; validating on organic arrival traces is the natural next test.
  • The asymmetry between prefill and decode energy suggests a straightforward extension: stage-level power metering could provide even more direct labels and possibly reduce the need for exhaustive subset replay.
  • For carbon accounting, the same attribution vector multiplies by an intensity factor, so token-based carbon reports inherit the same roughly 25% misallocation; JCalib-style calibration could correct those reports with no new telemetry.
  • A testable extension is workload-specific calibration for chat-dominated traffic, where token error is smallest and calibrated gains are modest.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper addresses request-level GPU energy attribution in batched LLM serving. It builds an offline harness that, for groups of n=8 requests, replays every subset on vLLM under two protocols (static batching and continuous batching with a fixed 0.5 s inter-arrival gap), integrates NVML GPU power after subtracting idle power, and computes exact Shapley energy per request. This measured reference is used to audit token-proportional and standalone-proportional allocation and to train JCalib, a ridge-regression calibrator over token-count features. Across 16 model/workload runs and 40,800 subset generations, token attribution deviates from Shapley by mean normalized L1 0.440 under static batching and 0.458 under continuous batching, while JCalib reaches 0.116 and 0.177, below the offline Solo baseline. A sampled-Shapley campaign at n=16 on H100 shows the gap persists and that one offline calibration transfers across GPU, model scale, and group size. The paper includes repeatability checks, cross-hardware validation, permutation tests, and reproducibility artifacts.

Significance. If the results hold, they provide the first measured request-level Shapley ground truth for LLM inference and demonstrate that token-proportional accounting, as used in benchmarks such as ML.ENERGY, systematically misallocates a substantial fraction of batch energy. The paper is unusually careful empirically: per-subset CV around 1% with 97.8% identical outputs across repeats, bootstrap confidence intervals and paired permutation tests, a supervision ablation that isolates the value of Shapley labels, and a quantified sampling floor for the n=16 extension. The main caveat—the reference is defined by a counterfactual replay protocol rather than organic production traffic—is explicitly acknowledged in §5.6. Under the stated protocol, the central claims are well supported.

major comments (3)
  1. [§5.6, Abstract, Conclusion] The audit and calibration claims are established relative to a Shapley reference whose characteristic function E(S) comes entirely from counterfactual replay: static batches, or continuous batching with a fixed 0.5 s inter-arrival gap, greedy decoding, and no preemption, chunked prefill, or prefix/KV-cache sharing (§4.1). Because Shapley values are averages of marginals [E(S∪{i})−E(S)], any systematic divergence between replayed coalition energy and energy under a production scheduler shifts both the reported Token–Shapley gap and JCalib's training targets. Section 5.6 concedes that organic-trace validation is future work. This is a scope limitation rather than an internal inconsistency, but the abstract and conclusion should be explicitly scoped to the replay protocol; otherwise the reader may infer production-level generality. A small trace-driven validation or a clear statement of the
  2. [Abstract; Table 10] The statement that a single offline calibration 'remains the most accurate deployable rule' is contradicted by one per-workload cell: under continuous batching at n=16 on Chat, Table 10 reports Token L1 0.269 versus JCalib 0.282. If the claim is intended as an average over workloads, it should be qualified as such and the exception reported in both the abstract and §5.4.
  3. [Appendix A.4] The protocol for 'within-run leave-one-group-out' is not fully specified: the number of training groups, how the disjoint split is made, and whether the ridge regularization λ is fixed a priori or selected on the training split. Without this detail the reported 0.116/0.177 numbers and the transfer ablations cannot be reproduced from the text alone. The reproducibility notes mention scripts, but the written protocol should be self-contained.
minor comments (5)
  1. [Abstract; §5.2] The phrase 'roughly a quarter of each batch's energy is charged to the wrong requests' follows from L1/2 = 0.22, since each misallocated joule is counted twice. Please state this arithmetic so the reader does not confuse normalized L1 with the fraction of energy mischarged.
  2. [§4.2, Eq. (6)] Define \tilde{I} explicitly as the identity matrix with the intercept diagonal entry zeroed. The prose describes it, but the equation should be self-contained.
  3. [§5.4, Table 3] The sentence 'Under static batching its apparent growth from n=8 is smaller than the sampling floor' would be easier to verify with the arithmetic: 0.197−0.140 = 0.057 < 0.079. Please add this.
  4. [§5.5] The claim 'roughly 69× below the gap' appears inconsistent with the reported ±0.007 repeatability variation and 0.440 mean Token L1, which gives about 63×. Please correct the factor or report the exact standard deviation.
  5. [Table 2 caption] State explicitly that 'Token L1' and 'Solo L1' are normalized L1 distances against exact Shapley as defined in Eq. (7), rather than leaving the reader to infer it from the text.

Circularity Check

0 steps flagged

No circularity: measured subset replay provides independent Shapley ground truth, and JCalib is a supervised estimator evaluated on held-out groups.

full rationale

The derivation chain is an empirical measurement pipeline rather than a derivation that folds its inputs into its outputs. E(S) is independently measured by physically replaying subsets under vLLM (Eq. 1), and exact Shapley is computed from those measured coalition energies via Eq. 3. Token-proportional attribution is an independent baseline computed from token counts via Eq. 4, so the headline 0.440/0.458 gap is a measured quantity against the Shapley reference, not an algebraic identity. JCalib is trained on measured Shapley shares and evaluated under leave-one-group-out, leave-model-out, and leave-workload-out splits (Tables 7 and Appendix A.4), and the supervision ablation in Table 8 shows that fitting the same features to batch energies yields 0.367 L1 instead of 0.116; the gain is therefore attributed to the Shapley supervision rather than to features constructed to encode the target. No load-bearing step invokes a self-citation or an imported uniqueness theorem: the Shapley criterion is justified by the classic external result (Shapley 1953), and the cited prior carbon-accounting work (Li et al. 2025) is used only as conceptual motivation, not as the source of the measured numbers. Section 5.6 explicitly limits the ground truth to the counterfactual replay protocol and states that validation on organic production traces remains future work; this manuscript text was flagged and weighed. It is a scope/external-validity caveat, not a circularity, because all claimed numbers are stated for the replay protocol and the protocol is reproducible, with repeatability (CV ~1%), cross-GPU reproduction, and bootstrap intervals reported. The paper is self-contained against measured physical data, so no circular step is exhibited.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The central claim rests on one normative axiom (Shapley fairness), two measurement-protocol assumptions (replay representativeness, active-energy definition), and the fitted JCalib weights. No physical entities are invented.

free parameters (2)
  • JCalib ridge weights β (fitted per run/world model) = learned values, not enumerated
    The central calibration claims depend on a supervised fit to measured Shapley shares; the model is evaluated on held-out groups, so the fitted weights are parameters rather than independent evidence.
  • Ridge regularization λ = 1
    Fixed by hand, not tuned; used in Eq. 6 for closed-form ridge regression. It is a minor hyperparameter choice with a stated value.
axioms (4)
  • domain assumption Shapley value is the normative fairness target for shared-cost allocation
    The paper stipulates this criterion in §3.2; all 'misattribution' claims are measured as deviation from Shapley, so the fairness target itself is an axiom.
  • domain assumption Counterfactual replay under the fixed protocol yields a characteristic function representative of production batching
    §4.1 and §5.6: E(S) for every subset is defined by replaying subsets on vLLM with fixed 0.5s arrival gaps; no organic production traces validate this representativeness.
  • domain assumption Active GPU energy E(S)=∫max(P_S(t)-P_idle,0) is the service output to attribute
    Eq. 1 subtracts idle power and clamps sensor noise; CPU/system power and embodied carbon are excluded. If idle subtraction interacts nonlinearly with utilization, Shapley shares would change.
  • domain assumption Greedy decoding and the resulting output-length drift perturb the ground truth only at second order
    §5.1: generated lengths vary with batch composition for 67% of occurrences; the authors estimate a 0.036 normalized L1 perturbation, providing evidence but leaving a modeling assumption.

pith-pipeline@v1.3.0-alltime-deepseek · 16670 in / 14206 out tokens · 137614 ms · 2026-08-04T01:39:33.720736+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Request-Level Energy Attribution for Batched LLM Serving." pith.science (2026). https://pith.science/paper/NVHIGTYC

@misc{pith2026260800026,
  author       = {Pith},
  title        = {Pith review of: Request-Level Energy Attribution for Batched LLM Serving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NVHIGTYC}},
  note         = {Machine review of arXiv:2608.00026}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Batched LLM serving improves throughput but complicates energy accounting. GPU power telemetry is aggregate, whereas sustainability reporting, chargeback, and workload analysis often require request-level energy charges. Existing inference-energy benchmarks report model-, phase-, or token-level energy, and recent carbon-accounting work motivates Shapley fairness conceptually. Neither provides measured request-level ground truth, so how far the accounting rules used in practice deviate from a fair allocation has remained unknown. We present JouleShare, an attribution framework with two components. An offline harness establishes this ground truth by replaying request subsets under vLLM with a reproducible protocol, integrating GPU power telemetry, and computing exact Shapley energy for each request. A lightweight calibration model, JCalib, then learns to predict Shapley shares from cheap request features for use at serving time. Across 16 model/workload runs, token-proportional attribution differs from exact Shapley by 0.440 normalized L1 on average under static batching and by 0.458 under continuous batching, a gap that reproduces across three data-center GPUs. JCalib reduces this error to 0.116 under static batching and 0.177 under continuous batching, below even a standalone-measurement baseline that is unavailable online, while preserving exact batch-energy efficiency. Sampled Shapley extends the measured reference to larger group sizes, where the gap persists and a single offline calibration remains the most accurate deployable rule. The results show that token attribution is not a reliable proxy for marginal energy under batched execution, and that measured Shapley ground truth can calibrate low-cost request features toward fairer attribution.

Figures

Figures reproduced from arXiv: 2608.00026 by Dongsheng Wang, Kunlin Li, Qi Luo, Yun Chen, Ziwen Wang.

Figure 1
Figure 1. Figure 1: Under batching, token attribution over-charges the [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: JouleShare architecture. The offline harness replays request subsets under vLLM with GPU power telemetry to compute exact Shapley energy, which supervises JCalib for attribution at serving time. Feature group Per-request features Raw counts prefill pi, decode di, total pi+di Log counts log(1+pi), log(1+di), log(1+pi+di) Within-group share pi/ Pp, di/ Pd, (pi+di)/ P(p+d) Ratio to group mean pi/p¯, di/d¯, (p… view at source ↗
Figure 4
Figure 4. Figure 4: Normalized L1 against exact Shapley versus group [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

12 extracted references · 5 linked inside Pith

  1. [4]

    InNeurIPS Datasets and Bench- marks

    The ML.ENERGY Benchmark: Toward Automated Inference Energy Measure- ment and Optimization. InNeurIPS Datasets and Bench- marks. Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; Hesse,C.;andSchulman,J.2021. TrainingVerifierstoSolve Math Word Problems.arXiv preprint arXiv:2110.14168. Dod...

    Pith/arXiv arXiv 2021

  2. [8]

    Littlechild,S.C.;andThompson,G.F.1977

    Fair, Practical, and Efficient Carbon Accounting for LLMServing.ACMSIGMETRICSPerformanceEvaluation Review, 53(2): 99–103. Littlechild,S.C.;andThompson,G.F.1977. AircraftLand- ing Fees: A Game Theory Approach.The Bell Journal of Economics, 8(1): 186–204. Mistral AI Team

    1977

  3. [9]

    https://huggingface.co/mistralai/Mistral-7B-Instruct- v0.3

    Mistral-7B-Instruct-v0.3 Model Card. https://huggingface.co/mistralai/Mistral-7B-Instruct- v0.3. Niu,C.;Zhang,W.;Li,J.;Zhao,Y.;Wang,T.;Wang,X.;and Chen,Y.2026. TokenPowerBench:BenchmarkingthePower Consumption of LLM Inference.Proceedings of the AAAI Conference on Artificial Intelligence, 40(38): 32582–32590. Patel, P.; Choukse, E.; Zhang, C.; Goiri, I.; W...

    2026

  4. [12]

    InUSENIX Symposium on Operating Systems De- sign and Implementation (OSDI), 193–210

    DistServe: Disaggregating Prefill and Decoding for Goodput-Optimized Large Language Model Serving. InUSENIX Symposium on Operating Systems De- sign and Implementation (OSDI), 193–210. A Additional Experimental Details A.1 Full Energy Breakdown Table 4 gives the full energy breakdown behind the attribu- tionerrorsinTable2ofthemainpaper,permodelandwork- loa...

    arXiv 1989

  5. [846]

    LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding

    Bai,Y.;Lv,X.;Zhang,J.;Lyu,H.;Tang,J.;Huang,Z.;Du,Z.; Liu,X.;Zeng,A.;Hou,L.;Dong,Y.;Tang,J.;andLi,J.2024. LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 3119–3137. Caravaca, F.; Cuevas, Á.; and Cuevas, R

    2024

  6. [1970]

    Islam,M.A.;andRen,S.2016.ANewPerspectiveonEnergy AccountinginMulti-TenantDataCenters

    Ridge Regression: BiasedEstimationforNonorthogonalProblems.Technomet- rics, 12(1): 55–67. Islam,M.A.;andRen,S.2016.ANewPerspectiveonEnergy AccountinginMulti-TenantDataCenters. InUSENIXWork- shoponCoolTopicsonSustainableDataCenters(CoolDC). Kansal, A.; Zhao, F.; Liu, J.; Kothari, N.; and Bhattacharya, A.A.2010. VirtualMachinePowerMeteringandProvision- ing....

    2016

  7. [2019]

    In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 3645–3650

    Energy and Policy Considerations for Deep Learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 3645–3650. Yang, A.; Yang, B.; Zhang, B.; Hui, B.; Zheng, B.; Yu, B.; Li, C.; Liu, D.; Huang, F.; Wei, H.; Lin, H.; Yang, J.; Tu, J.; Zhang, J.; Yang, J.; Yang, J.; Zhou, J.; Lin, J.; Dang, K.; Lu, K.; Bao,...

    Pith/arXiv arXiv 2024

  8. [2021]

    Schwartz,R.;Dodge,J.;Smith,N.A.;andEtzioni,O.2020

    CarbonEmissionsandLargeNeuralNetworkTraining.arXiv preprint arXiv:2104.10350. Schwartz,R.;Dodge,J.;Smith,N.A.;andEtzioni,O.2020. Green AI.Communications of the ACM, 63(12): 54–63. Shapley, L. S

    Pith/arXiv arXiv 2020

  9. [2022]

    InProceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, 1877–1894

    Measuring the Car- bon Intensity of AI in Cloud Instances. InProceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, 1877–1894. Dong, M.; Lan, T.; and Zhong, L

    2022

  10. [2023]

    arXiv preprint arXiv:2308.16369

    SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills. arXiv preprint arXiv:2308.16369. anon8231489123

    Pith/arXiv arXiv

  11. [2024]

    arXiv preprint arXiv:2407.21783

    The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783. Han, L.; Kakadia, J.; Lee, B. C.; and Gupta, U

    Pith/arXiv arXiv

  12. [2025]

    Castro, J.; Gómez, D.; and Tejada, J

    From Prompts to Power: Measuring the Energy Footprint of LLM Inference.arXiv preprint arXiv:2511.05597. Castro, J.; Gómez, D.; and Tejada, J

    arXiv