{"id":"650e1240-825d-4d73-9f5e-dcae29d9b86f","arxiv_id":"2509.08867","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Under vLLM serving, per-request inference energy plateaus at 100 concurrent requests, tracks parameter count closely within the Pythia family, and shows little variation across 3B-scale architectures.","lead":"This paper measures how much energy large language models use during inference when served through vLLM, a production-style serving tool. It finds that energy per request drops as concurrency rises, flattens around 100 requests, and scales roughly with parameter count within a model family.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Per-request energy without token-count normalization confounds the architecture-null claim: Table II models differ in tokenizer and output length, so equal J/request is compatible with real J/token differences.","rationale":"The reader identified vLLM-versus-model confounding as the weakest assumption; that threat is real and is explicitly acknowledged in Section V.A. My stress-test finds a more direct threat in the measurement unit. The architecture claim is the central place where this paper contradicts prior results (a 47% efficiency spread) and supports the general statement that architecture has no significant effect on energy efficiency. Without token counts or fixed output lengths, the null result in Figure 4 is not a model-level null. The proposed concrete test can settle the point by reusing the existing benchmark and logging completion token counts. The plateau and size-scaling claims may still hold, so rejection is too strong; the conditional verdict remains appropriate. For that reason I recommend leaving the reader's CONDITIONAL verdict unchanged.","tokens_in":7369,"tokens_out":6593,"duration_ms":399670,"concrete_test":"Re-run the Section IV.C experiment on the four Table II models, logging per-request completion token counts and computing both J/request and J/token, with identical sampling settings and a fixed max_tokens (e.g., 64) to control output length. If the spread or rank order of J/token differs materially from J/request (for example, the coefficient of variation doubles or a model moves from middle to best or worst), the architecture-null conclusion is an artifact of the request-based metric.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing issue is not vLLM itself but the unnormalized metric used to compare architectures. In Section IV.C and Figure 4, GPU energy per request is compared across Pythia 2.8B, Dolly V2 2.8B, BLOOM 3B, and RedPajama 2.8B. The paper does not report max_tokens, sampling parameters, or generated token counts, and HellaSwag completions are not length-controlled. Since energy per request equals generated tokens per request times energy per token plus fixed overhead, a null result in J/request is compatible with large differences in J/token if the models emit different numbers of tokens. The Section V.A defense of J/request is an application-level argument: different tokenizers and conciseness are real product differences, but the paper then interprets the null as a model-efficiency result and contrasts it with Argerich et al.'s 47% J/token effect. The Pythia size-scaling claim is less exposed because that family shares a tokenizer and similar output behavior, but the architecture comparison, which is also the point where this paper disagrees with prior work, is not currently testing what it claims to test.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the \"LLM Efficiency Benchmark,\" a measurement framework that uses the vLLM serving backend to measure GPU energy consumption of LLM inference under conditions meant to approximate production serving. Experiments vary concurrent request volume (5 to 5000 requests), model size across the Pythia family (70M to 6.9B parameters), and architecture across four models near 3B parameters (Pythia 2.8B, Dolly V2 2.8B, BLOOM 3B, RedPajama 2.8B). Energy is measured with CodeCarbon and reported as joules per request. The main claims are: per-request GPU energy decreases with concurrent request volume and plateaus at 100 requests; within the Pythia family energy per request grows close to linearly with parameter count; and model architecture has no significant effect on energy per request for models near 3B parameters. The results are compared against an earlier study by Argerich et al. that used the Transformers library and measured energy per token.","tokens_in":7705,"tokens_out":3028,"duration_ms":28593,"significance":"If the central claims are correct, the benchmark would be a useful step toward evaluating LLM inference efficiency in serving-oriented conditions rather than in isolated lab scripts, and the comparison with vLLM would be informative for practitioners. The paper is a direct measurement study with no fitted parameters or derived prediction, which is a strength in terms of transparency; the comparison to Argerich et al. is an external benchmark. The most valuable contribution--the platform-level observation that vLLM's batching behavior changes the energy profile relative to single-request Transformers inference--is plausible and worth pursuing. However, the architecture-null claim and the size-scaling claim are currently undersupported because the primary metric is not normalized by generated tokens, variance information is largely absent, and the compared models differ in size and tuning status. The paper's own validity section acknowledges the model-versus-backend threat and the tokenizer argument, but the conclusions are nevertheless stated as model-level results.","major_comments":[{"comment":"The architecture comparison is based on energy per request without any control for the number of generated tokens. HellaSwag completions are not length-controlled, and the paper does not report max_tokens, sampling parameters, or per-model output lengths. Since the models use different tokenizers and may produce different numbers of tokens, equal J/request is compatible with substantial differences in J/token. The Section V.A defense of J/request is an application-level argument, but the paper then interprets the null result as a model-efficiency statement and contrasts it with Argerich et al.'s 47% J/token effect. This comparison is not currently testing what it claims to test; token-level normalization or at least reported output-length statistics is needed.","section":"§IV.C, Figure 4, and §V.A"},{"comment":"Most of the reported results lack error bars, repetition counts, and any statistical test. Only Figure 4 states that each model was tested 10 times; Figures 1-3 appear to be based on unreported numbers of repeated runs. The plateau at 100 requests and the 'close to linear' relationship in Figure 3 are asserted from point estimates. The paper should report the number of repetitions per configuration, standard deviations or confidence intervals, and, for the linearity claim, a regression or residual analysis. The 410M/1B deviation noted in Section IV.B makes the linearity claim particularly sensitive to measurement uncertainty.","section":"§IV.A, Figures 1-2, and §IV.B, Figure 3"},{"comment":"CodeCarbon is described as reading power values every 15 seconds. For small request counts such as 5 or 10 and small models like Pythia 70M, the total run duration may be comparable to or shorter than the sampling interval, which would make per-request energy estimates coarse or biased. The paper should report run durations, the number of power samples collected per run, and an assessment of whether the 15-second sampling interval is adequate for the shortest runs.","section":"§III.A and §IV.A"},{"comment":"The architecture comparison does not hold model size constant: BLOOM is 3B while the other three are 2.8B. In addition, Dolly V2 is an instruction-tuned model and the exact RedPajama checkpoint (base, instruction-tuned, or chat) is not specified. These differences confound architecture with model size and training objective. The paper should either match parameters and tuning status more carefully or explicitly treat these factors as limitations in the interpretation of the null result.","section":"Table II and §IV.C"},{"comment":"The claim that energy per request grows 'close to linear' with parameter count is not supported by a quantitative model. A linear fit with confidence intervals, or a comparison of linear versus alternative scaling models, should be provided. Additionally, the discussion attributes the 410M/1B deviation to layer count, but no sensitivity analysis or direct test of the layer-count explanation is presented; this explanation should be framed as a hypothesis rather than a conclusion.","section":"§IV.B and §V"}],"minor_comments":[{"comment":"The manuscript contains numerous typos and grammatical issues, including 'Alumni' in the author affiliation, 'preformed' for 'performed', 'Wether' for 'Whether', 'an growing' in the abstract, and 'scenario's' for 'scenarios'. A thorough copyedit is needed.","section":"Throughout"},{"comment":"The sentence 'the 6.9 billion parameter version of Pythia shows a stabilized energy consumption per request at 40 parameters' appears to contain a typo; it should presumably refer to a request-count value, not 40 parameters.","section":"§IV.A"},{"comment":"No link to the benchmark code or data is provided. Given that the paper introduces a 'benchmark', making the code and raw measurements available would substantially aid reproducibility.","section":"§III.B"},{"comment":"The paper does not specify the exact model revisions used for Dolly V2 and RedPajama, nor the HuggingFace identifiers. This should be added so that the experiments can be reproduced.","section":"§III.C"},{"comment":"The statement that 'existing research shows no correlation between parameter amount and energy efficiency' is followed by the claim that there is a sub-linear correlation within the same architecture; these statements should be reconciled and both should be cited precisely.","section":"§II.C"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and practical topic, and the idea of benchmarking under a production-style serving backend is valuable. My main concern is that the architecture-null result--the point where the paper disagrees with prior work--is built on an unnormalized metric and heterogeneous models, so the central claim as stated is not yet supported. The authors' own validity section shows awareness of the issue, which suggests it can be fixed with additional analysis and possibly additional measurements. I would like the revision to include token-count data or a token-normalized analysis, error bars/repetition counts for all figures, and a more cautious framing of the model-level conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: this is a genuinely useful measurement study, and the parts that matter most (vLLM changes the energy-vs-concurrency curve; plateau around 100 requests) are probably robust. The architecture comparison is the weak link: it is presented as a model-level null result but the metric is joules per request, not per token, and the models are poorly matched (2.8B vs 3B, base vs instruction-tuned, different tokenizers). On the plus side, the paper ships real observations not in the prior literature: per-request GPU energy under vLLM as a function of concurrent load across the Pythia family, the plateau at 100 requests, and the size-scaling curve with the 410M/1B anomaly. The methodology is transparent enough to reproduce, the authors pick a sensible serving backend, and they openly flag the core vLLM-vs-model validity threat. That last point matters: they know the confound and name it, which is more honest than most benchmark papers. The soft spots are real but not fatal to the whole paper. No error bars on Figures 1-3; the paper reports 10 repeats only for Figure 4. The workload is burst-only (all requests at once), so claiming production realism is a stretch. Energy per request is not normalized by generated tokens, and the paper does not report max_tokens or output lengths. The stress-test worry lands: J/request can look equal across architectures while J/token differs by a lot, and since this is exactly where the paper disagrees with Argerich et al., that claim does not test what it says. The Section V.A defense of J/request as an application-level metric is reasonable for a product lens, but the paper cannot have it both ways: either the null is about user-perceived serving efficiency (fine, but then do not call it a model-efficiency result), or it is about model efficiency, in which case token normalization is required. The size-scaling claim is mostly safe because the Pythia family shares a tokenizer and similar generation behavior, though the 410M outlier gets a plausible layer-count explanation from prior work. Citation pattern is fine; the comparison to Argerich et al. is external, and the self-citations are not inflated. Who this is for: practitioners choosing models under a vLLM workload, and people building the next generation of inference-energy benchmarks. A serious referee should see it, mainly to force the authors to either add token-level reporting or reframe the architecture claim. Recommendation to the editor: send to peer review, with a request for the token-normalized analysis and error bars; do not desk reject.","headline":"Useful vLLM-era energy measurements, but the architecture-null claim needs token-normalized controls before it can stand.","tokens_in":8098,"tokens_out":1257,"would_cite":true,"duration_ms":10428,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Per-request LLM energy plateaus at 100 concurrent requests","keywords":["large language models","inference energy efficiency","vLLM","LLM serving","energy benchmarking","Pythia model family","GPU energy measurement","concurrent request load"],"falsifier":"Run the same four roughly 3B models and the Pythia family at 100 concurrent requests with vLLM's optimization stack disabled or with a different serving backend; if architecture-level energy differences reappear or the parameter-size curve changes shape, the reported plateau, near-linear scaling, and architecture null result are properties of vLLM, not of the models.","tokens_in":7148,"feed_emoji":"⚡","tokens_out":6923,"duration_ms":56095,"temperature":0.7,"pith_summary":"The paper argues that energy-efficiency benchmarks for large language models should measure inference the way production systems actually serve it, and it demonstrates a benchmark built on the vLLM serving backend. It claims three regularities: GPU energy per request falls as simultaneous request volume rises and plateaus at about 100 requests; within the Pythia model family, energy per request grows close to linearly with parameter count; and among models near 3 billion parameters, architecture has no significant effect on energy per request. The practical point is that developers can estimate per-request energy cost from request load and model size, and that serving technology, not just model design, determines real-world efficiency. This matters because most earlier measurements came from lab setups that omit production serving optimizations.","feed_headline":"Per-request LLM energy plateaus at 100 concurrent requests","feed_subtitle":"Under vLLM serving, energy scales nearly linearly with model size; architecture barely matters.","key_machinery":"The load-bearing machinery is the benchmark itself: the vLLM serving backend, which optimizes throughput through paged attention (flexible, non-contiguous storage of key-value caches), dynamic batching, speculative decoding, and GPU acceleration; CodeCarbon, which reads GPU power sensors every 15 seconds to estimate per-component energy; and the experimental design that sends all requests concurrently without rate limiting, after 200 warm-up requests. The plateau at 100 requests is the signature of amortization: fixed per-request overheads shrink as the backend fills GPU batches until utilization saturates. The near-linear size scaling and the architecture null result are both read off this same per-request energy metric at the plateau load.","core_discovery":"On the paper's own terms, the central discovery is that a production-style serving setup changes the energy-efficiency picture that laboratory benchmarks had reported. Using vLLM with all requests fired at once after warm-up, the benchmark finds that per-request GPU energy decreases with concurrency until it stabilizes at 100 simultaneous requests for both Pythia models and the four roughly 3B-parameter models. Within the Pythia family, per-request energy increases close to linearly with parameter count, with one exception: the 410M and 1B models consume nearly the same energy, which the paper attributes to the 1B model having fewer layers (16 versus 24), so layer count rather than parameter count is the limiting factor. Across four models of comparable size but different architectures, energy per request differs only marginally, in contrast to an earlier study that found a 47% spread. The paper reads the flat architecture result as likely an effect of vLLM's optimizations, while leaving the mechanism open.","pith_inferences":["If vLLM's optimizations are what erase architectural efficiency differences, then energy rankings from offline, non-serving benchmarks may not transfer to production; production-aware benchmarks like this one are the relevant comparison for deployed systems.","The 410M/1B exception suggests the true driver of inference energy is the number of sequential layer computations, not raw parameter count; testing more same-family models with controlled layer counts would sharpen the scaling law.","Measuring energy per request rather than per token creates an incentive for concise outputs, since a model that answers in fewer tokens looks cheaper; pairing the metric with output quality or token-normalized cost would separate brevity gains from efficiency gains.","Since the paper only tested burst requests sent at once, varying arrival rates would reveal whether the plateau and architecture null result hold under realistic interleaved traffic; that is a direct next experiment."],"forward_implications":["Energy-efficiency comparisons between models are only meaningful at or above the plateau load; below 100 concurrent requests the measured cost reflects load rather than model efficiency.","For same-architecture families, per-request energy is roughly predictable from parameter count, with layer count as a secondary factor as seen in the 410M/1B pair.","At roughly 3 billion parameters, architecture choice carries little energy penalty under vLLM, so developers can choose among comparable models on accuracy and features without major energy trade-offs.","Increasing request concurrency up to the plateau reduces per-request GPU energy, making batching and queue management a practical lever for cutting inference energy."],"supporting_citations":[{"why":"Supplies the vLLM serving backend and paged-attention mechanism that define the production-style conditions under test.","marker":"[4]"},{"why":"Supplies the Pythia suite, whose models differ mainly in parameter count, forming the size-scaling arm of the benchmark.","marker":"[17]"},{"why":"Supplies the HellaSwag sentence-completion requests used as the workload for every run.","marker":"[16]"},{"why":"Supplies the CodeCarbon tracker that reads GPU sensors and defines the per-component energy measurement.","marker":"[10]"},{"why":"Provides the earlier lab-condition study whose parameter-scaling result is reproduced and whose architecture-efficiency finding is contradicted.","marker":"[1]"},{"why":"Provides earlier lab-condition measurements and the observation on sub-linear same-architecture scaling that motivates the production-style design.","marker":"[3]"}],"fun_headline_variants":["Serving LLMs flattens energy across architectures","Layer count, not parameter count, drives LLM energy","vLLM serving: energy per request stabilizes at 100 concurrent","Architecture barely matters for LLM energy under vLLM","Per-request LLM energy plateaus at 100 concurrent calls"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that vLLM's automatic batching, speculative decoding, and GPU acceleration treat every model and architecture equally, so the measured patterns describe the models rather than the serving backend.","fun_headline_variants_meta":{"raw":{"variants":["Serving LLMs flattens energy across architectures","Layer count, not parameter count, drives LLM energy","vLLM serving: energy per request stabilizes at 100 concurrent","Architecture barely matters for LLM energy under vLLM","Per-request LLM energy plateaus at 100 concurrent calls"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001036,"raw_usage":{"total_tokens":4336,"prompt_tokens":896,"completion_tokens":3440,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":512,"completion_tokens_details":{"reasoning_tokens":3364}},"tokens_in":512,"tokens_out":3440,"duration_ms":21571,"temperature":1.0,"reasoning_tokens":3364,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:00:57.555711+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same four roughly 3B models and the Pythia family at 100 concurrent requests with vLLM's optimization stack disabled or with a different serving backend; if architecture-level energy differences reappear or the parameter-size curve changes shape, the reported plateau, near-linear scaling, and architecture null result are properties of vLLM, not of the models.","supporting_citations":[{"cited_title":"Pythia: A suite for analyzing large language models across training and scaling, 2023","cited_arxiv_id":null,"evidence_quote":"Supplies the Pythia suite, whose models differ mainly in parameter count, forming the size-scaling arm of the benchmark."},{"cited_title":"Friedler, and Jonathan P","cited_arxiv_id":null,"evidence_quote":"Supplies the CodeCarbon tracker that reads GPU sensors and defines the per-component energy measurement."},{"cited_title":"Measuring and improving the energy efficiency of large language models inference","cited_arxiv_id":null,"evidence_quote":"Provides the earlier lab-condition study whose parameter-scaling result is reproduced and whose architecture-efficiency finding is contradicted."},{"cited_title":"Towards the systematic reporting of the energy and carbon footprints of machine learning.J","cited_arxiv_id":null,"evidence_quote":"Provides earlier lab-condition measurements and the observation on sub-linear same-architecture scaling that motivates the production-style design."}],"review_version":2}