Pith. sign in

REVIEW 4 major objections 6 minor 64 references

vLLM settings for attention and prefix caching change energy, latency, and even accuracy—with no single best setup.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 04:54 UTC pith:4AIZO2ZZ

load-bearing objection Solid factorial measurement study of three vLLM knobs: attention and prefix caching matter, model dominates the Pareto front, and configs can move accuracy—energy rankings need a grain of salt on A100 nvidia-smi. the 4 major comments →

arxiv 2607.09172 v1 pith:4AIZO2ZZ submitted 2026-07-10 cs.SE cs.AIcs.PF

Attention to Detail: Evaluating Energy, Performance, and Accuracy Trade-offs Across vLLM Configurations

Nada Zine , Tristan Coignion , Vincenzo Stoico , Cl\'ement Quinton , Romain Rouvoy , Patricia Lago This is my paper
classification cs.SE cs.AIcs.PF
keywords vLLMLLM inferenceenergy consumptionattention kernelsprefix cachingchunked prefillPareto frontieraccuracy
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether the knobs that practitioners turn when serving large language models with vLLM actually matter for energy use, speed, and answer quality. The authors run a full-factorial controlled experiment on three system-level options—attention kernel, prefix caching, and chunked prefill—across five open-weight models and five task types, for 9,000 runs and tens of thousands of measurements. They find that attention type and prefix caching often move energy and latency by large margins, that the best choice depends on the model and workload, and that chunked prefill barely matters under the engine’s default long-prompt budget and the workloads they tested. Model choice sets the main energy–latency–accuracy regime; configuration only slides points along that frontier. Unexpectedly, the same “system-level” options can also shift measured accuracy scores.

Core claim

Across a controlled full-factorial study of attention kernel, prefix caching, and chunked prefill on five models and five tasks, the first two options significantly affect energy and performance in model- and workload-dependent ways, with no universally optimal configuration; model choice dominates the global energy–latency–accuracy trade-off while configuration only yields local Pareto improvements; and these inference options can also change measured accuracy.

What carries the argument

A full-factorial controlled experiment over three vLLM options (three attention kernels × prefix caching on/off × chunked prefill on/off), five models, five task datasets, and thirty repetitions—analyzed with non-parametric ART ANOVA for main and interaction effects plus multi-objective Pareto frontiers.

Load-bearing premise

The study assumes that one-hertz software power samples on A100 GPUs and offline batch serving under fixed default budgets fairly represent how these configuration options behave in real deployments.

What would settle it

Repeat the same factorial design with continuous hardware power metering and online HTTP serving (or with a much lower max_num_batched_tokens so prefill chunking actually fires), and check whether attention and prefix caching still dominate, chunked prefill stays near-null, and accuracy still shifts.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Practitioners should treat attention kernel and prefix caching as first-class energy and latency levers and evaluate them jointly on their own workloads rather than relying on a single default.
  • When energy is the priority, FlashInfer is often the better attention backend; when latency is the priority, FlashAttention-3 is often better—so energy and latency can require different configs.
  • Published accuracy scores that omit the full inference-stack configuration may not transfer to another deployment of the same model.
  • Model selection remains the primary multi-objective decision; configuration tuning is secondary fine-grained optimization along the Pareto front.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Benchmark leaderboards and green-AI comparisons that fix the model but not the serving stack risk attributing accuracy or energy differences to the model when they partly come from the engine.
  • Inference-engine vendors may need task-aware or multi-objective defaults rather than one static configuration for all workloads.
  • If non-associative floating-point reductions in attention kernels and cache paths truly change sampled tokens, reproducibility checklists for LLM evaluation should require full engine configuration, not only decoding hyperparameters.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper reports a full-factorial controlled experiment on three vLLM system-level options—attention kernel (FlashAttention-2/3, FlashInfer), prefix caching (on/off), and chunked prefill (on/off)—across five open-weight dense LLMs (3B–32B) and five task datasets, with 30 randomized repetitions per cell (9,000 runs). Using ART ANOVA with Holm correction, Cliff’s δ, and Pareto aggregation, it claims that attention type and prefix caching are the dominant configuration drivers of energy, end-to-end latency, and TTFT; chunked prefill has limited effect under default serving settings; effects are strongly model- and task-dependent with no universal optimum; model choice dominates the energy–latency–accuracy Pareto front while configuration only yields local moves; and, unexpectedly, attention backend and prefix caching can shift measured accuracy on EE and LB.

Significance. If the results hold under stronger measurement and serving conditions, the paper supplies concrete, practitioner-facing evidence that inference-engine configuration is not a neutral implementation detail for energy, latency, or even benchmark scores. Strengths that should be credited include: a GQM-framed design with full factorial coverage of the three factors; 30 repetitions, random scheduling, warmup/calibration, and cache resets; non-parametric factorial analysis with multiplicity control and effect sizes; multi-objective Pareto reporting by task; an explicit threats section; and a full replication package. The accuracy finding, if real, would force empirical LLM studies to treat the inference stack as a first-class experimental variable. The work is a useful empirical SE contribution on configurable serving systems, not a new algorithm.

major comments (4)
  1. §IV and Threats §VII: energy is computed as E = mean power × time from nvidia-smi (GPU) and perf (CPU) at 1 Hz. The manuscript itself cites that on A100 nvidia-smi samples power for only ~25% of runtime and can err by up to ~65% on spiky loads. Attention kernels and prefix-caching paths change memory-access and compute intensity, so differential sampling bias across backends is plausible. Table IV’s large Cliff’s δ rankings (e.g., FlashInfer vs FlashAttention-3) and the energy axis of the Pareto front (Table V, Fig. 5) therefore rest on an incompletely validated construct. Repeatability (CV = 0.043) does not rule out systematic bias. Either validate relative energy orderings against a higher-fidelity meter / denser sampling on a subset of cells, or reframe energy claims as software-sampled estimates and de-emphasize absolute kJ rankings in the abstract and contributions.
  2. §V.A.5 and contribution (iv): statistically significant accuracy shifts under attention type and prefix caching are elevated to a main contribution, yet the Discussion only offers two untested explanations (finite-sample noise vs non-associative FP reductions) and defers disentanglement. Decoding uses temperature = 0.4 and top-p = 0.95 (§IV), so stochastic generation is a first-order confound. With accuracy only on EE and LB (10 model–task cells for accuracy tests in Table III), the claim that “inference options can also affect model accuracy” is not yet strong enough for a headline contribution. Add fixed-seed / greedy or logit-level comparisons on the significant cells, or demote the claim to a carefully scoped observation pending mechanism checks.
  3. RQ1 results, Discussion (“Why chunked prefill had negligible effects”), and abstract: the near-null chunked-prefill result is correctly attributed to default max_num_batched_tokens = 8192 and mostly short prompts, so the option is often never activated. That scoping is load-bearing for the claim that chunked prefill “has a limited effect.” The abstract and contribution list already qualify this, but Table III (2/1/1 significant tests) and stakeholder advice to “leave chunked prefill at default” still read as a general ranking of the three options. Either run a sensitivity cell with a lower batch-token budget on LB (or another long-context set) so active chunking is forced, or state more sharply that the study ranks the option-as-enabled-under-defaults, not chunked prefill as a mechanism.
  4. External Validity §VII and Study Design §III: all runs use vLLM’s offline batch interface with controlled batching and cold caches between datasets. The target audience is developers of production vLLM-served systems, where HTTP serving, concurrency, continuous batching, and warm prefix caches dominate. Latency/TTFT and prefix-caching benefits are especially sensitive to that gap. The threats section notes the issue; the main claims and implications (§VI) should either bound conclusions to offline/batch serving or add a small online-serving validation subset so the “no universal configuration” and prefix-caching advice are not over-extended.
minor comments (6)
  1. Fig. 1 configuration IDs (f2-p0-c0, …) are dense; a short legend in the caption listing all 12 codes would help readers who do not memorize the encoding.
  2. Table IV reports energy pairwise tests only; the text refers readers to the replication package for latency/TTFT/accuracy post-hocs. A compact appendix table of significant latency/TTFT contrasts would make the paper self-contained for the performance claims.
  3. §III.D: NQ is excluded from accuracy because it “contains data used for training rather than for benchmarking,” but contamination is model-dependent and not checked. A one-sentence caveat would avoid overstating the reason.
  4. Fig. 4 color scale and signed Δ (On−Off) are useful; stating units and the median aggregation rule again in the caption would match the care taken in Fig. 1.
  5. Related Work is thorough but could more sharply contrast this factorial system-level design with decoding-hyperparameter studies (e.g., Zine et al.) so the novelty boundary is one paragraph, not distributed.
  6. Minor consistency: model labels alternate between Qwen3-32B / Qwen-32B and Llama-3.1-8B-Instruct / Llama-8B; pick one short form after Table I.

Circularity Check

0 steps flagged

No circular derivation: controlled factorial measurement study whose claims rest on external operational metrics and new experimental data, not on self-definitional or fitted-as-prediction steps.

full rationale

The paper is an empirical software-engineering experiment (GQM, full-factorial ART ANOVA, Cliff’s δ, Pareto aggregation of medians/means). Energy is the standard operational definition E = mean power × duration from nvidia-smi/perf; TTFT and request latency come from vLLM timers; accuracy uses established pass@k and exact-match scoring on EE/LB. Configuration effects, interactions, and Pareto fronts are computed from 9 000 new runs; they are not forced by construction from fitted free parameters, nor do they reduce to a self-citation uniqueness theorem or renamed known pattern. Author self-citations appear only as related-work background (e.g., prior hyperparameter or green-LLM studies) and are not load-bearing for the central claims. Dependence on chosen defaults (max_num_batched_tokens=8192, offline batch mode, A100 sampling) is a validity/threats issue, not circularity. Hence score 0 with empty steps.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The central claims rest on standard empirical-SE measurement practice and fixed experimental knobs, not on new physical entities. Load-bearing free choices are decoding and serving defaults that define the regime (especially the 8192 batched-token budget that largely disables chunked prefill). Domain assumptions include software power sampling as a valid energy proxy and offline batch serving as informative for configuration effects. No new particles or mediators are invented.

free parameters (6)
  • temperature = 0.4
    Fixed at 0.4 for all runs; shapes sampling and can interact with measured accuracy variability.
  • top-p = 0.95
    Fixed nucleus sampling parameter for all generations.
  • max-tokens / task output caps = 2048 (default); task-specific overrides
    Global max-tokens 2048 and task-specific caps (e.g., NQ 512, WC 32000) bound generation length and thus energy/latency.
  • max_num_batched_tokens (vLLM default) = 8192
    Default 8192 largely determines whether chunked prefill activates; central to the limited-effect claim for that option.
  • power sampling rate = 1 Hz
    1 Hz sampling for perf and nvidia-smi defines energy integration resolution.
  • task subsample sizes = AT20/EE100/LB30/NQ1500/WC200
    Hand-chosen sample sizes (e.g., 30 LB, 1500 NQ, 200 WC) set statistical power and workload mix.
axioms (5)
  • domain assumption Energy for a run equals mean sampled power times duration (E = W × T).
    Stated in Experimental Variables; underpins all energy comparisons.
  • domain assumption Aligned Rank Transform ANOVA with Holm–Bonferroni correction is appropriate for non-normal multi-factor comparisons.
    Data Analysis section; supports significance claims for main and interaction effects.
  • ad hoc to paper Holding unstudied vLLM options at defaults isolates the three selected factors sufficiently for the stated conclusions.
    Selection of options and Threats to Validity; untested interactions with other serving knobs remain possible.
  • domain assumption Offline batch inference with controlled cache resets is an informative proxy for configuration effects relevant to serving practice.
    Experiment Execution and External Validity; HTTP/async serving may differ.
  • domain assumption pass@k and LongBench exact-match scoring are adequate accuracy constructs for detecting config-induced output changes.
    Accuracy metrics section; accuracy only evaluated on EE and LB.

pith-pipeline@v1.1.0-grok45 · 25059 in / 3445 out tokens · 42336 ms · 2026-07-13T04:54:45.394190+00:00 · methodology

0 comments
read the original abstract

Large Language Models are reshaping how software is developed and maintained. They are typically deployed in production using inference engines such as vLLM, which can efficiently serve pre-trained, highly configurable models. While prior work has focused on model architectures and hardware acceleration, the impact of inference engine configuration on energy consumption, performance, and output quality remains poorly understood. In this paper, we present a large-scale controlled study of three selected vLLM configuration options: attention kernel type, prefix caching, and chunked prefill. We evaluate all combinations of these configurations across 5 open-weight LLMs and 5 diverse inference tasks, totaling $9,000$ runs and $93,600$ measures. We analyze energy consumption, latency, and accuracy, and examine both main effects and interaction effects between configuration options and tasks. Our results show that the studied configuration options significantly impact energy and performance, mainly driven by attention type and prefix caching, while chunked prefill has a limited effect under the default vLLM serving configuration and evaluated workloads. These effects are highly model- and workload-dependent, and no configuration is universally optimal. We further show that model choice dominates global trade-offs, while configuration tuning provides local improvements along the Pareto frontier. Unexpectedly, inference options can also affect model accuracy.

Figures

Figures reproduced from arXiv: 2607.09172 by Cl\'ement Quinton, Nada Zine, Patricia Lago, Romain Rouvoy, Tristan Coignion, Vincenzo Stoico.

Figure 1
Figure 1. Figure 1: Total energy consumption (kJ) by model, task, [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Representative 2-way interaction effects on end-to-end [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Task and attention type interaction on total energy [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Difference in total energy consumption (prefix caching [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Two-dimensional projection of the EvoEval Pareto [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

64 extracted references · 1 canonical work pages

  1. [1]

    Gpts are gpts: Labor market impact potential of llms,

    T. Eloundou, S. Manning, P. Mishkin, and D. Rock, “Gpts are gpts: Labor market impact potential of llms,”Science, vol. 384, no. 6702, pp. 1306–1308, 2024

    2024

  2. [2]

    An analysis of large language models: their impact and potential applications,

    G. Bharathi Mohan, R. Prasanna Kumar, P. Vishal Krishh, A. Keerthi- nathan, G. Lavanya, M. K. U. Meghana, S. Sulthana, and S. Doss, “An analysis of large language models: their impact and potential applications,”Knowledge and Information Systems, vol. 66, no. 9, pp. 5047–5070, 2024

    2024

  3. [3]

    The impact of llm chatbots on learning outcomes in advanced driver assistance systems education,

    M. Murtaza, C.-T. Cheng, B. M. Albahlal, M. M. A. Muslam, and M. S. Raza, “The impact of llm chatbots on learning outcomes in advanced driver assistance systems education,”Scientific Reports, vol. 15, no. 1, p. 7260, 2025

    2025

  4. [4]

    Recalibrating global artificial intelligence e-waste estimates,

    A. d. Vries-Gao, “Recalibrating global artificial intelligence e-waste estimates,”Resources, Conservation and Recycling, vol. 229, p. 108872, 2026. [Online]. Available: https://www.sciencedirect.com/ science/article/pii/S0921344926000960

    2026

  5. [5]

    Towards greener llms: Bringing energy-efficiency to the forefront of llm infer- ence,

    J. Stojkovic, E. Choukse, C. Zhang, I. Goiri, and J. Torrellas, “Towards greener llms: Bringing energy-efficiency to the forefront of llm infer- ence,”arXiv preprint arXiv:2403.20306, 2024

    Pith/arXiv arXiv 2024

  6. [6]

    How hungry is ai? benchmarking energy, water, and carbon footprint of llm inference,

    N. Jegham, M. Abdelatti, C. Y . Koh, L. Elmoubarki, and A. Hendawi, “How hungry is ai? benchmarking energy, water, and carbon footprint of llm inference,”arXiv preprint arXiv:2505.09598, 2025

    arXiv 2025

  7. [7]

    Llmco2: Advancing accu- rate carbon footprint prediction for llm inferences,

    Z. Fu, F. Chen, S. Zhou, H. Li, and L. Jiang, “Llmco2: Advancing accu- rate carbon footprint prediction for llm inferences,”ACM SIGENERGY Energy Informatics Review, vol. 5, no. 2, pp. 63–68, 2025

    2025

  8. [8]

    Characterizing power management opportunities for llms in the cloud,

    P. Patel, E. Choukse, C. Zhang, ´I. Goiri, B. Warrier, N. Mahalingam, and R. Bianchini, “Characterizing power management opportunities for llms in the cloud,” inProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, 2024, pp. 207–222

    2024

  9. [9]

    Sustainable llm serving: Environmental implica- tions, challenges, and opportunities,

    Y . Ding and T. Shi, “Sustainable llm serving: Environmental implica- tions, challenges, and opportunities,” in2024 IEEE 15th International Green and Sustainable Computing Conference (IGSC). IEEE, 2024, pp. 37–38

    2024

  10. [10]

    A survey on in- ference engines for large language models: Perspectives on optimization and efficiency,

    S. Park, S. Jeon, C. Lee, S. Jeon, B.-S. Kim, and J. Lee, “A survey on in- ference engines for large language models: Perspectives on optimization and efficiency,”preprint arXiv:2505.01658, 2025

    arXiv 2025

  11. [11]

    Llm inference serving: Survey of recent advances and opportunities,

    B. Li, Y . Jiang, V . Gadepally, and D. Tiwari, “Llm inference serving: Survey of recent advances and opportunities,” in2024 IEEE High Performance Extreme Computing Conference (HPEC). IEEE, 2024, pp. 1–8

    2024

  12. [12]

    Efficient memory management for large language model serving with pagedattention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” inProceedings of the 29th symposium on operating systems principles, 2023, pp. 611–626

    2023

  13. [13]

    Huggingface’s trans- formers: State-of-the-art natural language processing,

    T. Wolf, L. Debut, V . Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowiczet al., “Huggingface’s trans- formers: State-of-the-art natural language processing,”arXiv preprint arXiv:1910.03771, 2019

    Pith/arXiv arXiv 1910

  14. [14]

    Using ollama,

    F. S. Marcondes, A. Gala, R. Magalh ˜aes, F. Perez de Britto, D. Dur ˜aes, and P. Novais, “Using ollama,” inNatural Language Analytics with Generative Large-Language Models: A Practical Approach with Ollama and Open-Source LLMs. Springer, 2025, pp. 23–35

    2025

  15. [15]

    ggml-org/llama.cpp,

    “ggml-org/llama.cpp,” Mar. 2026. [Online]. Available: https://github. com/ggml-org/llama.cpp

    2026

  16. [16]

    Seesaw: High-throughput llm inference via model re-sharding,

    Q. Su, W. Zhao, X. Li, M. Andoorveedu, C. Jiang, Z. Zhu, K. Song, C. Giannoula, and G. Pekhimenko, “Seesaw: High-throughput llm inference via model re-sharding,”preprint arXiv:2503.06433, 2025

    Pith/arXiv arXiv 2025

  17. [17]

    Towards green ai: Decoding the energy of llm inference in software development,

    L. Solovyeva and F. Castor, “Towards green ai: Decoding the energy of llm inference in software development,”preprint arXiv:2602.05712, 2026

    arXiv 2026

  18. [18]

    Pimp My LLM: Leveraging Variability Modeling to Tune Inference Hyperparameters,

    N. Zine, C. Quinton, and R. Rouvoy, “Pimp My LLM: Leveraging Variability Modeling to Tune Inference Hyperparameters,” inEASE’26 - 30th International Conference on Evaluation and Assessment in Software Engineering, Glasgow, United Kingdom, Jun. 2026. [Online]. Available: https://hal.science/hal-05567430

    2026

  19. [19]

    Wohlin, P

    C. Wohlin, P. Runeson, M. H ¨ost, M. Ohlsson, B. Regnell, and A. Wessl ´en,Experimentation in Software Engineering, ser. Computer Science. Springer, 2012

    2012

  20. [20]

    Replication package of this study,

    Anonymous, “Replication package of this study,” Aug. 2026. [Online]. Available: https://doi.org/10.5281/zenodo.19253906

    work page doi:10.5281/zenodo.19253906 2026

  21. [21]

    Sustainable ai: Environmental implications, challenges and opportunities,

    C.-J. Wu, R. Raghavendra, U. Gupta, B. Acun, N. Ardalani, K. Maeng, G. Chang, F. Aga, J. Huang, C. Baiet al., “Sustainable ai: Environmental implications, challenges and opportunities,”Proceedings of machine learning and systems, vol. 4, pp. 795–813, 2022

    2022

  22. [22]

    Dy- namollm: Designing llm inference clusters for performance and energy efficiency,

    J. Stojkovic, C. Zhang, ´I. Goiri, J. Torrellas, and E. Choukse, “Dy- namollm: Designing llm inference clusters for performance and energy efficiency,” in2025 IEEE International Symposium on High Perfor- mance Computer Architecture (HPCA). IEEE, 2025, pp. 1348–1362

    2025

  23. [23]

    Llm-pilot: Characterize and opti- mize performance of your llm inference services,

    M. Lazuka, A. Anghel, and T. Parnell, “Llm-pilot: Characterize and opti- mize performance of your llm inference services,” inSC24: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 2024, pp. 1–18

    2024

  24. [24]

    From words to watts: Benchmarking the energy costs of large language model inference,

    S. Samsi, D. Zhao, J. McDonald, B. Li, A. Michaleas, M. Jones, W. Bergeron, J. Kepner, D. Tiwari, and V . Gadepally, “From words to watts: Benchmarking the energy costs of large language model inference,” inIEEE High Performance Extreme Computing Conference (HPEC). IEEE, 2023, pp. 1–9

    2023

  25. [25]

    Power hungry processing: Watts driving the cost of ai deployment?

    S. Luccioni, Y . Jernite, and E. Strubell, “Power hungry processing: Watts driving the cost of ai deployment?” inProceedings of the 2024 ACM conference on fairness, accountability, and transparency, 2024, pp. 85– 99

    2024

  26. [26]

    Investigating energy efficiency and performance trade-offs in llm inference across tasks and dvfs settings,

    P. J. Maliakel, S. Ilager, and I. Brandic, “Investigating energy efficiency and performance trade-offs in llm inference across tasks and dvfs settings,”arXiv preprint arXiv:2501.08219, 2025

    arXiv 2025

  27. [27]

    Offline energy-optimal llm serv- ing: Workload-based energy models for llm inference on heterogeneous systems,

    G. Wilkins, S. Keshav, and R. Mortier, “Offline energy-optimal llm serv- ing: Workload-based energy models for llm inference on heterogeneous systems,”ACM SIGENERGY Energy Informatics Review, vol. 4, no. 5, pp. 113–119, 2024

    2024

  28. [28]

    Sustainable llm inference for edge ai: Evaluating quantized llms for energy efficiency, output accuracy, and inference latency,

    E. J. Husom, A. Goknil, M. Astekin, L. K. Shar, A. K ˚asen, S. Sen, B. A. Mithassel, and A. Soylu, “Sustainable llm inference for edge ai: Evaluating quantized llms for energy efficiency, output accuracy, and inference latency,”arXiv preprint arXiv:2504.03360, 2025

    Pith/arXiv arXiv 2025

  29. [29]

    Understanding efficiency: Quantization, batching, and serving strategies in llm energy use,

    J. Delavande, R. Pierrard, and S. Luccioni, “Understanding efficiency: Quantization, batching, and serving strategies in llm energy use,”arXiv preprint arXiv:2601.22362, 2026

    arXiv 2026

  30. [30]

    Small prompts, big energy and co 2 impact: Benchmarking ollama llms on cpu and gpu,

    A. Kolovska, M. Gusev, and D. Mileski, “Small prompts, big energy and co 2 impact: Benchmarking ollama llms on cpu and gpu,” in2025 33rd Telecommunications Forum (TELFOR). IEEE, 2025, pp. 1–4

    2025

  31. [31]

    A thorough examination of decoding methods in the era of llms, 2024,

    C. Shi, H. Yang, D. Cai, Z. Zhang, Y . Wang, Y . Yang, and W. Lam, “A thorough examination of decoding methods in the era of llms, 2024,” URL https://arxiv. org/abs/2402.06925

    Pith/arXiv arXiv 2024

  32. [32]

    Decoding decoded: Understanding hyperparameter effects in open-ended text generation,

    E. G. Arias, M. Li, C. Heumann, and M. Aßenmacher, “Decoding decoded: Understanding hyperparameter effects in open-ended text generation,” inProceedings of the 31st International Conference on Computational Linguistics, 2025, pp. 9992–10 020

    2025

  33. [33]

    Impact of decoding strategies on gpu energy usage in large language model text generation,

    A. Nik, M. A. Riegler, and P. Halvorsen, “Impact of decoding strategies on gpu energy usage in large language model text generation,”Scientific Reports, 2025

    2025

  34. [34]

    Green my llm: Studying the key factors affecting the energy consumption of code assistants,

    T. Coignion, C. Quinton, and R. Rouvoy, “Green my llm: Studying the key factors affecting the energy consumption of code assistants,”arXiv preprint arXiv:2411.11892, 2024

    Pith/arXiv arXiv 2024

  35. [35]

    The impact of hyperparameters on large language model inference performance: An evaluation of vllm and huggingface pipelines,

    M. Martinez, “The impact of hyperparameters on large language model inference performance: An evaluation of vllm and huggingface pipelines,” inProceedings of the 33rd ACM International Conference on the Foundations of Software Engineering, 2025, pp. 1672–1678

    2025

  36. [36]

    Energy considerations of large language model inference and efficiency optimizations,

    J. Fernandez, C. Na, V . Tiwari, Y . Bisk, S. Luccioni, and E. Strubell, “Energy considerations of large language model inference and efficiency optimizations,”preprint arXiv:2504.17674, 2025

    Pith/arXiv arXiv 2025

  37. [37]

    Benchmarking Energy Efficiency of Large Language Models Using vLLM,

    K. Pronk and Q. Zhao, “Benchmarking Energy Efficiency of Large Language Models Using vLLM,” Sep. 2025

    2025

  38. [38]

    Energy Efficient or Exhaus- tive? Benchmarking Power Consumption of LLM Inference Engines,

    C. Niu, W. Zhang, Y . Zhao, and Y . Chen, “Energy Efficient or Exhaus- tive? Benchmarking Power Consumption of LLM Inference Engines,” SIGENERGY Energy Inform. Rev., vol. 5, no. 2, pp. 56–62, 2025

    2025

  39. [39]

    Shull, J

    F. Shull, J. Singer, and D. I. Sjøberg,Guide to advanced empirical software engineering. Springer, 2007

    2007

  40. [40]

    A. Guldner andet al., “Development and evaluation of a reference measurement model for assessing the resource and energy efficiency of software products and components-green software measurement model (gsmm),”Future Generation Computer Systems, vol. 155, pp. 402–418, 2024

    2024

  41. [41]

    A Beginner’s Guide to Power and Energy Measurement and Estimation for Computing and Machine Learning,

    A. Jagannadharao, N. Beckage, S. Biswas, H. Egan, J. Gafur, T. Metsch, D. Nafus, G. Raffa, and C. Tripp, “A Beginner’s Guide to Power and Energy Measurement and Estimation for Computing and Machine Learning,” Dec. 2024

    2024

  42. [42]

    The tame project: Towards improvement-oriented software environments,

    V . R. Basili and H. D. Rombach, “The tame project: Towards improvement-oriented software environments,”IEEE Transactions on Software Engineering, vol. 14, no. 6, pp. 758–773, 1988

    1988

  43. [43]

    vllm – configuration options,

    “vllm – configuration options,” 2026. [Online]. Available: https: //docs.vllm.ai/en/latest/configuration

    2026

  44. [44]

    Flashattention: Fast and memory-efficient exact attention with io-awareness,

    T. Dao, D. Fu, S. Ermon, A. Rudra, and C. R ´e, “Flashattention: Fast and memory-efficient exact attention with io-awareness,”Advances in neural information processing systems, vol. 35, pp. 16 344–16 359, 2022

    2022

  45. [45]

    Flashattention-2: Faster attention with better parallelism and work partitioning,

    T. Dao, “Flashattention-2: Faster attention with better parallelism and work partitioning,”arXiv preprint arXiv:2307.08691, 2023

    Pith/arXiv arXiv 2023

  46. [46]

    Flashinfer: Efficient and customizable attention engine for llm inference serving,

    Z. Ye, L. Chen, R. Lai, W. Lin, Y . Zhang, S. Wang, T. Chen, B. Kasikci, V . Grover, A. Krishnamurthyet al., “Flashinfer: Efficient and customizable attention engine for llm inference serving,”arXiv preprint arXiv:2501.01005, 2025

    Pith/arXiv arXiv 2025

  47. [47]

    On the suitability of hug- ging face hub for empirical studies,

    A. Ait, J. L. C ´anovas Izquierdo, and J. Cabot, “On the suitability of hug- ging face hub for empirical studies,”Empirical Software Engineering, vol. 30, no. 2, p. 57, 2025

    2025

  48. [48]

    Text Generation Models – Hugging Face — huggingface.co,

    “Text Generation Models – Hugging Face — huggingface.co,” https:// huggingface.co/models?pipeline tag=text-generation&sort=downloads, 2025

    2025

  49. [49]

    Top leaderboard ranking= top coding proficiency, always? evoeval: Evolving coding benchmarks via llm,

    C. S. Xia, Y . Deng, and L. Zhang, “Top leaderboard ranking= top coding proficiency, always? evoeval: Evolving coding benchmarks via llm,” arXiv preprint arXiv:2403.19114, 2024

    Pith/arXiv arXiv 2024

  50. [50]

    Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks,

    Y . Bai, S. Tu, J. Zhang, H. Peng, X. Wang, X. Lv, S. Cao, J. Xu, L. Hou, Y . Donget al., “Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025, pp. 3639–3664

    2025

  51. [51]

    Natural questions: a benchmark for question answering research,

    T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Leeet al., “Natural questions: a benchmark for question answering research,”Transactions of the Association for Computational Linguistics, vol. 7, pp. 453–466, 2019

    2019

  52. [52]

    Lighteval: A lightweight framework for llm evaluation,

    N. Habib, C. Fourrier, H. Kydl ´ıˇcek, T. Wolf, and L. Tunstall, “Lighteval: A lightweight framework for llm evaluation,” 2023. [Online]. Available: https://github.com/huggingface/lighteval

    2023

  53. [53]

    Taming the long-tail: Efficient reasoning rl training with adaptive drafter,

    Q. Hu, S. Yang, J. Guo, X. Yao, Y . Lin, Y . Gu, H. Cai, C. Gan, A. Klimovic, and S. Han, “Taming the long-tail: Efficient reasoning rl training with adaptive drafter,”preprint arXiv:2511.16665, 2025

    arXiv 2025

  54. [54]

    Wildchat: 1m chatgpt interaction logs in the wild,

    W. Zhao, X. Ren, J. Hessel, C. Cardie, Y . Choi, and Y . Deng, “Wildchat: 1m chatgpt interaction logs in the wild,”preprint arXiv:2405.01470, 2024

    Pith/arXiv arXiv 2024

  55. [55]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockmanet al., “Evaluating large language models trained on code,”arXiv preprint arXiv:2107.03374, 2021

    Pith/arXiv arXiv 2021

  56. [56]

    Pareto multi objective optimization,

    P. Ngatchou, A. Zarei, and A. El-Sharkawi, “Pareto multi objective optimization,” inProceedings of the 13th International Conference on, Intelligent Systems Application to Power Systems, 2005, pp. 84–91

    2005

  57. [57]

    The aligned rank transform for nonparametric factorial analyses using only anova procedures,

    J. O. Wobbrock, L. Findlater, D. Gergle, and J. J. Higgins, “The aligned rank transform for nonparametric factorial analyses using only anova procedures,” inProceedings of the SIGCHI conference on human factors in computing systems, 2011, pp. 143–146

    2011

  58. [58]

    A simple sequentially rejective multiple test procedure,

    S. Holm, “A simple sequentially rejective multiple test procedure,” Scandinavian journal of statistics, pp. 65–70, 1979

    1979

  59. [59]

    Dominance statistics: Ordinal analyses to answer ordinal questions

    N. Cliff, “Dominance statistics: Ordinal analyses to answer ordinal questions.”Psychological bulletin, vol. 114, no. 3, p. 494, 1993

    1993

  60. [60]

    A critique and improvement of the cl common language effect size statistics of mcgraw and wong,

    A. Vargha and H. D. Delaney, “A critique and improvement of the cl common language effect size statistics of mcgraw and wong,”Journal of Educational and Behavioral Statistics, vol. 25, no. 2, pp. 101–132, 2000

    2000

  61. [61]

    The Curious Case of Neural Text Degeneration,

    A. Holtzman, J. Buys, L. Du, M. Forbes, and Y . Choi, “The Curious Case of Neural Text Degeneration,” Feb. 2020

    2020

  62. [62]

    vllm – optimization and tuning,

    “vllm – optimization and tuning,” 2026. [Online]. Available: https: //docs.vllm.ai/en/latest/configuration/optimization.html

    2026

  63. [63]

    Guidelines for Empirical Studies in Software Engineering involving Large Language Models,

    S. Baltes, F. Angermeir, C. Arora, M. M. Bar ´on, C. Chen, L. B ¨ohme, F. Calefato, N. Ernst, D. Falessi, B. Fitzgerald, D. Fucci, M. Kalinowski, S. Lambiase, D. Russo, M. Lungu, L. Prechelt, P. Ralph, R. van Tonder, C. Treude, and S. Wagner, “Guidelines for Empirical Studies in Software Engineering involving Large Language Models,” Sep. 2025

    2025

  64. [64]

    Part-time Power Measurements: Nvidia-smi’s Lack of Attention,

    Z. Yang, K. Adamek, and W. Armour, “Part-time Power Measurements: Nvidia-smi’s Lack of Attention,” inSC24: International Conference for High Performance Computing, Networking, Storage and Analysis, Nov. 2024, pp. 1–17

    2024