Pith. sign in

REVIEW 4 major objections 5 minor 51 references

GPT-OSS-20B: A Comprehensive Deployment-Centric Analysis of OpenAI's Open-Weight Mixture of Experts Model

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a 20.9B-parameter mixture-of-experts model, GPT-OSS-20B, outperforms dense models Qwen3-32B and Yi-34B in single-GPU decode throughput, memory, and energy at a 2,048-token context, at the cost of a higher time-to-firs

desk verdict Routine but honest benchmark study; new numbers for GPT-OSS-20B, but the MoE-vs-dense conclusion is only true for the eager transformers harness, not production. read the letter →

arxiv 2508.16700 v2 pith:LA7SLZTY submitted 2025-08-22 cs.AR cs.AIcs.DCcs.PF

Deepak Kumar , Divakar Yadav , Yash Patel This is my paper
classification cs.ARcs.AIcs.DCcs.PF
keywords mixtureofexpertsLLMinferencesingle-GPUdeploymentdecodethroughputenergyefficiencypeakVRAMactiveparameterGPT-OSS-20B
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a 20.9B-parameter mixture-of-experts model, GPT-OSS-20B, is a better single-GPU deployment choice than two dense models of comparable total size, Qwen3-32B and Yi-34B, when the goal is throughput, memory, and energy rather than accuracy. On one H100 in bf16 at a 2,048-token context with 64 generated tokens, the authors measure about 32% higher decode throughput and 26% lower energy per 1,000 generated tokens than Qwen3-32B, alongside roughly 32% lower peak VRAM, at the cost of a higher time to first token. The comparison matters because production inference is often constrained by exactly these resources, and MoE models are supposed to pay memory and energy costs only for parameters that are actually used. The paper does not evaluate accuracy; it is a deployment-focused study.

What carries the argument

The load-bearing object is the sparse activation pattern of the MoE architecture: only about 3.61B of the 20.9B parameters are active per token, so the model's weight footprint and per-token compute are closer to a 3.6B dense model than a 20.9B one. To make this advantage visible, the paper introduces Active Parameter Efficiency (APE), which normalizes throughput, energy, and time-to-first-token by active parameter count; under this lens GPT-OSS-20B delivers roughly 12 times higher throughput per active billion parameters than the dense baselines. The measurement protocol also matters: exact post-template context lengths, a GPU-memory allocator peak that keeps the persistent key-value cache

What would settle it

Run the same three models on the same H100 with an optimized serving engine and measure decode throughput, energy per 1,000 generated tokens, and peak VRAM at a 2,048-token context with 64 generated tokens. If Qwen3-32B or Yi-34B match or beat GPT-OSS-20B under that engine, the reported advantage is explained by the unoptimized harness; if GPT-OSS-20B still leads, the architecture-level conclusion stands.

Watch

Extended reading notes

Core claim

The central claim is that activating only 3.61B of GPT-OSS-20B's 20.9B parameters during inference yields concrete deployment advantages: at a fixed 2,048-token context and 64-token decode, measured decode throughput is 31.27 tokens/s versus 23.73 for Qwen3-32B and 26.30 for Yi-34B; tokens per Joule are about 0.10 versus 0.08 and 0.07; energy per 1,000 generated tokens is 9,764 J versus 13,155 J and 13,464 J; and peak VRAM is roughly 43.5 GB versus 63.7 GB and 66.5 GB. These advantages come with a higher true time-to-first-token (459.7 ms versus 369.5 ms and 368.3 ms), which the authors attribute to MoE routing overhead. The paper does not evaluate accuracy; it is a deployment-focused study.

Load-bearing premise

The comparison assumes that running all three models through the same minimal eager deep-learning harness on one GPU, with no optimized serving engine, is a fair and representative deployment comparison; if framework overhead dominates the absolute numbers, the measured gaps may be artifacts of the harness rather than the architectures.

Editorial extensions

If this is right

  • A practitioner serving a model on a single GPU in bf16 can expect a MoE checkpoint of this total size to use roughly a third less peak memory and a quarter less energy per generated token than a dense 30B-class model.
  • The higher time-to-first-token means interactive, prefill-heavy workloads may still prefer dense models, while throughput-bound batch generation favors the MoE.
  • The per-active-parameter efficiency gap suggests sparsity, not just total parameter count, should enter deployment cost models.
  • At longer contexts, GPT-OSS-20B's throughput advantage shrinks: it is roughly 40% below its 512-token baseline at 4K context, so context-length assumptions should be part of the deployment decision.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the reported absolute throughput of about 31 tokens/s for only 3.6B active parameters on an H100 is far below hardware capability, so framework and routing overhead dominate; the MoE-versus-dense ordering could change under an optimized serving engine that overlaps expert computation.
  • Because only GPT-OSS-20B is MoE in this comparison, the claim is about this model against these two dense baselines, not a general MoE-versus-dense theorem; testing other MoE checkpoints would show whether the advantage transfers.
  • The paper's energy numbers come from a coarse device-level sampler on short runs; direct measurement with an instrumented power meter or longer runs could shift the absolute J/1K values, although the relative direction would likely persist.
  • A natural extension is to add task-accuracy evaluations at matched deployment settings, since deployment efficiency without task quality is only half of the model-selection problem.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents a single-GPU (H100, bf16) deployment-focused comparison of the open-weight MoE model GPT-OSS-20B against dense baselines Qwen3-32B and Yi-34B. It reports true TTFT, decode throughput (TPOT), end-to-end latency percentiles, peak VRAM with persistent KV cache, and energy (tokens/J, J/1K) under controlled post-template contexts. The central claim is that at 2,048-token context with 64 generated tokens, GPT-OSS-20B delivers higher decode throughput and tokens per Joule and lower peak VRAM and energy per 1,000 tokens than the dense baselines, while having higher TTFT. The paper also introduces an Active Parameter Efficiency (APE) normalization and includes decoding, context-length, precision, and serving-stack ablations. Code and consolidated results are released.

Significance. If the central measurements hold, the paper provides a useful, reproducible data point on single-GPU deployment trade-offs between a 20B-class MoE and 30B-class dense models. The strengths are the exact post-template context control, the use of allocator peak memory with PKV held, normalized energy reporting, and the release of code/data. The APE metric, while definitional, offers a clear way to contextualize throughput by active parameters. However, the broader conclusion that MoE models are more viable in production single-GPU settings rests critically on the representativeness of the eager PyTorch/transformers harness and on the internal consistency of the measurements; both currently require major qualification.

major comments (4)
  1. [§3.1, Table 1, Table 12, §5] The central deployment claim is based entirely on a minimal eager PyTorch/transformers harness. The absolute TPOT for GPT-OSS-20B (31.27 tok/s for ~3.61B active parameters on an H100) is far below what the hardware should sustain, indicating that framework and dispatch overhead dominate the measurement. The paper explicitly excludes vLLM (Table 12) and argues that minimal scripts avoid framework-induced variability, but relative rankings across models can still depend on implementation overhead, especially because MoE routing and many small experts incur more Python/dispatch cost than dense models of similar active size. The conclusion that GPT-OSS-20B is more deployment-viable than dense baselines is therefore not established beyond this specific eager stack. The authors should either add an optimized-engine comparison (e.g., vLLM, TensorRT-LLM) or substantially qualify the conclusion t
  2. [§4.1, Table 2, Table 10, Table 9] The reported throughput numbers are internally inconsistent. Table 2 gives GPT-OSS-20B TPOT of 31.27 tok/s at 2048 context and 38.18 at 512; Table 10 gives p50 tok/s of 30.25 at 2048 and 36.29 at 512 for the same model under greedy decoding. Table 9 lists a greedy p50 of 39.45 tok/s without stating the context length; if this is the same 2048/64 condition, it conflicts with both Table 2 and Table 10. These discrepancies make it difficult to know which numbers are canonical and undermine the robustness claims. The authors must unify the protocol and clarify why the same condition yields different medians across tables.
  3. [§3.5, Table 6] The energy measurements are central to the paper's claims (25–38% improvements in tokens/J and J/1K), but they are based on nvidia-smi sampling around short runs, with only medians of 5 runs and no confidence intervals, standard deviations, or per-run distributions. The paper itself notes nvidia-smi is coarse. Without uncertainty estimates, the reader cannot assess whether the reported energy differences are significant. At minimum, the authors should report per-run variance and, ideally, validate the sampling methodology against a higher-fidelity power monitor (e.g., EIT). This is a load-bearing gap for the energy efficiency conclusion.
  4. [§3.3, §4.1] The methodology includes an ad-hoc correction: 'When a TPOT value is suspiciously equal to 1/TTFT, we correct it by recomputing from the measured decode wall time.' TTFT includes prefill, so its reciprocal is not a valid throughput; the need for such a correction suggests some reported values were initially miscomputed. The manuscript should disclose which runs were corrected, show the raw and corrected values, and justify the correction rule. As written, this undermines trust in the reliability of the TPOT tables.
minor comments (5)
  1. [§3.5] The text says energy is sampled 'before/after each decode run' and also 'instantaneous power ... averaged across the run'. These are not the same; please clarify the exact sampling procedure and whether the before/after readings are used to estimate the run-average power.
  2. [Table 6, §4.2] Units are inconsistent: the table header says 'Tokens/W' while the text uses 'tokens per Joule'. Please standardize to Joules throughout and ensure the energy formulas match the stated units.
  3. [Table 9] The table caption says 'ctx fixed' but does not state the context length. Since throughput varies strongly with context (Table 2), the context must be specified for comparison with other tables.
  4. [§4.1, Table 3] The claim that GPT-OSS-20B's higher TTFT is 'due to MoE routing overhead' is not supported by any profiling or kernel-level evidence. The TTFT scaling across contexts is also steeper for GPT-OSS than for the dense models, which may indicate prefill implementation differences rather than routing alone. Please soften the attribution or provide supporting data.
  5. [§3.3] The term 'true TTFT' is not standard; define it clearly. Also, reporting p50/p95 over 'N runs' without specifying N in the text makes the confidence in the estimates unclear.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: benchmark claims are direct measurements; APE is a stated normalization, not a fitted prediction.

full rationale

This paper is an empirical deployment benchmark, not a derivation. Its central claims—higher decode throughput, lower energy per 1,000 tokens, and lower peak VRAM for GPT-OSS-20B versus dense baselines—are direct measurements reported in Tables 1, 5, and 6 under a consistent H100/bf16 harness. There is no fitted parameter that is subsequently renamed as a prediction, and no load-bearing self-citation: the references are to external work, and no uniqueness theorem or ansatz is imported from the authors' prior publications. The Active Parameter Efficiency (APE) metric in Section 3.6 is explicitly a normalization ('APE-TPOT = TPOT / Active Params (B)', 'APE-Energy = Tokens/J / Active Params (B)'), so its values are deterministic arithmetic transformations of the raw measurements; the paper presents APE as complementing raw deployment metrics rather than as the sole evidence for its conclusions. The acknowledged limitations—nvidia-smi sampling coarseness (Section 3.5), short-run energy estimates (Section 4.2 caveats), and the absence of vLLM runs (Table 12)—affect the generalizability of the empirical comparison, but they do not make any result circular. Therefore no specific circular step can be quoted, and the appropriate score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters are fitted; the 'Active Parameter Efficiency' metric is a definitional ratio, not a fitted quantity. The key assumptions are about the measurement harness and power sampling being fair and representative. No new physical entities are proposed.

assumptions (3)
  • domain assumption nvidia-smi instantaneous power sampling is representative enough for comparing energy across the three models.
    Stated in Section 3.5 and in the Caveats of Section 4.2; the authors acknowledge the sampling is approximate and micro-bursts are not captured.
  • domain assumption Identical transformers generate() harness with eager mode is a fair, apples-to-apples deployment setting for all three models.
    Section 3.1 and 3.3; the low absolute TPOT suggests harness overhead dominates, yet the relative comparisons are built on this assumption.
  • domain assumption Trimming or padding prompts to the exact post-template context length equalizes prefill cost across models.
    Section 3.2; assumes padded tokens have negligible compute cost, which is plausible but not verified in the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GPT-OSS-20B: A Comprehensive Deployment-Centric Analysis of OpenAI's Open-Weight Mixture of Experts Model." pith.science (2026). https://pith.science/paper/LA7SLZTY

@misc{pith2026250816700,
  author       = {Pith},
  title        = {Pith review of: GPT-OSS-20B: A Comprehensive Deployment-Centric Analysis of OpenAI's Open-Weight Mixture of Experts Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LA7SLZTY}},
  note         = {Machine review of arXiv:2508.16700}
}
read the original abstract

We present a single-GPU (H100, bf16) evaluation of GPT-OSS-20B (Mixture-of-Experts; 20.9B total, approx. 3.61B active) against dense baselines Qwen3-32B and Yi-34B across multiple dimensions. We measure true time-to-first-token (TTFT), full-decode throughput (TPOT), end-to-end latency percentiles, peak VRAM with past key values (PKV) held, and energy via a consistent nvidia-smi-based sampler. At a 2048-token context with 64-token decode, GPT-OSS-20B delivers higher decode throughput and tokens per Joule than dense baselines Qwen3-32B and Yi-34B, while substantially reducing peak VRAM and energy per 1000 generated tokens; its TTFT is higher due to MoE routing overhead. With only 17.3% of parameters active (3.61B of 20.9B), GPT-OSS-20B provides about 31.8% higher decode throughput and 25.8% lower energy per 1000 generated tokens than Qwen3-32B at 2048/64, while using 31.7% less peak VRAM. Normalized by active parameters, GPT-OSS-20B shows markedly stronger per-active-parameter efficiency (APE), underscoring MoE's deployment advantages. We do not evaluate accuracy; this is a deployment-focused study. We release code and consolidated results to enable replication and extension.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

51 extracted references · 33 canonical work pages

  1. [1]

    Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., & Dean, J. (2017). Outra- geously large neural networks: The sparsely-gated mixture-of-experts layer . International Conference on Learning Representations (ICLR)

    work page 2017

  2. [2]

    OpenAI. (2025). Introducing GPT-OSS. OpenAI Blog. Retrieved fromhttps://openai.com/index/ introducing-gpt-oss/

    work page 2025

  3. [3]

    OpenAI. (2025). GPT-OSS Model Card . Retrieved from https://cdn.openai.com/pdf/ 419b6906-9da6-406c-a19d-1bb078ac7637/oai_gpt-oss_model_card.pdf

    work page 2025

  4. [4]

    Liu, Y., Zhang, H., Chen, X., & others. (2023). A survey of resource-efficient large language models . arXiv preprint arXiv:2312.00678

    arXiv 2023

  5. [5]

    J., Cheng, C., Coleman, D., Kanter, D., Mattson, P., Schmuelling, C., & others

    Reddi, V. J., Cheng, C., Coleman, D., Kanter, D., Mattson, P., Schmuelling, C., & others. (2020).MLPerf Inference Benchmark. Proceedings of Machine Learning and Systems (MLSys)

    work page 2020

  6. [6]

    Q., Sablayrolles, A., Mensch, A., & others

    Jiang, A. Q., Sablayrolles, A., Mensch, A., & others. (2023). Mistral 7B . arXiv preprint arXiv:2310.06825

    arXiv 2023

  7. [7]

    Qwen Team. (2025). Qwen3 Technical Report. arXiv preprint arXiv:2505.09388

    arXiv 2025

  8. [8]

    01.AI. (2024). Yi: Open Foundation Models by 01.AI. arXiv preprint arXiv:2403.04652

    arXiv 2024

Show all 51 references
  1. [9]

    Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., & Chen, Z. (2021). GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. In Proceedings of the 9th International Conference on Learning Representations (ICLR 2021)

    2021

  2. [10]

    (2021).Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

    Fedus, W., Zoph, B., & Shazeer, N. (2021).Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. arXiv preprint arXiv:2101.03961

    2021 arXiv

  3. [11]

    Q., Sablayrolles, A., Roux, A., & others

    Jiang, A. Q., Sablayrolles, A., Roux, A., & others. (2024). Mixtral of Experts . arXiv preprint arXiv:2401.04088

    2024 arXiv

  4. [12]

    (2024).DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models

    Dai, D., Shao, S., Zhang, Y., & others. (2024).DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models. arXiv preprint arXiv:2401.06066

    2024 arXiv

  5. [13]

    xAI. (2024). Grok-1 Model Card. Retrieved from https://x.ai/news/grok/model-card

    2024

  6. [14]

    Y., Ermon, S., Rudra, A., & R ´e, C

    Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & R ´e, C. (2022). FlashAttention: Fast and Memory- Efficient Exact Attention with IO-Awareness . Advances in Neural Information Processing Systems (NeurIPS 2022). Retrieved fromhttps://proceedings.neurips.cc/paper_files/paper/2022/ ha...

    2022

  7. [15]

    H., Gonzalez, J

    Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention . In Proceedings of the 29th ACM Symposium on Operating Systems Principles (SOSP ’23...

    2023

  8. [16]

    (2023).LLaMA: Open and Efficient Foundation Language Models

    Touvron, H., Lavril, T., Izacard, G., & others. (2023).LLaMA: Open and Efficient Foundation Language Models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  9. [17]

    (2022).ST-MoE: Designing Stable and Transferable Sparse Expert Models

    Zoph, B., Shazeer, N., et al. (2022).ST-MoE: Designing Stable and Transferable Sparse Expert Models. Advances in Neural Information Processing Systems (NeurIPS 2022)

    2022

  10. [18]

    Y., Awan, A

    Rajbhandari, S., Li, C., Yao, Z., Zhang, M., Aminabadi, R. Y., Awan, A. A., Rasley, J., & He, Y. (2022). DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale. Proceedings of the International Conference on Machine Learning (ICML)...

    2022

  11. [19]

    Cai, W., Jiang, J., Wang, F., Tang, J., Kim, S., & Huang, J. (2024). A Survey on Mixture of Experts in Large Language Models. arXiv preprint arXiv:2407.06204

    2024 arXiv

  12. [20]

    (2025).Unveiling Super Experts in Mixture-of- Experts Large Language Models

    Su, Z., Li, Q., Zhang, H., Qian, Y., Xie, Y., & Yuan, K. (2025).Unveiling Super Experts in Mixture-of- Experts Large Language Models. arXiv preprint arXiv:2507.23279

    2025

  13. [21]

    Huang, W., Liao, J., Liu, J., He, J., Tan, H., Zhang, J., Li, J., Liu, J., & Qi, X. (2025). Mixture Compressor for Mixture-of-Experts LLMs Gains More. Proceedings of the International Conference on Learning Representations (ICLR)

    2025

  14. [22]

    Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., & others. (2024). A survey on evaluation of large language models . ACM Transactions on Intelligent Systems and Technology, 15(3), 1-45

    2024

  15. [23]

    Saleh, Y., Abu Talib, M., Nasir, Q., & Dakalbab, F. (2025). Evaluating large language models: a systematic review of efficiency, applications, and future directions . Frontiers in Computer Science, 7, 1523699

    2025

  16. [24]

    Poddar, S., Koley, P., Misra, J., Ganguly, N., & Ghosh, S. (2025). Towards Sustain- able NLP: Insights from Benchmarking Inference Energy in Large Language Models . In Proc. NAACL-HLT 2025 (Long Papers), 12688–12704. Association for Computational Linguistics. https://doi.org/1...

    2025 doi

  17. [25]

    (2024).Characterizing Power Management Opportunities for LLMs in the Cloud

    Patel, P., Choukse, E., Zhang, C., Shah, A., Goiri,´I., Maleki, S., & Bianchini, R. (2024).Characterizing Power Management Opportunities for LLMs in the Cloud. In Proc. ASPLOS 2024. ACM. (Shows power behavior vs. input length up to 8k tokens and distinct prompt/token phases)

    2024

  18. [26]

    (2024).Splitwise: Effi- cient Generative LLM Inference Using Phase Splitting

    Patel, P., Choukse, E., Zhang, C., Shah, A., Goiri,´I., Maleki, S., & Bianchini, R. (2024).Splitwise: Effi- cient Generative LLM Inference Using Phase Splitting. In Proc. ISCA 2024. IEEE/ACM. (Demonstrates phase-specific power and Perf/W improvements.)

    2024

  19. [27]

    Wilkins, G., Keshav, S., & Mortier, R. (2024). Offline Energy-Optimal LLM Serving: Workload-Based Energy Models for LLM Inference on Heterogeneous Systems . In Proc. ACM HotCarbon 2024. ACM. (Models energy vs. input/output tokens.)

    2024

  20. [28]

    Y., Ermon, S., Rudra, A., & R´e, C

    Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & R´e, C. (2023).FlashAttention-2: Faster Attention with Better Parallelism. ICLR 2024 (Poster). Retrieved fromhttps://openreview.net/forum?id=mZn2Xyh9Ec 12

    2023

  21. [29]

    Shah, J., Bikshandi, G., Zhang, Y., Thakkar, V., Ramani, P., & Dao, T. (2024). FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision . In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), 2024

    2024

  22. [30]

    Maliakel, P. J. (2025). Investigating Energy Efficiency and Performance Trade-offs in LLM Inference Across Tasks and DVFS Settings. arXiv preprint arXiv:2501.08219

    2025

  23. [31]

    (2025).Energy Considerations of Large Language Model Inference and Efficiency Optimizations

    Fernandez, J., Na, C., Tiwari, V., Bisk, Y., Luccioni, S., & Strubell, E. (2025).Energy Considerations of Large Language Model Inference and Efficiency Optimizations. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    2025

  24. [32]

    (2025).Energy costs of communicating with AI

    Dauner, M., & Socher, G. (2025).Energy costs of communicating with AI. Frontiers in Communication, 10, 1523. doi:10.3389/fcomm.2025.01523

    2025

  25. [33]

    Ye, Z., Zhao, Y., Zhao, Y., & others. (2025). FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving . Proceedings of Machine Learning and Systems (MLSys 2025). Retrieved from https://homes.cs.washington.edu/˜arvind/papers/flashinfer.pdf

    2025

  26. [34]

    NVIDIA. (2024). TensorRT-LLM Developer Guide. Retrieved from https://nvidia.github.io/ TensorRT-LLM/

    2024

  27. [35]

    Dettmers, T., Lewis, M., Shleifer, S., & Zettlemoyer, L. (2022). LLM.int8(): 8-bit Matrix Multi- plication for Transformers at Scale . Advances in Neural Information Processing Systems (NeurIPS 2022). Retrieved from https://proceedings.neurips.cc/paper_files/paper/2022/hash/ c...

    2022

  28. [36]

    Frantar, E., et al. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pretrained Transformers. Advances in Neural Information Processing Systems (NeurIPS 2022)

    2022

  29. [37]

    Xiao, X., Wei, S., Chen, Y., & others. (2023). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. International Conference on Machine Learning (ICML 2023). PMLR v202. Retrieved from https://proceedings.mlr.press/v202/xiao23c.html

    2023

  30. [38]

    Lin, J., Tang, J., Han, S., & others. (2024). AWQ: Activation-aware Weight Quan- tization for On-Device LLMs . Proceedings of Machine Learning and Systems (MLSys 2024). Retrieved from https://proceedings.mlsys.org/paper_files/paper/2024/hash/ 42a452cbafa9dd64e9ba4aa95cc1ef21-A...

    2024

  31. [39]

    Lin, J., Zhao, S., Chen, X., & Han, S. (2025). QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving . Proceedings of Machine Learning and Systems (MLSys 2025). Retrieved from https://mlsys.org/virtual/2025/poster/3288

    2025

  32. [40]

    & Lespiau, J

    Hoffmann, J., Borgeaud, S., Mensch, A., Sifre, L., Cai, T., Rutherford, D., ... & Lespiau, J. B. (2022). Training Compute-Optimal Large Language Models. In Advances in Neural Information Processing Sys- tems (NeurIPS 2022). Retrieved from https://proceedings.neurips.cc/paper_f...

    2022

  33. [41]

    I., Jeong, E., Park, J., & others

    Yu, G. I., Jeong, E., Park, J., & others. (2022). Orca: A Distributed Serving System for Transformer- Based Generative Models. OSDI 2022. Retrieved from https://www.usenix.org/system/files/ osdi22-yu.pdf 13

    2022

  34. [42]

    Diakun, O., & Czarnul, P. (2025). Investigation of CUDA Graphs Performance for Selected Parallel Applications. In Computational Science – ICCS 2025, Lecture Notes in Computer Science, vol. 389, pp. 130–137. Springer. Retrieved from https://doi.org/10.1007/978-3-031-97635-3_16

    2025 doi

  35. [43]

    Yang, Z., Ad ´amek, K., & Armour, W. (2024). A Detailed Study of NVIDIA GPU’s Built-In Power Sensor. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC 2024). IEEE. doi:10.1109/SC41406.2024.00028

    2024 arXiv

  36. [44]

    Henderson, P., Hu, J., Romoff, J., Brunskill, E., Jurafsky, D., & Pineau, J. (2020). Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning . Journal of Machine Learning Research, 21(248), 1–43

    2020

  37. [45]

    Pope, D., Smith, J., Lee, A., & Johnson, M. (2023). Efficient Benchmarking of LLM Inference . In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC 2023). IEEE. doi:10.1145/3581784.3607051

    2023

  38. [46]

    A., Ivanov, S., Oseledets, I

    Budennyy, S. A., Ivanov, S., Oseledets, I. V., & Zhukov, L. E. (2022). eco2AI: Carbon Emissions Tracking of Machine Learning Models. Doklady Mathematics, 106(6), 338–342. Retrieved fromhttps: //link.springer.com/article/10.1134/S1064562422060230

    2022 doi

  39. [47]

    Leviathan, Y., Kalman, M., Matias, Y., & Dean, J. (2023). Fast Inference from Transformers via Speculative Decoding . In Proceedings of the 40th International Conference on Machine Learning (ICML 2023). PMLR 202, 19274-19286. Retrieved fromhttps://proceedings.mlr.press/v202/ l...

    2023

  40. [48]

    Liu, Y., Li, X., Wang, Z., Chen, J., & Zhang, H. (2024). Speculative Decoding via Early-Exiting for Faster LLM Inference with Thompson Sampling Control Mechanism. In Findings of the Association for Computational Linguistics: ACL 2024, pages 2345–2357. Association for Computati...

    2024

  41. [49]

    Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P., Glaese, A., Balle, B., Kasirzadeh, A., Biles, C., & others. (2021). Ethical and social risks of harm from language models. In Advances in Neural Information Processing Systems (NeurIPS 2021). Retrieved fr...

    2021 arXiv

  42. [50]

    A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M., Bohg, J., Bosselut, A., Brunskill, E., & others

    Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M., Bohg, J., Bosselut, A., Brunskill, E., & others. (2021). On the Opportunities and Risks of Foundation Models. Journal of Machine Learning Research, 22(1), 1–199. Retrieved from https://...

    2021 arXiv

  43. [51]

    M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A

    Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A. W., Firat, O., Zoph, B., & others. (2022). GLaM: Efficient Scaling of Language Models with Mixture-of-Experts. In Proceedings of the 39th International Conference on Machine Learning (I...

    2022

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.