Pith. sign in

REVIEW 4 major objections 7 minor 54 references

Meta-Metrics and Best Practices for System-Level Inference Performance Benchmarking

T0 review · 4 major / 7 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FMwork shows that a sparse, structured set of LLM inference configurations can replace exhaustive sweeps, cutting benchmarking effort by up to 24x while keeping results within roughly 4% of a full measurement campaign.

desk verdict A useful, well-scoped benchmarking methodology with a real empirical artifact, but the headline 24x/96.6% fidelity claim is a single-configuration, in-sample demonstration rather than a validated general result. read the letter →

arxiv 2508.10251 v1 pith:ERDC7PYR submitted 2025-08-14 cs.PF

Shweta Salaria , Zhuoran Liu , Nelson Mimura Gonzalez This is my paper
classification cs.PF
keywords FMworkLLM inference benchmarkingmeta-metricsparameter selectionpower-of-two samplingcost-performance analysissynthetic workloadsinter-token latency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FMwork sets out to make LLM inference benchmarking tractable by treating the experimental space as something to be sampled strategically rather than exhausted. It introduces meta-metrics that quantify how much time and money a benchmarking campaign costs, and how faithfully a chosen subset of measurements represents a much larger ground-truth space. The core practical claim is that batch sizes chosen as powers of two, combined with an output size of 128 tokens instead of 1024, reproduce a full Llama 3.1 8B sweep on an H100 with a global relative difference around 0.034-0.045, while cutting benchmarking effort by roughly 24x, with a further 2.7x saving from the output-size reduction alone. A sympathetic reader would care because the framework offers a quantitative, reusable way to decide which experiments actually need to be run before committing expensive GPU time.

What carries the argument

The load-bearing object is a pair of meta-metrics: the normalized relative difference delta between ground truth G and predicted set P, and an efficiency factor E = (1 - $\Delta$) / (C_P / C_G) that combines fidelity with cost. Parameter selection is driven by a sampling heuristic—batch sizes as powers of two, with unmeasured values filled by simple linear interpolation—and the observation that output size 128 is sufficient to characterize inter-token latency. Synthetic input and synthetic model generators also carry weight: they let the framework control workload and architecture geometry without depending on a particular benchmark corpus, and the paper reports a geomean difference of only 2.3

What would settle it

Run a full sweep on the same Llama 3.1 8B / H100 / vLLM setup with every batch size from 1 to 128 at output sizes 16, 128, and 1024, then compare power-of-two-subset-plus-linear-interpolation predictions against the measured non-power-of-two batch sizes; if the global relative difference exceeds the reported roughly 0.034-0.045, or if specific regimes such as batch sizes just past GPU saturation show errors above 10%, the representational accuracy claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that benchmarking cost and benchmarking fidelity can be separated, measured, and traded off explicitly. It defines a ground-truth set G of parameter combinations and a measured set M, then builds a predicted set P by interpolating over the gaps; fidelity is scored by a normalized relative difference between G and P, and cost is scored by the time or resources required. Applying this to a Llama 3.1 8B model on a single H100 with vLLM, the paper finds that using only powers of two for batch sizes and filling in the rest by simple linear interpolation keeps the global relative difference at 0.034-0.045 across all input sizes, while shrinking total sweep time from 32

Load-bearing premise

The claim rests on the assumption that performance at unmeasured batch sizes can be recovered by simple linear interpolation between the selected points, and that the chosen ground-truth grid of input sizes, output sizes, and batch sizes faithfully represents the full experimental space; if the performance surface is not smooth—say, at memory-saturation or kernel-transition boundaries—the reported 24x and 2.7x savings will not transfer.

Editorial extensions

If this is right

  • Benchmarking campaigns can shift from exhaustive sweeps to log-spaced sampling: power-of-two batch sizes plus interpolation reproduces full sweeps within a few percent relative difference.
  • Output size can be treated as a tunable cost lever; for TTFT and ITL characterization, 128 tokens is enough, giving a 2.7x saving when moving from 1024 tokens.
  • The efficiency factor E gives a single number for comparing experimental designs, letting teams choose configurations by explicitly trading time and cost against representational accuracy.
  • Synthetic inputs can substitute for real workloads for throughput characterization, with a reported geomean difference of about 2.37%, so controlled experiments need not distort results.
  • The same meta-metric machinery can be applied to cost-performance and parallelism decisions, such as picking tensor-parallel degree based on throughput per device and dollars per million tokens rather than raw throughput.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the performance surface is broadly smooth in batch size, the power-of-two heuristic is likely a symptom of hardware-utilization saturation curves being roughly log-linear; the sparsity may transfer to other backends and models, but should be revalidated wherever kernels or memory layouts change.
  • The linear-interpolation assumption is most fragile at regime transitions such as memory saturation or batch sizes crossing a GPU's parallelism granularity; an adaptive strategy that adds probe points where local curvature is high could make the 24x claim more robust.
  • The meta-metrics could be applied directly to energy-aware benchmarking: the cost term in the efficiency factor could be replaced with watts or carbon per experiment, producing a green-benchmarking analogue of the same trade-off.
  • A testable extension is to use the same delta/E framework to select input sizes, precision levels, and parallelism configurations automatically, not just batch and output sizes, potentially automating the entire experimental design.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper introduces FMwork, a framework for reducing the cost of LLM inference benchmarking while preserving representational fidelity. It defines meta-metrics for experimental cost and accuracy, a ground-truth set G of measured configurations, a measured subset P, and an efficiency metric E. The central parameter-selection strategy is to sample batch sizes at powers of two and linearly interpolate the omitted values. Using Llama 3.1 8B at BF16 on a single H100 with vLLM 0.7.2, the paper reports that this strategy, together with output size 128 rather than 1024, reproduces the full sweep with global relative differences of 0.034–0.045, yielding up to 24x efficiency and an additional 2.7x speedup. The paper also presents a cost-performance analysis for Llama 3.3 70B, synthetic input generation, and synthetic model generation.

Significance. If the headline results hold, the paper addresses a real and important problem: the combinatorial cost of LLM inference benchmarking. The meta-metric framing is useful, and the concrete recipe — power-of-two batch-size sampling with linear interpolation and reduced output size — is simple and potentially actionable. The paper is also reproducible in spirit: it describes the experimental setup in detail and points to an open-source GitHub repository. The arithmetic in Section IV.B is internally consistent with the reported runtimes. However, the central quantitative claims are currently supported only by an in-sample evaluation on one model/GPU/backend, and the abstract misaligns the 96.6% accuracy figure with the 2.7x speedup comparison. These issues do not invalidate the methodology, but they must be addressed before the 24x/96% headline can be taken as a validated property rather than a single case-study demonstration.

major comments (4)
  1. [Section IV.B, Figure 2] The 'powers of two' batch-size rule is selected by inspecting errors computed on the ground-truth set G, and the same G is then used to report the 0.034 and 0.045 relative differences. This is selection on the evaluation set: the reported accuracy is an optimistic in-sample estimate. To support the claim that the rule preserves roughly 96% representational accuracy, the paper should hold out part of G during rule selection (e.g., one or more input sizes, output sizes, or batch-size regions) and report the error on the held-out part, or evaluate on at least one different model/backend/hardware. Without such a validation, the 24x/96% headline is a demonstration on a single performance surface rather than a general best practice.
  2. [Abstract and Section IV.B] The abstract pairs '2.7x gain' with '96.6% accuracy', but these numbers come from different comparisons. In Section IV.B, 96.6% corresponds to 1 - 0.034, the error for the input-size-1024-only sweep with output size 128. The 2.7x speedup, however, is 3483/1301 across all input sizes, where the output-size-128 error is 0.045 (95.5%). The '24x' figure is also an efficiency factor relative to the full output-size-1024 sweep (32270/1301 x (1 - 0.045)), whereas the same-output-size comparison gives about 9x. Please correct the abstract and present the comparisons transparently, stating for each comparison the speedup, the accuracy, and the exact reference set.
  3. [Section III, Eq. (3)] Equation (3) as printed, E(G,P) = 1 - Δ(G,P) / C_P / C_G, is syntactically ambiguous and, under the usual left-to-right reading, does not equal the values computed in Section IV.B, where the authors use (C_G/C_P) x (1 - Δ). The efficiency metric should be written as E = (1 - Δ) / (C_P/C_G), or equivalently E = (1 - Δ) x (C_G/C_P), with C_P and C_G defined explicitly. Since this is the core meta-metric, the formula should be unambiguous and match the numerical results.
  4. [Section IV.B, Tables/Figures 2-3] All headline times, errors, and efficiency factors are reported as point values with no repetitions, confidence intervals, or statement of run-to-run variability. TTFT and ITL measurements on shared or even dedicated GPUs are noisy, and the 2.7x speedup (3483/1301) and the 3.4% error could shift non-negligibly across runs. For a benchmarking-methodology paper, the measurement procedure (number of repeats, warm-up, outlier handling) and ideally confidence intervals on Δ and E should be reported. This is needed to judge whether the reported 24x/96% figures are stable.
minor comments (7)
  1. [Abstract] Typo: 'benchmkaring' should be 'benchmarking'.
  2. [Throughout] The framework name is inconsistently written as 'FMwork' and 'FMWork'; please unify.
  3. [Section IV.B] The input-size sweep is described as 'starting at 128 and incrementing in multiples of 2', but the upper bound is never stated. Please give the exact set of input sizes used to build G.
  4. [Figure 2 caption] The caption 'Columns (26 to 2)' is confusing. Clarify what the column numbers mean and how each subset P is constructed (e.g., total number of sampled batch sizes, and which specific values).
  5. [Section IV.B] The statement 'The same subset of powers of 2 can achieve 0.6% global difference for TTFT projections' is not supported by any table or figure in the text. Indicate where this number comes from or compute it explicitly.
  6. [Section IV.C] Typo in Figure 3: 'bb i.e. concurrency level' should be 'batch size'.
  7. [Section IV.D and IV.E] The 'geomean percentage difference' is used for Figure 4 and Figure 5 but never defined. Define the formula and, ideally, report the per-configuration spread.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the efficiency and accuracy claims are measured against an explicitly defined ground-truth set, with no equation reducing to its own input and no load-bearing self-citation.

full rationale

FMwork's derivation chain is transparent and does not rely on circular reasoning. In Section III, the ground-truth set G is explicitly defined as 'a relatively large subset of the complete experimental space,' and the accuracy metric Δ is computed as the relative difference between G and the interpolated set P (Eq. 2). The efficiency factor E (Eq. 3) combines that measured accuracy with the runtime ratio CP/CG. The headline numbers in Section IV.B—24x efficiency, 2.7x speedup, and 96.6% accuracy—are arithmetic consequences of the measured runtimes (e.g., 32270 s vs 1301 s) and the interpolation errors computed against G; they do not follow by definition from a fitted parameter or from a self-citation. The paper does not invoke any uniqueness theorem, ansatz borrowed from prior work by the same authors, or rename a known result as a new derivation. The in-sample nature of the accuracy evaluation (G is used both to motivate the powers-of-two choice and to measure error) is a legitimate external-validity limitation, but it does not make the reported numbers equivalent to their inputs by construction. Similarly, the slight mismatch between the 96.6% figure (input-size-1024 error 0.034) and the 24x figure (all-input error 0.045) is a presentation inconsistency, not a circular step. The derivations are self-contained under the paper's explicitly scoped conditions (single model, GPU, and backend).

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central quantitative claims rest on the representativeness of G, the smoothness of the metric surface, and the transferability of synthetic workloads; none of these is independently established outside the paper's own data.

free parameters (3)
  • batch-size subset rule = powers of two from 1 to 128
    Hand-selected sampling strategy tested on ground-truth G; the headline accuracy numbers depend on it.
  • output size target = 128 tokens
    Chosen to reduce runtime while preserving accuracy; the 95.5%-96.6% accuracy claims are conditional on this choice.
  • latency threshold for real-time analysis = 50 ms/token
    Chosen as a good reference for interactive use in Section IV.C; used to classify acceptable TP configurations.
assumptions (4)
  • domain assumption Linear interpolation between selected batch sizes accurately predicts unmeasured batch sizes.
    Section IV.B: 'we take a subset of batch sizes from G and interpolate the other values using simple linear regression.' The accuracy claims rely on this smoothness assumption.
  • domain assumption The ground-truth set G is representative of the complete experimental space.
    Section III defines G as 'representing a relatively large subset of the complete experimental space'; all fidelity percentages are relative to G, not to the true full space.
  • domain assumption TTFT does not depend on output size.
    Section IV.B: 'TTFT projections are independent of output size.' Used to justify collecting TTFT with output size 1.
  • domain assumption Synthetic inputs and synthetic models preserve the performance behavior of real workloads and real models.
    Sections IV.D and IV.E use small geomean differences (2.37% for inputs, 10% for models) to claim representativeness; generalizing beyond tested workloads is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Meta-Metrics and Best Practices for System-Level Inference Performance Benchmarking." pith.science (2026). https://pith.science/paper/ERDC7PYR

@misc{pith2026250810251,
  author       = {Pith},
  title        = {Pith review of: Meta-Metrics and Best Practices for System-Level Inference Performance Benchmarking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ERDC7PYR}},
  note         = {Machine review of arXiv:2508.10251}
}
read the original abstract

Benchmarking inference performance (speed) of Foundation Models such as Large Language Models (LLM) involves navigating a vast experimental landscape to understand the complex interactions between hardware and software components. However, evaluating every possible test configuration is impractical, unfeasible and unnecessary. To address this challenge, we introduce FMwork, a comprehensive and methodical approach to creating a controlled testing environment that accurately reflects and characterizes performance. FMwork comprises a set of benchmkaring best practices with three key components: 1) meta-metrics, 2) parameter selection, and 3) strategic cost-performance evaluation. Meta-metrics account for time and resources spent on benchmarking and the relative accuracy of the results compared to a larger body of measurements, representing the complete experimental space. FMwork operationalizes the meta-metrics and provides efficient strategies for parameter selection and cost-performance analysis. Using the framework, we show up to 24x improvement (speedup and/or resource savings) running sweeps of experiments compared to the ground truth. Even already considering a subset of experiments as reference point (using the power of two for batch sizes), reducing experimental output size from 1024 to 128 tokens yields another 2.7x gain while keeping 96.6% accuracy for an evaluation using Llama 3.1 8B model.

Figures

Figures reproduced from arXiv: 2508.10251 by the authors.

Figure 1
Figure 1. FMwork : A structured approach to LLM inference [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Colored cells indicate relative difference (error; the [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Cost-performance analysis of llama-3.3-70b. Tables [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Performance comparison between synthetic and real [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Throughput (in sequences processed per second) for [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Throughput (in sequences processed per second) of encoder models with different geometries. [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Incremental factor of throughput of encoder models [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

54 extracted references · 36 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Vidur: A large-scale simulation framework for llm inference

    Amey Agrawal, Nitin Kedia, Jayashree Mohan, Ashish Panwar, Nipun Kwatra, Bhargav Gulavani, Ramachan- dran Ramjee, and Alexey Tumanov. Vidur: A large-scale simulation framework for llm inference. Proceedings of Machine Learning and Systems , 6:351–366, 2024

    work page 2024

  3. [3]

    The llama 4 herd: The beginning of a new era of natively multimodal ai innovation, April 2025

    Meta AI. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation, April 2025. Accessed: 2025-04-05

    work page 2025

  4. [4]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints

    Joshua Ainslie, James Lee-Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebr ´on, and Sumit Sang- hai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245, 2023

    arXiv 2023

  5. [5]

    Claude: A breakthrough ai assistant based on constitutional ai

    Anthropic. Claude: A breakthrough ai assistant based on constitutional ai. https://www.anthropic.com/claude.html,

    work page

  6. [6]

    On the opportunities and risks of foundation models

    Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bern- stein, Jeannette Bohg, Antoine Bosselut, Emma Brun- skill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 , 2021

    arXiv 2021

  7. [7]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  8. [8]

    Llm-inference-bench: Inference benchmark- ing of large language models on ai accelerators

    Krishna Teja Chitty-Venkata, Siddhisanket Raskar, Bharat Kale, Farah Ferdaus, Aditya Tanikanti, Ken Raf- fenetti, Valerie Taylor, Murali Emani, and Venkatram Vishwanath. Llm-inference-bench: Inference benchmark- ing of large language models on ai accelerators. In SC24-W: Workshops of the International Conference for High Performance Computing, Networking,...

    work page 2024

Show all 54 references
  1. [9]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research , 24(240):1–113, 2023

    2023

  2. [10]

    Flashattention-2: Faster attention with bet- ter parallelism and work partitioning

    Tri Dao. Flashattention-2: Faster attention with bet- ter parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023

    2023 arXiv

  3. [11]

    The case for 4-bit precision: k-bit inference scaling laws

    Tim Dettmers and Luke Zettlemoyer. The case for 4-bit precision: k-bit inference scaling laws. In International Conference on Machine Learning , pages 7750–7774. PMLR, 2023

    2023

  4. [12]

    Bert: Pre-training of deep bidirec- tional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. In Pro- ceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human l...

    2019

  5. [13]

    A comprehensive evaluation of novel ai accelerators for deep learning workloads

    Murali Emani, Zhen Xie, Siddhisanket Raskar, Varuni Sastry, William Arnold, Bruce Wilson, Rajeev Thakur, Venkatram Vishwanath, Zhengchun Liu, Michael E Papka, et al. A comprehensive evaluation of novel ai accelerators for deep learning workloads. In 2022 IEEE/ACM international...

    2022

  6. [14]

    Open llm leaderboard

    Hugging Face. Open llm leaderboard. https://huggingface.co/spaces/HuggingFaceH4/open llm leaderboard, 2023. Accessed: 2023-05-15

    2023

  7. [15]

    Granite 3.0 language models, 2024

    IBM Granite Team. Granite 3.0 language models, 2024

    2024

  8. [16]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  9. [17]

    Gpipe: Ef- ficient training of giant neural networks using pipeline parallelism

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Ef- ficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019

    2019

  10. [18]

    Quantized neural net- works: Training neural networks with low precision weights and activations

    Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Quantized neural net- works: Training neural networks with low precision weights and activations. journal of machine learning research, 18(187):1–30, 2018

    2018

  11. [19]

    Text generation inference: A toolkit for efficient text generation with transformers

    Hugging Face. Text generation inference: A toolkit for efficient text generation with transformers. https:// github.com/huggingface/text-generation-inference, 2023. Accessed: 2025-03-09

    2023

  12. [20]

    LLM inference at scale with TGI, September 2024

    Martin Iglesias Goyanes. LLM inference at scale with TGI, September 2024. Accessed: 2025-03-09

    2024

  13. [21]

    Mixtral of experts

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, De- vendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024

    2024 arXiv

  14. [22]

    Echoswift: An inference benchmarking and configuration discovery tool for large language models (llms)

    Karthik Krishna and Ramana Bandili. Echoswift: An inference benchmarking and configuration discovery tool for large language models (llms). In Companion of 11 the 15th ACM/SPEC International Conference on Per- formance Engineering, pages 158–162, 2024

    2024

  15. [23]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, p...

    2023

  16. [24]

    A comprehensive study on quantization techniques for large language models

    Jiedong Lang, Zhehao Guo, and Shuyu Huang. A comprehensive study on quantization techniques for large language models. In 2024 4th International Conference on Artificial Intelligence, Robotics, and Communication (ICAIRC), pages 224–231. IEEE, 2024

    2024

  17. [25]

    Llm-pilot: Characterize and optimize performance of your llm inference services

    Malgorzata Lazuka, Andreea Anghel, and Thomas Par- nell. Llm-pilot: Characterize and optimize performance of your llm inference services. In SC24: International Conference for High Performance Computing, Network- ing, Storage and Analysis , pages 1–18. IEEE, 2024

    2024

  18. [26]

    Bart: Denois- ing sequence-to-sequence pre-training for natural lan- guage generation, translation, and comprehension

    Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denois- ing sequence-to-sequence pre-training for natural lan- guage generation, translation, and comprehension. arXiv preprint arXiv:1910.13461, 2019

    1910 arXiv

  19. [27]

    Efficient llms training and inference: An introduc- tion

    Rui Li, Deji Fu, Chunyu Shi, Zhilan Huang, and Gang Lu. Efficient llms training and inference: An introduc- tion. IEEE Access, 2024

    2024

  20. [28]

    {AlpaServe}: Statistical multiplexing with model parallelism for deep learning serving

    Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E Gonzalez, et al. {AlpaServe}: Statistical multiplexing with model parallelism for deep learning serving. In 17th USENIX Symposium on Op- erating Systems ...

    2023

  21. [29]

    Awq: Activation- aware weight quantization for on-device llm compression and acceleration

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei- Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation- aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87–100, 2024

    2024

  22. [30]

    Deepseek-v2: A strong, econom- ical, and efficient mixture-of-experts language model

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, econom- ical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434 , 2024

    2024 arXiv

  23. [31]

    Efficient large-scale language model training on gpu clusters using megatron-lm

    Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedin...

    2021

  24. [32]

    Mastering llm techniques: Inference optimization

    NVIDIA. Mastering llm techniques: Inference optimization. https://developer.nvidia.com/blog/ mastering-llm-techniques-inference-optimization/,

  25. [33]

    TensorRT-LLM: A TensorRT toolbox for large language models

    NVIDIA. TensorRT-LLM: A TensorRT toolbox for large language models. https://github.com/NVIDIA/ TensorRT-LLM, 2023. Accessed: 2025-03-09

    2023

  26. [34]

    Accessed: 2023-03-09

    2023

  27. [35]

    Efficiently scaling transformer inference

    Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems, 5:606–624, 2023

    2023

  28. [36]

    Pytorch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zem- ing Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019

    2019

  29. [37]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research , 21(140):1–67, 2020

    2020

  30. [38]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019

    2019

  31. [39]

    Zero: Memory optimizations toward train- ing trillion parameter models

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward train- ing trillion parameter models. In SC20: International Conference for High Performance Computing, Network- ing, Storage and Analysis , pages 1–16. IEEE, 2020

    2020

  32. [40]

    Deepspeed-moe: Advanc- ing mixture-of-experts inference and training to power next-generation ai scale

    Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. Deepspeed-moe: Advanc- ing mixture-of-experts inference and training to power next-generation ai scale. In International conference on machine lear...

    2022

  33. [41]

    Scott Gardner, Itay Hubara, Sachin Idgunji, Thomas B

    Vijay Janapa Reddi, Christine Cheng, David Kanter, Peter Mattson, Guenther Schmuelling, Carole-Jean Wu, Brian Anderson, Maximilien Breughe, Mark Charlebois, William Chou, Ramesh Chukka, Cody Coleman, Sam Davis, Pan Deng, Greg Diamos, Jared Duke, Dave Fick, J. Scott Gardner, It...

    2019

  34. [42]

    Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 3...

    2020

  35. [43]

    From words to watts: Benchmarking the energy costs of large language model inference

    Siddharth Samsi, Dan Zhao, Joseph McDonald, Baolin Li, Adam Michaleas, Michael Jones, William Bergeron, Jeremy Kepner, Devesh Tiwari, and Vijay Gadepally. From words to watts: Benchmarking the energy costs of large language model inference. In 2023 IEEE High Performance Extrem...

    2023

  36. [44]

    Mlperf inference benchmark

    Vijay Janapa Reddi, Christine Cheng, David Kanter, Peter Mattson, Guenther Schmuelling, Carole-Jean Wu, Brian Anderson, Maximilien Breughe, Mark Charlebois, William Chou, et al. Mlperf inference benchmark. In 2020 ACM/IEEE 47th Annual International Symposium on Computer Archit...

    2020

  37. [45]

    Megatron-lm: Training multi-billion parameter language models using model parallelism

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019

    1909 arXiv

  38. [46]

    Flashattention-3: Fast and accurate attention with asynchrony and low-precision

    Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention-3: Fast and accurate attention with asynchrony and low-precision. Advances in Neural Information Processing Systems , 37:68658–68685, 2024

    2024

  39. [47]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023

    2023 arXiv

  40. [48]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Bap- tiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023

    2023 arXiv

  41. [49]

    Sup- porting very large models using automatic dataflow graph partitioning

    Minjie Wang, Chien-chin Huang, and Jinyang Li. Sup- porting very large models using automatic dataflow graph partitioning. In Proceedings of the Fourteenth EuroSys Conference 2019, pages 1–17, 2019

    2019

  42. [50]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017

    2017

  43. [51]

    Trans- formers: State-of-the-art natural language processing

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chau- mond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R ´emi Louf, Morgan Funtowicz, et al. Trans- formers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical meth- o...

    2020

  44. [52]

    Exploring activation patterns of parame- ters in language models

    Yudong Wang, Damai Dai, Zhe Yang, Jingyuan Ma, and Zhifang Sui. Exploring activation patterns of parame- ters in language models. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 39, pages 25416–25424, 2025

    2025

  45. [54]

    Megascale-infer: Serving mixture-of-experts at scale with disaggregated expert parallelism

    Ruidong Zhu, Ziheng Jiang, Chao Jin, Peng Wu, Cesar A Stuardo, Dongyang Wang, Xinlei Zhang, Huaping Zhou, Haoran Wei, Yang Cheng, et al. Megascale-infer: Serving mixture-of-experts at scale with disaggregated expert parallelism. arXiv preprint arXiv:2504.02263 , 2025. 13

    2025 arXiv

  46. [2023]

    [Accessed: 2023-05-15]

    2023

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.