Pith. sign in

REVIEW 5 major objections 6 minor 21 references

Meeting SLOs, Slashing Hours: Automated Enterprise LLM Optimization with OptiKIT

T0 review · 5 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read OptiKIT is a distributed pipeline that automatically quantizes, benchmarks, and tunes LLMs for deployment, reporting more than 2x per-GPU throughput in production with near full-precision quality.

desk verdict A well-built production pipeline, but the Benchmarker's steady-state equation is backwards, so the headline throughput gains cannot be trusted until that detector is fixed and the experiments re-run. read the letter →

arxiv 2601.20408 v2 pith:QDMZQQAD submitted 2026-01-28 cs.DC cs.AI

Nicholas Santavas , Kareem Eissa , Patrycja Cieplicka , Piotr Florek , Matteo Nulli , Stefan Vasilev , Seyyed Hadi Hashemi , Antonios Gasteratos
show 1 more author
Shahram Khadivi
This is my paper · ORCID
classification cs.DCcs.AI
keywords LLMoptimizationquantizationSLO-drivenbenchmarkingBayesianhyperparametertuningGPUthroughputautomatedpipelineenterprisedeploymentcalibrationdata
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

OptiKIT is a distributed pipeline that takes a raw large-language model and, with no optimization expert in the loop, returns a quantized model plus a runtime configuration that meets the application's latency targets. The paper's central claim is that automating the whole optimization lifecycle—calibration, quantization, statistical quality checks, SLO-driven load testing, and runtime tuning—is enough to make enterprise-grade LLM deployment routine: in production it reports more than 2x per-GPU throughput improvement over a default FP16 serving setup, with FP8 and INT8 models recovering over 99% of full-precision benchmark accuracy on average. This matters because the real bottleneck for enterprise AI is not GPU hardware alone but the scarce craft knowledge needed to compress and tune models; a system that removes that dependency changes how many teams can ship LLM features within a fixed compute budget.

What carries the argument

The load-bearing machinery is a staged actor-pool pipeline on a distributed cluster, whose three core subsystems form the actual optimization loop. The Optimizer is a backend-agnostic, recipe-based compression engine: a recipe is a declarative specification (int w8a8, int w4a16, fp8 dynamic) bundling a quantization scheme, calibration-data requirements, and layer-selection policy, and each trial draws its own calibration subset to capture sampling variance. The Benchmarker sweeps candidate request rates with exponential search and accepts a rate only if the fitted linear regression of completion timestamps on arrival timestamps has slope $\beta$ near 1, $|\beta - 1| \le \tau_\beta$, meaning the queue is neither emptying nor backing up; this is the mechanism that decides what "SLO-compliant throughput" means. The Tuner wraps this measurement in a Bayesian search (tree-structured Parzen estimators) over runtime parameters such as tensor-parallel size, max concurrency, and max batched tokens, optimizing per-GPU throughput with a large negative penalty for SLO violations. The loop returns a model artifact plus a tuned configuration, and the actor-pool design destroys and re-creates pools between stages to reclaim GPUs deterministically.

What would settle it

Run an independent closed-loop load test at the rates OptiKIT reports as SLO-compliant—for instance Mistral Small 3 24B with FP8 at tensor-parallelism 4, sustained for several hours with realistic arrival variability—and check whether p95 latency stays under 1500 ms and queue length stays flat; or re-run the full sweep with the stability tolerance $\tau_\beta$ set to 0.01 and 0.1 and compare the reported per-GPU gains. If the gains collapse or the latency targets break, the steady-state detector is the load-bearing assumption.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that model compression and serving optimization compose cleanly into a single automated workflow, and that the combined workflow is what unlocks the gains. Individually, quantization buys some throughput and tuning buys some throughput, but the end-to-end configuration—measured by OptiKIT's own SLO-compliant per-GPU throughput—reaches gains such as 2.87x for Mistral Small 3 24B on a latency-p95 workload and 2.13x for Qwen 2.5 7B, while also making previously infeasible tensor-parallel configurations meet their latency targets. The statistical claim is that a generic calibration dataset is sufficient: across 7B, 24B, and 70B models, FP8 Dynamic and INT W8A8 quantization stay within about 1% of full-precision scores on GSM8K, IFEval, and Do-Not-Answer, whereas INT W4A16 is workable but more variable. The paper frames the result as democratization: a non-expert team submits a model and gets a deployable artifact, not a report of recommendations.

Load-bearing premise

The reported throughput gains rest on the Benchmarker's steady-state detector, which accepts a trial as stable when the fitted slope $\beta$ lies within a hand-chosen tolerance (typically 0.02–0.05), and the paper does not validate that detector against an independent load generator or report how the 2x gains depend on that tolerance.

Editorial extensions

If this is right

  • A non-expert team can go from a raw model to a production deployment configuration in roughly 15–25 hours of automated pipeline time, versus the 80–100 person-hours the paper estimates for manual optimization.
  • The same GPU fleet can serve more than twice the traffic per GPU, or serve a larger model within an existing resource envelope, while keeping near-full-precision quality.
  • A single generic calibration dataset is enough for FP8/INT8 quantization across at least the 7B, 24B, and 70B families tested, so calibration-data curation ceases to be a per-model expert task.
  • Latency-critical workloads benefit most from the tuning stage: configurations that fail strict SLOs at low tensor-parallelism become feasible after tuning and/or quantization, giving operators cheaper paths to meet latency targets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the headline gain should be tested for sensitivity to the stability tolerance $\tau_\beta$: re-running the sweep with $\tau_\beta$ set to 0.01 and 0.1 would show whether the reported 2x is a fragile artifact of the detector or a stable property of the configurations.
  • The paper reports that tuning gains are largest under strict SLOs; this suggests a natural routing rule where latency-critical interactive workloads go through the full pipeline, while bulk throughput workloads may need quantization only, which would save pipeline time.
  • The sequential synchronization barrier between stages, which the paper flags as a limitation, is the obvious next scalability lever; removing it becomes even more important if the same framework extends to pruning and distillation, since the joint search space grows combinatorially.
  • Because every tuning trial is measured against synthetic load, the paper's configurations are certified only against that pattern; a shadow-deployment validation step under real arrival variability would be a direct, testable extension of the claim.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper presents OptiKIT, an end-to-end distributed framework for automated LLM optimization developed at eBay. OptiKIT automates model compression through quantization recipes, statistical quality evaluation, regression-based inference benchmarking under SLOs, and Bayesian runtime tuning, and it is integrated with enterprise infrastructure via Ray-based actor pools. The authors report that OptiKIT achieves more than 2x per-GPU throughput improvement over a default FP16 vLLM baseline while preserving near-full-precision quality on GSM8K, IFEval, and Do-Not-Answer, and that it cuts engineering time from 80-100 to 15-25 hours per model. The paper also states that the system is open-sourced.

Significance. If the claims hold, OptiKIT would be a valuable contribution to production LLM serving. It addresses a real operational bottleneck, combines quantization and serving-parameter tuning in a single automated workflow, evaluates on externally recognized quality benchmarks, and reports trial-level variance for statistical quality. The architecture is described in enough detail to be a useful reference for similar systems, and the authors explicitly acknowledge open questions such as long-context fidelity and domain-tuned models. However, the headline throughput claim is supported by only two of four reported workloads, and the benchmark acceptance criterion contains a direction inconsistency that calls into question every SLO-compliant throughput number. The strengths here are real, but the central claim cannot be taken at face value without resolving that issue.

major comments (5)
  1. [Section 4.3, Eq. (1), Figure 4] The overload criterion is algebraically inverted as written. With r_i and c_i the arrival and completion timestamps, fitting r_i = α + β c_i gives β ≈ λ_s / λ_a under rate mismatch (assuming r_i ≈ i/λ_a and c_i ≈ i/λ_s), so queue growth corresponds to β < 1, whereas the paper states that β > 1 denotes overload. Under the acceptance rule |β−1| ≤ τβ with τβ ∈ [0.02, 0.05], a mildly overloaded trial with β ≈ 0.96 would be certified as steady state. Since every SLO-compliant rate in Tables 5 and 7 is selected by this detector, the reported throughput gains cannot be validated until the regression direction is corrected (or the definitions of r_i and c_i are swapped) and the experiments are rerun; the paper should also report how the headline gains vary with τβ.
  2. [Abstract, Section 7.1, Table 5] The unqualified claim that OptiKIT 'in production delivers more than 2x GPU throughput improvement' is not supported by the paper's own data. Table 5 reports end-to-end gains of 2.13x for Qwen, 2.87x for Mistral with the p95 latency SLO, 1.55x for Mistral with the TTFT/TPOT SLO, and 1.25x for Llama 3 70B. Only two of the four workloads exceed 2x, so the abstract and conclusion should be rephrased to state the full range and to explicitly acknowledge the workloads that do not meet the 2x threshold.
  3. [Section 5.4, Appendix B, Tables 6-7] The throughput results are reported as point estimates without replication counts, error bars, or confidence intervals. Given that the benchmark acceptance depends on a hand-set stability tolerance, the paper should report the number of repeated measurements, the variance across runs, and the sensitivity of the per-GPU gains to the choice of τβ. Without this information, the reader cannot assess whether the reported 1.25x-2.87x range is statistically distinguishable from the baseline or whether the gains are robust to small changes in the benchmark criterion.
  4. [Section 6, Figure 1] The human-hour comparison ('80-100 hours manual vs 15-25 hours OptiKIT') is presented as a headline result in Figure 1 and in the title, but it is only described as 'estimated on internal data.' The estimation procedure, sample size, task scope, and uncertainty are not specified. The paper should either provide the underlying data and a counting methodology or clearly downgrade the claim to an illustrative estimate so that the reader can judge its reliability.
  5. [Section 5.3, Table 3, Discussion] The claim of 'near full-precision performance' with 'average recovery rates exceeding 99%' obscures notable per-task degradations: Llama 3.3 70B drops to 95.4% recovery on Do-Not-Answer and Mistral 24B INT8 drops to 93.5% on IFEval. If 95% recovery is considered production-ready, the paper should state the acceptance threshold explicitly, since the text elsewhere describes these results as robust and production-ready without defining what level of quality degradation is acceptable.
minor comments (6)
  1. [Section 5.3] There is a duplicated sentence fragment: 'Mistral exhibited the greatest degradation on the IFEval task ... compared to the full-precision counterpart. as well as the full-precision counterpart.' This should be cleaned up.
  2. [Section 4.2] The sentence 'The StatEval package (Figures 2, 3) package is a core component' repeats the word 'package'; this should be corrected to 'The StatEval package (Figures 2, 3) is a core component.'
  3. [Algorithm 1] The terms 'asynchronous closed-loop trial' and 'asynchronous open-loop trial' are used without definitions, and the update rule 'LB←E[latency−1]' is unclear. These should be explained in the text so the reader can understand the search logic.
  4. [Appendix B, Table 6] Table 6 is not referenced in the main text, while Table 7 is referenced in Section 5.4. The authors should either cite Table 6 where it is relevant or remove it to avoid confusion.
  5. [References] The calibration dataset is cited as 'Magic, 2024' with a URL, but the in-text citation should use the full author/name ('Neural Magic, 2024') consistently, as is done elsewhere.
  6. [Abstract and Section 7.1] The paper claims that the system is open-sourced, but no repository URL, artifact identifier, or release information appears anywhere in the manuscript. For a systems paper whose reproducibility argument rests on the open-source claim, this omission should be fixed.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: throughput and quality claims are measured against external baselines; no self-citation chain or fit-to-prediction reduction.

full rationale

The derivation chain in OPTIKIT is empirical rather than analytic: the claimed >2x per-GPU throughput improvement (Section 5.4, Tables 5-7) is obtained by comparing FP16 default-vLLM baselines against quantized/tuned configurations, with quality recovery checked on external benchmarks GSM8K, IFEval, and Do-Not-Answer (Section 4.2, Table 3). No target quantity is defined in terms of an input parameter, no fitted parameter is renamed as a prediction, and the paper does not rely on self-citations: references to compression techniques (GPTQ, SmoothQuant, RTN) and calibration data (Neural Magic) are external. The Benchmarker's steady-state regression (Eq. 1) is an internal measurement instrument; using it to select SLO-compliant rates is an operational choice, not a circular derivation. A separate concern, that the interpretation of beta > 1 as overload in Eq. 1 appears algebraically inverted under standard queueing and that the tolerance tau_beta is hand-set, is a measurement-validity and correctness risk rather than a circularity, because the reported gains still depend on external baselines and external quality benchmarks. The paper also discloses open limitations (e.g., long-context fidelity not evaluated, future work on global scheduling), which further supports a non-circular reading. Therefore no circularity step meets the evidentiary bar.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

All measurements of the headline throughput gains are produced by the system's own Benchmarker and depend on hand-set tolerances and workload specifications. The quality results rest on standard open benchmarks and the external Neural Magic calibration dataset. No new theoretical entities are introduced.

free parameters (5)
  • τβ steady-state tolerance = 0.02-0.05
    Section 4.3: a trial is stable if |β−1|≤τβ; the tolerance is hand-set depending on noise and sampling granularity, directly affecting the sustainable throughput reported.
  • λ SLO penalty coefficient = -1000
    Equation 2: λ applies a large negative penalty for SLO violations; this choice influences which configurations the tuner selects.
  • context buffer factor = 1.15
    Section 4.4: max context size is computed as (input_len + output_len) × 1.15; this heuristic affects memory allocation and thus achievable concurrency.
  • calibration sample counts = 256 (W8A8), 512 (W4A16), 5 trials
    Section 5.2: five independent trials with 256 or 512 random calibration samples; these are experimental choices that influence measured recovery ratios.
  • human-hours estimates = 80-100 manual vs 15-25 automated
    Figure 1 and Section 6: the claimed engineering-time savings are 'estimated on internal data' with no methodology, yet they anchor the ROI argument.
assumptions (5)
  • domain assumption The workload patterns in Table 2 are representative of production inference for eBay's applications.
    Section 5.1 and Table 2: the measured gains depend on these input/output token ratios and latency SLOs; different workloads could yield different gains.
  • standard math The linear regression r_i = α + β c_i + ε_i captures steady-state behavior.
    Equation 1: the Benchmarker uses β≈1 to detect stability; this is a modeling assumption about queue behavior, not a proven equivalence.
  • domain assumption vLLM with default settings is a valid baseline and the Benchmarker's load generation is an accurate proxy for production traffic.
    Section 5.4 and Appendix B: all throughput comparisons measure SLO compliance under synthetic loads generated by the Benchmarker itself.
  • ad hoc to paper The Neural Magic calibration dataset (Magic, 2024) generalizes to these models and tasks.
    Section 5.2: the INT calibrations use the external default dataset; the paper raises the open question of whether domain-aligned data would be needed, acknowledging this assumption in Section 6.
  • domain assumption Results on H100 GPUs transfer to the heterogeneous infrastructure (H200, A100) mentioned in Figure 3.
    Section 5.1: experiments run only on H100, yet the framework is claimed to manage heterogeneous clusters; no cross-hardware validation is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Meeting SLOs, Slashing Hours: Automated Enterprise LLM Optimization with OptiKIT." pith.science (2026). https://pith.science/paper/QDMZQQAD

@misc{pith2026260120408,
  author       = {Pith},
  title        = {Pith review of: Meeting SLOs, Slashing Hours: Automated Enterprise LLM Optimization with OptiKIT},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QDMZQQAD}},
  note         = {Machine review of arXiv:2601.20408}
}
read the original abstract

Enterprise LLM deployment faces a critical scalability challenge: organizations must optimize models systematically to scale AI initiatives within constrained compute budgets, yet the specialized expertise required for manual optimization remains a niche and scarce skillset. This challenge is particularly evident in managing GPU utilization across heterogeneous infrastructure while enabling teams with diverse workloads and limited LLM optimization experience to deploy models efficiently. We present OPTIKIT, a distributed LLM optimization framework that democratizes model compression and tuning by automating complex optimization workflows for non-expert teams. OPTIKIT provides dynamic resource allocation, staged pipeline execution with automatic cleanup, and seamless enterprise integration. In production, it delivers more than 2x GPU throughput improvement while empowering application teams to achieve consistent performance improvements without deep LLM optimization expertise. We share both the platform design and key engineering insights into resource management, pipeline orchestration, and integration patterns that enable large-scale, production-grade democratization of model optimization. Finally, we open-source the system to enable external contributions and broader reproducibility.

Figures

Figures reproduced from arXiv: 2601.20408 by the authors.

Figure 1
Figure 1. OPTIKIT time and throughput gains. The top figure shows the engineering time saved in model optimization through OPTIKIT vs human hours. In the bottom figure the optimal TPS (Transactions Per Second i.e., Throughput) after the OPTIKIT cycle has terminated vs the baseline TPS. We report results on three model families. Human hours are estimated on internal data. dependencies on a small pool of experts. In this paper,… view at source ↗
Figure 2
Figure 2. OPTIKIT full pipeline. The figure shows the full OPTIKIT flow. We begin by fetching any base/instruct model along with calibration data if needed, and apply model compression through the user selected technique. We then proceed to perform a statistical evaluation of the optimized model to ensure the validity of our compression strategy. If the performance is up to standards, we determine the set of parameter space f… view at source ↗
Figure 3
Figure 3. OPTIKIT system architecture. The figure illustrates the modular orchestration of distributed LLM optimization workflows. The central orchestration layer manages workflow submission, resource allocation, and experiment tracking via Ray Actors, integrating with external data and model sources (HDFS, MMS, EMS) and underlying heterogeneous Ray clusters (H200, H100, A100 nodes). Supporting libraries for compression, benc… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Regression diagnostics for a Benchmarker trial. A fitted slope β ≈ 1 (green) indicates steady-state operation queuing while β > 1 (red) denotes an overload regime. act as critical proxies of e-commerce production metrics for rapid experimentation cycles. These benchmar…
Figure 5
Figure 5. Figure 5: Total OPTIKIT Runtime per Model. We show for each model family the total optimization flow time. Mistral Small 3 24B has two usage scenarios, respectively with 3k/0.2k and 1.5k/1.5k input/output sizes. For Qwen 2.5 7B, performance degradation was mini￾mal—typically bel…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 4 canonical work pages

  1. [4]

    Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J

    URL https://arxiv.org/ abs/2408.04323. Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training Verifiers to Solve Math Word Problems,

    arXiv

  2. [6]

    Values are normalized per-GPU RPS (SLO- compliant)

    Normalized per-GPU throughput forFP16 tuningacross tensor parallelism levels. Values are normalized per-GPU RPS (SLO- compliant). Gains are shown vs. FP16 baseline. Missing baselines ( ∗∗) indicate configurations not measured or not SLO-compliant. TP Baseline (FP16) FP16 (Tuned) Gain (Tuned / Baseline) Qwen 2.5 7B (Input 1200, Output 80, Latency P95 500 m...

    work page 2000

  3. [7]

    Han, S., Mao, H., and Dally, W. J. Deep compres- sion: Compressing deep neural networks with pruning, trained quantization and huffman coding.arXiv preprint arXiv:1510.00149,

    arXiv

  4. [8]

    co/datasets/neuralmagic/ LLM-compression-calibration

    URL https://huggingface. co/datasets/neuralmagic/ LLM-compression-calibration. Micikevicius, P., Narang, S., Alben, J., Diamos, G., Elsen, E., Garc´ıa, J., Micikevicius, V ., Mirza, M., Subramanian, S., and Zhu, H. Fp8 formats for deep learning.arXiv preprint arXiv:2209.05433,

    arXiv

  5. [9]

    Mistral AI Team

    URL https:// arxiv.org/abs/2104.08378. Mistral AI Team. Mistral Small 3,

    arXiv

  6. [10]

    Meeting SLOs, Slashing Hours: Automated Enterprise LLM Optimization with OptiKIT NVIDIA Corporation

    Accessed: 2024-04-15. Meeting SLOs, Slashing Hours: Automated Enterprise LLM Optimization with OptiKIT NVIDIA Corporation. Tensorrt-llm: High-performance inference for large language models. https: //developer.nvidia.com/tensorrt-llm,

    work page 2024

  7. [11]

    Outliers and calibration sets have diminishing ef- fect on quantization of modern llms.arXiv preprint arXiv:2405.20835,

    Paglieri, D., Dash, S., Rockt¨aschel, T., and Parker-Holder, J. Outliers and calibration sets have diminishing ef- fect on quantization of modern llms.arXiv preprint arXiv:2405.20835,

    arXiv

  8. [12]

    A survey on inference engines for large language mod- els: Perspectives on optimization and efficiency.arXiv preprint arXiv:2505.01658,

    Park, S., Jeon, S., Lee, C., Jeon, S., Kim, B.-S., and Lee, J. A survey on inference engines for large language mod- els: Perspectives on optimization and efficiency.arXiv preprint arXiv:2505.01658,

    arXiv

Show all 21 references
  1. [13]

    Sun, M., Liu, Z., Bair, A., and Kolter, J

    URL https: //arxiv.org/abs/2412.15115. Sun, M., Liu, Z., Bair, A., and Kolter, J. Z. A simple and effective pruning approach for large language models. InThe Twelfth International Conference on Learning Representations,

    arXiv

  2. [14]

    Model compression and efficient inference for large language models: A survey

    Wang, W., Chen, W., Luo, Y ., Long, Y ., Lin, Z., Zhang, L., Lin, B., Cai, D., and He, X. Model compression and efficient inference for large language models: A survey. arXiv preprint arXiv:2402.09748, 2024a. Wang, Y ., Li, H., Han, X., Nakov, P., and Baldwin, T. ”do- not-answ...

    arXiv

  3. [15]

    and Aletras, N

    Williams, M. and Aletras, N. On the impact of calibration data in post-training quantization and pruning.arXiv preprint arXiv:2311.09755,

    arXiv

  4. [16]

    Zhang, Y ., Zhao, L., Lin, M., Sun, Y ., Yao, Y ., Han, X., Tanner, J., Liu, S., and Ji, R

    URL https://arxiv.org/abs/2504.15303. Zhang, Y ., Zhao, L., Lin, M., Sun, Y ., Yao, Y ., Han, X., Tanner, J., Liu, S., and Ji, R. Dynamic sparse no training: Training-free fine-tuning for sparse llms.arXiv preprint arXiv:2310.08915,

    arXiv

  5. [17]

    Taming the titans: A survey of efficient llm inference serving.arXiv preprint arXiv:2504.19720,

    Zhen, R., Li, J., Ji, Y ., Yang, Z., Liu, T., Xia, Q., Duan, X., Wang, Z., Huai, B., and Zhang, M. Taming the titans: A survey of efficient llm inference serving.arXiv preprint arXiv:2504.19720,

    arXiv

  6. [18]

    Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y ., Zhou, D., and Hou, L

    URL https://arxiv.org/abs/2312.07104. Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y ., Zhou, D., and Hou, L. Instruction-Following Evaluation for Large Language Models,

    arXiv

  7. [19]

    A survey on effi- cient inference for large language models.arXiv preprint arXiv:2404.14294,

    Zhou, Z., Ning, X., Hong, K., Fu, T., Xu, J., Li, S., Lou, Y ., Wang, L., Yuan, Z., Li, X., et al. A survey on effi- cient inference for large language models.arXiv preprint arXiv:2404.14294,

    arXiv

  8. [21]

    FP16 baseline across models, tensor parallelism, and bitwidths

    Normalized per-GPU throughput and improvement vs. FP16 baseline across models, tensor parallelism, and bitwidths. Values are normalized per-GPU RPS (SLO-compliant). Missing baselines (∗∗) indicate configurations not measured or not SLO-compliant. TP Bitwidth Baseline (FP16) Qu...

    2000

  9. [2020]

    Chavan, A., Magazine, R., Kushwaha, S., Debbah, M., and Gupta, D

    URL https:// arxiv.org/abs/2005.14165. Chavan, A., Magazine, R., Kushwaha, S., Debbah, M., and Gupta, D. Faster and lighter llms: A survey on current challenges and way forward.arXiv preprint arXiv:2402.01799,

    2005 arXiv

  10. [2021]

    Spqr: A sparse-quantized representation for near-lossless llm weight compression.arXiv preprint arXiv:2306.03078,

    Dettmers, T., Svirschevski, R., Egiazarian, V ., Kuznedelev, D., Frantar, E., Ashkboos, S., Borzunov, A., Hoefler, T., and Alistarh, D. Spqr: A sparse-quantized representation for near-lossless llm weight compression.arXiv preprint arXiv:2306.03078,

    arXiv

  11. [2023]

    Gptq: Accurate post-training quantization for generative pre- trained transformers.arXiv preprint arXiv:2210.17323,

    Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre- trained transformers.arXiv preprint arXiv:2210.17323,

    arXiv

  12. [2024]

    URLhttps://arxiv.org/abs/2407.21783. AI, R. H. and vLLM Project. LLM Compres- sor, 8

    arXiv

  13. [2025]

    Brown, T

    URL https: //arxiv.org/abs/2505.09388. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-V oss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter...

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.