Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

iServe: An Intent-based Serving System for LLMs

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read iServe claims LLM serving can be automated by profiling a lightweight fingerprint instead of the full model, then automatically choosing the deployment configuration that best matches the developer's stated intent.

desk verdict Genuinely novel fingerprint-based profiling for LLM serving, but the headline throughput/latency claims rest on a batch-size-1 evaluation that needs a continuous-batching baseline before they are credible. read the letter →

arxiv 2501.13111 v1 pith:FYAZNP5H submitted 2025-01-08 cs.SE cs.LG

Dimitrios Liakopoulos , Tianrui Hu , Prasoon Sinha , Neeraja J. Yadwadkar This is my paper · ORCID
classification cs.SEcs.LG
keywords intent-basedservingLLMinferencemodelfingerprintingdeploymentconfigurationsearchprofilingcostload-awareplacementparallelismquantization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

iServe aims to remove the manual, expert step of choosing how an LLM is deployed. The paper's core claim is that the latency and memory of a decoder-only LLM can be estimated across hundreds of deployment configurations by profiling a "fingerprint" — the same architecture with at most two hidden layers — because both metrics scale linearly with layer count. If true, a serving platform could automatically pick the parallelism, quantization, and pruning combination that best matches a developer's intent (lower latency, lower cost, or a target for either) without expensive full-model profiling. The paper reports that this reduces latency by 77.62%, SLO violations by 7.09x, and profiling GPU-hours by 6.05x compared to prior systems.

What carries the argument

The fingerprint is a reconstructed model that keeps the embedding layer, hidden layers, normalization layers, linear layers, and softmax of the original but with at most two hidden layers, so it is much lighter: Llama-2-70B's fingerprint is 41.59x smaller and 9.75x faster. The argument is carried by two decompositions. Memory splits into weights ($W$), activations ($A$), and pipeline activations ($A_P$) whose replication under tensor and pipeline parallelism is known, and latency splits into time-to-first-token and time-per-output-token, each decomposed into per-hidden-layer and "other" components. Observing the fingerprint under three parallelism configurations supplies enough equations to solve for the memory components, and observing two output lengths on two fingerprint sizes supplies enough equations to solve for the latency components; linear scaling then predicts every configuration of the full LLM.

What would settle it

Run a full model with 4, 8, 16, 32, and 64 layers under a fixed configuration and compare measured latency and memory to the linear extrapolation from the one- and two-layer fingerprints; if the residual grows nonlinearly or changes sign across layer counts, the extrapolation is unsound. A quicker check is a mixture-of-experts model whose per-layer behavior differs, since the linearity claim assumes homogeneous layers.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is a fingerprint-based serving pipeline: a system that profiles a one- or two-layer stand-in of an LLM under a few configurations, solves small systems of equations to recover per-layer memory and latency components, extrapolates to the full model, and then greedily ranks configurations by the user's intent and maps the chosen deployment onto available GPUs. The mechanism rests on the empirical observation that memory and latency grow linearly with the number of hidden layers, which the paper validates on six decoder-only LLMs. With this, iServe reports that it best meets user intent across varied workloads, cutting latency by 77.62%, cost by 86.70%, SLO violations by 7.09x, and profiling cost by 6.05x compared to state-of-the-art baselines.

Load-bearing premise

The claim collapses if latency and memory do not actually scale linearly with the number of identical hidden layers, so that a one- or two-layer fingerprint mispredicts the full LLM's behavior.

Editorial extensions

If this is right

  • Developers could declare intent (minimize latency or cost, or meet a target for either) and get a deployment choice automatically, instead of hand-searching hundreds of configurations.
  • Profiling for a new LLM could run on spare GPUs in the serving cluster rather than dedicated hardware, because fingerprints are small and fast enough to avoid disrupting inference.
  • The system can adapt placement to fluctuating load, packing LLMs onto fewer GPUs at low load and spreading them at high load to reduce latency.
  • SLO attainment improves substantially when latency SLOs are given, and cost, memory, and GPU-hours intents can all be optimized from the same profile data.
  • The same estimated metrics support multiple intents simultaneously, so one profiling run yields the configuration map for latency, cost, memory, and GPU-hour objectives.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the linear scaling with layer count generalizes beyond the six decoder-only LLMs tested, the fingerprint idea could extend to mixture-of-experts or vision-language models, though those architectures would need extra profiling for their non-identical layers.
  • The intent interface suggests a natural next step: adding accuracy as a first-class intent, which the paper explicitly leaves as future work.
  • A testable consequence of the method is that estimation error should stay roughly constant in absolute terms as models grow, since errors for smaller models are a larger fraction of their footprint; the paper's error analysis already points in that direction.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. iServe is an LLM serving system that automates deployment configuration selection across parallelism, quantization, and pruning. The central idea is an 'LLM fingerprint': a reconstruction of a model with 1-2 hidden layers that is profiled on a few configurations, after which latency and memory are linearly extrapolated to the full model. A deployment controller ranks configurations by user intent (minimize latency, cost, memory, GPU-hours, or meet SLO targets) and a load-aware placement policy assigns models to GPUs. The evaluation, on an 8-GPU A6000 cluster with six LLMs and Azure traces, reports a 77.62% latency reduction, 7.09x fewer SLO violations, 4.72x higher GPU throughput, and 6.05x lower profiling cost relative to TensorRT-LLM static-parallelism baselines, Accelerate, and AlpaServe.

Significance. If the claims hold, iServe addresses a real pain point: the cost of profiling deployment configurations for LLMs. The fingerprint idea is elegant, and the paper ships a substantial empirical study: design-space exploration of profiling methods (L1/L2, M1/M2/M3), placement policies, multiple intents, two Azure traces at four load levels, and profiling accuracy measured against ground-truth full-LLM measurements. The profiling error is low (4.91% latency, 6.92% memory), and the evaluation is not circular: fingerprint predictions are checked against independent full-LLM measurements. However, the contribution's central comparative claims are currently restricted to an unrepresentative batch-size-1 regime, and the linear extrapolation assumption is only validated on dense decoder-only models. With additional experiments or narrower claims, this could be a strong systems paper.

major comments (3)
  1. [§5.3, §6, §8] The evaluation disables batching for all systems (stated in §5.3 and §6: 'we disable batching in this work' and 'we exclude runtime preemption, swapping, and batching') and compares only against TensorRT-LLM PP/TP static-configuration baselines, Accelerate, and AlpaServe. None of these baselines uses continuous batching or paged KV-cache, which are standard techniques in production LLM serving systems such as vLLM and TensorRT-LLM's in-flight batching. The headline claims in the abstract (77.62% latency reduction, 7.09x fewer SLO violations, 4.72x GPU throughput improvement) are therefore measured in a regime where GPUs are underutilized and throughput is not comparable to the state of the art. The manuscript acknowledges this limitation but does not quantify its effect; this is load-bearing because the central claim is explicitly comparative.
  2. [§5, Figure 7] The fingerprint-based extrapolation assumes exact linear scaling of memory footprint and inference latency with the number of hidden layers. This assumption is validated on six dense decoder-only LLMs, but the Introduction and Section 2 present Mixtral MoE as a target architecture. MoE layers contain routing and multiple experts, and encoder-decoder or multimodal models contain heterogeneous layer types; the linear model is empirical and not derived from architectural invariants. The low estimation errors for the evaluated dense models (Section 8.4) are reassuring, but they do not establish the general scaling law. The paper should either validate the linearity on at least one non-dense or heterogeneous architecture, or explicitly restrict the scope and adjust the abstract's 'various LLMs' claim.
  3. [§5.3, §6] The paper's claim in §6 that the profiler 'can easily predict LLM latency and memory needs for different batch sizes using simple modeling (e.g., linear regressors)' is unsupported; no batching-enabled experiment is presented, and Equation (1) does not include batching or queueing effects. Since the evaluation's SLO attainment results (§8.3) are derived in the same batch-size-1 setting, the reported 7.09x SLO-violation improvement may not carry over to production workloads. Please either add a batching-enabled experiment or remove or substantially temper this claim.
minor comments (4)
  1. [§8, Metrics] The 'GPU throughput' metric used throughout Figures 11-16 is reported in req/s, but the denominator (per GPU, per server, or cluster-wide) and the time averaging are not defined. Please state the definition once in the Metrics paragraph.
  2. [§6, Hybrid placement policy] The hybrid placement policy depends on a load threshold θ, but the paper does not state how θ is set or whether it is tuned per trace. Please provide the value or the selection procedure to make the evaluation reproducible.
  3. [§8.2] The main text reports results for the 'cost' intent without specifying which of the three cost formulations from §4.1 is used. Please state the formulation used in Figure 13 and confirm whether the other two formulations yield similar conclusions.
  4. [§8.4] The sentence 'iServe average latency and memory estimation error is 4.91% (< 50ms error) and 6.92% (1.74GB)' is unclear: are these mean or median absolute errors, and across which configurations and models? Please clarify.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity; the fingerprint extrapolations are checked against independent full-LLM measurements, and the only self-citation is non-load-bearing.

full rationale

The central derivation is iServe's fingerprint-based estimation of full-LLM latency and memory. iServe assumes a linear relationship between hidden-layer count and these metrics (Section 5: 'We find a linear relationship between the number of hidden layers and memory footprint/inference latency (Figure 7)') and extrapolates from 1-2 layer fingerprints via Equations 2-3 and method M2. These estimates are not equal to the target values by construction; they are empirical extrapolations that could be wrong, and the paper validates them against actual full-LLM profiling (Figure 9; Section 8.4 reports 4.91% average latency error and 6.92% memory error). The end-to-end configuration decisions are evaluated by measuring served latency, throughput, SLO attainment, cost, and GPU-hours (Section 8), so the headline improvements are not read back from iServe's own estimates. The only author-overlapping citation, Hermod [55], is used in Section 6 solely to justify the synthetic-trace methodology ('Following the methodology of previous work [55, 61], we generate a 30-minute synthetic trace'); it is not load-bearing for any central claim and does not reduce a result to the citation. The paper's explicit limitation that batching is disabled (Section 5.3: 'similar to previous systems [61], we disable batching in this work'; Section 6: 'though batching is disabled, iServe's Profiler can easily predict...') is an external-validity concern for the throughput comparison, not a circularity, because the predicted metrics are still measured independently rather than defined by the assumption. Overall, the derivation chain is self-contained against independent full-LLM measurements, so no circular step is identified; the score reflects only the minor, non-load-bearing self-citation.

Assumptions & free parameters 3 free parameters · 3 assumptions · 1 invented entities

The central mechanism relies on an empirical linear-scaling law, a design choice of fingerprint sizes, and a hand-tuned placement threshold. These are the main fitted or chosen elements; no new physical entities are postulated.

free parameters (3)
  • Load threshold theta in hybrid placement policy = unspecified (likely tuned on evaluation traces)
    Section 6 defines packable GPUs using a threshold theta, but the paper never states its value or how it is chosen. The hybrid policy's behavior and results depend on this parameter.
  • Fingerprint hidden-layer counts (1 and 2) = 1 and 2 hidden layers
    Design choice in Section 5.3. The paper does not justify why these two sizes are optimal for extrapolation; a linear model requires two points, but other choices might change accuracy.
  • Linear scaling coefficients for latency and memory vs layer count = Empirically derived from fingerprint measurements (Figure 7)
    The extrapolation from fingerprint to full LLM assumes latency and memory grow linearly in the number of hidden layers. Coefficients are fit to measurements, and the resulting estimates are validated, but the linear model is an empirical fit, not a physically derived law.
assumptions (3)
  • domain assumption Latency and memory footprint scale linearly with the number of hidden layers for decoder-only LLMs.
    Stated in Section 5 ('We find a linear relationship...') and used as the basis for fingerprint extrapolation. Empirically supported by Figure 7, but not derived from architecture or hardware principles.
  • domain assumption At batch size 1, pipeline parallelism overhead is negligible (<0.05%), so TTFT and TPOT measured at PP=1 can be reused for any PP >= 2.
    Section 5.3 L1 argues PP overhead is below 0.05% for Llama-2-70B and assumes this holds for all studied LLMs and configurations. This allows profiling only PP=1 configurations.
  • domain assumption The memory footprint components (weights W, activations A, pipeline activations A_p) replicate as in Figure 8 across TP and PP degrees.
    Standard parallelism memory model, attributed to [68]. iServe uses this decomposition to compute memory for all configurations from three fingerprint observations.
invented entities (1)
  • LLM fingerprint independent evidence
    purpose: A lightweight proxy model with one or two hidden layers used to profile latency and memory of the full LLM across deployment configurations.
    Predictions from the fingerprint are validated against full-LLM measurements in Section 8.4 and Figure 19, so the proxy has an external falsifiable check. It is a computational artifact, not a new physical entity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of iServe: An Intent-based Serving System for LLMs." pith.science (2026). https://pith.science/paper/FYAZNP5H

@misc{pith2026250113111,
  author       = {Pith},
  title        = {Pith review of: iServe: An Intent-based Serving System for LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FYAZNP5H}},
  note         = {Machine review of arXiv:2501.13111}
}
read the original abstract

Large Language Models (LLMs) are becoming ubiquitous across industries, where applications demand they fulfill diverse user intents. However, developers currently face the challenge of manually exploring numerous deployment configurations - combinations of parallelism and compression techniques that impact resource usage, latency, cost, and accuracy - to meet these intents. Assessing the impact of these configurations on user metrics requires extensive, costly profiling for each model. Existing approaches avoid this expense by using fixed, static configurations, but this often leads to sub-optimal performance and higher costs. Moreover, none of these solutions dynamically adapt to changing user intents to balance latency and cost, effectively. We present iServe, an automated, intent-based system for distributed LLM inference. Instead of manually selecting deployment configurations, developers simply specify their intent - such as minimizing latency, reducing cost, or meeting specific targets for either. iServe introduces fingerprints, lightweight representations of LLMs, to efficiently estimate how different configurations impact latency and memory usage. Based on these insights and GPU availability, iServe dynamically selects the optimal configuration to align with the user's intent. For various LLMs and query arrival rates, iServe best meets user intents compared to state-of-the-art systems by reducing latency by 77.62% and SLO violations by 7.09x while improving GPU throughput by 4.72x. Moreover, iServe's fingerprint-based profiling reduces profiling cost by 6.05x (GPU-hours) compared to baselines.

Figures

Figures reproduced from arXiv: 2501.13111 by the authors.

Figure 1
Figure 1. Each LLM has hundreds of deployment configura￾tions with trade-offs in latency and memory consumption. The best configuration is specific to the user intent. See § 1. intents, experts introduced various (a) compression tech￾niques that reduce the size of the models (e.g., quantiza￾tion [27, 28, 37, 62, 118, 125], pruning [36, 71, 101]) and (b) parallelism techniques that distribute the models across mul￾tiple GPUs (… view at source ↗
Figure 2
Figure 2. Architectures of a decoder-only LLM (left) and its fingerprint (right). The fingerprint has the same components as the full LLM, but only 1-2 hidden layers. See § 2. range of LLMs and query arrival rates from the Azure LLM inference serving traces [84], iServe best meets user in￾tent: iServe reduces latency by 77.62%, cost by 86.70%, and SLO violations by 7.09× while improving GPU throughput by 4.72× on average comp… view at source ↗
Figure 4
Figure 4. The impact of quantization on latency is LLM￾specific. See § 3.2. No Pruning Wanda 2:4 LLM-Pruner 25% LLM-Pruner 50% 0 3 Lat (s) Llama-2-7B No Pruning Wanda 2:4 LLM-Pruner 25% LLM-Pruner 50% 0 5 Lat (s) Llama-2-13B No Pruning Unstructured Structured [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗
Figures from the paper (17 more)
Figure 6
Figure 6. Figure 6: Architecture of iServe. See § 4. education platforms and automated FAQs focus on cost￾efficiency. Meanwhile, mobile apps and IoT devices require minimal GPU usage. However, to leverage LLMs, developers are forced to either (a) manually deploy them and reverse￾engineer …
Figure 7
Figure 7. Figure 7: Latency and memory consumption are linearly proportional to the LLM’s number of hidden layers. See § 5. × NTP∙NPP Config No Parallel TP PP TP + PP W W/NTP∙NPP A/NPP AP Per GPU Balanced Offloading Total Memory membase = (W+A) membase + (NTP-1) × A membase + NPP × AP mem…
Figure 8
Figure 8. Figure 8: LLM parallelism memory overheads. Weights (𝑊 ), activations (𝐴), and pipeline activations (𝐴𝑝 ) are replicated on 𝑁𝑇 𝑃 (TP degree) and/or 𝑁𝑃𝑃 (PP degree) GPUs. See § 5.2. 5.1 Generating LLM Fingerprints Instead of profiling the full LLM, iServe creates and profiles the…
Figure 9
Figure 9. Figure 9: Design exploration of profiling methods. We show profiling accuracy (row 1) and cost (row 2) for each method. iServe uses (L2,M2) to reduce profiling cost. See § 5.3. Of the two fingerprint-based memory profiling methods (M2-M3), M3’s accuracy is best: it observes ever…
Figure 10
Figure 10. Figure 10: Design exploration of placement policies. The hybrid policy balances packing LLMs onto fewer GPUs to reduce the # of GPUs consumed and dispersing LLMs across GPUs to reduce latency. See § 6 [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]
Figure 11
Figure 11. Figure 11: Given the intent to minimize latency, iServe consistently minimizes latency (row 1), maximizes GPU throughput (row 2), and minimizes the # of GPUs consumed (row 3) as the query arrival rate (rate scale factor) fluctuates under Azure’s two LLM inference traces: (a) cod…
Figure 13
Figure 13. Figure 13: iServe adapts to the intent, minimizing cost (col. 1), memory (col. 2), and GPU-hours (col. 3). See § 8.2. 2×Llama-2-7B, 2×Falcon-40B, and 2×GPT-J-6B. iServe con￾tinues to minimize latency: it reduces p95 latency by 63.55- 99.96% compared to our baselines. Unlike the …
Figure 16
Figure 16. Figure 16: iServe scales and maximizes the cluster through￾put and number of LLMs deployed in a fixed cluster size. iServe sustains high performance across intents: minimize latency (l), cost (c), GPU-hour (g), and memory (m). See § 8.5. Error analysis. iServe average latency an…
Figure 15
Figure 15. Figure 15: iServe reduces LLM profiling cost (memory us￾age, time, and GPU-hours) compared to AlpaServe. See § 8.4. memory footprint by 74.36-83.92% compared to the base￾lines. Minimizing cost (memory × latency) and GPU-hours requires balancing resource allocation and latency. H…
Figure 17
Figure 17. Figure 17: The effect of quantization on memory reduction becomes more significant as the level of quantization be￾comes more aggressive. No Pruning Wanda 2:4 LLM-Pruner 25% LLM-Pruner 50% 0 10 Mem (GB) Llama-2-7B No Pruning Wanda 2:4 LLM-Pruner 25% LLM-Pruner 50% 0 20 Llama-2-1…
Figure 18
Figure 18. Figure 18: The memory of LLMs can be reduced with prun￾ing techniques. Model Parallelism Quantization Pruning Falcon-7B Pipeline Falcon-40B INT8, INT4 - Pipeline, Tensor GPT-J-6B Llama-2-7B INT8, INT4, GPTQ4, INT8 KV Cache SparseGPT, Wanda, Wanda 2:4, Wanda 4:8 Llama-2-13B Llama…
Figure 19
Figure 19. Figure 19: iServe predicts the latency and memory require￾ments across deployment configurations with minimal error. 10% 20% 30% 40% 50% Average GPU Loads 0 50 90 SLO-Att (%) Serving only Serving w/ Profiling [PITH_FULL_IMAGE:figures/full_fig_p018_19.png]
Figure 20
Figure 20. Figure 20 [PITH_FULL_IMAGE:figures/full_fig_p018_20.png]
Figure 21
Figure 21. Figure 21: Given the intent to minimize cost, iServe can minimize cost (row 1), maximize GPU throughput (row 2), and minimize # GPU used (row 3) across all query arrival rates under both of Azure’s traces: (a) code and (b) conversation. 10 1 10 0 Norm. Memory RS=0.4 RS=0.3 RS=0.…
Figure 22
Figure 22. Figure 22: Given the intent to minimize memory, iServe can minimize memory (row 1), maximize GPU throughput (row 2), and minimize # GPU used (row 3) across all query arrival rates under both of Azure’s traces: (a) code and (b) conversation. 10 2 10 1 10 0 Norm. GPU-hrs RS=0.4 RS…
Figure 23
Figure 23. Figure 23: Given the intent to minimize GPU-hours, iServe can minimize GPU-hours (row 1), maximize GPU throughput (row 2), and minimize # GPU used (row 3) across all query arrival rates under both of Azure’s traces: (a) code and (b) conversation. 10 5 10 3 10 0 Norm. Value Laten…
Figure 24
Figure 24. Figure 24: iServe meets the intent to minimize latency, cost, memory, or GPU-hours in serving a large set of small LLMs. 2 [PITH_FULL_IMAGE:figures/full_fig_p019_24.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling

    cs.DC 2026-07 conditional novelty 6.5 of 10

    Balancing only first-turn agent requests while keeping follow-ups cache-aware raises LLM cluster TPS 10-16% (colocation) and prefill TPS 2-34% (disaggregation) over SOTA schedulers on production agent traces.

Reference graph

Works this paper leans on

134 extracted references · 49 canonical work pages · cited by 1 Pith paper

  1. [1]

    https://developer.nvidia.com/blog/optimizing- nvidia-tensorrt-conversion-for-real-time-inference-on- autonomous-vehicles/, 2020

    Optimizing NVIDIA TensorRT Conversion for Real-time Inference on Autonomous Vehicles. https://developer.nvidia.com/blog/optimizing- nvidia-tensorrt-conversion-for-real-time-inference-on- autonomous-vehicles/, 2020. Accessed: 2024-10-15

    2020

  2. [2]

    https://github.com/horseee/LLM-Pruner/issues/33, 2023

    Issue #33. https://github.com/horseee/LLM-Pruner/issues/33, 2023. Accessed: 2023-10-16

    2023

  3. [3]

    https://sharegpt.com/, 2023

    ShareGPT. https://sharegpt.com/, 2023. Accessed: 2024-10-16

    2023

  4. [4]

    https://resources.nvidia

    Amazon Accelerates Customer Satisfaction With NVIDIA Triton Inference Server and NVIDIA TensorRT. https://resources.nvidia. com/en-us-inference-customer-story/nvidia-amazon-custom , 2024. Accessed: 2024-10-17

    2024

  5. [5]

    https://resources.nvidia.com/en-us-inference- customer-story/american-express-prevents-fraud , 2024

    American Express Prevents Fraud and Foils Cybercrime With NVIDIA AI Solutions. https://resources.nvidia.com/en-us-inference- customer-story/american-express-prevents-fraud , 2024. Accessed: 2024-10-17

    2024

  6. [6]

    https://opennmt.net/CTranslate2/, 2024

    CTranslate2. https://opennmt.net/CTranslate2/, 2024. Accessed: 2024-10-06

    2024

  7. [7]

    https://github.com/huggingface/safetensors, 2024

    Safetensors. https://github.com/huggingface/safetensors, 2024. Ac- cessed: 2024-10-08

    2024

  8. [8]

    https://www.unix.com/man-page/linux/7posix/tcp.h/, 2024

    tcp.h. https://www.unix.com/man-page/linux/7posix/tcp.h/, 2024. Accessed: 2024-08-17

    2024

Show all 134 references
  1. [9]

    Etalon: Holistic Performance Evaluation Framework for LLM Inference Systems

    Amey Agrawal, Anmol Agarwal, Nitin Kedia, Jayashree Mohan, Sou- vik Kundu, Nipun Kwatra, Ramachandran Ramjee, and Alexey Tu- manov. Etalon: Holistic Performance Evaluation Framework for LLM Inference Systems. arXiv preprint arXiv:2407.07000, 2024

    2024 arXiv

  2. [10]

    Vidur: A Large-Scale Simulation Framework For LLM Infer- ence

    Amey Agrawal, Nitin Kedia, Jayashree Mohan, Ashish Panwar, Nipun Kwatra, Bhargav Gulavani, Ramachandran Ramjee, and Alexey Tu- manov. Vidur: A Large-Scale Simulation Framework For LLM Infer- ence. Proceedings of Machine Learning and Systems , 6:351–366, May 2024

    2024

  3. [11]

    Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve

    Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. In 18th USENIX Symposium on Operating Systems Design and Implementation (...

    2024

  4. [12]

    Friedman, Thomas Williams, Ramesh K

    Sohaib Ahmad, Hui Guan, Brian D. Friedman, Thomas Williams, Ramesh K. Sitaraman, and Thomas Woo. Proteus: A High- Throughput Inference-Serving System with Accuracy Scaling. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languag...

    2024

  5. [13]

    Llama 3B on TensorRT-LLM

    Cerebrium AI. Llama 3B on TensorRT-LLM. https://docs.cerebrium. ai/v4/examples/tensorRT, 2024. Accessed: October 16, 2024

    2024

  6. [14]

    Serverless Pricing

    Mystic AI. Serverless Pricing. https://www.mystic.ai/pricing/ serverless, 2023. Accessed: 2024-10-18

    2023

  7. [15]

    The Falcon Series of Open Language Models

    Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessan- dro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. The Falcon Series ...

    2023 arXiv

  8. [16]

    DeepSpeed Inference: En- abling Efficient Inference of Transformer Models at Unprecedented Scale

    Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Am- mar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, and Yuxiong He. DeepSpeed Inference: En- abling Efficient Inference of Transformer Models at Unprecedented Scale. arXiv prepr...

    2022 arXiv

  9. [17]

    Introducing claude

    Anthropic. Introducing claude. https://www.anthropic.com/news/ introducing-claude, 2023. Accessed 2024-05-02

    2023

  10. [18]

    Varuna: scalable, low-cost training of massive deep learning models

    Sanjith Athlur, Nitika Saran, Muthian Sivathanu, Ramachandran Ramjee, and Nipun Kwatra. Varuna: scalable, low-cost training of massive deep learning models. In Proceedings of the Seventeenth European Conference on Computer Systems, EuroSys ’22, page 472–487, New York, NY, USA,...

    2022

  11. [19]

    OpenLLM: An open platform for operating large language models in production

    BentoML. OpenLLM: An open platform for operating large language models in production. https://github.com/bentoml/OpenLLM, 2023

    2023

  12. [20]

    Accelerating BERT with semi-structured (2:4) spar- sity

    Jesse Cai. Accelerating BERT with semi-structured (2:4) spar- sity. https://pytorch.org/tutorials/prototype/semi_structured_sparse. html, 2023. Accessed: 2023-10-16

    2023

  13. [21]

    Effi- cient and Economic Large Language Model Inference with Attention Offloading

    Shaoyuan Chen, Yutong Lin, Mingxing Zhang, and Yongwei Wu. Effi- cient and Economic Large Language Model Inference with Attention Offloading. arXiv preprint arXiv:2405.01814, 2024

    2024 arXiv

  14. [22]

    Cloud-Native AI Engineering Practice: Accelerating LLM Inference with FasterTransformer

    Alibaba Cloud. Cloud-Native AI Engineering Practice: Accelerating LLM Inference with FasterTransformer. https://www.alibabacloud. com/blog/cloud-native-ai-engineering-practice-accelerating-llm- inference-with-fastertransformer_600424, 2024. Accessed: October 16, 2024

    2024

  15. [23]

    Microsoft Copilot: Your AI Companion.https: //copilot.microsoft.com/, 2024

    Microsoft Corporation. Microsoft Copilot: Your AI Companion.https: //copilot.microsoft.com/, 2024. Accessed: 2024-10-20

    2024

  16. [24]

    Franklin, Joseph E

    Daniel Crankshaw, Xin Wang, Guilio Zhou, Michael J. Franklin, Joseph E. Gonzalez, and Ion Stoica. Clipper: A Low-Latency Online Prediction Serving System. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17), pages 613–627, Boston, MA, March 2017. ...

    2017

  17. [25]

    A Complete Survey on LLM-based AI Chatbots.arXiv preprint arXiv:2406.16937, 2024

    Sumit Kumar Dam, Choong Seon Hong, Yu Qiao, and Chaoning Zhang. A Complete Survey on LLM-based AI Chatbots.arXiv preprint arXiv:2406.16937, 2024

    2024 arXiv

  18. [26]

    Scheduling slack time in fixed priority pre-emptive systems

    Robert I Davis, Ken W Tindell, and Alan Burns. Scheduling slack time in fixed priority pre-emptive systems. In Proceedings of the Real-Time Systems Symposium, pages 222–231. IEEE, 1993

    1993

  19. [27]

    LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22. 13

  20. [28]

    QLoRA: Efficient Finetuning of Quantized LLMs

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettle- moyer. QLoRA: Efficient Finetuning of Quantized LLMs. In Pro- ceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, 2023

    2023

  21. [29]

    The case for 4-bit precision: k-bit inference scaling laws

    Tim Dettmers and Luke Zettlemoyer. The case for 4-bit precision: k-bit inference scaling laws. In Proceedings of the 40th International Conference on Machine Learning , ICML’23. JMLR.org, 2023

    2023

  22. [30]

    QAQ: Qual- ity Adaptive Quantization for LLM KV Cache

    Shichen Dong, Wen Cheng, Jiayu Qin, and Wei Wang. QAQ: Qual- ity Adaptive Quantization for LLM KV Cache. arXiv preprint arXiv:2403.04643, 2024

    2024 arXiv

  23. [31]

    MOPAR: A Model Partitioning Framework for Deep Learning Inference Services on Serverless Platforms

    Jiaang Duan, Shiyou Qian, Dingyu Yang, Hanwen Hu, Jian Cao, and Guangtao Xue. MOPAR: A Model Partitioning Framework for Deep Learning Inference Services on Serverless Platforms. arXiv preprint arXiv:2404.02445, 2024

    2024 arXiv

  24. [32]

    GPT-J-6B

    EleutherAI. GPT-J-6B. https://huggingface.co/EleutherAI/gpt-j-6b. Accessed: 2024-09-20

    2024

  25. [33]

    Nanotron: Lightweight quantization for transformers

    Hugging Face. Nanotron: Lightweight quantization for transformers. https://github.com/huggingface/nanotron, 2023

    2023

  26. [34]

    FairScale: A general purpose modular PyTorch library for high performance and large scale training

    FairScale authors. FairScale: A general purpose modular PyTorch library for high performance and large scale training. https://github. com/facebookresearch/fairscale, 2021

    2021

  27. [35]

    TurboTrans- formers: an efficient GPU serving system for transformer models

    Jiarui Fang, Yang Yu, Chengduo Zhao, and Jie Zhou. TurboTrans- formers: an efficient GPU serving system for transformer models. In Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming , PPoPP ’21, page 389–402, New York, NY, USA, 202...

    2021

  28. [36]

    SparseGPT: Massive Language Mod- els Can Be Accurately Pruned in One-Shot

    Elias Frantar and Dan Alistarh. SparseGPT: Massive Language Mod- els Can Be Accurately Pruned in One-Shot. In Proceedings of the 40th International Conference on Machine Learning , ICML’23. JMLR.org, 2023

    2023

  29. [37]

    GPTQ: Accurate Post-Training Quantization for Generative Pre- trained Transformers

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate Post-Training Quantization for Generative Pre- trained Transformers. In International Conference on Learning Repre- sentations, 2023

    2023

  30. [38]

    MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models.arXiv preprint arXiv:2408.11743, 2024

    Elias Frantar, Roberto L Castro, Jiale Chen, Torsten Hoefler, and Dan Alistarh. MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models.arXiv preprint arXiv:2408.11743, 2024

    2024 arXiv

  31. [39]

    ServerlessLLM: Locality- Enhanced Serverless Inference for Large Language Models

    Yao Fu, Leyang Xue, Yeqi Huang, Andrei-Octavian Brabete, Dmitrii Ustiugov, Yuvraj Patel, and Luo Mai. ServerlessLLM: Locality- Enhanced Serverless Inference for Large Language Models. arXiv preprint arXiv:2401.14351, 2024

    2024 arXiv

  32. [40]

    Megablocks: Efficient Sparse Training with Mixture-of-Experts

    Trevor Gale, Deepak Narayanan, Cliff Young, and Matei Zaharia. Megablocks: Efficient Sparse Training with Mixture-of-Experts. Pro- ceedings of Machine Learning and Systems , 5:288–304, 2023

    2023

  33. [41]

    Paxml: A composable and scalable framework for JAX-based neural network models

    Google. Paxml: A composable and scalable framework for JAX-based neural network models. https://github.com/google/paxml, 2023

    2023

  34. [42]

    SaxML: Scalable and composable framework for ML model serving

    Google. SaxML: Scalable and composable framework for ML model serving. https://github.com/google/saxml, 2023

    2023

  35. [43]

    Mélange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity

    Tyler Griggs, Xiaoxuan Liu, Jiaxiang Yu, Doyoung Kim, Wei-Lin Chiang, Alvin Cheung, and Ion Stoica. Mélange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity. arXiv preprint arXiv:2404.14527, 2024

    2024 arXiv

  36. [44]

    Serving DNNs like clockwork: performance predictability from the bottom up

    Arpan Gujarati, Reza Karimi, Safya Alzayat, Wei Hao, Antoine Kauf- mann, Ymir Vigfusson, and Jonathan Mace. Serving DNNs like clockwork: performance predictability from the bottom up. In Pro- ceedings of the 14th USENIX Conference on Operating Systems Design and Implementation...

    2020

  37. [45]

    OliVe: Accelerat- ing Large Language Models via Hardware-friendly Outlier-Victim Pair Quantization

    Cong Guo, Jiaming Tang, Weiming Hu, Jingwen Leng, Chen Zhang, Fan Yang, Yunxin Liu, Minyi Guo, and Yuhao Zhu. OliVe: Accelerat- ing Large Language Models via Hardware-friendly Outlier-Victim Pair Quantization. In Proceedings of the 50th Annual International Symposium on Comput...

    2023

  38. [46]

    Hinton, Oriol Vinyals, and Jeffrey Dean

    Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the Knowledge in a Neural Network. ArXiv, abs/1503.02531, 2015

    2015 arXiv

  39. [47]

    DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference

    Connor Holmes, Masahiro Tanaka, Michael Wyatt, Ammar Ah- mad Awan, Jeff Rasley, Samyam Rajbhandari, Reza Yazdani Am- inabadi, Heyang Qin, Arash Bakhtiari, Lev Kurilenko, and Yuxiong He. DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference...

    2024 arXiv

  40. [48]

    LoRA: Low-Rank Adaptation of Large Language Models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations, 2022

    2022

  41. [49]

    Can Perplexity Reflect Large Language Model’s Ability in Long Text Understanding? In The Second Tiny Papers Track at ICLR 2024 , 2024

    Yutong Hu, Quzhe Huang, Mingxu Tao, Chen Zhang, and Yansong Feng. Can Perplexity Reflect Large Language Model’s Ability in Long Text Understanding? In The Second Tiny Papers Track at ICLR 2024 , 2024

    2024

  42. [50]

    Le, Yonghui Wu, and Zhifeng Chen

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism . Curran Associates Inc., Red Hook, NY, USA, 2019

    2019

  43. [51]

    https://huggingface.co/

    Hugging Face. https://huggingface.co/. Accessed: 2024-10-20

    2024

  44. [52]

    https://lmdeploy.readthedocs.io/en/v0.4.0/ quantization/kv_quant.html

    INT8 KV Cache. https://lmdeploy.readthedocs.io/en/v0.4.0/ quantization/kv_quant.html. Accessed: 2024-10-20

    2024

  45. [53]

    Compressing LLMs: The Truth is Rarely Pure and Never Simple

    Ajay Jaiswal, Zhe Gan, Xianzhi Du, Bowen Zhang, Zhangyang Wang, and Yinfei Yang. Compressing LLMs: The Truth is Rarely Pure and Never Simple. InInternational Conference on Learning Representations, 2024

    2024

  46. [54]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Men- sch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-An...

    2024 arXiv

  47. [55]

    Yadwadkar, and Christos Kozyrakis

    Kostis Kaffes, Neeraja J. Yadwadkar, and Christos Kozyrakis. Her- mod: principled and practical scheduling for serverless functions. In Proceedings of the 13th Symposium on Cloud Computing , SoCC ’22, page 289–305, New York, NY, USA, 2022. Association for Computing Machinery

    2022

  48. [56]

    Efficient Memory Management for Large Language Model Serving with PagedAttention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Op- erating Systems Principles,...

    2023

  49. [57]

    BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension

    Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Ab- delrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettle- moyer. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Dan Jurafsky, Joyce Chai...

    2020

  50. [58]

    Easy and Efficient Transformer: Scalable Inference Solution For Large NLP Model

    Gongzheng Li, Yadong Xi, Jingzhen Ding, Duan Wang, Ziyang Luo, Rongsheng Zhang, Bai Liu, Changjie Fan, Xiaoxi Mao, and Zeng Zhao. Easy and Efficient Transformer: Scalable Inference Solution For Large NLP Model. In Anastassia Loukina, Rashmi Gangadhara- iah, and Bonan Min, edit...

    2022

  51. [59]

    A Speed Odyssey for Deployable Quantization of LLMs

    Qingyuan Li, Ran Meng, Yiduo Li, Bo Zhang, Liang Li, Yifan Lu, Xiangxiang Chu, Yerui Sun, and Yuchen Xie. A Speed Odyssey for Deployable Quantization of LLMs. arXiv preprint arXiv:2311.09550, 2023

    2023 arXiv

  52. [60]

    Gonzalez, and Ion Stoica

    Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. AlpaServe [Source Code]: Function in model_parallelism.py, line 323-353. https://github.com/alpa- projects/mms/blob/dba47b18e9...

    2023

  53. [61]

    Gon- zalez, and Ion Stoica

    Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E. Gon- zalez, and Ion Stoica. AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving. In 17th USENIX Symposium on Operating ...

    2023

  54. [62]

    AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. In P. Gibbons, G. Pekhimenko, and C. De Sa, editors, Proceedings...

    2024

  55. [63]

    QUART: Latency-Aware FaaS System for Pipelining Large Model Inference

    Yanying Lin, Yanbo Li, Shijie Peng, Yingfei Tang, Shutian Luo, Haiy- ing Shen, Chengzhong Xu, and Kejiang Ye. QUART: Latency-Aware FaaS System for Pipelining Large Model Inference. In 2024 IEEE 44th International Conference on Distributed Computing Systems (ICDCS) , pages 1–12, 2024

    2024

  56. [64]

    SuperScaler: Supporting Flexible DNN Par- allelization via a Unified Abstraction

    Zhiqi Lin, Youshan Miao, Guodong Liu, Xiaoxiang Shi, Quanlu Zhang, Fan Yang, Saeed Maleki, Yi Zhu, Xu Cao, Cheng Li, Mao Yang, Lintao Zhang, and Lidong Zhou. SuperScaler: Supporting Flexible DNN Par- allelization via a Unified Abstraction. arXiv preprint arXiv:2301.08984, 2023

    2023 arXiv

  57. [65]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 34892–34916. Curran Associates, Inc., 2023

    2023

  58. [66]

    AutoPipe: A fast pipeline parallelism approach with balanced partitioning and micro-batch slicing

    Weijie Liu, Zhiquan Lai, Shengwei Li, Yabo Duan, Keshi Ge, and Dongsheng Li. AutoPipe: A fast pipeline parallelism approach with balanced partitioning and micro-batch slicing. In 2022 IEEE Interna- tional Conference on Cluster Computing (CLUSTER) , pages 301–312. IEEE, 2022

    2022

  59. [67]

    Deja Vu: contextual sparsity for efficient LLMs at inference time

    Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Ré, and Beidi Chen. Deja Vu: contextual sparsity for efficient LLMs at inference time. In Proceedings of the 40th International Conference on Machi...

    2023

  60. [68]

    RTP: Rethinking Tensor Parallelism with Memory Deduplication

    Cheng Luo, Tianle Zhong, and Geoffrey Fox. RTP: Rethinking Tensor Parallelism with Memory Deduplication. arXiv preprint arXiv:2311.01635, 2023

    2023 arXiv

  61. [69]

    SPLIT: QoS-Aware DNN Inference on Shared GPU via Evenly- Sized Model Splitting

    Diaohan Luo, Tian Yu, Yuewen Wu, Heng Wu, Tao Wang, and Wenbo Zhang. SPLIT: QoS-Aware DNN Inference on Shared GPU via Evenly- Sized Model Splitting. In Proceedings of the 52nd International Con- ference on Parallel Processing , ICPP ’23, page 605–614, New York, NY, USA, 2023. ...

    2023

  62. [70]

    The era of 1-bit llms: All large language models are in 1.58 bits.arXiv preprint arXiv:2402.17764, 2024

    Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, and Furu Wei. The era of 1-bit llms: All large language models are in 1.58 bits.arXiv preprint arXiv:2402.17764, 2024

    2024 arXiv

  63. [71]

    LLM-Pruner: On the Structural Pruning of Large Language Models

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. LLM-Pruner: On the Structural Pruning of Large Language Models. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, 2023

    2023

  64. [72]

    LLM-Powered Conversational Voice Assis- tants: Interaction Patterns, Opportunities, Challenges, and Design Guidelines

    Amama Mahmood, Junxiang Wang, Bingsheng Yao, Dakuo Wang, and Chien-Ming Huang. LLM-Powered Conversational Voice Assis- tants: Interaction Patterns, Opportunities, Challenges, and Design Guidelines. arXiv preprint arXiv:2309.13879, 2023

    2023 arXiv

  65. [73]

    SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verifica- tion

    Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. SpecInfer: Accelerating Large Language Model Serving with Tr...

    2024

  66. [74]

    SpotServe: Serving Generative Large Lan- guage Models on Preemptible Instances

    Xupeng Miao, Chunan Shi, Jiangfei Duan, Xiaoli Xi, Dahua Lin, Bin Cui, and Zhihao Jia. SpotServe: Serving Generative Large Lan- guage Models on Preemptible Instances. In Proceedings of the 29th ACM International Conference on Architectural Support for Program- ming Languages a...

    2024

  67. [75]

    Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism

    Xupeng Miao, Yujie Wang, Youhe Jiang, Chunan Shi, Xiaonan Nie, Hailin Zhang, and Bin Cui. Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism. Proc. VLDB Endow., 16(3):470–479, nov 2022

    2022

  68. [76]

    How trustwor- thy are open-source LLMs? an assessment under malicious demon- strations shows their vulnerabilities

    Lingbo Mo, Boshi Wang, Muhao Chen, and Huan Sun. How trustwor- thy are open-source LLMs? an assessment under malicious demon- strations shows their vulnerabilities. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North America...

    2024

  69. [77]

    LLM Foundry: Tools for training and fine-tuning large language models

    MosaicML. LLM Foundry: Tools for training and fine-tuning large language models. https://github.com/mosaicml/llm-foundry, 2023

    2023

  70. [78]

    Fractional GPU Allocation for Less Demanding or In- teractive Workloads

    NetApp. Fractional GPU Allocation for Less Demanding or In- teractive Workloads. https://docs.netapp.com/us-en/netapp- solutions/ai/osrunai_fractional_gpu_allocation_for_less_ demanding_or_interactive_workloads.html, 2023. Accessed: 2024-10-18

    2023

  71. [79]

    FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement

    Xiaonan Nie, Xupeng Miao, Zilong Wang, Zichao Yang, Jilong Xue, Lingxiao Ma, Gang Cao, and Bin Cui. FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement. Proceedings of the ACM on Management of Data , 1:1–19, 05 2023

    2023

  72. [80]

    TensorRT-LLM

    NVIDIA. TensorRT-LLM. https://github.com/NVIDIA/TensorRT- LLM. Accessed: 2024-10-20

    2024

  73. [81]

    ExeGPT: Constraint-Aware Resource Scheduling for LLM Inference

    Hyungjun Oh, Kihong Kim, Jaemin Kim, Sungkyun Kim, Junyeol Lee, Du seong Chang, and Jiwon Seo. ExeGPT: Constraint-Aware Resource Scheduling for LLM Inference. In Proceedings of the 29th ACM International Conference on Architectural Support for Program- ming Languages and Opera...

    2024

  74. [82]

    OpenAI. ChatGPT. https://www.openai.com, 2024. June 16 version

    2024

  75. [83]

    InstInfer: In- Storage Attention Offloading for Cost-Effective Long-Context LLM Inference

    Xiurui Pan, Endian Li, Qiao Li, Shengwen Liang, Yizhou Shan, Ke Zhou, Yingwei Luo, Xiaolin Wang, and Jie Zhang. InstInfer: In- Storage Attention Offloading for Cost-Effective Long-Context LLM Inference. arXiv preprint arXiv:2409.04992, 2024

    2024 arXiv

  76. [84]

    Splitwise: Efficient gen- erative LLM inference using phase splitting

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient gen- erative LLM inference using phase splitting. In ISCA, June 2024

    2024

  77. [85]

    Accelerating Inference with Sparsity Using the NVIDIA Ampere Architecture and NVIDIA TensorRT

    Jeff Pool, Abhishek Sawarkar, and Jay Rodge. Accelerating Inference with Sparsity Using the NVIDIA Ampere Architecture and NVIDIA TensorRT. https://developer.nvidia.com/blog/accelerating-inference- with-sparsity-using-ampere-and-tensorrt/ , 2021. Accessed: 2024-10- 20. 15

    2021

  78. [86]

    vAttention: Dynamic Memory Manage- ment for Serving LLMs without PagedAttention

    Ramya Prabhu, Ajay Nayak, Jayashree Mohan, Ramachandran Ram- jee, and Ashish Panwar. vAttention: Dynamic Memory Manage- ment for Serving LLMs without PagedAttention. arXiv preprint arXiv:2405.04437, 2024

    2024 arXiv

  79. [87]

    Ray-LLM: Scalable and distributed framework for serv- ing large language models

    Ray Project. Ray-LLM: Scalable and distributed framework for serv- ing large language models. https://github.com/ray-project/ray-llm, 2023

    2023

  80. [88]

    Improving language understanding by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018

    2018

  81. [89]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Ex- ploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. volume 21. JMLR.org, January 2020

    2020

  82. [90]

    ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. In Proceedings of the International Conference for High Per- formance Computing, Networking, Storage and Analysis , SC ’20. IEEE Press, 2020

    2020

  83. [91]

    Yadwadkar, and Christos Kozyrakis

    Francisco Romero, Qian Li, Neeraja J. Yadwadkar, and Christos Kozyrakis. INFaaS: Automated Model-less Inference Serving. In 2021 USENIX Annual Technical Conference (USENIX ATC 21) , pages 397–411. USENIX Association, July 2021

    2021

  84. [92]

    Deploy large models at high per- formance using FasterTransformer on Amazon SageMaker

    Amazon Web Services. Deploy large models at high per- formance using FasterTransformer on Amazon SageMaker. https://aws.amazon.com/blogs/machine-learning/deploy-large- models-at-high-performance-using-fastertransformer-on- amazon-sagemaker/, 2024. Accessed: October 16, 2024

    2024

  85. [93]

    Shallue, Jaehoon Lee, Joseph Antognini, Jascha Sohl- Dickstein, Roy Frostig, and George E

    Christopher J. Shallue, Jaehoon Lee, Joseph Antognini, Jascha Sohl- Dickstein, Roy Frostig, and George E. Dahl. Measuring the Effects of Data Parallelism on Neural Network Training. Journal of Machine Learning Research, 20:1–49, 2019

    2019

  86. [94]

    Nexus: a GPU cluster engine for accelerating DNN-based video anal- ysis

    Haichen Shen, Lequn Chen, Yuchen Jin, Liangyu Zhao, Bingyu Kong, Matthai Philipose, Arvind Krishnamurthy, and Ravi Sundaram. Nexus: a GPU cluster engine for accelerating DNN-based video anal- ysis. In Proceedings of the 27th ACM Symposium on Operating Systems Principles, SOSP ...

    2019

  87. [95]

    Gonzalez, and Ion Stoica

    Ying Sheng, Shiyi Cao, Dacheng Li, Banghua Zhu, Zhuohan Li, Danyang Zhuo, Joseph E. Gonzalez, and Ion Stoica. Fairness in Serving Large Language Models. In Proceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation , July 10–12 2024

    2024

  88. [96]

    FlexGen: high-throughput generative inference of large language models with a single GPU

    Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. FlexGen: high-throughput generative inference of large language models with a single GPU. In Proceedings of the 40th Inter- national Conference...

    2023

  89. [97]

    Megatron-LM: Training Multi- Billion Parameter Language Models Using Model Parallelism

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training Multi- Billion Parameter Language Models Using Model Parallelism. arXiv preprint arXiv:1909.08053, 2020

    1909 arXiv

  90. [98]

    A Hybrid Tensor- Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training

    Siddharth Singh, Olatunji Ruwase, Ammar Ahmad Awan, Samyam Rajbhandari, Yuxiong He, and Abhinav Bhatele. A Hybrid Tensor- Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training. In Proceedings of the 37th International Conference on Su- percomputing, ICS ’23....

    2023

  91. [99]

    DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency

    Jovan Stojkovic, Chaojie Zhang, Íñigo Goiri, Josep Torrellas, and Esha Choukse. DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency. arXiv preprint arXiv:2408.00741, 2024

    2024

  92. [100]

    Llumnix: Dynamic Scheduling for Large Lan- guage Model Serving

    Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, and Wei Lin. Llumnix: Dynamic Scheduling for Large Lan- guage Model Serving. In Proceedings of the 24th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) , 2024

    2024

  93. [101]

    Zico Kolter

    Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A Simple and Effective Pruning Approach for Large Language Models. In The Twelfth International Conference on Learning Representations , 2024

    2024

  94. [102]

    AdaPipe: Optimizing Pipeline Parallelism with Adaptive Recomputation and Partitioning

    Zhenbo Sun, Huanqi Cao, Yuanwei Wang, Guanyu Feng, Shengqi Chen, Haojie Wang, and Wenguang Chen. AdaPipe: Optimizing Pipeline Parallelism with Adaptive Recomputation and Partitioning. In Proceedings of the 29th ACM International Conference on Archi- tectural Support for Progra...

    2024

  95. [103]

    Accelerate: Training and inference at scale made simple, efficient and adaptable

    Gugger Sylvain, Debut Lysandre, Wolf Thomas, Schmid Philipp, Mueller Zachary, and Mangrulkar Sourab. Accelerate: Training and inference at scale made simple, efficient and adaptable. https: //github.com/huggingface/accelerate, 2022. Accessed: 2024-08-17

    2022

  96. [104]

    Redco: A Lightweight Tool to Automate Distributed Training of LLMs on Any GPU/TPUs

    Bowen Tan, Yun Zhu, Lijuan Liu, Hongyi Wang, Yonghao Zhuang, Jindong Chen, Eric Xing, and Zhiting Hu. Redco: A Lightweight Tool to Automate Distributed Training of LLMs on Any GPU/TPUs. arXiv preprint arXiv:2310.16355, 2023

    2023 arXiv

  97. [105]

    Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Siamak Shakeri, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler

    Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Siamak Shakeri, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler. UL2: Unifying Language Learning Paradigms. In International Conference ...

    2023

  98. [106]

    MLC team. MLC-LLM. https://github.com/mlc-ai/mlc-llm, 2023

    2023

  99. [107]

    composer

    The Mosaic ML Team. composer. https://github.com/mosaicml/ composer/, 2021. Accessed: 2024-08-16

    2021

  100. [108]

    Baby Llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty

    Inar Timiryasov and Jean-Loup Tastet. Baby Llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty. arXiv preprint arXiv:2308.02019, 2023

    2023 arXiv

  101. [109]

    Llama 2: Open Foundation and Fine-Tuned Chat Models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Alma- hairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Can- ton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fer- nandes, Jeremy...

    2023 arXiv

  102. [110]

    Using Fractional GPUs

    TrueFoundry. Using Fractional GPUs. https://docs.truefoundry.com/ docs/using-fractional-gpus , 2023. Accessed: 2024-10-18

    2023

  103. [111]

    Tenplex: Dynamic Parallelism for Deep Learning using Parallelizable Tensor Collections

    Marcel Wagenlander, Guo Li, Bo Zhao, Luo Mai, and Peter Pietzuch. Tenplex: Dynamic Parallelism for Deep Learning using Parallelizable Tensor Collections. arXiv preprint arXiv:2312.05181, 2023

    2023 arXiv

  104. [112]

    Tesseract: Parallelize the Tensor Parallelism Efficiently

    Boxiang Wang, Qifan Xu, Zhengda Bian, and Yang You. Tesseract: Parallelize the Tensor Parallelism Efficiently. In Proceedings of the 51st International Conference on Parallel Processing , ICPP ’22. ACM, August 2022

    2022

  105. [113]

    Truong, Simran Arora, Mantas Mazeika, Dan Hendrycks, Zinan Lin, Yu Cheng, Sanmi Koyejo, Dawn Song, and Bo Li

    Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, Sang T. Truong, Simran Arora, Mantas Mazeika, Dan Hendrycks, Zinan Lin, Yu Cheng, Sanmi Koyejo, Dawn Song, and Bo Li. Decod- ingTrust: A Compreh...

  106. [114]

    BitNet: Scaling 1-bit Transformers for Large Language Models.arXiv preprint arXiv:2310.11453, 2023

    Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, and Furu Wei. BitNet: Scaling 1-bit Transformers for Large Language Models.arXiv preprint arXiv:2310.11453, 2023

    2023 arXiv

  107. [115]

    Fast Distributed Inference Serving for Large Language Models

    Bingyang Wu, Yinmin Zhong, Zili Zhang, Shengyu Liu, Fangyue Liu, Yuanhang Sun, Gang Huang, Xuanzhe Liu, and Xin Jin. Fast Distributed Inference Serving for Large Language Models. arXiv e-prints, page arXiv:2305.05920, May 2023

    2023 arXiv

  108. [116]

    A Generic, High-Performance, Compression-Aware Framework for Data Parallel DNN Training

    Hao Wu, Shiyi Wang, Youhui Bai, Cheng Li, Quan Zhou, Jun Yi, Feng Yan, Ruichuan Chen, and Yinlong Xu. A Generic, High-Performance, Compression-Aware Framework for Data Parallel DNN Training. IEEE Transactions on Parallel and Distributed Systems , pages 1–20, 2023

    2023

  109. [117]

    Understanding INT4 quantization for language models: latency speedup, composability, and failure cases

    Xiaoxia Wu, Cheng Li, Reza Yazdani Aminabadi, Zhewei Yao, and Yuxiong He. Understanding INT4 quantization for language models: latency speedup, composability, and failure cases. In Proceedings of the 40th International Conference on Machine Learning , ICML’23. JMLR.org, 2023

    2023

  110. [118]

    Smoothquant: Accurate and efficient post-training quantization for large language models

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087–38099. PMLR, 2023

    2023

  111. [119]

    Moe- infinity: Activation-aware expert offloading for efficient moe serving

    Leyang Xue, Yao Fu, Zhan Lu, Luo Mai, and Mahesh Marina. Moe- infinity: Activation-aware expert offloading for efficient moe serving. arXiv preprint arXiv:2401.14361, 2024

    2024 arXiv

  112. [120]

    PowerInfer-2: Fast Large Language Model Inference on a Smartphone

    Zhenliang Xue, Yixin Song, Zeyu Mi, Le Chen, Yubin Xia, and Haibo Chen. PowerInfer-2: Fast Large Language Model Inference on a Smartphone. arXiv preprint arXiv:2406.06282, 2024

    2024 arXiv

  113. [121]

    MOSEC: Model Serving made Efficient in the Cloud

    Keming Yang, Zichen Liu, and Philip Cheng. MOSEC: Model Serving made Efficient in the Cloud. https://github.com/mosecorg/mosec,

  114. [122]

    Talk2Care: An LLM-based Voice Assistant for Communication between Healthcare Providers and Older Adults

    Ziqi Yang, Xuhai Xu, Bingsheng Yao, Ethan Rogers, Shao Zhang, Stephen Intille, Nawar Shara, Guodong Gordon Gao, and Dakuo Wang. Talk2Care: An LLM-based Voice Assistant for Communication between Healthcare Providers and Older Adults. Proc. ACM Interact. Mob. Wearable Ubiquitous...

    2024

  115. [123]

    ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation

    Zhewei Yao, Xiaoxia Wu, Cheng Li, Stephen Youn, and Yuxiong He. ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation. arXiv preprint arXiv:2303.08302, 2023

    2023 arXiv

  116. [124]

    Orca: A Distributed Serving System for Transformer-Based Generative Models

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A Distributed Serving System for Transformer-Based Generative Models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22) , pages 521–538, Carlsbad, CA, July 2022...

    2022

  117. [125]

    Radway, Eric Han, and Priyanka Raina

    Jeffrey Yu, Kartik Prabhu, Yonatan Urman, Robert M. Radway, Eric Han, and Priyanka Raina. 8-bit Transformer Inference and Fine- tuning for Edge Accelerators. In Proceedings of the 29th ACM Interna- tional Conference on Architectural Support for Programming Languages and Operat...

    2024

  118. [126]

    MArk: Exploiting Cloud Services for Cost-Effective, SLO-Aware Machine Learning Inference Serving

    Chengliang Zhang, Minchen Yu, Wei Wang, and Feng Yan. MArk: Exploiting Cloud Services for Cost-Effective, SLO-Aware Machine Learning Inference Serving. In 2019 USENIX Annual Technical Con- ference (USENIX ATC 19), pages 1049–1062, Renton, WA, July 2019. USENIX Association

    2019

  119. [127]

    Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. PE- GASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization. In Proceedings of the 37th International Conference on Machine Learning, ICML’20. JMLR.org, 2020

    2020

  120. [128]

    vPipe: A Virtualized Acceler- ation System for Achieving Efficient and Scalable Pipeline Parallel DNN Training

    Shixiong Zhao, Fanxin Li, Xusheng Chen, Xiuxian Guan, Jianyu Jiang, Dong Huang, Yuhao Qing, Sen Wang, Peng Wang, Gong Zhang, Cheng Li, Ping Luo, and Heming Cui. vPipe: A Virtualized Acceler- ation System for Achieving Efficient and Scalable Pipeline Parallel DNN Training. IEEE...

    2022

  121. [129]

    PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. PyTorch FSDP: Experiences on Sc...

    2023 arXiv

  122. [130]

    Learn to be efficient: Build structured sparsity in large language models

    Haizhong Zheng, Xiaoyan Bai, Beidi Chen, Fan Lai, and Atul Prakash. Learn to be efficient: Build structured sparsity in large language models. arXiv preprint arXiv:2402.06126, 2024

    2024 arXiv

  123. [131]

    Xing, Joseph E

    Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P. Xing, Joseph E. Gonzalez, and Ion Stoica. Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learn- ing. In 16th USENIX Sy...

    2022

  124. [132]

    SparDA: Accelerating Dynamic Sparse Deep Neural Networks via Sparse-Dense Transformation

    Ningxin Zheng, Huiqiang Jiang, Quanlu Zhang, Zhenhua Han, Yuqing Yang, Lingxiao Ma, Fan Yang, Lili Qiu, Mao Yang, and Li- dong Zhou. SparDA: Accelerating Dynamic Sparse Deep Neural Networks via Sparse-Dense Transformation. CoRR, 2023

    2023

  125. [133]

    DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serv- ing

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xu- anzhe Liu, Xin Jin, and Hao Zhang. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serv- ing. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI...

    2024

  126. [2021]

    Accessed: 2024-09-16

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.