REVIEW 11 cited by
DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
The past several years have witnessed the success of transformer-based models, and their scale and application scenarios continue to grow aggressively. The current landscape of transformer models is increasingly diverse: the model size varies drastically with the largest being of hundred-billion parameters; the model characteristics differ due to the sparsity introduced by the Mixture-of-Experts; the target application scenarios can be latency-critical or throughput-oriented; the deployment hardware could be single- or multi-GPU systems with different types of memory and storage, etc. With such increasing diversity and the fast-evolving pace of transformer models, designing a highly performant and efficient inference system is extremely challenging. In this paper, we present DeepSpeed Inference, a comprehensive system solution for transformer model inference to address the above-mentioned challenges. DeepSpeed Inference consists of (1) a multi-GPU inference solution to minimize latency while maximizing the throughput of both dense and sparse transformer models when they fit in aggregate GPU memory, and (2) a heterogeneous inference solution that leverages CPU and NVMe memory in addition to the GPU memory and compute to enable high inference throughput with large models which do not fit in aggregate GPU memory. DeepSpeed Inference reduces latency by up to 7.3X over the state-of-the-art for latency-oriented scenarios and increases throughput by over 1.5x for throughput-oriented scenarios. Moreover, it enables trillion parameter scale inference under real-time latency constraints by leveraging hundreds of GPUs, an unprecedented scale for inference. It can inference 25x larger models than with GPU-only solutions, while delivering a high throughput of 84 TFLOPS (over $50\%$ of A6000 peak).
Forward citations
Cited by 11 Pith papers
-
SCOPE: Compress Mathematical Reasoning Steps for Efficient Automated Process Annotation
SCOPE compresses sampled math reasoning traces into a prefix tree of AST-normalized code steps, yielding PRM training labels at O(N) cost and improving Best-of-N and ProcessBench scores over automated baselines.
-
Predict-then-Diffuse: Adaptive Response Length for Compute-Budgeted Inference in Diffusion LLMs
Predict-then-Diffuse predicts response lengths for diffusion LLMs via an auxiliary model and safety buffer to reduce FLOP waste while preserving output quality.
-
Hardware-Efficient Attention for Fast Decoding
Grouped-Tied Attention and Grouped Latent Attention reduce KV-cache memory and speed up LLM decoding by up to 2x while matching the quality of GQA and MLA at up to 1.47B parameters.
-
iServe: An Intent-based Serving System for LLMs
iServe uses small proxy models, called fingerprints, to estimate latency and memory for many LLM deployment configurations and then selects the configuration that best satisfies a user's intent.
-
FastSwitch: Optimizing Context Switching Efficiency in Fairness-aware Large Language Model Serving
FastSwitch reduces context-switching overhead in fairness-aware LLM serving with coarse-grained KV cache allocation, asynchronous multithreaded swapping, and KV cache reuse, reporting 1.4-11.2x tail-latency speedups o...
-
LLaSA: Large Language and Structured Data Assistant
A single hypergraph encoder and G-Former, pretrained on 25 million tables, can be appended to different LLMs to improve their performance on table, knowledge graph, and database structured knowledge grounding tasks.
-
Trustworthy Reasoning: Evaluating and Enhancing Factual Accuracy in LLM Intermediate Thought Processes
The paper reports that a fact-checker trained on replaced entities, used both as a reward and as the evaluation metric, raises measured step-factuality of small open LLMs by up to 49.9 percentage points.
-
EvolveSearch: An Iterative Self-Evolving Search Agent
An iterative loop of RL and filtered SFT on the agent's own rollouts improves a 7B web-search agent by a few accuracy points on multi-hop QA benchmarks.
-
Glinthawk: A Two-Tiered Architecture for Offline LLM Inference
A two-tier inference system that offloads attention and KV cache to cheap CPU nodes raises offline LLM throughput about 6x and lowers hardware cost about 2.8x in a T4-based prototype.
-
Taming the Titans: A Survey of Efficient LLM Inference Serving
A survey that classifies LLM inference serving techniques into instance, cluster, emerging-scenario, and miscellaneous categories, with a companion GitHub index and four research directions.
-
Aero-LLM: A Distributed Framework for Secure UAV Communication and Intelligent Decision-Making
The paper assembles off-the-shelf language models into a distributed drone security pipeline, but its evaluation is too limited to support the claimed cyber-defense benefits.
Discussion (0). Continue with ORCID to comment.