Pith. sign in

REVIEW 17 cited by

Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2401.11181 v1 pith:LQ67J327 submitted 2024-01-20 cs.DC

Cunchen Hu , Heyang Huang , Liangliang Xu , Xusheng Chen , Jiang Xu , Shuang Chen , Hao Feng , Chenxi Wang
show 4 more authors
Sa Wang Yungang Bao Ninghui Sun Yizhou Shan
This is my paper · ORCID
classification cs.DC
keywords inferencedecodeinterferenceaveragecharacteristicslargephaseprefill
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Transformer-based large language model (LLM) inference serving is now the backbone of many cloud services. LLM inference consists of a prefill phase and a decode phase. However, existing LLM deployment practices often overlook the distinct characteristics of these phases, leading to significant interference. To mitigate interference, our insight is to carefully schedule and group inference requests based on their characteristics. We realize this idea in TetriInfer through three pillars. First, it partitions prompts into fixed-size chunks so that the accelerator always runs close to its computationsaturated limit. Second, it disaggregates prefill and decode instances so each can run independently. Finally, it uses a smart two-level scheduling algorithm augmented with predicted resource usage to avoid decode scheduling hotspots. Results show that TetriInfer improves time-to-first-token (TTFT), job completion time (JCT), and inference efficiency in turns of performance per dollar by a large margin, e.g., it uses 38% less resources all the while lowering average TTFT and average JCT by 97% and 47%, respectively.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 17 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Rethinking AI Cloud Infrastructure for Agentic Serving Systems with the Aries Experimentation Framework

    cs.DC 2026-07 conditional novelty 7.0 of 10

    Agent serving faces non-inference bottlenecks (up to 48% of latency), a 4.4x serving-capacity loss from long context, and 4.9x cost amplification from snapshot-based sandbox suspension.

  2. Sangam: Efficiently Serving Diffusion LLMs with the AR Stack

    cs.DC 2026-07 conditional novelty 6.5 of 10

    Deficit token-budget scheduling plus hybrid prefill overflow makes AR-style colocated and disaggregated serving work for cached diffusion LLMs with block-sized, recurring prefills.

  3. CoCoScale: Leveraging Layer-wise Scaling to Unlock the Potential of Online LLM Serving

    cs.DC 2026-07 conditional novelty 6.5 of 10

    Layer-wise data-parallel replication of hot Transformer layers onto reclaimed idle GPUs reduces LLM serving cold-start latency 97.9–99.3% and average latency 20.7–28.1% while attaining 100% SLO on production traces.

  4. TensorCast: The Missing Tensor Management Layer in Large Language Model Infrastructure

    cs.DC 2026-08 conditional novelty 6.0 of 10

    TensorCast packages tensor lifecycle management into a programmable service layer and reports it can match specialized LLM systems while enabling new cross-component optimization policies.

  5. Energy-Efficient LLM Serving via Disaggregated Attention--FFN and Flexible Frequency Scaling

    cs.DC 2026-08 conditional novelty 6.0 of 10

    AFlex combines operator-level disaggregation with per-operator DVFS to cut LLM serving energy per token by up to 49% without violating P90 TTFT/TPOT SLOs.

  6. SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cache Transfer

    cs.DC 2026-07 conditional novelty 6.0 of 10

    Selective three-path KV cache transfer (profile-proactive, parallel on-demand, speculative) cuts time-to-second-token up to 4.3× vs full transfer on bandwidth-limited cloud GPUs with near-baseline accuracy and decode speed.

  7. AutoSLO: Practical Latency SLOs on Cloud Data Warehouses -- Extended Version

    cs.DB 2026-07 conditional novelty 6.0 of 10

    AutoSLO meets query latency SLOs on multi-cluster cloud warehouses while cutting cost ~26% versus the next-best baseline by combining forecast-driven planning, reactive scaling, and concurrency-aware routing.

  8. Efficient Multi-round LLM Inference over Disaggregated Serving

    cs.DC 2026-02 conditional novelty 6.0 of 10

    AMPD adaptively routes incremental prefill tasks between prefill and decode workers and reorders queued prefill jobs to improve SLO attainment for multi-round LLM inference under prefill-decode disaggregation.

  9. Hetis: Serving LLMs in Heterogeneous GPU Clusters with Fine-grained and Dynamic Parallelism

    cs.DC 2025-09 conditional novelty 6.0 of 10

    Hetis dynamically splits LLM attention across heterogeneous GPUs at the granularity of individual heads while keeping dense computation on a selected subset, and reports up to 2.25x throughput gains over Splitwise and Hexgen.

  10. Taming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inference

    cs.DC 2025-08 conditional novelty 6.0 of 10

    HeteroScale coordinates scaling of prefill and decode pools using decode TPS as a single robust signal, reporting a 26.6 percentage point GPU utilization gain in production.

  11. Nexus:Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Serving

    cs.DC 2025-07 conditional novelty 6.0 of 10

    Nexus performs proactive intra-GPU disaggregation of prefill and decode, using an analytical cost model and greedy search to dynamically partition SMs, achieving up to 2.2x throughput gains over vLLM.

  12. Memory Offloading for Large Language Model Inference with Latency SLO Guarantees

    cs.DC 2025-02 conditional novelty 6.0 of 10

    Select-N meets LLM latency SLOs by choosing and continuously adjusting an offloading interval that controls how much model state lives in CPU memory, increasing host memory usage and throughput versus prior offloading...

  13. HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environment

    cs.DC 2025-02 conditional novelty 6.0 of 10

    A scheduling algorithm for disaggregated LLM inference on heterogeneous GPUs that improves throughput by up to 2x and cuts cost by 30% relative to state-of-the-art serving systems.

  14. Beyond the Buzz: A Pragmatic Take on Inference Disaggregation

    cs.DC 2025-06 conditional novelty 5.0 of 10

    Disaggregated serving (separate prefill and decode GPU pools) expands the throughput-interactivity Pareto frontier mainly for prefill-heavy workloads and models larger than about 10B parameters, provided the prefill-t...

  15. HACK: Homomorphic Acceleration via Compression of the Key-Value Cache for Disaggregated LLM Inference

    cs.DC 2025-02 conditional novelty 5.0 of 10

    HACK performs attention matrix multiplications directly on 2-bit quantized key-value cache entries, eliminating per-iteration dequantization and cutting end-to-end latency in disaggregated LLM serving.

  16. Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs

    cs.DC 2025-02 conditional novelty 5.0 of 10

    A MILP scheduler that jointly optimizes GPU composition, deployment configuration, and workload assignment reports 20-41% cost-efficiency gains over homogeneous GPU clusters for LLM serving.

  17. Hybrid Offline-online Scheduling Method for Large Language Model Inference Optimization

    cs.DC 2025-02 reject novelty 4.0 of 10

    A hybrid offline-online scheduler claims to raise LLM inference hardware utilization by about 9 points, but its MIP model and 'Lagrangian' rules contain mathematical flaws.

Pith tools