Pith. sign in

REVIEW 13 cited by

LLM Inference Serving: Survey of Recent Advances and Opportunities

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2407.12391 v1 pith:7X2XVFCQ submitted 2024-07-17 cs.DC cs.AI

Baolin Li , Yankai Jiang , Vijay Gadepally , Devesh Tiwari This is my paper · ORCID
classification cs.DCcs.AI
keywords surveyrecentservingabreastadvancementsadvancesalteringcomprehensive
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

This survey offers a comprehensive overview of recent advancements in Large Language Model (LLM) serving systems, focusing on research since the year 2023. We specifically examine system-level enhancements that improve performance and efficiency without altering the core LLM decoding mechanisms. By selecting and reviewing high-quality papers from prestigious ML and system venues, we highlight key innovations and practical considerations for deploying and scaling LLMs in real-world production environments. This survey serves as a valuable resource for LLM practitioners seeking to stay abreast of the latest developments in this rapidly evolving field.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 13 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Adaptive LLM Routing under Budget Constraints

    cs.LG 2025-08 conditional novelty 6.0 of 10

    LLM routing is framed as a budget-constrained contextual bandit, solved by a preference-prior initialized LinUCB variant with an online multi-choice knapsack cost policy.

  2. Faster Machine Translation Ensembling with Reinforcement Learning and Competitive Correction

    cs.CL 2025-01 conditional novelty 6.0 of 10

    A DQN-based candidate selection and a competitive correction block improve MT ensembling quality while reducing inference cost on English-Hindi and Hindi-English tasks.

  3. APEX: An Extensible and Dynamism-Aware Simulator for Automated Parallel Execution in LLM Serving

    cs.DC 2024-11 conditional novelty 6.0 of 10

    A CPU-based simulator that searches hybrid data, pipeline, and tensor parallelism plans for LLM serving, claiming up to 3.37x faster plans than heuristics in simulation and 10.7% average speedup-prediction error.

  4. InstantInfer: Enabling Fast LLM Cold Start with Communicating Finite Automata

    cs.DC 2026-07 conditional novelty 5.0 of 10

    InstantInfer refactors vLLM's cold start into a concurrent state-machine pipeline, speeding up startup by up to 7.2×.

  5. Unveiling the Landscape of LLM Deployment in the Wild: An Empirical Study

    cs.CR 2025-05 conditional novelty 5.0 of 10

    An internet-wide scan of 320,102 public-facing LLM services across 15 frameworks finds widespread plain-HTTP exposure, weak TLS, and unauthenticated access to model and system APIs.

  6. Universal Model Routing for Efficient LLM Inference

    cs.CL 2025-02 conditional novelty 5.0 of 10

    UniRoute represents each language model by its error rates on a few prompt clusters, letting a router choose among models it has never seen during training.

  7. DeServe: Towards Affordable Offline LLM Inference via Decentralization

    cs.DC 2025-01 conditional novelty 5.0 of 10

    DeServe combines KV cache offloading and microbatch scheduling to maintain high LLM offline inference throughput in high-latency decentralized environments.

  8. Less is More: Optimizing Function Calling for LLM Execution on Edge Devices

    cs.PF 2024-11 conditional novelty 5.0 of 10

    Selectively reducing the number of tools presented to an LLM, using embedding similarity over individual tools or clusters, improves function-calling success and efficiency on edge devices.

  9. Joint Partitioning and Placement of Foundation Models for Real-Time Edge AI

    cs.DC 2025-11 reject novelty 4.0 of 10

    A framework for runtime re-splitting and re-placement of foundation model layers across edge nodes is proposed, but its claimed latency gains are inherited from prior work rather than measured.

  10. Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniques

    cs.LG 2025-06 unverdicted novelty 4.0 of 10

    A survey of LLM routing and hierarchical inference techniques that proposes an unvalidated unified evaluation metric called the Inference Efficiency Score.

  11. Challenging GPU Dominance: When CPUs Outperform for On-Device LLM Inference

    cs.DC 2025-05 conditional novelty 4.0 of 10

    On an iPhone 15 Pro, CPU-only llama.cpp inference achieved 17 tokens/s with a 1B model, beating the 12.8 tokens/s GPU path, but only for small models and without a confirmed cause.

  12. Taming the Titans: A Survey of Efficient LLM Inference Serving

    cs.CL 2025-04 conditional novelty 3.0 of 10

    A survey that classifies LLM inference serving techniques into instance, cluster, emerging-scenario, and miscellaneous categories, with a companion GitHub index and four research directions.

  13. A Survey on Inference Optimization Techniques for Mixture of Experts Models

    cs.LG 2024-12 conditional novelty 3.0 of 10

    A structured survey of MoE inference optimization that categorizes existing techniques into model, system, and hardware levels and summarizes reported speedups and memory savings.

Pith tools