REVIEW 6 cited by
Etalon: Holistic Performance Evaluation Framework for LLM Inference Systems
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Serving large language models (LLMs) in production can incur substantial costs, which has prompted recent advances in inference system optimizations. Today, these systems are evaluated against conventional latency and throughput metrics (eg. TTFT, TBT, Normalised Latency and TPOT). However, these metrics fail to fully capture the nuances of LLM inference, leading to an incomplete assessment of user-facing performance crucial for real-time applications such as chat and translation. In this paper, we first identify the pitfalls of current performance metrics in evaluating LLM inference systems. We then propose Etalon, a comprehensive performance evaluation framework that includes fluidity-index -- a novel metric designed to reflect the intricacies of the LLM inference process and its impact on real-time user experience. Finally, we evaluate various existing open-source platforms and model-as-a-service offerings using Etalon, discussing their strengths and weaknesses. Etalon is available at https://github.com/project-etalon/etalon.
Forward citations
Cited by 6 Pith papers
-
Accelerating LLM Inference with Flexible N:M Sparsity via A Fully Digital Compute-in-Memory Accelerator
A pruning method and a digital compute-in-memory accelerator that jointly support layer-wise flexible N:M sparsity, improving LLM perplexity and zero-shot accuracy over fixed N:M baselines while cutting simulated infe...
-
Diagnose Before You Compress: Prediction-Independent Bottleneck Witness Refinement for LLM Serving Traces
BPW builds compact LLM serving replay suites that keep at least two direct witnesses for each of four bottleneck components and beats 16 baseline policies on quality metrics.
-
iServe: An Intent-based Serving System for LLMs
iServe uses small proxy models, called fingerprints, to estimate latency and memory for many LLM deployment configurations and then selects the configuration that best satisfies a user's intent.
-
RAGDoll: Efficient Offloading-based Online RAG System on a Single GPU
RAGDoll pipelines retrieval and generation, jointly manages memory across disk, RAM, and GPU, and adaptively sizes batches to cut average RAG latency by up to 3.6x on a single GPU.
-
Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniques
A survey of LLM routing and hierarchical inference techniques that proposes an unvalidated unified evaluation metric called the Inference Efficiency Score.
-
Taming the Titans: A Survey of Efficient LLM Inference Serving
A survey that classifies LLM inference serving techniques into instance, cluster, emerging-scenario, and miscellaneous categories, with a companion GitHub index and four research directions.
Discussion (0). Continue with ORCID to comment.