Pith. sign in

REVIEW 9 cited by

Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2404.16283 v2 pith:BKCSA6KY submitted 2024-04-25 cs.DC cs.LG

Jiachen Liu , Jae-Won Chung , Zhiyu Wu , Fan Lai , Myungjin Lee , Mosharaf Chowdhury This is my paper
classification cs.DCcs.LG
keywords userandesexperienceservicesservingstreamingtextfirst
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Large language models (LLMs) are now at the core of conversational AI services such as real-time translation and chatbots, which provide live user interaction by incrementally streaming text to the user. However, existing LLM serving systems fail to provide good user experience because their optimization metrics are not always aligned with user experience. In this paper, we first introduce and define the notion of Quality-of-Experience (QoE) for text streaming services by considering each user's end-to-end interaction timeline. Based on this, we propose Andes, a QoE-aware LLM serving system that enhances user experience by ensuring that users receive the first token promptly and subsequent tokens at a smooth, digestible pace, even during surge periods. This is enabled by Andes's preemptive request scheduler that dynamically prioritizes requests at the token granularity based on each request's expected QoE gain and GPU resource usage. Our evaluations demonstrate that, compared to state-of-the-art LLM serving systems, Andes improves the average QoE by up to $4.7\times$ given the same GPU resource, or saves up to 61% GPU resources while maintaining the same high QoE.

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Leaky Language Models: Stealing Architecture and Inference Optimizations via Per-Token Timing

    cs.CR 2026-07 conditional novelty 7.0 of 10

    Per-token generation timing leaks speculative decoding and draft-model context length from Gemini, and recovers layer count and hidden size of Llama-family models with top-5 accuracy up to 65% when both are unknown.

  2. Beyond Prediction: Tail-Aware Scheduling for LLM Inference

    cs.LG 2026-06 unverdicted novelty 7.0 of 10

    Presents a distribution-aware scheduling framework for LLM inference that reduces P99 TTLT by 35-50% and TTFT by 34-47% versus SRPT with perfect length knowledge using statistical signals instead of predictions.

  3. Regulating Branch Parallelism in LLM Serving

    cs.DC 2026-05 unverdicted novelty 7.0 of 10

    TAPER regulates LLM branch parallelism by admitting extra branches opportunistically when predicted externality fits slack, delivering 1.48-1.77x higher goodput than eager or fixed-cap baselines on Qwen3-32B while kee...

  4. CacheFlow: Efficient LLM Serving with 3D-Parallel KV Cache Restoration

    cs.DC 2026-04 unverdicted novelty 7.0 of 10

    CacheFlow cuts TTFT by 10-62% in batched LLM serving via 3D-parallel KV cache restoration and a two-pointer scheduler that overlaps recompute and I/O.

  5. EnerInfer: Energy-Aware On-Device LLM Inference

    cs.SE 2026-06 unverdicted novelty 6.0 of 10

    EnerInfer uses model-structure-aware predictions and online feedback to select energy-efficient NPU and memory frequencies for on-device LLM inference while preserving QoE and managing thermal limits.

  6. LiveServe: Interaction-Aware Serving for Real-Time Omni-Modal LLMs

    cs.DC 2026-06 unverdicted novelty 6.0 of 10

    LiveServe exposes audio playback and barge-in signals to the scheduler and KV manager, lowering P90 audio TTFP by 1.55x on average and raising completed-request throughput by 1.15x on two Omni-LMs.

  7. GPU-to-Grid: Voltage Regulation via GPU Utilization Control

    eess.SY 2026-02 conditional novelty 6.0 of 10

    GPU batch-size control, driven by real-time voltage and latency feedback, can serve as a fast distribution-grid voltage regulation resource.

  8. Compass: SLO-aware Query Planner for Compound AI Serving at Scale

    cs.DB 2025-04 unverdicted novelty 6.0 of 10

    Compass decomposes multi-query multi-SLO planning for compound AI serving, exploits plan similarities, uses selective profiling, and applies bipartite matching at runtime to deliver 2.4-5.1x higher goodput and 3.8-4.5...

  9. ServeGen: Workload Characterization and Generation of Large Language Model Serving in Production

    cs.DC 2025-05 unverdicted novelty 5.0 of 10

    ServeGen characterizes production LLM inference workloads across model types and generates realistic per-client composed workloads that reduce under-provisioning by 50% in a production validation.

Pith tools