LeoAM reports a 3.46x average latency speedup for long-context LLM inference on one commodity GPU by adaptively chunking KV data and loading compact key abstracts from disk instead of full KV values.
Title resolution pending
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.OS 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Breaking the Boundaries of Long-Context LLM Inference: Adaptive KV Management on a Single Commodity GPU
LeoAM reports a 3.46x average latency speedup for long-context LLM inference on one commodity GPU by adaptively chunking KV data and loading compact key abstracts from disk instead of full KV values.