Pith. sign in

REVIEW 12 cited by

vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.04437 v3 pith:HFW7PH7U submitted 2024-05-07 cs.LG cs.OS

Ramya Prabhu , Ajay Nayak , Jayashree Mohan , Ramachandran Ramjee , Ashish Panwar This is my paper · ORCID
classification cs.LGcs.OS
keywords memoryvirtualpagedattentionallocationcachephysicalservingvattention
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

PagedAttention is a popular approach for dynamic memory allocation in LLM serving systems. It enables on-demand allocation of GPU memory to mitigate KV cache fragmentation -- a phenomenon that crippled the batch size (and consequently throughput) in prior systems. However, in trying to allocate physical memory at runtime, PagedAttention ends up changing the virtual memory layout of the KV cache from contiguous to non-contiguous. Such a design leads to non-trivial programming and performance overheads. We present vAttention -- an approach that mitigates fragmentation in physical memory while retaining the contiguity of KV cache in virtual memory. We achieve this by decoupling the allocation of virtual and physical memory using CUDA virtual memory management APIs. We also introduce various LLM-specific optimizations to address the limitations of CUDA virtual memory support. Overall, vAttention is a simpler, portable, and performant alternative to PagedAttention: it supports various attention kernels out-of-the-box and improves LLM serving throughput by up to 1.23x compared to the use of PagedAttention-based kernels of FlashAttention and FlashInfer.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents

    cs.LG 2026-07 conditional novelty 6.0 of 10

    KV-cache eviction, prompt compression, recurrent state bounding, and agent memory consolidation are unified as one rate-distortion problem with a shared lower bound, shared failure mode, and transferable mechanisms.

  2. Keyless Attention: Value-Space Routing and Value-Only Caching for Efficient Transformers

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    Keyless Attention eliminates the key projection in transformers via value-space routing, cutting KV cache size by 50% with competitive or superior performance on language modeling and zero-shot benchmarks.

  3. Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Serving

    cs.DC 2026-03 conditional novelty 6.0 of 10

    Pattern-aware speculative tool execution cuts agent end-to-end latency by roughly half and observed tool latency by about 1.8× by overlapping predicted tools with LLM generation.

  4. VoltanaLLM: Energy-Efficient and SLO-Aware Disaggregated LLM Serving via Adaptive Frequency Control and State-Space Routing

    cs.DC 2025-09 conditional novelty 6.0 of 10

    VoltanaLLM saves up to 36.3% GPU energy in LLM serving by per-iteration, phase-aware frequency scaling and state-space routing that preserve latency SLOs.

  5. HGCA: Hybrid GPU-CPU Attention for Long Context LLM Inference

    cs.LG 2025-07 conditional novelty 6.0 of 10

    HGCA splits attention between GPU (dense, recent KV) and CPU (sparse, salient KV) and merges partial results with exact log-sum-exp fusion, scaling long-context decoding on commodity GPUs.

  6. Hardware-based Heterogeneous Memory Management for Large Language Model Inference

    cs.AR 2025-04 conditional novelty 6.0 of 10

    H2M2 uses an asymmetric HBM+LPDDR memory with compute on both sides and a dynamic kernel-mapping algorithm to speed up LLM generation, beating an LPDDR-only system by up to 2.94x but lagging an 8-HBM system by 1.57x.

  7. iServe: An Intent-based Serving System for LLMs

    cs.SE 2025-01 conditional novelty 6.0 of 10

    iServe uses small proxy models, called fingerprints, to estimate latency and memory for many LLM deployment configurations and then selects the configuration that best satisfies a user's intent.

  8. cuRAMSES: Scalable AMR Optimizations for Large-Scale Cosmological Simulations

    astro-ph.GA 2026-04 conditional novelty 5.0 of 10

    Recursive k-section domain decomposition, Morton-key hashing, and GPU dispatch cut communication and memory bottlenecks in RAMSES while preserving conservation to ~0.5%.

  9. AutoChemSchematic AI: Agentic Physics-Aware Automation for Chemical Manufacturing Scale-Up

    cs.LG 2025-05 reject novelty 4.0 of 10

    The framework trains small models on synthetic AI-generated data to produce PFD/PID text, then validates two examples by manual DWSIM setup, leaving the industrial-viability claim unproven.

  10. EcoServe: Enabling Cost-effective LLM Serving with Proactive Intra- and Inter-Instance Orchestration

    cs.DC 2025-04 conditional novelty 4.0 of 10

    A time-sliced, cyclically staggered scheduling strategy, partial disaggregation, is shown to improve SLO-aware goodput for 30B to 72B LLMs on commodity-interconnect clusters relative to co-located and fully disaggrega...

  11. Position: AI Scaling: From Up to Down and Out

    cs.LG 2025-02 conditional novelty 4.0 of 10

    AI scaling is reframed as three paradigms: Scaling Up, Scaling Down, and Scaling Out, with future gains predicted to come from down and out.

  12. Token-Operations-Oriented Inference Optimization Techniques for Large Models

    cs.SE 2026-06 unverdicted novelty 3.0 of 10

    The paper introduces a four-layer technical architecture for token-operations-oriented inference optimization in large models and reviews key technologies and industry status at each layer.

Pith tools