CRAwLeR creates and validates Danish and Polish benchmarks for cross-reference-aware legal chunk retrieval, where current methods reach only 55-59% Recall@10 and the gap is attributed to the contextualizing LLM.
hub
Prompt cache: Modular attention reuse for low-latency inference
14 Pith papers cite this work, alongside 12 external citations. Polarity classification is still indexing.
14
Pith papers citing it
12
external citations · external index
hub tools
citation-role summary
background 2
baseline 1
citation-polarity summary
representative citing papers
Execution-state capsules enable graph-bound full-state checkpointing and sub-millisecond restore for LLMs including KV and recurrent states, yielding 3.9x-27x TTFT speedups in on-device physical-AI serving.
On a real multi-node H100 cluster the authors show that for MLA, routing the ~1 KB compressed query row is cheaper than moving cache chunks and supply a topology-aware cost model accurate to ~7% on IBGDA fabrics.
Establishes the first rigorous framework for continuous semantic caching of LLM responses using ε-net discretization and kernel ridge regression, with sublinear regret bounds.
PrefixWall mitigates APC side channels in multi-tenant LLM systems via selective prefix isolation, delivering up to 70% higher cache reuse and 30% lower latency than full-isolation baselines.
KVCodec uses GPU-native video codecs and pipelined fetching to compress and transmit KV caches, delivering up to 3.51x faster TTFT than prior methods while preserving accuracy.
KernelSight-LM simulates LLM inference at kernel granularity with cross-generation (12.1% per-kernel error) and target-measured (3.8% error) tiers, yielding end-to-end median errors of 15.4%/12.8%/3.0% and 14.3%/6.2%/2.7% for TTFT/TPOT/throughput across six model families.
Prevalence study of 10k GitHub repos identifies duplication, low revision, and missing permissions in LLM agent configs; proposes Rel(AI)Build control plane with content addressing, tiered permissions, state machine, and Jaccard drift detection.
SGLang is a new system that speeds up structured LLM programs by up to 6.4x using RadixAttention for KV cache reuse and compressed finite state machines for output decoding.
MiniPIC enables multiple position-independent caching methods inside vLLM via unrotated KV storage, per-request RoPE application, and three primitives, delivering 49% prefill throughput gains and up to 100x lower cached-span TTFT on 2WikiMultihopQA.
SIFT precomputes selective attention indices via local and cross-attention invariance to speed RAG prefill 1.71x while keeping accuracy within 1% of full recompute, storing only bit vectors 24,000x smaller than KV tensors.
Audits if OpenRouter's gateway architecture with shared credentials inadvertently enables cross-user prompt cache leakage despite provider isolation guarantees.
Workload-aware optimizations for LLM serving in AML and fraud detection yield substantial gains in throughput, latency, and GPU utilization on synthetic compliance prompts.
citing papers explorer
-
CRAwLeR -- Cross-Reference Aware Legal Retrieval
CRAwLeR creates and validates Danish and Polish benchmarks for cross-reference-aware legal chunk retrieval, where current methods reach only 55-59% Recall@10 and the gap is attributed to the contextualizing LLM.
-
Execution-State Capsules: Graph-Bound Execution-State Checkpoint and Restore for Low-Latency, Small-Batch, On-Device Physical-AI Serving
Execution-state capsules enable graph-bound full-state checkpointing and sub-millisecond restore for LLMs including KV and recurrent states, yielding 3.9x-27x TTFT speedups in on-device physical-AI serving.
-
Move the Query, Not the Cache: Characterizing Cross-Instance Latent Attention Redistribution Across GPU Fabrics
On a real multi-node H100 cluster the authors show that for MLA, routing the ~1 KB compressed query row is cheaper than moving cache chunks and supply a topology-aware cost model accurate to ~7% on IBGDA fabrics.
-
Continuous Semantic Caching for Low-Cost LLM Serving
Establishes the first rigorous framework for continuous semantic caching of LLM responses using ε-net discretization and kernel ridge regression, with sublinear regret bounds.
-
PrefixWall: Mitigating Prefix Caching Side Channels in Shared LLM Systems
PrefixWall mitigates APC side channels in multi-tenant LLM systems via selective prefix isolation, delivering up to 70% higher cache reuse and 30% lower latency than full-isolation baselines.
-
Efficient Remote KV Cache Reuse with GPU-native Video Codec
KVCodec uses GPU-native video codecs and pipelined fetching to compress and transmit KV caches, delivering up to 3.51x faster TTFT than prior methods while preserving accuracy.
-
KernelSight-LM: A Kernel-Level LLM Inference Simulator
KernelSight-LM simulates LLM inference at kernel granularity with cross-generation (12.1% per-kernel error) and target-measured (3.8% error) tiers, yielding end-to-end median errors of 15.4%/12.8%/3.0% and 14.3%/6.2%/2.7% for TTFT/TPOT/throughput across six model families.
-
A Deterministic Control Plane for LLM Coding Agents
Prevalence study of 10k GitHub repos identifies duplication, low revision, and missing permissions in LLM agent configs; proposes Rel(AI)Build control plane with content addressing, tiered permissions, state machine, and Jaccard drift detection.
-
SGLang: Efficient Execution of Structured Language Model Programs
SGLang is a new system that speeds up structured LLM programs by up to 6.4x using RadixAttention for KV cache reuse and compressed finite state machines for output decoding.
-
MiniPIC: Flexible Position-Independent Caching in <100LOC
MiniPIC enables multiple position-independent caching methods inside vLLM via unrotated KV storage, per-request RoPE application, and three primitives, delivering 49% prefill throughput gains and up to 100x lower cached-span TTFT on 2WikiMultihopQA.
-
SIFT: Selective-Index For Fast Compute of RAG Prefill by Exploiting Attention Invariance
SIFT precomputes selective attention indices via local and cross-attention invariance to speed RAG prefill 1.71x while keeping accuracy within 1% of full recompute, storing only bit vectors 24,000x smaller than KV tensors.
-
CacheProbe: Auditing Prompt Cache Isolation in Gateway APIs
Audits if OpenRouter's gateway architecture with shared credentials inadvertently enables cross-user prompt cache leakage despite provider isolation guarantees.
-
Rethinking LLMOps for Fraud and AML: Building a Compliance-Grade LLM Serving Stack
Workload-aware optimizations for LLM serving in AML and fraud detection yield substantial gains in throughput, latency, and GPU utilization on synthetic compliance prompts.
- HieraSparse: Hierarchical Semi-Structured Sparse KV Attention