Pith. sign in

REVIEW 6 cited by

Punica: Multi-Tenant LoRA Serving

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.18547 v1 pith:HNXIJQYP submitted 2023-10-28 cs.DC cs.LG

Lequn Chen (1) , Zihao Ye (1) , Yongji Wu (2) , Danyang Zhuo (2) , Luis Ceze (1) , Arvind Krishnamurthy (1) ((1) University of Washington , (2) Duke University) This is my paper · ORCID
classification cs.DCcs.LG
keywords lorapunicamodelsservingclustermultipleallowsdifferent
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Low-rank adaptation (LoRA) has become an important and popular method to adapt pre-trained models to specific domains. We present Punica, a system to serve multiple LoRA models in a shared GPU cluster. Punica contains a new CUDA kernel design that allows batching of GPU operations for different LoRA models. This allows a GPU to hold only a single copy of the underlying pre-trained model when serving multiple, different LoRA models, significantly enhancing GPU efficiency in terms of both memory and computation. Our scheduler consolidates multi-tenant LoRA serving workloads in a shared GPU cluster. With a fixed-sized GPU cluster, our evaluations show that Punica achieves 12x higher throughput in serving multiple LoRA models compared to state-of-the-art LLM serving systems while only adding 2ms latency per token. Punica is open source at https://github.com/punica-ai/punica .

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PoLoRA: A Preconditioned Orthogonalized LoRA Optimizer

    cs.LG 2026-07 conditional novelty 6.0 of 10

    A preconditioned, product-aware spectral optimizer for LoRA reaches Adam's final loss in 1.2–1.7x fewer steps on instruction-tuning, with ~3% per-step overhead.

  2. HMI: Hierarchical Knowledge Management for Efficient Multi-Tenant Inference in Pretrained Language Models

    cs.LG 2025-04 conditional novelty 6.0 of 10

    HMI serves up to 10,000 tenant-specific BERT and GPT instances on one GPU by replacing lower-layer computation with trigram lookup tables and sharing upper layers with per-task adapters.

  3. PARA: Parameter-Efficient Fine-tuning with Prompt Aware Representation Adjustment

    cs.CL 2025-02 conditional novelty 6.0 of 10

    PARA generates prompt-conditioned scaling vectors for Q, V, and FFN activations, outperforming (IA)^3 and LoRA-style baselines on several benchmarks with similar parameter counts and lower multi-tenant inference latency.

  4. BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batching

    cs.LG 2024-11 conditional novelty 6.0 of 10

    BlendServe combines resource-aware batching with prefix sharing using a resource-aware prefix tree and dual scanner, achieving up to 1.44x throughput vs vLLM/SGLang in offline LLM inference.

  5. JoyNexus: Service-Oriented Multi-Tenant Post-Training for VLA Models

    cs.DC 2026-07 conditional novelty 5.0 of 10

    A service-oriented multi-tenant architecture with schema-compatible group batching reduces aggregate GPU time for VLA post-training by about 28% in simulation.

  6. EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devices

    cs.DC 2025-07 conditional novelty 5.0 of 10

    EdgeLoRA combines automatic adapter routing, LRU caching with a memory pool, and grouped LoRA batching to serve thousands of LoRA adapters on edge devices with up to 4x higher throughput than llama.cpp.

Pith tools