Pith. sign in

REVIEW 9 cited by

CaraServe: CPU-Assisted and Rank-Aware LoRA Serving for Generative LLM Inference

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2401.11240 v1 pith:5P36TSLH submitted 2024-01-20 cs.DC

Suyi Li , Hanfeng Lu , Tianyuan Wu , Minchen Yu , Qizhen Weng , Xusheng Chen , Yizhou Shan , Binhang Yuan
show 1 more author
Wei Wang
This is my paper · ORCID
classification cs.DC
keywords caraserveloraadaptersbasegpusmodelservingactivated
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
abstract

Pre-trained large language models (LLMs) often need specialization for domain-specific tasks. Low-Rank Adaptation (LoRA) is a popular approach that adapts a base model to multiple tasks by adding lightweight trainable adapters. In this paper, we present CaraServe, a system that efficiently serves many LoRA adapters derived from a common base model. CaraServe maintains the base model on GPUs and dynamically loads activated LoRA adapters from main memory. As GPU loading results in a cold-start that substantially delays token generation, CaraServe employs a CPU-assisted approach. It early starts the activated adapters on CPUs for prefilling as they are being loaded onto GPUs; after loading completes, it then switches to the GPUs for generative LoRA inference. CaraServe develops a highly optimized synchronization mechanism to efficiently coordinate LoRA computation on the CPU and GPU. Moreover, CaraServe employs a rank-aware scheduling algorithm to optimally schedule heterogeneous LoRA requests for maximum service-level objective (SLO) attainment. We have implemented CaraServe and evaluated it against state-of-the-art LoRA serving systems. Our results demonstrate that CaraServe can speed up the average request serving latency by up to 1.4$\times$ and achieve an SLO attainment of up to 99%.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PLoRA: An NDP-Enhanced Pooled-Memory System for Cost-Efficient Multi-LoRA Serving

    cs.AR 2026-08 conditional novelty 6.0 of 10

    PLoRA places LoRA adapters and KV cache in CXL pooled memory with near-data processors, cutting decode latency by about 6.6x versus S-LoRA in simulation.

  2. Data Driven Optimization of GPU efficiency for Distributed LLM-Adapter Serving

    cs.DC 2026-02 conditional novelty 6.0 of 10

    A simulator-plus-ML pipeline places LoRA adapters onto GPUs so a given workload needs fewer GPUs (60% claimed on average) without request starvation or memory errors.

  3. Efficient Serving of LLM Applications with Probabilistic Demand Modeling

    cs.DC 2025-06 conditional novelty 6.0 of 10

    Hermes models LLM applications as probabilistic demand graphs and uses the Gittins scheduling policy plus backend prewarming to cut average completion time by over 70% in testbed runs.

  4. Improving the Serving Performance of Multi-LoRA Large Language Models via Efficient LoRA and KV Cache Management

    cs.AR 2025-04 conditional novelty 6.0 of 10

    A dependency-aware unified caching manager and cost-model swapper reduce Time-To-First-Token for multi-LoRA LLM serving by roughly 50 to 60 percent over vLLM and S-LoRA.

  5. InstantInfer: Enabling Fast LLM Cold Start with Communicating Finite Automata

    cs.DC 2026-07 conditional novelty 5.0 of 10

    InstantInfer refactors vLLM's cold start into a concurrent state-machine pipeline, speeding up startup by up to 7.2×.

  6. EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devices

    cs.DC 2025-07 conditional novelty 5.0 of 10

    EdgeLoRA combines automatic adapter routing, LRU caching with a memory pool, and grouped LoRA batching to serve thousands of LoRA adapters on edge devices with up to 4x higher throughput than llama.cpp.

  7. HSplitLoRA: A Heterogeneous Split Parameter-Efficient Fine-Tuning Framework for Large Language Models

    cs.LG 2025-05 conditional novelty 4.0 of 10

    HSplitLoRA combines split learning with LoRA, selects important weights via a gradient-weight product metric, adapts LoRA ranks and model split points to device budgets, and aggregates heterogeneous adapters by concatenation.

  8. Adaptive Parameter-Efficient Federated Fine-Tuning on Heterogeneous Devices

    cs.DC 2024-12 conditional novelty 4.0 of 10

    Assigning federated fine-tuning devices different numbers of LoRA layers near the output, with ranks increasing toward the output, reaches target accuracy 1.5-2.8x faster and with up to 42.3% less communication than e...

  9. Taming the Titans: A Survey of Efficient LLM Inference Serving

    cs.CL 2025-04 conditional novelty 3.0 of 10

    A survey that classifies LLM inference serving techniques into instance, cluster, emerging-scenario, and miscellaneous categories, with a companion GitHub index and four research directions.

Pith tools