Pith. sign in

REVIEW 5 cited by

HeteGen: Heterogeneous Parallel Inference for Large Language Models on Resource-Constrained Devices

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.01164 v1 pith:WKFOJ2EC submitted 2024-03-02 cs.PF cs.DC

Xuanlei Zhao , Bin Jia , Haotian Zhou , Ziming Liu , Shenggan Cheng , Yang You This is my paper · ORCID
classification cs.PFcs.DC
keywords inferencedeviceshetegenheterogeneousllmsparallelbottleneckscomputing
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

In recent times, the emergence of Large Language Models (LLMs) has resulted in increasingly larger model size, posing challenges for inference on low-resource devices. Prior approaches have explored offloading to facilitate low-memory inference but often suffer from efficiency due to I/O bottlenecks. To achieve low-latency LLMs inference on resource-constrained devices, we introduce HeteGen, a novel approach that presents a principled framework for heterogeneous parallel computing using CPUs and GPUs. Based on this framework, HeteGen further employs heterogeneous parallel computing and asynchronous overlap for LLMs to mitigate I/O bottlenecks. Our experiments demonstrate a substantial improvement in inference speed, surpassing state-of-the-art methods by over 317% at most.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. EcoServe: Designing Carbon-Aware AI Inference Systems

    cs.DC 2025-02 conditional novelty 6.0 of 10

    EcoServe combines four strategies (reuse, rightsize, reduce, recycle) in an ILP optimizer to cut modeled carbon emissions for LLM serving by up to 47% while keeping SLOs.

  2. BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batching

    cs.LG 2024-11 conditional novelty 6.0 of 10

    BlendServe combines resource-aware batching with prefix sharing using a resource-aware prefix tree and dual scanner, achieving up to 1.44x throughput vs vLLM/SGLang in offline LLM inference.

  3. WindVE: Collaborative CPU-NPU Vector Embedding

    cs.DC 2025-04 conditional novelty 5.0 of 10

    WindVE offloads peak vector-embedding queries to idle CPUs through a regression-calibrated queue manager, achieving up to 22.3% higher concurrency than a GPU/NPU-only baseline under the same SLOs.

  4. RAGDoll: Efficient Offloading-based Online RAG System on a Single GPU

    cs.DC 2025-04 conditional novelty 5.0 of 10

    RAGDoll pipelines retrieval and generation, jointly manages memory across disk, RAM, and GPU, and adaptively sizes batches to cut average RAG latency by up to 3.6x on a single GPU.

  5. Deploying Foundation Model Powered Agent Services: A Survey

    cs.DC 2024-12 accept novelty 4.0 of 10

    This survey proposes a layered framework (execution, resource, model, agent, application) for deploying foundation-model-powered agent services across edge-cloud environments, and reviews optimization techniques at ea...

Pith tools