Pith. sign in

REVIEW 5 cited by

Efficient LLM Inference on CPUs

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2311.00502 v2 pith:MWRLQGOO submitted 2023-11-01 cs.LG cs.AIcs.CL

Haihao Shen , Hanwen Chang , Bo Dong , Yu Luo , Hengyu Meng This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords cpusinferencellmsapproachlargememorymodelsaccelerate
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Large language models (LLMs) have demonstrated remarkable performance and tremendous potential across a wide range of tasks. However, deploying these models has been challenging due to the astronomical amount of model parameters, which requires a demand for large memory capacity and high memory bandwidth. In this paper, we propose an effective approach that can make the deployment of LLMs more efficiently. We support an automatic INT4 weight-only quantization flow and design a special LLM runtime with highly-optimized kernels to accelerate the LLM inference on CPUs. We demonstrate the general applicability of our approach on popular LLMs including Llama2, Llama, GPT-NeoX, and showcase the extreme inference efficiency on CPUs. The code is publicly available at: https://github.com/intel/intel-extension-for-transformers.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Accelerating Retrieval-Augmented Generation

    cs.CL 2024-12 conditional novelty 6.0 of 10

    Exact nearest neighbor search, accelerated by a near-memory CXL device called IKS, can make retrieval-augmented generation faster and more accurate end-to-end than approximate search.

  2. RIS-Kernel: A Model-Agnostic Architecture for Long-Context LLM Inference via Sparse Attention

    cs.LG 2026-07 reject novelty 5.0 of 10

    RIS-Kernel claims O(N log N) sparse attention via ensembled random masks for long-context CPU inference, but the complexity claim conflicts with the paper's fixed-density mask design and the accuracy evidence is stati...

  3. TEESlice: Protecting Sensitive Neural Network Models in Trusted Execution Environments When Attackers have Pre-Trained Models

    cs.CR 2024-11 conditional novelty 5.0 of 10

    TEESlice trains small private slices on top of a public backbone inside a TEE, leaving only the public backbone and encrypted features on the GPU, and reports black-box-level attack resistance at about 10x lower TEE c...

  4. Deploying Foundation Model Powered Agent Services: A Survey

    cs.DC 2024-12 accept novelty 4.0 of 10

    This survey proposes a layered framework (execution, resource, model, agent, application) for deploying foundation-model-powered agent services across edge-cloud environments, and reviews optimization techniques at ea...

  5. Inference performance evaluation for LLMs on edge devices with a novel benchmarking framework and metric

    cs.PF 2025-08 reject novelty 2.0 of 10

    A benchmarking study of LLM inference on three edge devices with a proposed MBU metric that reduces to a standard throughput normalization.

Pith tools