Pith. sign in

REVIEW 4 major objections 7 minor 35 references

Challenging GPU Dominance: When CPUs Outperform for On-Device LLM Inference

T0 review · 4 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read On an iPhone, a CPU beats the GPU at running a 1B model

desk verdict The CPU-beats-GPU headline is plausible but unsupported: the 'GPU-enabled' baseline was likely never on the GPU at the pinned llama.cpp commit. read the letter →

arxiv 2505.06461 v1 pith:LEGVSBB7 submitted 2025-05-09 cs.DC cs.LG

Haolin Zhang , Jeff Huang This is my paper · ORCID
classification cs.DCcs.LG
keywords on-deviceLLMinferenceCPUvsGPUllama.cppiPhone15ProGEMMbottleneckthreadscalingquantizationmobile
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper challenges the assumption that GPU acceleration is always faster than CPU execution for small language models on phones. Using llama.cpp on an iPhone 15 Pro, the authors report that running a 1-billion-parameter model on two CPU threads at F16 precision generates 17 tokens per second, while the GPU-enabled default configuration manages only 12.8. They also measure that matrix multiplication dominates both the prefill phase (87.6% of time) and the decode phase (76.2%), and they show that CPU performance peaks when thread count matches the two performance cores. The paper's broader point is that choosing a backend for on-device inference should be an empirical, workload-aware decision rather than a reflexive GPU-first one.

What carries the argument

The load-bearing object is the llama.cpp compute graph: the model is a directed graph of tensor operations (GEMMs, norms, attention), executed node by node through the ggml tensor library, with each node dispatched to a backend chosen at compile time (CPU via NEON, GPU via Metal). Because each decoder layer's operations run serially, the cost structure is set by the seven matrix multiplications per layer, and the paper identifies GGML_OP_MUL_MAT, the operation node for general matrix multiplication, as the dominant cost. This graph view supplies both the profiling target (which operations and which of the seven GEMMs take the time) and the optimization lever (scheduling independent nodes in parallel).

What would settle it

Run the same LLaMA 3.2-1B-F16 model on the same iPhone 15 Pro build while recording GPU utilization through Instruments or by instrumenting Metal buffer copies; if the default GPU path spends most of its time waiting on the CPU or copies the model weights every token, or if an explicitly tuned Metal kernel pushes GPU throughput above 17 tokens per second, the claimed CPU-superiority result is an artifact of the default build rather than a hardware trade-off.

Watch

Extended reading notes

Core claim

The central claim is an existence proof: for a small on-device LLM, a carefully tuned CPU-only configuration can beat the phone's GPU. Specifically, LLaMA 3.2-1B in F16 on the iPhone 15 Pro with two CPU threads delivers 17.0 tokens per second versus 12.8 for the GPU-enabled default. The same pattern holds for Qwen2-0.5B, where CPU with several threads exceeds the GPU baseline, while for models of 1.5B parameters and larger the GPU generally wins. The authors attribute the CPU advantage to reduced kernel-launch and memory-transfer overhead on small workloads and to matching thread count to the chip's performance cores, while cautioning that the low-level mechanism 'remains without a detailed low-level explanation' because iOS profiling tools are limited.

Load-bearing premise

The comparison assumes the default 'GPU-enabled' llama.cpp configuration actually offloads the heavy computation to the iPhone GPU and is a reasonably optimized baseline; the paper does not report GPU utilization, layers offloaded, or Metal kernel timing, and it concedes the CPU advantage lacks a detailed low-level explanation.

Editorial extensions

If this is right

  • For sub-1B and 1B models on phones, deployment strategies should treat CPU-only execution as a first-class option rather than a fallback, since it can beat the default GPU path at equal precision.
  • Thread count should be tuned to the device's performance-core count; adding threads beyond that point stops helping and can degrade throughput.
  • Because GEMM dominates both prefill and decode, engineering effort aimed at faster small-batch matrix multiplication (SIMD tuning, cache-aware blocking) should transfer directly to end-to-end inference speed.
  • Quantization to Q4 gives a 1.5x to 2.5x speedup across most configurations, so precision choice is as important as backend choice for on-device latency.
  • Hybrid CPU+GPU execution is not automatically beneficial: the authors' attempt to split graph nodes across both backends dropped throughput to about 6 tokens per second, suggesting synchronization overhead can exceed the GPU's compute advantage on small batches.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the fixed 7-token prompt is replaced with longer or batched inputs, the balance may shift: larger matrices give the GPU more work to amortize kernel launch overhead, so CPU's advantage may shrink or reverse on long-context prefill.
  • The authors' graph-parallel scheduling idea (running Q, K, V, and FFN GEMMs concurrently on the CPU) is a testable template for other engines and could be combined with quantization to widen the CPU margin further.
  • A direct comparison against Apple's own optimized runtime and against a GPU-tuned Metal implementation in a newer llama.cpp release, rather than the default build, would reveal how much of the 17-vs-12.8 gap is intrinsic hardware behavior and how much is the default Metal path being under-optimized.
  • The same measurement methodology could be applied to Android devices with different core topologies to see whether the 'performance-core count' rule generalizes across SoCs.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper presents an empirical benchmark of on-device LLM inference on an iPhone 15 Pro using llama.cpp at a pinned commit, comparing a CPU-only mode (1-6 threads) against a "GPU-Enabled (Default)" mode across six models and three precisions, with throughput in tokens per second as the metric. The headline claim is that for Llama-3.2-1B at F16, a two-thread CPU configuration achieves 17.0 tk/s versus 12.8 tk/s for the GPU-enabled configuration. The paper also profiles operation-level time shares, reporting that matrix multiplication dominates prefill (87.6%) and decode (76.2%) for Llama-3.2-1B-F16, and experiments with graph-level and tensor-level parallelism to improve CPU throughput. The manuscript candidly acknowledges that the CPU advantage lacks a detailed low-level explanation and that GPU profiling was unstable.

Significance. If the CPU-vs-GPU result is substantiated, the paper would provide a useful existence proof that a carefully configured CPU path can outperform a default GPU path for small on-device LLMs, challenging a common GPU-first assumption. The GEMM dominance measurement for Llama-3.2-1B-F16 is a valuable data point for engine optimization, and the thread-oversubscription finding is practically relevant. I also credit the authors for pinning the exact commit, describing thermal mitigation, running five repetitions, and being explicit about profiling limitations. However, the significance is substantially muted because the central comparison rests on an unverified GPU baseline, the throughput figures are reported without any variance, and the paper's own remarks concede that the key causal explanations were not established.

major comments (4)
  1. [§4.3, §5.2, Fig. 4(b)] The 'GPU-Enabled (Default)' configuration is not actually demonstrated to use the GPU. The paper never reports the value of n_gpu_layers (-ngl), never shows ggml backend-assignment logs, and never provides GPU utilization or Metal kernel timings. At the pinned commit 8648c52, the default value of n_gpu_layers is 0, meaning a run with the default CLI would execute on the CPU only. The reported GPU value of 12.8 tk/s for Llama-3.2-1B-F16 is close to the 1-thread CPU value of 12.0 tk/s, which makes this concern concrete rather than hypothetical. Since the headline 'CPU beats GPU' claim depends entirely on the 12.8 tk/s figure actually being produced by the GPU backend, please provide the exact command-line invocation (including any -ngl flag), the 'llm_load_tensors: offloading' output, and evidence that Metal kernels were executed. If the default configuration was used without an explicit -ngl override, the claim should be withdrawn or the experiment rerun with a verified GPU offload setting.
  2. [§4.4, Fig. 4] The paper reports only the average of five runs, with no error bars, standard deviation, or per-run values. Throughput measurements on a phone are strongly affected by thermal state and background activity; the ice-cooling procedure is described but no temperature is logged or controlled for, so the statement 'to ensure statistical reliability' is not supported by the data shown. In addition, the benchmark protocol is under-specified: the fixed prompt is 'The meaning of life is ' (7 tokens), but the number of generated tokens per run is not stated, and the mention of 128-token context windows in §5.1 is not sufficient to reproduce the measurement. Please report per-run values or variance, state the generation length, and clarify whether the reported tk/s is decode-only or includes prefill.
  3. [§6.1, Figs. 5-6] The profiling analysis claims that GGML_OP_MUL_MAT accounts for 87.6% of prefill and 76.2% of decode time for Llama-3.2-1B-F16, but the surrounding text says this analysis was performed in a CPU-only environment, while Figure 6 includes GPU bars. Given §8.2's statement that Metal Debugger profiling repeatedly crashed during data transfer, it is unclear whether any GPU-side timings were successfully captured and how they were obtained. Please specify which backend each sub-figure corresponds to and how the GPU measurements were made, or remove the GPU bars if they are not based on actual measurements.
  4. [Abstract, §5.4 Remark, §7.2] There is a mismatch between the abstract's claim that 'GPU memory transfer overhead and CPU thread optimization play a critical role' and the paper's own admission in §5.4 that the CPU advantage 'remains without a detailed low-level explanation.' Similarly, §7.2 states that the graph-parallel version 'nearly matching default GPU-enabled execution,' which inherits the unverified GPU baseline from §4.3. Please either provide direct evidence for the asserted overhead mechanisms (e.g., Metal buffer allocation or memory-transfer measurements) or temper the abstract and conclusion to match the admitted lack of explanation.
minor comments (7)
  1. [§1] In the opening paragraph, 'Section 6 discussed profiling results' should be 'Section 6 discusses profiling results.'
  2. [Footnote 1] The notation 'Q41' appears to refer to Q4_1 quantization; it is confusing and should be made consistent with the Q4/Q8 notation used elsewhere.
  3. [Fig. 1] The full compute-graph figure is extremely large and will likely be unreadable in a two-column page; consider replacing it with a trimmed version showing one decoder block.
  4. [§4.3] The description 'GPU-Enabled (Default): The model utilized the iPhone's integrated GPU' should be rephrased to 'was intended to utilize' to reflect the verification issue raised in Major Comment 1.
  5. [§8.3] The limitation that the study is device-specific is only stated in §8.3; consider adding a sentence to the abstract or conclusion noting that the findings are from a single device.
  6. [§3] The text says 'The full graph can be accessed on GitHub' but no URL is provided; please add the link or remove the pointer.
  7. [Algorithm 1] The pseudocode uses undefined macros such as INIT and APPLY_WEIGHT; please define them or replace them with plain descriptive text.

Circularity Check

0 steps flagged · score 0.0 of 10

Empirical benchmark with no derivation chain; no fitted parameters, no self-citation load-bearing, and no definitional reduction of the headline result.

full rationale

This paper is a measurement study, not a derivation. The central claim (CPU-only 2-thread F16 at 17.0 tk/s vs. GPU-enabled 12.8 tk/s for Llama-3.2-1B) is an observed throughput comparison obtained from running llama.cpp on an iPhone 15 Pro, as described in Sections 4.3-4.5 and reported in Figure 4b. No equation is derived, no parameter is fitted to a subset of the data and then renamed as a prediction, and no result is imported by self-citation: the references are to external model papers, quantization methods, and inference engines, with no load-bearing citation to the authors' own prior work. The GEMM dominance percentages (87.6% prefill, 76.2% decode) are direct profiling measurements reported in Section 6.1, not quantities forced by the methodology. The paper's own limitations (Section 5.4 Remark that the CPU advantage 'remains without a detailed low-level explanation'; Section 8.2 Metal Debugger instability) weaken the explanatory depth and create measurement-validity risk, but lack of explanation is not circularity. Even the skeptic's concern that the 'GPU-Enabled (Default)' baseline may not have offloaded to the GPU is a question about whether the measured number was correctly labeled, not a case where the claimed result is equivalent to its inputs by construction. Therefore no circular step can be exhibited, and the honest finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The report depends on no fitted parameters and invents no entities. Its load-bearing premises are the fairness of the default GPU baseline, the stability of the thermal environment, and the representativeness of the llama.cpp configuration, none of which are validated with measurements in the paper.

assumptions (3)
  • domain assumption llama.cpp's default GPU-enabled configuration on iOS effectively uses the Metal GPU for the tested models and represents a reasonable GPU baseline.
    Section 4.3 defines GPU-Enabled as 'default enabled setting', but the paper does not verify how many graph nodes are offloaded to GPU or whether the Metal backend is well tuned; the CPU-vs-GPU comparison depends on this premise.
  • domain assumption The iPhone 15 Pro was in a steady thermal state during all runs such that 5-run averages are comparable across backends.
    Section 4.1 says the device was placed in an ice-cooled environment to mitigate throttling, but device temperature is not monitored; if thermal state varied across runs, the CPU/GPU comparison could be biased.
  • domain assumption llama.cpp commit 8648c52 is representative of the framework's performance on iOS, and no unlisted configuration differences (e.g., batch size, context length) affect the comparison.
    Section 4.1 only specifies the commit and the prompt; other llama.cpp flags (n_batch, n_ctx, mmap, MLock, GPU layers) are not stated, so the reader must assume the default settings were used uniformly.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Challenging GPU Dominance: When CPUs Outperform for On-Device LLM Inference." pith.science (2026). https://pith.science/paper/LEGVSBB7

@misc{pith2026250506461,
  author       = {Pith},
  title        = {Pith review of: Challenging GPU Dominance: When CPUs Outperform for On-Device LLM Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LEGVSBB7}},
  note         = {Machine review of arXiv:2505.06461}
}
read the original abstract

The common assumption in on-device AI is that GPUs, with their superior parallel processing, always provide the best performance for large language model (LLM) inference. In this work, we challenge this notion by empirically demonstrating that, under certain conditions, CPUs can outperform GPUs for LLM inference on mobile devices. Using a 1-billion-parameter LLM deployed via llama.cpp on the iPhone 15 Pro, we show that a CPU-only configuration (two threads, F16 precision) achieves 17 tokens per second, surpassing the 12.8 tokens per second obtained with GPU acceleration. We analyze the architectural factors driving this counterintuitive result, revealing that GPU memory transfer overhead and CPU thread optimization play a critical role. Furthermore, we explore the impact of thread oversubscription, quantization strategies, and hardware constraints, providing new insights into efficient on-device AI execution. Our findings challenge conventional GPU-first thinking, highlighting the untapped potential of optimized CPU inference and paving the way for smarter deployment strategies in mobile AI. However, fully explaining the observed CPU advantage remains difficult due to limited access to low-level profiling tools on iOS.

Figures

Figures reproduced from arXiv: 2505.06461 by the authors.

Figure 2
Figure 2. Left (GPU-enabled): Nodes are offloaded to the GPU backend. Right (CPU-only execution): Nodes are processed in parallel across four CPU threads. Power9 Vector, Loongarch ASX for CPU acceleration, and CUDA, MUSA HIP, Vulkan, SYCL and Metal for GPU execution. This flexibility allows llama.cpp to achieve effi￾cient performance across different computing platforms. 4 Methodology 4.1 Hardware and Software Setup To evalua… view at source ↗
Figure 3
Figure 3. Execution sequence of nodes within transformer blocks. Each operation is processed sequentially. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Performance Comparison of Different Models Across Backends [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figures from the paper (5 more)
Figure 6
Figure 6. Figure 6: MatMul performance comparison between prefill [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 8
Figure 8. Figure 8: Comparison of the execution flow in llama.cpp [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 7
Figure 7. Figure 7: Simplified compute graph of decoder block in [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 9
Figure 9. Figure 9: Transition from Version 1 (Top, graph-level [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 10
Figure 10. Figure 10: Transition from Version 2 (Top, graph + ten [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 11 canonical work pages

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin et al. “Phi-3 technical report: A highly capable language model locally on your phone”. In: arXiv preprint arXiv:2404.14219 (2024)

    arXiv 2024

  2. [2]

    GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints

    Joshua Ainslie et al. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. 2023. arXiv: 2305.13245 [cs.CL] . URL: https://arxiv.org/abs/2305.13245

    arXiv 2023

  3. [3]

    Stable LM 2 1.6B Technical Report

    Marco Bellagente et al. Stable LM 2 1.6B Technical Report. 2024. arXiv: 2402.17834 [cs.CL]. URL: https://arxiv.org/abs/2402.17834. 11 Challenging GPU Dominance: When CPUs Outperform for On-Device LLM Inference

    arXiv 2024

  4. [4]

    MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devices

    Xiangxiang Chu et al. MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devices. 2023. arXiv: 2312.16886 [cs.CV]. URL: https://arxiv.org/abs/2312.16886

    arXiv 2023

  5. [5]

    Implementing small language mod- els (slms) with RAG on embedded devices lead- ing to cost reduction, data privacy, and offline use

    Kamil Czerski. Implementing small language mod- els (slms) with RAG on embedded devices lead- ing to cost reduction, data privacy, and offline use. Jan. 2025. URL: https : / / deepsense . ai / blog / implementing - small - language - models - slms - with - rag - on - embedded - devices - leading - to - cost - reduction - data-privacy-and-offline-use

    work page 2025

  6. [6]

    Flashattention: Fast and memory- efficient exact attention with io-awareness

    Tri Dao et al. “Flashattention: Fast and memory- efficient exact attention with io-awareness”. In: Ad- vances in Neural Information Processing Systems 35 (2022), pp. 16344–16359

    work page 2022

  7. [7]

    GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transform- ers

    Elias Frantar et al. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transform- ers. 2023. arXiv: 2210 . 17323 [cs.LG] . URL: https://arxiv.org/abs/2210.17323

    arXiv 2023

  8. [8]

    K-Quants: Optimized Quanti- zation for Llama.cpp

    Georgi Gerganov. K-Quants: Optimized Quanti- zation for Llama.cpp. GitHub pull request #1684

    work page

Show all 35 references
  1. [9]

    llama.cpp

    Georgi Gerganov. llama.cpp. https://github. com/ggerganov/llama.cpp. 2023

    2023

  2. [10]

    The Llama 3 Herd of Models

    Aaron Grattafiori et al. The Llama 3 Herd of Models

  3. [11]

    Apple Intelligence Foundation Language Models

    Tom Gunter et al. Apple Intelligence Foundation Language Models . 2024. arXiv: 2407 . 21075 [cs.AI]. URL: https://arxiv.org/abs/2407. 21075

    2024

  4. [12]

    Apple Intelligence for developers

    Apple Inc. Apple Intelligence for developers. URL: https : / / developer . apple . com / apple - intelligence/

  5. [13]

    Phi-2: The surprising power of small language models

    Mojan Javaheripi et al. “Phi-2: The surprising power of small language models”. In: Microsoft Research Blog 1.3 (2023), p. 3

    2023

  6. [14]

    Jiang et al

    Albert Q. Jiang et al. Mistral 7B. 2023. arXiv: 2310. 06825 [cs.CL] . URL: https : / / arxiv . org / abs/2310.06825

    2023 arXiv

  7. [15]

    MNN: A Universal and Effi- cient Inference Engine

    Xiaotang Jiang et al. MNN: A Universal and Effi- cient Inference Engine. 2020. arXiv: 2002.12418 [cs.CV]. URL: https://arxiv.org/abs/2002. 12418

    2020 arXiv

  8. [16]

    LLM Inference Serving: Survey of Recent Advances and Opportunities

    Baolin Li et al. LLM Inference Serving: Survey of Recent Advances and Opportunities. 2024. arXiv: 2407 . 12391 [cs.DC]. URL: https : / / arxiv . org/abs/2407.12391

    2024 arXiv

  9. [17]

    Transformer-Lite: High-efficiency Deployment of Large Language Models on Mobile Phone GPUs

    Luchang Li et al. Transformer-Lite: High-efficiency Deployment of Large Language Models on Mobile Phone GPUs. 2024. arXiv: 2403.20041 [cs.CL]. URL: https://arxiv.org/abs/2403.20041

    2024 arXiv

  10. [18]

    Large Language Models on Mobile Devices: Measurements, Analysis, and Insights

    Xiang Li et al. “Large Language Models on Mobile Devices: Measurements, Analysis, and Insights”. In: Proceedings of the Workshop on Edge and Mo- bile Foundation Models. EdgeFM ’24. Minato-ku, Tokyo, Japan: Association for Computing Machin- ery, 2024, pp. 1–6. ISBN : 9798400706...

    2024

  11. [19]

    Mobilellm: Optimizing sub- billion parameter language models for on-device use cases

    Zechun Liu et al. “Mobilellm: Optimizing sub- billion parameter language models for on-device use cases”. In: arXiv preprint arXiv:2402.14905 (2024)

    2024 arXiv

  12. [20]

    Small Language Models: Survey, Measurements, and Insights

    Zhenyan Lu et al. Small Language Models: Survey, Measurements, and Insights . 2024. arXiv: 2409 . 15790 [cs.CL] . URL: https : / / arxiv . org / abs/2409.15790

    2024 arXiv

  13. [21]

    MediaPipe: A Framework for Building Perception Pipelines

    Camillo Lugaresi et al. MediaPipe: A Framework for Building Perception Pipelines . 2019. arXiv: 1906 . 08172 [cs.DC]. URL: https : / / arxiv . org/abs/1906.08172

    2019 arXiv

  14. [22]

    The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits

    Shuming Ma et al. The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits . 2024. arXiv: 2402 . 17764 [cs.CL]. URL: https : / / arxiv . org/abs/2402.17764

    2024 arXiv

  15. [23]

    Pham et al

    Thang M. Pham et al. SlimLM: An Efficient Small Language Model for On-Device Document Assis- tance. 2024. arXiv: 2411 . 09944 [cs.CL]. URL: https://arxiv.org/abs/2411.09944

    2024 arXiv

  16. [24]

    ExecuTorch

    PyTorch. ExecuTorch. https : / / github . com / pytorch/executorch. 2024

    2024

  17. [25]

    PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU

    Yixin Song et al. PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU. 2024. arXiv: 2312 . 12456 [cs.LG] . URL: https : / / arxiv.org/abs/2312.12456

    2024 arXiv

  18. [26]

    Gemini: A Family of Highly Capable Multimodal Models

    Gemini Team et al. Gemini: A Family of Highly Capable Multimodal Models . 2024. arXiv: 2312. 11805 [cs.CL] . URL: https : / / arxiv . org / abs/2312.11805

    2024 arXiv

  19. [27]

    Gemma: Open Models Based on Gemini Research and Technology

    Gemma Team et al. Gemma: Open Models Based on Gemini Research and Technology. 2024. arXiv: 2403 . 08295 [cs.CL]. URL: https : / / arxiv . org/abs/2403.08295

    2024 arXiv

  20. [28]

    LLaMA: Open and Efficient Foundation Language Models

    Hugo Touvron et al. LLaMA: Open and Efficient Foundation Language Models. 2023. arXiv: 2302. 13971 [cs.CL] . URL: https : / / arxiv . org / abs/2302.13971

    2023 arXiv

  21. [29]

    SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Lan- guage Models

    Guangxuan Xiao et al. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Lan- guage Models. 2024. arXiv: 2211.10438 [cs.CL]. URL: https://arxiv.org/abs/2211.10438

    2024 arXiv

  22. [30]

    On-Device Language Models: A Comprehensive Review

    Jiajun Xu et al. On-Device Language Models: A Comprehensive Review. 2024. arXiv: 2409.00088 [cs.CL]. URL: https://arxiv.org/abs/2409. 00088

    2024 arXiv

  23. [31]

    Qwen2 Technical Report

    An Yang et al. Qwen2 Technical Report . 2024. arXiv: 2407 . 10671 [cs.CL] . URL: https : / / arxiv.org/abs/2407.10671. 12 Challenging GPU Dominance: When CPUs Outperform for On-Device LLM Inference

    2024 arXiv

  24. [32]

    TinyLlama: An Open-Source Small Language Model

    Peiyuan Zhang et al. TinyLlama: An Open-Source Small Language Model. 2024. arXiv: 2401.02385 [cs.CL]. URL: https://arxiv.org/abs/2401. 02385

    2024 arXiv

  25. [33]

    OPT: Open Pre-trained Trans- former Language Models

    Susan Zhang et al. OPT: Open Pre-trained Trans- former Language Models . 2022. arXiv: 2205 . 01068 [cs.CL] . URL: https : / / arxiv . org / abs/2205.01068. 13

    2022 arXiv

  26. [2023]

    URL: https://github.com/ggerganov/ llama.cpp/pull/1684

  27. [2024]

    URL: https: //arxiv.org/abs/2407.21783

    arXiv: 2407.21783 [cs.AI]. URL: https: //arxiv.org/abs/2407.21783

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.