Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

Scaling On-Device GPU Inference for Large Generative Models

T0 review · 5 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ML Drift, a GPU inference engine, uses tensor virtualization to deliver order-of-magnitude prefill speedups on mobile GPUs and to run generative models 10-100x larger than previous on-device engines.

desk verdict A genuinely useful mobile GPU inference engine whose abstract overstates the order-of-magnitude claim; the real result is mobile-prefill-only 5-11x speedups, with decode and desktop gains far smaller. read the letter →

arxiv 2505.00232 v1 pith:JC54UBGX submitted 2025-05-01 cs.LG cs.AI

Jiuqiang Tang , Raman Sarokin , Ekaterina Ignasheva , Grant Jensen , Lin Chen , Juhyun Lee , Andrei Kulik , Matthias Grundmann This is my paper · ORCID
classification cs.LGcs.AI
keywords GPUinferenceon-devicetensorvirtualizationlargelanguagemodelsStableDiffusionOpenCLmobileoperatorfusion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces ML Drift, a GPU inference engine that decouples the logical shape of a tensor from its physical representation on the GPU through a mechanism called tensor virtualization, allowing each kernel to read and write tensors in whatever memory layout suits it best. The authors claim that this design, combined with device-specialized shader generation, operator fusion, memory-reuse planning, and stage-aware LLM optimization, delivers an order-of-magnitude speedup in the compute-bound prefill phase on mobile GPUs compared with existing open-source engines. As a result, they argue, on-device generative models can be 10 to 100 times larger in parameter count than what previously fit on phones and laptops. If true, this matters because privacy-sensitive and latency-critical generative workloads could run locally on personal devices rather than on servers.

What carries the argument

The central mechanism is tensor virtualization: an abstraction layer that separates a logical tensor's axes (batch, height, width, depth, channels) from the physical GPU objects that store it (1D buffers, 2D textures, 3D textures, texture arrays). A coordinate-translation helper generated during shader code generation converts a logical access like (b, x, y, s) into the physical address of the chosen layout, so kernel authors never handle fragmentation across multiple textures. This flexibility lets the engine pick a memory layout per kernel and per device, which the paper identifies as a primary driver of matrix-multiplication speedups; around it sit device specialization (backend-specific shader generators for OpenCL, Metal, WebGPU), memory-reuse planning, automatic operator fusion, and stage-aware LLM quantization paths that use int8 for prefill and fuse activations for decode.

What would settle it

Run Llama 3.1 8B with the paper's mixed 8/4/4 quantization and a fixed 1024-token prompt on a phone with the same GPU and driver version used in the paper, and compare prefill tokens/s between ML Drift and the paper's open-source baseline; if the speedup is below 10x, the order-of-magnitude prefill claim is refuted. On a desktop GPU, the paper itself reports prefill 4-7x slower than a baseline that uses the hardware's native matrix-multiply accelerators, so any claim of a universal 10x improvement fails that test.

Watch

Extended reading notes

Core claim

ML Drift's central claim is that a layered set of GPU-specific optimizations - tensor virtualization, coordinate translation generated at shader-compile time, per-device kernel selection, memory-reuse allocation, operator fusion, and a KV cache layout designed for convolution-based matrix multiplication - enables mobile and laptop GPUs to reach prefill throughput of hundreds to thousands of tokens per second on 2B- to 8B-parameter language models, outperforming the open-source engines benchmarked in the paper by roughly 5 to 11 times on prefill while staying competitive on token-by-token decode. The same stack runs Stable Diffusion 1.4 end-to-end (20 iterations, 512x512, FP16) in under 9 seconds on a high-end Android phone and about 3.4 seconds on a recent laptop, with memory reuse cutting intermediate-tensor footprint from 4.31 GB to 387 MB. The paper presents these results as evidence that mobile GPUs can serve as a general platform for large generative models, not just as a fallback for tiny networks.

Load-bearing premise

The paper's headline speedup is measured on the compute-bound prefill stage of language models running on mobile GPUs; if the representative workload were token-by-token decode, a desktop GPU, or a full end-to-end text-to-image run, the measured gains would be much smaller, so the 'order of magnitude' claim stands or falls with that workload choice.

Editorial extensions

If this is right

  • Current flagship phones can run 8-billion-parameter language models at usable interactive speeds: the paper reports about 412 tokens/s prefill and 12.7 tokens/s decode for Llama 3.1 8B on a high-end phone.
  • Text-to-image generation moves from cloud to device: Stable Diffusion 1.4 completes 20 diffusion iterations in under 9 seconds on a current Android phone and about 3.4 seconds on a recent laptop, making local generation practical for interactive use.
  • The same engine spans multiple GPU APIs, so a single model artifact can be deployed across phone, laptop, and desktop platforms; the paper demonstrates functional backends for OpenCL, Metal, and WebGPU, with performance varying by backend and device.
  • LLM inference should be staged: the paper shows that compute-bound prefill needs different kernels and quantization than memory-bound decode, and that this separation is what unlocks the large prefill speedups while decode gains remain modest (about 1.2-1.9x from quantization choice).
  • Memory-reuse planning is a precondition for the whole approach: cutting intermediate activations from 4.31 GB to 387 MB is what allows diffusion and large LLMs to fit within phone memory limits.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The order-of-magnitude claim is workload- and platform-specific: the paper's own data show decode speedups of only about 1.2-1.9x, and on a desktop GPU the OpenCL prefill path is 4-7x slower because it cannot use the hardware's tensor cores. A fair summary would be 'an order of magnitude faster mobile prefill,' not 'an order of magnitude faster inference.'
  • Tensor virtualization is a general principle that could transfer to heterogeneous processors, for example by using a similar abstraction between NPU, CPU, and GPU memory hierarchies; the paper only gestures at this through zero-copy interoperability as future work.
  • The baselines in the paper are the most popular open-source engines, but not necessarily the fastest production engines with equivalent quantization; comparing against an equally tuned engine with INT4 weight-only quantization would test whether the speedup comes from the abstractions or from less optimized baselines.
  • If future versions exploit vendor matrix-multiply extensions (available through OpenCL or WebGPU shader extensions), the desktop prefill gap could close and the technique could extend to discrete GPUs; this is a testable prediction, not a claim the paper makes.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces ML Drift, a GPU inference framework that extends an existing on-device GPU inference engine with tensor virtualization, coordinate translation, device specialization, memory management, operator fusion, stage-aware LLM optimizations, and a custom KV cache layout. The system targets OpenCL, Metal, and WebGPU backends and is evaluated on Stable Diffusion 1.4 and on Gemma and Llama LLMs across Qualcomm, Arm, Intel, NVIDIA, and Apple GPUs. The abstract and conclusion claim an order-of-magnitude performance improvement over existing open-source GPU inference engines and state that ML Drift enables on-device generative workloads 10 to 100x larger than existing on-device models.

Significance. If the headline claim were true as stated, ML Drift would be a significant contribution to on-device generative inference, especially for mobile prefill-dominated workloads. The paper's genuine strengths are the breadth of real-hardware evaluations (five mobile GPUs, Intel integrated GPUs, RTX 4090, Apple Silicon), the concrete engineering techniques (tensor virtualization, coordinate translation, stage-aware kernels), and the transparent disclosure of the Tensor Core limitation on desktop prefill. The 5-11x mobile prefill speedups in Figure 6 are plausible and useful. However, the central claim as written is not supported by the paper's own data: decode gains are only 1.2-1.9x, desktop prefill is 4-7x slower, and diffusion gains are 1.3-2.7x. The paper is credible as an engineering report for a narrow workload class, but the abstract overgeneralizes that result, and the baseline methodology needs tightening to make the speedup ratios robust.

major comments (5)
  1. [Abstract; Section 4.2 (Figures 6-7, Tables 3-4)] The unqualified abstract claim of an order-of-magnitude improvement over open-source GPU inference engines is contradicted by the paper's own benchmarks. Figure 6 shows 5-11x only for mobile prefill on Adreno; Section 4.2 reports decode gains of at most 1.9x; Figure 7 reports 4-7x slower prefill on the RTX 4090; and Table 3 reports 1.3-2.7x for diffusion on an Intel laptop. Since decode, desktop, and diffusion workloads are all within the paper's stated scope, the headline claim must be scoped to compute-bound mobile prefill with the specialized int8 kernels, or supported by an end-to-end representative workload.
  2. [Section 4.2 / Figure 6] The baseline comparison is not fully controlled. The versions or commits of llama.cpp, MLC LLM, ollama, torchchat, and MLX LM are not pinned; the GPU backend used by llama.cpp's benchmark tool is not stated; and the quantization schemes differ between ML Drift (q8 and 8/4/4) and the baselines (GGUF q4f16). These differences directly affect the speedup ratios, especially the 5-11x prefill numbers in Figure 6. Please provide pinned versions, explicit backend configurations for every baseline, and ideally repeated-run statistics or error bars.
  3. [Abstract; Introduction; Section 5] The claim that ML Drift 'enables on-device execution of workflows which contain 10 to 100x more parameters than existing on-device generative AI models' is not supported by the presented evidence. The largest model evaluated is Llama 3.1 8B (Tables 2 and 4), and the paper provides no quantitative definition of 'existing on-device generative AI models' for comparison. The 93% memory savings in Section 3.5 concern activation memory, not model parameter count. Please either remove or substantially rephrase this claim, or supply the comparison it relies on.
  4. [Section 3.2; Section 5] The central technical claim that tensor virtualization and coordinate translation add negligible runtime overhead is asserted but never measured. Section 3.2 says the mapping is resolved at shader code-generation time, while Section 5 admits that an ablation study to quantify the overhead and individual contributions 'will be conducted.' Without such a measurement, the paper's core mechanism is not isolated from the other optimizations (memory management, operator fusion, stage-aware kernels), so its role in the reported speedups is untested. Please add a microbenchmark or code-generation-level comparison, or explicitly label the claim as a design rationale rather than a measured property.
  5. [Section 4.1 / Table 3] The diffusion comparison against Intel's reported Lunar Lake result is made on a different platform (Ultra 7 258V versus Intel's 288V). The 14.4% speedup is therefore not a direct comparison. Please state the platform mismatch in the comparison sentence and add the source [22] as a reproducibility reference, or remove the percentage comparison.
minor comments (5)
  1. [Table 3] The table header 'ML Drift ML Drift OpenCL WebGPU DirectML' appears to contain a duplicated 'ML Drift' label; please fix the header to clearly distinguish the two ML Drift backends from ONNX Runtime DirectML.
  2. [Section 4.2] The sentence 'For the GPUs that support 8-bit cooperative matrix extensions' does not identify which GPUs or which extensions are meant; please give an explicit example or reference.
  3. [Section 3.5] The GREEDY BY SIZE policy is referenced as [43] but not defined; a one-sentence definition of the policy would make the memory-management section self-contained.
  4. [Tables 2-4] All latency and throughput cells report a single value without standard deviations or run counts; adding at least a statement of how many runs were averaged would improve confidence in the numbers.
  5. [Section 5] The phrases 'unparalleled flexibility' and 'one to two orders of magnitude larger' repeat the abstract's unsupported scope; they should be aligned with the scoped performance claims recommended above.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: performance claims are empirical comparisons against external baselines; self-citations are antecedent work, not load-bearing derivations.

full rationale

The paper's central claims are benchmark results, not derivations from their own conclusions. ML Drift's speedups are measured against external open-source engines (llama.cpp, MLC LLM, ONNX Runtime/DirectML, CoreML, torchchat, ollama, MLX LM) on fixed hardware and models. The abstract's order-of-magnitude statement is supported only in the mobile prefill regime and is contradicted for decode/desktop by the paper's own Section 4.2/Figure 7/Table 3, but that is an internal-scope/overstatement issue, not circularity. Self-citations appear ([26] as the base engine architecture, [9] for prior Stable Diffusion optimization, [43] for memory reuse), but none is used to justify the headline result: [26] and [43] are cited as starting points/inspiration, and [9] is used as a comparative 'previously reported' baseline, not as the source of the claimed speedup. No equation defines X in terms of Y, no fitted parameter is renamed a prediction, and no uniqueness theorem from the authors is invoked to force a choice. The reported 5-11x prefill gains are empirical and falsifiable by re-running the benchmarks, so there is no reduction to inputs.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No numerical free parameters are fitted to data; the system's performance rests on engineering assumptions about layout efficiency, offline-per-device GPU object selection, and negligible overhead for coordinate translation. These are not physical postulates, but they are load-bearing design assumptions for the benchmark claims.

assumptions (3)
  • domain assumption Four-element SIMD slice-aware layouts (PHWC4-style) are an efficient storage assumption across all supported GPU APIs.
    Section 3.1 states that PHWC4 layouts exploit the GPU's 4-element SIMD; this foundational layout choice underlies all kernels and is inherited from the authors' prior work [26].
  • ad hoc to paper Tensor virtualization and coordinate translation add negligible runtime overhead because they are resolved at shader code generation time.
    Sections 3.2 and 3.3 assert the overhead is negligible without a measured ablation; if layout translation occurs on a hot path or code generation is repeated per model, the performance claims would weaken.
  • domain assumption The empirically determined optimal GPU object for each device and kernel remains optimal at runtime on the tested hardware.
    Section 3.1 says the optimal GPU object is selected during offline testing and chosen at initialization; no sensitivity analysis or alternative selection rule is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scaling On-Device GPU Inference for Large Generative Models." pith.science (2026). https://pith.science/paper/JC54UBGX

@misc{pith2026250500232,
  author       = {Pith},
  title        = {Pith review of: Scaling On-Device GPU Inference for Large Generative Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JC54UBGX}},
  note         = {Machine review of arXiv:2505.00232}
}
read the original abstract

Driven by the advancements in generative AI, large machine learning models have revolutionized domains such as image processing, audio synthesis, and speech recognition. While server-based deployments remain the locus of peak performance, the imperative for on-device inference, necessitated by privacy and efficiency considerations, persists. Recognizing GPUs as the on-device ML accelerator with the widest reach, we present ML Drift--an optimized framework that extends the capabilities of state-of-the-art GPU-accelerated inference engines. ML Drift enables on-device execution of generative AI workloads which contain 10 to 100x more parameters than existing on-device generative AI models. ML Drift addresses intricate engineering challenges associated with cross-GPU API development, and ensures broad compatibility across mobile and desktop/laptop platforms, thereby facilitating the deployment of significantly more complex models on resource-constrained devices. Our GPU-accelerated ML/AI inference engine achieves an order-of-magnitude performance improvement relative to existing open-source GPU inference engines.

Figures

Figures reproduced from arXiv: 2505.00232 by the authors.

Figure 1
Figure 1. Tensor virtualization enables flexible memory layouts. A [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Memory savings for Stable Diffusion 1.4 using G [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Automatic operator fusion examples for large model in [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (2 more)
Figure 6
Figure 6. Figure 6: Comparative analysis of LLM performance (tokens/s) [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 8
Figure 8. Figure 8: Comparative analysis of LLM performance (tokens/s) on [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. EdgeWisePersona: A Dataset for On-Device User Profiling from Natural Language Interactions

    cs.HC 2025-05 conditional novelty 6.0 of 10

    EdgeWisePersona is a new synthetic dataset and benchmark for reconstructing structured smart-home user routines from multi-session dialogues, on which large LLMs clearly outperform small on-device models.

Reference graph

Works this paper leans on

59 extracted references · 56 canonical work pages · cited by 1 Pith paper

  1. [22]

    Intel Core Ultra Series 2 Media Deck

    Intel Corporation. Intel Core Ultra Series 2 Media Deck. https : / / download . intel . com / newsroom / 2024/client- computing/Intel- Core- Ultra- Series-2-Media-Deck.pdf , 2024. 6

    work page 2024

  2. [1]

    The Khronos Group Inc., 2019

    The OpenCL C++ 1.0 Specification . The Khronos Group Inc., 2019. 2

    work page 2019

  3. [2]

    The Khronos Group Inc., 2025

    Vulkan 1.4.309 - A Specification(with all registered exten- sions). The Khronos Group Inc., 2025. 2

    work page 2025

  4. [3]

    AMD ROCm Soft- ware

    Advanced Micro Devices, Inc. AMD ROCm Soft- ware. https : / / www . amd . com / en / products / software/rocm.html . [Online; accessed March 17, 2025]. 2

    work page 2025

  5. [4]

    LLM in a flash: Effi- cient Large Language Model Inference with Limited Mem- ory, 2024

    Keivan Alizadeh, Iman Mirzadeh, Dmitry Belenko, Karen Khatamifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. LLM in a flash: Effi- cient Large Language Model Inference with Limited Mem- ory, 2024. 2

    work page 2024

  6. [5]

    Core ML Stable Diffusion

    Apple Inc. Core ML Stable Diffusion. https : / / github.com/apple/ml-stable-diffusion. [On- line; accessed March 17, 2025]. 6

    work page 2025

  7. [6]

    Apple Inc. Core ML. https://developer.apple. com / documentation / coreml. [Online, accessed March 17, 2025]. 2

    work page 2025

  8. [7]

    Compute Library

    Arm Ltd. Compute Library. https : / / www . arm . com/products/development-tools/embedded- and - software / compute - library. [Online; ac- cessed March 17, 2025]. 2

    work page 2025

Show all 59 references
  1. [8]

    TVM: An automated End-to-End optimizing com- piler for deep learning

    Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, Carlos Guestrin, and Arvind Krishna- murthy. TVM: An automated End-to-End optimizing com- piler for deep learning. In 13th USENIX Symposium on Ope...

    2018

  2. [9]

    Speed Is All You Need: On-Device Acceleration of Large Diffusion Models via GPU-Aware Optimizations, 2023

    Yu-Hui Chen, Raman Sarokin, Juhyun Lee, Jiuqiang Tang, Chuo-Ling Chang, Andrei Kulik, and Matthias Grundmann. Speed Is All You Need: On-Device Acceleration of Large Diffusion Models via GPU-Aware Optimizations, 2023. 2, 6

    2023

  3. [10]

    LMDeploy: A Toolkit for Com- pressing, Deploying, and Serving LLM

    LMDeploy Contributors. LMDeploy: A Toolkit for Com- pressing, Deploying, and Serving LLM. https : / / github.com/InternLM/lmdeploy, 2023. 2

    2023

  4. [11]

    Fu, Stefano Ermon, Atri Rudra, and Christopher R´e

    Tri Dao, Daniel Y . Fu, Stefano Ermon, Atri Rudra, and Christopher R´e. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, 2022. 7

    2022

  5. [12]

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition ...

    2010 arXiv

  6. [13]

    GPTQ: Accurate Post-Training Quantization for Gen- erative Pre-trained Transformers, 2023

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Al- istarh. GPTQ: Accurate Post-Training Quantization for Gen- erative Pre-trained Transformers, 2023. 2

    2023

  7. [14]

    Gemma 2: Improving Open Language Mod- els at a Practical Size, 2024

    Gemma Team. Gemma 2: Improving Open Language Mod- els at a Practical Size, 2024. 6

    2024

  8. [15]

    Gemma: Open Models Based on Gemini Research and Technology, 2024

    Gemma Team. Gemma: Open Models Based on Gemini Research and Technology, 2024. 6

    2024

  9. [16]

    llama.cpp

    ggml.ai. llama.cpp. https://github.com/ggml- org/llama.cpp. [Online; accessed March 17, 2025]. 2, 6

    2025

  10. [17]

    ggml.ai. GGUF. https://github.com/ggml-org/ ggml/blob/master/docs/gguf.md . [Online; ac- cessed March 17, 2025]. 7

    2025

  11. [18]

    LiteRT Overview

    Google LLC. LiteRT Overview. https://ai.google. dev/edge/litert. [Online, accessed March 17, 2025]. 2

    2025

  12. [19]

    Huawei HiAI

    Huawei Technologies Co., Ltd. Huawei HiAI. https: //developer.huawei.com/consumer/en/hiai . [Online; accessed March 17, 2025]. 2

    2025

  13. [20]

    Apple Inc. Metal. https://developer.apple.com/ documentation/metal. [Online; accessed March 17, 2025]. 2

    2025

  14. [21]

    OpenVINO

    Intel Corporation. OpenVINO. https://www.intel. com / content / www / us / en / developer / tools / openvino-toolkit/overview.html . [Online; ac- cessed March 17, 2025]. 2

    2025

  15. [23]

    IREE Organization. IREE. https://iree.dev/. [On- line, accessed March 17, 2025]. 2

    2025

  16. [24]

    MNN: A Universal and Efficient Inference Engine

    Xiaotang Jiang, Huan Wang, Yiliu Chen, Ziqi Wu, Lichuan Wang, Bin Zou, Yafeng Yang, Zongyang Cui, Yuezhi Cai, Tianhang Yu, Chengfei Lv, and Zhihua Wu. MNN: A Universal and Efficient Inference Engine. ArXiv, abs/2002.12418, 2020. 2

    2002 arXiv

  17. [25]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the ACM SIGOPS 29th Symposium on Op- erating Syste...

    2023

  18. [26]

    On-Device Neu- ral Net Inference with Mobile GPUs

    Juhyun Lee, Nikolay Chirkov, Ekaterina Ignasheva, Yury Pisarchyk, Mogan Shieh, Fabio Riccardi, Raman Sarokin, Andrei Kulik, and Matthias Grundmann. On-Device Neu- ral Net Inference with Mobile GPUs. In Proceedings of the CVPR Workshop on Efficient Deep Learning for Computer Vi...

    2019

  19. [27]

    OpenGL ES Version 3.1

    Jon Leech, editor. OpenGL ES Version 3.1 . The Khronos Group Inc., 2016. 2

    2016

  20. [28]

    Fast In- ference from Transformers via Speculative Decoding, 2023

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast In- ference from Transformers via Speculative Decoding, 2023. 7

    2023

  21. [29]

    AWQ: Activation-aware Weight Quantization for LLM Compression and Accelera- tion

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei- Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ: Activation-aware Weight Quantization for LLM Compression and Accelera- tion. In MLSys, 2024. 2

    2024

  22. [30]

    The Llama 3 Herd of Models, 2024

    Llama Team. The Llama 3 Herd of Models, 2024. 6

    2024

  23. [31]

    DeepMon: Mobile GPU-based Deep Learning Framework for Continuous Vision Applications

    Huynh Nguyen Loc, Youngki Lee, and Rajesh Krishna Balan. DeepMon: Mobile GPU-based Deep Learning Framework for Continuous Vision Applications. Proceed- ings of the 15th Annual International Conference on Mobile Systems, Applications, and Services, 2017. 2

    2017

  24. [32]

    NeuroPilot

    MediaTek Inc. NeuroPilot. https://neuropilot. mediatek.com. [Online; accessed March 17, 2025]. 2

    2025

  25. [33]

    ExecuTorch

    Meta Platforms, Inc. ExecuTorch. https://pytorch. org / executorch - overview. [Online; accessed March 17, 2025]. 2

    2025

  26. [34]

    Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

    Meta Platforms, Inc. Llama 3.2: Revolutionizing edge AI and vision with open, customizable models. https: / / ai . meta . com / blog / llama - 3 - 2 - connect - 2024- vision- edge- mobile- devices/. [Online; accessed March 17, 2025]. 6

    2024

  27. [35]

    DirectML Overview

    Microsoft Corporation, Inc. DirectML Overview. https: //learn.microsoft.com/en- us/windows/ai/ directml/dml. [Online; accessed March 17, 2025]. 2

    2025

  28. [36]

    Get started with ONNX Run- time Mobile

    Microsoft Corporation, Inc. Get started with ONNX Run- time Mobile. https://onnxruntime.ai/docs/ get- started/with- mobile.html . [Online; ac- cessed March 17, 2025]. 2

    2025

  29. [37]

    Stable Diffusion Op- timization with DirectML

    Microsoft Corporation Inc. Stable Diffusion Op- timization with DirectML. https : / / github . com/microsoft/Olive/tree/main/examples/ directml/stable_diffusion . [Online; accessed March 17, 2025]. 6

    2025

  30. [38]

    MLC team. MLC-LLM. https://github.com/mlc- ai/mlc-llm. [Online; accessed March 17, 2025]. 2, 6

    2025

  31. [39]

    MLX Community. MLX LM. https://github.com/ ml- explore/mlx- lm . [Online; accessed March 17, 2025]. 6

    2025

  32. [40]

    NVIDIA Tensor Cores

    NVIDIA Corporation. NVIDIA Tensor Cores. https: / / www . nvidia . com / en - us / data - center / tensor-cores. [Online; accessed March 17, 2025]. 7

    2025

  33. [41]

    TensorRT

    NVIDIA Corporation. TensorRT. https : //developer.nvidia.com/tensorrt-getting- started. [Online; accessed March 17, 2025]. 2

    2025

  34. [42]

    ollama. ollama. https://github.com/ollama/ ollama. [Online; accessed March 17, 2025]. 2, 6

    2025

  35. [43]

    Efficient Memory Manage- ment for Deep Neural Net Inference

    Yury Pisarchyk and Juhyun Lee. Efficient Memory Manage- ment for Deep Neural Net Inference. In Proceedings of the MLSys Workshop on Resource-Constrained Machine Learn- ing, 2020. 4

    2020

  36. [44]

    Snapdragon Neural Processing Engine SDK

    Qualcomm Inc. Snapdragon Neural Processing Engine SDK. https://www.qualcomm.com/developer/ software / neural - processing - sdk - for - a . [Online; accessed March 17, 2025]. 2

    2025

  37. [45]

    QualComm AI Hub Llama-v3.2-3B-Chat

    Qualcomm Inc. QualComm AI Hub Llama-v3.2-3B-Chat. https://aihub.qualcomm.com/models/llama_ v3_2_3b_chat_quantized . [Online; accessed March 17, 2025]. 7

    2025

  38. [46]

    World’s first on-device demonstration of Stable Diffusion on an Android phone

    Qualcomm Inc. World’s first on-device demonstration of Stable Diffusion on an Android phone. https : / / www . qualcomm . com / news / onq / 2023 / 02 / worlds - first - on - device - demonstration - of-stable-diffusion-on-android . [Online; ac- cessed March 17, 2025]. 6

    2023

  39. [47]

    Ruan, Yucheng Qin, Xun Zhou, Ruihang Lai, Hongyi Jin, Yixin Dong, Bohan Hou, Meng-Shiun Yu, Yiyan Zhai, Sudeep Agarwal, Hangrui Cao, Siyuan Feng, and Tianqi Chen

    Charlie F. Ruan, Yucheng Qin, Xun Zhou, Ruihang Lai, Hongyi Jin, Yixin Dong, Bohan Hou, Meng-Shiun Yu, Yiyan Zhai, Sudeep Agarwal, Hangrui Cao, Siyuan Feng, and Tianqi Chen. WebLLM: A High-Performance In-Browser LLM Inference Engine, 2024. 2

    2024

  40. [48]

    XLA: Compiling Machine Learning for Peak Performance, 2020

    Amit Sabne. XLA: Compiling Machine Learning for Peak Performance, 2020. 2

    2020

  41. [49]

    Introducing Stable Diffusion 3.5

    Stability AI, Ltd. Introducing Stable Diffusion 3.5. https: //stability.ai/news/introducing- stable- diffusion-3-5. [Online; accessed March 17, 2025]. 8

    2025

  42. [50]

    Introducing torchchat: Accelerating Local LLM Inference on Laptop, Desktop and Mobile

    Team PyTorch. Introducing torchchat: Accelerating Local LLM Inference on Laptop, Desktop and Mobile. https: //pytorch.org/blog/torchchat- local- llm- inference. [Online; accessed March 17, 2025]. 2, 6

    2025

  43. [51]

    Tencent Holdings Ltd. NCNN. https://github.com/ Tencent/ncnn. [Online; accessed March 17, 2025]. 2

    2025

  44. [52]

    Dawn, a WebGPU implemen- tation

    The Dawn and Tint Authors. Dawn, a WebGPU implemen- tation. https://dawn.googlesource.com/dawn? pli=1. [Online; accessed March 17, 2025]. 7

    2025

  45. [53]

    W3C. WebGPU. https://www.w3.org/TR/webgpu. [Online; accessed March 17, 2025]. 2

    2025

  46. [54]

    SmoothQuant: Accurate and Effi- cient Post-Training Quantization for Large Language Mod- els

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. SmoothQuant: Accurate and Effi- cient Post-Training Quantization for Large Language Mod- els. In Proceedings of the 40th International Conference on Machine Learning, 2023. 2

    2023

  47. [55]

    Xiaomi Corporation. MACE. https://github.com/ XiaoMi/mace. [Online; accessed March 17, 2025]. 2

    2025

  48. [56]

    LLMCad: Fast and Scalable On-device Large Language Model Inference, 2023

    Daliang Xu, Wangsong Yin, Xin Jin, Ying Zhang, Shiyun Wei, Mengwei Xu, and Xuanzhe Liu. LLMCad: Fast and Scalable On-device Large Language Model Inference, 2023. 2

    2023

  49. [57]

    Fast On-device LLM Inference with NPUs

    Daliang Xu, Hao Zhang, Liming Yang, Ruiqi Liu, Gang Huang, Mengwei Xu, and Xuanzhe Liu. Fast On-device LLM Inference with NPUs. In International Conference on Architectural Support for Programming Languages and Op- erating Systems, 2024

    2024

  50. [58]

    PowerInfer-2: Fast Large Language Model Inference on a Smartphone

    Zhenliang Xue, Yixin Song, Zeyu Mi, Le Chen, Yubin Xia, and Haibo Chen. PowerInfer-2: Fast Large Language Model Inference on a Smartphone. ArXiv, abs/2406.06282, 2024. 2

    2024 arXiv

  51. [59]

    Gonzalez, Clark Barrett, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. SGLang: Efficient Execution of Structured Language Model Programs, 2024. 2

    2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.