Pith. sign in

REVIEW 5 major objections 6 minor 31 references

MNN-LLM: A Generic Inference Engine for Fast Large Language Model Deployment on Mobile Devices

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MNN-LLM claims that memory-bound decoding, Flash-stored embeddings, and int4/int8 quantization deliver up to 8.6x prefill and 2.3x decode speedups over llama.cpp on phone CPUs.

desk verdict A capable re-integration of known mobile-LLM tricks with plausible but under-documented speedups; worth reviewing, not worth citing yet. read the letter →

arxiv 2506.10443 v1 pith:KMN4DR2M submitted 2025-06-12 cs.LG

Zhaode Wang , Jingbang Yang , Xinyu Qian , Shiwen Xing , Xiaotang Jiang , Chengfei Lv , Shengyu Zhang This is my paper
classification cs.LG
keywords mobileLLMinferenceDRAM-Flashhybridstorageasymmetricquantizationmemory-bounddecodeprefillandphaseshardware-drivendatareorderKVcacheLoRA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces MNN-LLM, an inference engine that runs large language models on phone CPUs and GPUs with lower memory use and higher speed than existing mobile LLM frameworks. Its central claim is that prefill is compute-bound while decode is memory-bound, so the two phases need different memory and compute optimizations. MNN-LLM stores rarely used embedding weights in Flash instead of DRAM, quantizes layer and output-head weights asymmetrically to int4/int8, and rearranges matrix data to match the phone's instruction set and GPU texture hardware. The reported result is up to 8.6x faster prefill than llama.cpp and 20.5x faster than fastllm on a flagship phone CPU, with decoding 2.3x and 8.9x faster, respectively. If the measurements hold, this is evidence that practical LLM deployment on ordinary phones is feasible without cloud help.

What carries the argument

The load-bearing mechanism is the memory-bound decode model combined with a byte-accounting argument: during decoding, if speed is set by memory bandwidth, then storing unused embedding rows in Flash costs almost nothing and quantizing the weights cuts time almost linearly. The engine makes this concrete with a tiled matrix multiplication ordering chosen by solving a register-usage minimization problem, together with a DRAM-Flash prefetch schedule for the KV cache that hides Flash latency as long as the cached length stays below roughly 3072 K tokens. It also uses geometric computation, treating data rearrangement as an affine address map $f(\vec{x}) = \vec{offset} + \vec{stride}\vec{x}$ and fusing compatible Regions, to remove long-tail operator overhead by about 3%.

What would settle it

Run MNN-LLM on the same phone with embedding weights in Flash versus in DRAM and measure decode latency; the paper's model predicts a difference near 0.014%, while a compute-bound decode would show a larger or qualitatively different overhead. A second check is to compare W8A8 and W4A8 decode throughput on a device with unusually high memory bandwidth relative to compute: if halving the weight bytes does not roughly halve per-token latency, the memory-bound premise that carries the speedup claims fails.

Watch

Extended reading notes

Core claim

The paper argues that the memory-bound character of the decode phase is an exploitable asset: each generated token requires reading the full layer and output-head weights but only one row from the embedding table, so the embedding table can live in Flash at a cost of about 1.4 permyriad (0.014%) of decode time, cutting DRAM use by roughly 15% (about 2.18 GB for Qwen2 7B in bfloat16). It combines this with asymmetric low-bit quantization—int4/int8 weights with int8 activations on CPUs, W4A16/W8A16 on GPUs, int8 for the accuracy-sensitive LM head, and fp8 for the value cache—so the bytes moved per decode step shrink and, because decode is memory-bound, token rate rises roughly in proportion. To keep prefill fast, the engine tiles and rearranges matrices according to each instruction set's register width (ARM sdot, ARM i8mm, AVX2, AVX512), balances workload across big.LITTLE cores, uses float16 for non-sensitive operations while keeping Softmax and query-key products in float32, fuses address-mapping regions for long-tail operators, and reorders LoRA computations to cut memory traffic dramatically. The quantitative payoff claimed is up to 8.6x prefill and 2.3x decode speedups over llama.cpp, and 20.5x and 8.9x over fastllm, on CPU.

Load-bearing premise

The speed and memory claims rest on the premise that the decode phase is memory-bound, so the time per generated token is set by how many bytes the hardware must load, not by how much arithmetic is performed.

Editorial extensions

If this is right

  • Quantized 1.5B to 8B models can run within phone DRAM constraints, with embedding tables shunted to Flash and the bulk of weights stored as int4/int8.
  • Because decode speed is claimed to scale with bytes moved, W4A8 CPU quantization should give near-linear token-rate gains over an otherwise identical W8 baseline on memory-bound workloads.
  • KV-cache overflow to Flash makes long-context generation possible up to a few thousand cached tokens before decode latency starts to increase by about 1 ms per extra 1K tokens beyond the prefetchable range.
  • The same byte-accounting argument generalizes to any model whose embedding and output-head tables dominate parameter count but contribute only one row per decode step.
  • Reordering LoRA computation as $A \mapsto LoRA_A \cdot (LoRA_B \cdot A)$ cuts memory access enough to make multi-LoRA serving on mobile practical.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension of the byte-accounting argument is a scaling law for a fixed phone: decode tokens per second should be inversely proportional to quantized parameter bytes plus active KV-cache bytes; verifying this across model sizes and bit widths would isolate the memory-bound claim.
  • The same hybrid Flash-DRAM recipe could transfer to laptops and tablets, where the DRAM-to-Flash speed gap is similar, though the compute-to-memory balance differs and would need its own measurement.
  • The 1.4 permyriad estimate assumes one embedding row per decode step; batched or speculative decoding would read multiple rows per step and erode the savings, so the strategy is tuned for single-token autoregressive generation.
  • A useful boundary test is a model with a small vocabulary: the embedding offload saves little, so the paper's memory gains should shrink, which would map exactly where the technique pays.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. MNN-LLM is a mobile inference engine built on the MNN framework that targets on-device large language model deployment. The paper proposes DRAM-Flash hybrid storage for embeddings and parts of the KV cache, combined int4/int8/fp8 quantization, hardware-driven data reordering for CPU and GPU kernels, multicore workload balancing, mixed-precision floating-point operations, geometric computation for long-tail operators, and a LoRA computation-order optimization. Evaluations on a Xiaomi 14 against llama.cpp, MLC-LLM, and fastllm report CPU prefill speedups up to 8.6x over llama.cpp and 20.5x over fastllm, with decode speedups of 2.3x and 8.9x, respectively, as well as GPU comparisons.

Significance. If the reported speedups are reproducible, MNN-LLM would be a practically valuable engineering contribution to on-device LLM inference, particularly for CPU-bound mobile deployments. The paper has concrete strengths: the tiling objective, the LoRA reordering table, and the 1.4-permyriad Flash-overhead arithmetic are presented as explicit, checkable calculations rather than fitted claims, and the speedups are measured on real hardware. However, the current evaluation does not establish the central attributions: quantization schemes are not controlled, no accuracy or memory-usage measurements are provided, and the reported decode speedups are in tension with the paper's memory-bound premise. The engineering is plausible, but the evidence as written is not yet sufficient to support the paper's central claims.

major comments (5)
  1. [Section 6, Figure 5] The CPU benchmarking comparison is not controlled. The quantization bit-widths, group sizes, and precision schemes used by llama.cpp and fastllm are never stated, and the text explicitly says that MLC-LLM was run with symmetric quantization while MNN-LLM used asymmetric quantization. Since quantization choice directly determines both speed and quality, the reported prefill speedups of 8.6x and 20.5x cannot be attributed to MNN-LLM's kernel or memory optimizations rather than to quantization differences. The evaluation should report the exact quantization configuration of every engine, include standard deviations over multiple runs, and justify the 16-token decoding limit.
  2. [Sections 4.1, 4.2, and 6] The paper's own decode results are in tension with its core memory-bound premise. For Qwen2-7B on CPU with a 64-token prompt, Figure 5 shows MNN-LLM decoding at 13.0 tokens/s versus llama.cpp at 6.4 tokens/s, a 2.0x gap that persists across prompt lengths. If decode were strictly bandwidth-saturated and both engines streamed the same quantized weights per token from the same DRAM, no software kernel change could produce such a gap without reducing the number of bytes moved. The paper must either report bytes-per-token moved by each engine, demonstrate that the baselines use different precision, or acknowledge that decode is partly compute-bound on this device. Without this, the speedups in Section 6 do not validate the memory-side analyses in Sections 4.1 and 4.2.
  3. [Section 4.2] The paper claims that asymmetric int4/int8 quantization and the mixed-precision strategy preserve model accuracy, but no accuracy evidence appears anywhere in the manuscript. There are no perplexity numbers, task-accuracy measurements, or qualitative output comparisons for the W4A8, W8A8, W4A16, W8A16, or fp8 KV-cache configurations. Because quantization is a central contribution and the comparison with MLC-LLM uses different quantization schemes, the reader cannot judge whether the reported speedups are obtained at an unacceptable quality cost. The paper should report accuracy metrics for each model and quantization scheme used in the benchmarks.
  4. [Section 5.1, Eq. (2)-(4)] The tiling optimization formulation is internally inconsistent. If R is the number of vector registers and e_p and h_p are element tile sizes, the constraint e_p + h_p + e_p*h_p <= R is violated by every row of Table 2 (for example, 12+8+96=116 for ARM i8sdot, far exceeding a typical 32-register NEON register file). If e_p and h_p instead denote register groups or vector counts, the text must say so explicitly. The stated objective also omits the l/l_p factor that would appear in a fully tiled memory-access count. The derivation should be corrected, or the section should be reframed as a heuristic with the actual register-usage accounting.
  5. [Section 6, GPU comparison paragraph] The GPU results paragraph is self-contradictory: it first says "MNN-LLM's performance slightly declines compared to MLC-LLM" and then states that MNN-LLM achieves "2.8x and 1.7x improvements over MLC-LLM." These statements cannot both be true for the same configuration, and the contradiction makes the GPU conclusions impossible to interpret. The authors should rewrite this paragraph and ensure the claims match the data in Figure 5.
minor comments (6)
  1. [Section 4.1] The sentence "The UFS 4.0 read speed is approximately 15 microseconds slower than LPDDR5X" is missing the quantity being compared; the rest of the argument suggests it refers to loading one 7 KB embedding entry, but this should be stated explicitly.
  2. [Section 4.1] The claim that "each additional 1 K of length adds approximately 1 ms of delay" is inconsistent with the stated 1 GB/s Flash read speed and roughly 1 KB of KV data per token; the delay should be on the order of microseconds per KB, not milliseconds.
  3. [Section 5.3] There is a typo in "float16 NOEN instructions" where "NOEN" should be "NEON."
  4. [Section 5.4] The claimed "improving performance by approximately 3%" from geometric computation is presented without any ablation or measurement description, so the reader cannot assess how this number was obtained.
  5. [Figure 5] The 'X' markers in Figure 5 are not explained in the caption or text; the reader cannot tell whether they denote unsupported configurations, timed-out runs, or omitted measurements.
  6. [Section 6] The evaluation identifies the device only as "Xiaomi 14" without specifying the SoC, RAM, OS version, or thermal/power settings, even though Section 5.2 references the Snapdragon 8 Gen 3; these details are needed for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's optimizations are self-contained calculations against stated hardware constraints, and the headline speedups are measured against external frameworks rather than derived from the assumptions.

full rationale

The paper's analytical steps are not circular. The 1.4-permyriad overhead estimate in Section 4.1 for storing embeddings in Flash is an arithmetic calculation conditioned on the explicitly stated memory-bound premise and on reported DRAM/Flash bandwidth figures; it is not used as evidence for any speedup claim. Section 4.2's quantization discussion reasons that if decode is memory-bound then bytes loaded per token dominate, which motivates int4/int8 weights, but the actual speedups in Section 6 are measured against llama.cpp, MLC-LLM, and fastllm, not computed from the bit-width ratios. The hardware-driven data-reorder objective in Section 5.1 is a stated memory-access minimization with explicit tile-size constraints; the paper does not fit a parameter to a target speedup and then present that speedup as a prediction. The only self-citations are to MNN (Ref. [15]) as the base framework and Walle (Ref. [17]) for the geometric-computation abstraction; both are described with their own equations and are not invoked as an external theorem that forces the design. The memory-bound assumption may be a correctness risk if decode is partly compute-bound, but that is a validity concern, not circularity. The central performance claims are externally benchmarked, so the derivation chain is not equivalent to its inputs.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The listed assumptions are engineering premises from Sections 4 and 5; none of them is a fitted scientific constant. The central speedup claim relies most heavily on the memory-bound assumption and the prefetching overlap assumption, both stated but not validated on the test device.

free parameters (1)
  • Assumed Flash read speed for prefetch budget = 1 GB/s
    Used in Section 4.1 to compute the 3072 K KV-cache threshold at which Flash prefetching stops hiding latency; the number is chosen by hand, not measured on the Xiaomi 14, and the claim that 3 MB of KV can be loaded within the computation time depends on it.
assumptions (5)
  • domain assumption Decode-phase LLM inference on mobile devices is memory-bound, so inference time is directly proportional to parameter size loaded per step.
    Invoked in Sections 4.1 and 4.2 to justify storing embeddings in Flash and to predict quantization speedups. If the decode phase is compute-bound or cache-dominated, the 1.4 permyriad overhead estimate and the direct proportionality do not hold.
  • domain assumption Prefetching during the MLP phase and the qkv projection can hide Flash read latency when prefetch time is less than or equal to computation time.
    Section 4.1 and Figure 2. Requires asynchronous Flash reads and sufficient compute time in the current layer to mask the reads, which is assumed but not measured on the target device.
  • domain assumption The loop tiling optimization objective (minimizing memory-access count with register constraint ep+hp+hp*ep <= R and lp = instruction width) captures the relevant performance optimum for mobile CPU matrix multiplication.
    Section 5.1, Equations (2)-(4). The objective ignores cache capacity, instruction scheduling, and memory subsystem effects beyond register reuse, so the derived tile sizes may not be truly optimal.
  • standard math Scaling the query by 1/sqrt(dk) instead of scaling the attention scores is numerically equivalent and safe for float16 computations.
    Section 5.3. This is standard linearity of attention, but the paper does not analyze rounding behavior at long contexts; it is a reasonable domain assumption for preventing overflow.
  • domain assumption Region fusion rules for geometric compute preserve the semantics of arbitrary data rearrangement operators.
    Section 5.4, based on prior Walle work [17]. The paper assumes that fusing Regions via loop transformations does not change results, which is plausible but not proven for all LLM graph patterns.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MNN-LLM: A Generic Inference Engine for Fast Large Language Model Deployment on Mobile Devices." pith.science (2026). https://pith.science/paper/KMN4DR2M

@misc{pith2026250610443,
  author       = {Pith},
  title        = {Pith review of: MNN-LLM: A Generic Inference Engine for Fast Large Language Model Deployment on Mobile Devices},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KMN4DR2M}},
  note         = {Machine review of arXiv:2506.10443}
}
read the original abstract

Large language models (LLMs) have demonstrated exceptional performance across a variety of tasks. However, their substantial scale leads to significant computational resource consumption during inference, resulting in high costs. Consequently, edge device inference presents a promising solution. The primary challenges of edge inference include memory usage and inference speed. This paper introduces MNN-LLM, a framework specifically designed to accelerate the deployment of large language models on mobile devices. MNN-LLM addresses the runtime characteristics of LLMs through model quantization and DRAM-Flash hybrid storage, effectively reducing memory usage. It rearranges weights and inputs based on mobile CPU instruction sets and GPU characteristics while employing strategies such as multicore load balancing, mixed-precision floating-point operations, and geometric computations to enhance performance. Notably, MNN-LLM achieves up to a 8.6x speed increase compared to current mainstream LLM-specific frameworks.

Figures

Figures reproduced from arXiv: 2506.10443 by the authors.

Figure 1
Figure 1. DRAM-Flash Hybrid Storage for LLM model param [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Comparison of KV loading times for DRAM, DRAM [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The reduction dimensions in the computation of [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Parallel computing between 1 prime cores and 3 per [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Prefill and decode speeds of MNN-LLM, llama.cpp, [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

31 extracted references · 16 canonical work pages

  1. [1]

    Keivan Alizadeh, Iman Mirzadeh, Dmitry Belenko, Karen Khatamifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. 2024. LLM in a flash: Efficient Large Language Model Inference with Limited Memory. arXiv:2312.11514 [cs.CL] https://arxiv.org/abs/2312.11514

    arXiv 2024

  2. [2]

    ARM. 2024. Arm Big.LITTLE. https://www.arm.com/zh-TW/technologies/big- little

    work page 2024

  3. [3]

    ARM. 2024. Arm NEON. https://www.arm.com/technologies/neon

    work page 2024

  4. [4]

    ARM. 2024. Dot Product. https://developer.arm.com/documentation/100069/ 0609/A64-SIMD-Vector-Instructions/SDOT--vector-

    work page 2024

  5. [5]

    ARM. 2024. Matrix Multiplication extension. https://developer.arm.com/ documentation/101754/0622/armclang-Reference/Other-Compiler-specific- Features/Supported-architecture-features/Matrix-Multiplication-extension

    work page 2024

  6. [6]

    ARM. 2024. The Armv8.2 architecture extension. https://developer.arm.com/ documentation/109697/latest/Feature-descriptions/The-Armv8-2-architecture- extension

    work page 2024

  7. [7]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

    arXiv 2023

  8. [8]

    Junjie Bai, Fang Lu, Ke Zhang, et al. 2019. ONNX: Open Neural Network Exchange. https://github.com/onnx/onnx

    work page 2019

Show all 31 references
  1. [9]

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. arXiv:2210.17323 [cs.LG] https://arxiv.org/abs/2210.17323

    2023 arXiv

  2. [10]

    Georgi Gerganov. 2024. ggerganov/llama.cpp: Port of Facebook’s LLaMA model in C/C++. https://github.com/ggerganov/llama.cpp

    2024

  3. [11]

    Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Y...

    2023 arXiv

  4. [12]

    Emna Hammami and Yosr Slama. 2017. An Overview on Loop Tiling Techniques for Code Generation. In 2017 IEEE/ACS 14th International Conference on Computer Systems and Applications (AICCSA). 280–287. https://doi.org/10.1109/AICCSA. 2017.168

    2017 doi

  5. [13]

    Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Wei- jun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam

    Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Wei- jun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. 2017. Mo- bileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv:1704.04861 [cs.CV] https://arxiv.org/abs/1704.04861

    2017 arXiv

  6. [14]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685 [cs.CL] https://arxiv.org/abs/2106.09685

    2021 arXiv

  7. [15]

    Xiaotang Jiang, Huan Wang, Yiliu Chen, Ziqi Wu, Lichuan Wang, Bin Zou, Yafeng Yang, Zongyang Cui, Yu Cai, Tianhang Yu, Chengfei Lv, and Zhihua Wu. 2020. MNN: A Universal and Efficient Inference Engine. CoRR abs/2002.12418 (2020). arXiv:2002.12418 https://arxiv.org/abs/2002.12418

    2020 arXiv

  8. [16]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

  9. [17]

    Chengfei Lv, Chaoyue Niu, Renjie Gu, Xiaotang Jiang, Zhaode Wang, Bin Liu, Ziqi Wu, Qiulin Yao, Congyu Huang, Panos Huang, Tao Huang, Hui Shu, Jinde Song, Bin Zou, Peng Lan, Guohuan Xu, Fei Wu, Shaojie Tang, Fan Wu, and Guihai Chen. 2022. Walle: An End-to-End, General-Purpose,...

    2022

  10. [18]

    OpenAI. 2023. ChatGPT. https://openai.com/chatgpt Available at: https: //openai.com/chatgpt

    2023

  11. [19]

    on device

    Qualcomm. 2023. The future of AI is “on device”. https://cms.tinyml.org/wp- content/uploads/ew2023/Kyuwoong-Hwang_tinyML-Asia-2023.pdf

    2023

  12. [20]

    qualcomm. 2024. Adreno Graphics Processing Units. https://www.qualcomm. com/products/features/adreno

    2024

  13. [21]

    qualcomm. 2024. Snapdragon 8 Gen 3 Mobile Platform. https://www.qualcomm. com/products/mobile/snapdragon/smartphones/snapdragon-8-series-mobile- platforms/snapdragon-8-gen-3-mobile-platform

    2024

  14. [22]

    Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2016. You Only Look Once: Unified, Real-Time Object Detection. arXiv:1506.02640 [cs.CV] https://arxiv.org/abs/1506.02640

    2016 arXiv

  15. [23]

    MLC team. 2024. MLC-LLM. https://github.com/mlc-ai/mlc-llm

    2024

  16. [24]

    Qwen Team. 2024. Qwen2.5: A Party of Foundation Models. https://qwenlm. github.io/blog/qwen2.5/

    2024

  17. [25]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. Attention Is All You Need. arXiv:1706.03762 [cs.CL] https://arxiv.org/abs/1706.03762

    2023 arXiv

  18. [26]

    Zhenliang Xue, Yixin Song, Zeyu Mi, Le Chen, Yubin Xia, and Haibo Chen

  19. [27]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Cheng- peng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzhe...

    2024 arXiv

  20. [28]

    Biao Zhang and Rico Sennrich. 2019. Root Mean Square Layer Normalization. arXiv:1910.07467 [cs.LG] https://arxiv.org/abs/1910.07467

    2019 arXiv

  21. [29]

    ztxz16. 2023. fastllm. https://github.com/ztxz16/fastllm. 7

    2023

  22. [2020]

    CoRR abs/2001.08361 (2020)

    Scaling Laws for Neural Language Models. CoRR abs/2001.08361 (2020). arXiv:2001.08361 https://arxiv.org/abs/2001.08361

    2020 arXiv

  23. [2024]

    arXiv:2406.06282 [cs.LG] https://arxiv.org/abs/2406.06282

    PowerInfer-2: Fast Large Language Model Inference on a Smartphone. arXiv:2406.06282 [cs.LG] https://arxiv.org/abs/2406.06282

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.