Pith. sign in

REVIEW 2 minor 18 references

Above the Inner Loop: Exceeding Accelerate at LLM Prefill GEMM on the M1 AMX

T0 review · 0 major / 2 minor · reviewed 2026-06-25 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A hand-written direct-AMX kernel exceeds all three Accelerate fp32 paths on M1 for every LLM prefill GEMM shape at S=128 by using fine multi-thread panels and weight pre-packing.

desk verdict Gains over Accelerate on M1 AMX for these GEMMs come from multi-block scheduling and pre-packing, not inner-loop changes, with concrete bit-exact measurements and an end-to-end win. read the letter →

arxiv 2606.25426 v1 pith:KZL2D3KB submitted 2026-06-24 cs.PF

Deyvik Bhan This is my paper · ORCID
classification cs.PF
keywords GEMMAMXAppleSiliconLLMprefillAcceleratefp32matrixmultiplicationperformanceoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper establishes that gains over Accelerate for fp32 GEMM in LLM prefill on the M1 AMX arise from two deployment levers rather than any faster inner loop: fine multi-thread panels that fill the second on-chip AMX block for K >= N shapes, and pre-packing constant weights at load time for N > K shapes. A bit-exact kernel applying both beats cblas_sgemm, BNNSMatMul, and BNNS Graph at all twelve tested shapes from GPT-2 to Llama-7B scale, with a 1.17x lead over the fastest Accelerate path and geometric means of 1.58x and 2.0x over the other two. When substituted for cblas_sgemm inside llama.cpp, the kernel raises measured full-forward throughput from 291 to 420 tokens/s at 128-token prefill while keeping every output bit-identical. The work supplies a shape-resolved characterization showing the inner loop is already load-issue bound once any operand load interleaves with the FMA32 stream, so further gains must come from scheduling and packing above that loop.

What carries the argument

Fine multi-thread panels that fill both on-chip AMX blocks together with pre-packing of the constant weight matrix at load time, which together avoid the load-issue bound that caps single-thread throughput at 610-680 GFLOPS once operand loads interleave with the FMA32 stream.

What would settle it

A direct measurement, on the exact twelve GEMM shapes and thread counts from the llama.cpp prefill experiment, of whether single-thread throughput stays inside the 610-680 GFLOPS band once operand loads are present and whether the full kernel still exceeds the three Accelerate paths by the reported margins while producing bit-identical outputs.

Watch

Extended reading notes

Core claim

The central claim is that a bit-exact direct-AMX kernel employing both fine multi-thread panels filling the second AMX block and pre-packing of constant weights at load time is the fastest fp32 GEMM path on the M1, exceeding cblas_sgemm, BNNSMatMul, and BNNS Graph at all twelve LLM prefill GEMM shapes at S=128 while remaining bit-identical to Accelerate, and that these two levers above the inner loop account for the measured speedups of 1.17x over BNNS Graph and end-to-end 1.44x in llama.cpp.

Load-bearing premise

The microbenchmark result that single-thread throughput falls to the 610-680 GFLOPS band as soon as any operand load interleaves with the FMA32 stream accurately reflects the behavior inside the actual LLM prefill GEMM shapes and thread counts used in the end-to-end llama.cpp experiment.

Editorial extensions

If this is right

  • The kernel exceeds all three Accelerate fp32 paths at every one of the twelve tested GEMM shapes for S=128 prefill.
  • BNNS Graph produces reduced-precision results (error up to 1.4e-3) at nine of the twelve shapes while the direct-AMX kernel stays bit-exact at all twelve.
  • Substituting the kernel for cblas_sgemm inside llama.cpp raises measured full-forward throughput from 291 to 420 tokens/s at 128-token prefill.
  • Mis-tuning the single column-panel width costs nearly 2x in throughput.
  • The inner loop is load-issue bound once operand loads interleave with FMA32, so gains must come from the two higher-level levers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same two levers could be applied to other constant-weight matrix operations that dominate inference beyond the twelve shapes tested here.
  • The M1 characterization implies that similar scheduling and packing opportunities may exist on later Apple Silicon chips that retain the AMX coprocessor.
  • Replacing library calls only for the prefill phase while leaving other kernels untouched could be tested for incremental gains in full-model serving.
  • The panel-width sensitivity result suggests that automatic tuning of panel size for each (M,N,K) triple would be a low-effort next step.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

0 major / 2 minor

Summary. The paper claims that a bit-exact direct-AMX fp32 GEMM kernel on the Apple M1, using fine multi-thread panels to fill the second AMX block and pre-packing constant weights, exceeds all three Accelerate fp32 paths for LLM prefill GEMMs at S=128. The gains are attributed to deployment levers rather than a faster inner loop, as microbenchmarks show the inner loop is load-issue bound with throughput dropping to 610-680 GFLOPS when loads interleave with FMA32. The kernel leads BNNS Graph by 1.17x (1.09x where both fp32), with geometric means of 1.58x over BNNSMatMul and 2.0x over cblas_sgemm, and improves llama.cpp prefill throughput from 291 to 420 tokens/s (1.44x).

Significance. If the results hold, this provides a useful shape-resolved characterization of M1 AMX performance for ML workloads, emphasizing that gains come from multi-threading and pre-packing rather than inner-loop optimization. Strengths include the use of direct hardware measurements, enforcement of bit-exactness with Accelerate, and validation through end-to-end integration in llama.cpp, making the findings reproducible and practically relevant for optimizing GEMM on Apple Silicon.

minor comments (2)
  1. The abstract does not detail how bit-exactness was enforced across all paths or the exact thread-to-AMX mapping; adding this would strengthen support for the two-block utilization claim without affecting the measured speedups.
  2. The microbenchmark isolation (single-thread load interleaving) is presented as reflecting the deployed multi-threaded case; a brief cross-check or note on why the load-issue bound transfers to the per-panel threading and pre-packing configuration would improve clarity.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for the positive assessment, detailed summary of our contributions, and recommendation for minor revision. The emphasis on the practical relevance, bit-exactness enforcement, and end-to-end validation in llama.cpp is appreciated. Since no specific major comments were raised, we provide no point-by-point responses below and confirm that the manuscript stands as presented, with any minor editorial adjustments to be made in revision.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; derivation rests on direct measurements

full rationale

The paper derives its structural conclusion (inner loop is load-issue bound; gains arise only from multi-panel threading and weight pre-packing) from explicit single-thread microbenchmarks that measure throughput collapse under operand-load interleaving, plus direct end-to-end timing on twelve LLM prefill shapes and a drop-in llama.cpp replacement. No equations, fitted parameters, or self-citations appear in the provided text; the reported speedups are not obtained by renaming or re-using quantities defined from the authors' own prior results. The derivation is therefore self-contained against external hardware benchmarks.

Assumptions & free parameters 1 free parameters · 1 assumptions · 0 invented entities

The central performance claim rests on one hardware-behavior assumption extracted from microbenchmarks and one tunable scheduling parameter whose value must be chosen per shape.

free parameters (1)
  • column-panel width
    Single tuning parameter whose mis-tuning is stated to cost nearly 2x throughput; value is chosen to fill the second AMX block for the target shapes.
assumptions (1)
  • domain assumption Single-thread AMX throughput drops to the 610-680 GFLOPS band as soon as any operand load interleaves with the FMA32 stream
    Invoked to conclude that inner-loop optimization cannot be the source of the observed gains.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Above the Inner Loop: Exceeding Accelerate at LLM Prefill GEMM on the M1 AMX." pith.science (2026). https://pith.science/paper/KZL2D3KB

@misc{pith2026260625426,
  author       = {Pith},
  title        = {Pith review of: Above the Inner Loop: Exceeding Accelerate at LLM Prefill GEMM on the M1 AMX},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KZL2D3KB}},
  note         = {Machine review of arXiv:2606.25426}
}
read the original abstract

On Apple Silicon the fp32 GEMMs dominating LLM prefill are dispatched by Accelerate to a matrix coprocessor (AMX) on the M1-M3. We ask where a hand-written kernel's throughput over Accelerate comes from on the M1 AMX, and reach a structural conclusion: not a faster inner loop. By microbenchmark the inner loop is load-issue bound -- once any operand load interleaves with the FMA32 stream, single-thread throughput falls to a 610-to-680 GFLOPS band, under half the load-free rate. The gain comes from two deployment-level levers Accelerate underuses: fine multi-thread panels filling the M1's second on-chip AMX block (winning the K >= N shapes), and pre-packing the constant weight at load (winning the N > K shapes). A bit-exact direct-AMX kernel using both is the fastest bit-exact fp32 GEMM path we find on the M1: it exceeds all three Accelerate fp32 paths (cblas_sgemm, BNNSMatMul, and the BNNS Graph compiler) at all twelve LLM prefill GEMMs at S = 128 (GPT-2 to Llama-7B scale), leading the fastest, BNNS Graph, by 1.17 -- and by 1.09 at the three shapes where it too holds fp32 -- with geometric means of 1.58 over BNNSMatMul and about 2.0x over cblas_sgemm. Every output is bit-identical to Accelerate, whereas BNNS Graph is bit-exact at only three of twelve shapes, the rest at reduced precision (error up to 1.4e-3). Dropped into llama.cpp for its cblas_sgemm prefill matmuls, it raises measured full-forward throughput from 291 to 420 tokens/s (1.44x, bit-identical) at 128-token prefill -- end-to-end, not a GEMM-only ratio. The contribution is this shape-resolved M1-AMX characterization (microbenchmark, two-block aggregate, per-core occupancy probe), leaving fine-panel scheduling and pre-packing as the only two levers above an inner loop at the hardware limit; mis-tuning the single column-panel width costs nearly 2x.

Figures

Figures reproduced from arXiv: 2606.25426 by the authors.

Figure 1
Figure 1. Prefill GEMM throughput (fp32, bit-exact) at the GPT-2- [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Throughput of the proposed pre-packed kernel at the QKV [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 7 canonical work pages

  1. [1]

    Demysti- fying ARM SME to Optimize General Matrix Multiplica- tions,

    C. Deng, W. Yang, J. Fang, and D. Dong, “Demysti- fying ARM SME to Optimize General Matrix Multiplica- tions,” arXiv:2512.21473, December 2025

    work page arXiv 2025

  2. [2]

    Performance Analysis of the Apple AMX Matrix Accelerator,

    J. Zhou, “Performance Analysis of the Apple AMX Matrix Accelerator,” M.Eng. thesis, Department of Elec- trical Engineering and Computer Science, Massachusetts Institute of Technology, September 2025. A vailable at https://commit.csail.mit.edu/papers/2025/Jonathan_ Zhou_SB_Thesis.pdf

    2025

  3. [3]

    Apple vs. Oranges: Evaluating the Apple Silicon M-Series SoCs for HPC Performance and Efficiency,

    P. Hübner, A. Hu, I. Peng, and S. Markidis, “Apple vs. Oranges: Evaluating the Apple Silicon M-Series SoCs for HPC Performance and Efficiency,” arXiv:2502.05317, 2025

    work page arXiv 2025

  4. [4]

    Bare-Metal Tensor Vir- tualization: Overcoming the Memory Wall in Edge-AI Inference on ARM64,

    B. Kilictas and F. Alpay, “Bare-Metal Tensor Vir- tualization: Overcoming the Memory Wall in Edge-AI Inference on ARM64,” arXiv:2601.03324, January 2026

    work page arXiv 2026

  5. [5]

    corsix/amx: reverse-engineered Apple AMX instruction reference,

    P. Cawley (corsix), “corsix/amx: reverse-engineered Apple AMX instruction reference,” GitHub repository, ht tps://github.com/corsix/amx, accessed May 2026

    2026

  6. [6]

    dougallj: Apple silicon reverse- engineering notes,

    D. Johnson, “dougallj: Apple silicon reverse- engineering notes,” GitHub repository, https://github.c om/dougallj, accessed May 2026

    2026

  7. [7]

    Anatomy of high- performance matrix multiplication,

    K. Goto and R. A. van de Geijn, “Anatomy of high- performance matrix multiplication,” ACM Transactions on Mathematical Software, vol. 34, no. 3, article 12, May 2008

    2008

  8. [8]

    llama.cpp: efficient LLM inference in pure C/C++,

    G. Gerganov and the llama.cpp contributors, “llama.cpp: efficient LLM inference in pure C/C++,” GitHub repository, https://github.com/ggml-org/llam a.cpp, accessed May 2026

    2026

Show all 18 references
  1. [9]

    amx-benchmarks: throughput measure- ments for the Apple matrix coprocessor,

    P. Turner, “amx-benchmarks: throughput measure- ments for the Apple matrix coprocessor,” GitHub reposi- tory, https://github.com/philipturner/amx-benchmarks , accessed May 2026

    2026

  2. [10]

    Accelerating Sparse Ternary GEMM for Quantized ML on Apple Silicon,

    B. Lipshitz, A. Melone, C. Maraziaris, and M. Bilal, “Accelerating Sparse Ternary GEMM for Quantized ML on Apple Silicon,” arXiv:2510.06957, 2025

    2025

  3. [11]

    Benchmarking and Characterization of Large Language Model Inference on Apple Silicon,

    A. Benazir and F. X. Lin, “Benchmarking and Characterization of Large Language Model Inference on Apple Silicon,” Proceedings of the ACM on Measure- ment and Analysis of Computing Systems (POMACS), vol. 9, no. 3, article 48, pp. 1–26, 2 December 2025. DOI: 10.1145/3771563

    2025 doi

  4. [12]

    Benchmarking On- Device Machine Learning on Apple Silicon with MLX,

    O. A. Ajayi and O. Odunayo, “Benchmarking On- Device Machine Learning on Apple Silicon with MLX,” presented at the 2024 Deep Learning Indaba, Dakar, Sene- gal; arXiv:2510.18921, October 2025

    2024

  5. [13]

    ONNX Runtime: cross-platform infer- ence engine,

    Microsoft, “ONNX Runtime: cross-platform infer- ence engine,” GitHub repository, https://github.com/m icrosoft/onnxruntime , accessed May 2026. MLAS Arm64 SGEMM kernels at onnxruntime/core/mlas/lib/aarch64/ SgemmKernelNeon.S and the NEON/DOT/I8MM/SVE dispatch in onnxruntime/co...

    2026

  6. [14]

    XNNPACK: high-efficiency neural- network inference operators,

    Google, “XNNPACK: high-efficiency neural- network inference operators,” GitHub repository, https://github.com/google/XNNPACK ; and “Memory- efficient inference with XNNPACK weights cache,” TensorFlow Blog, June 2022. (XNNPACK repacks and caches constant weights once for reuse.)

    2022

  7. [15]

    oneDNN: weight pre-packing for matmul and inner-product primitives,

    Intel, “oneDNN: weight pre-packing for matmul and inner-product primitives,” oneAPI Deep Neural Network Library documentation, accessed May 2026. (Constant weights are converted once to a blocked layout and reused, avoiding per-call repacking.)

    2026

  8. [16]

    What’s new in BNNS Graph,

    Apple, “What’s new in BNNS Graph,” WWDC25 session 276, and “Support real-time ML inference on the CPU,” WWDC24 session 10211, https://developer.apple. com/videos/play/wwdc2025/276/. (BNNS Graph, macOS 15+, automatically repacks weights for cache locality.)

  9. [17]

    PQC-AMX: Accelerating Saber and FrodoKEM on the Apple M1 and M3 SoCs,

    D. L. Gazzoni Filho, G. Brandão, G. Adj, A. Al- blooshi, I. A. Canales-Martínez, J. Chávez-Saab, and J. López, “PQC-AMX: Accelerating Saber and FrodoKEM on the Apple M1 and M3 SoCs,” in 2024 IEEE 31st Symposium on Computer Arithmetic (ARITH), pp. 9– 16, 2024; IACR ePrint 2024/...

    2024

  10. [18]

    A comparative per- formance and efficiency analysis of Apple’s M architec- tures: A GEMM case study,

    S. Catalán, R. Rodríguez-Sánchez, C. García Sánchez, and L. Piñuel Moreno, “A comparative per- formance and efficiency analysis of Apple’s M architec- tures: A GEMM case study,” Future Generation Com- puter Systems, vol. 180, article 108393, 2026. DOI: 10.1016/j.future.2026.108393

    2026 doi

Pith tools

Reviewed June 25, 2026 · model on record in the stance chip above.