REVIEW 3 major objections 5 minor 20 references
Profiling Apple Silicon Performance for ML Training
T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read The paper explains Apple Silicon's LLM training gap as a BLAS kernel problem, dominated by weak FP16 acceleration and compounded by page faults and slow kernel launches.
desk verdict A genuinely useful Apple-vs-NVIDIA benchmark dataset with an overreach: the paper's own FP32 end-to-end runs can't be explained by its headline FP16 BLAS gap. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the BLAS kernel, the basic linear algebra subroutine that implements the matrix-matrix, matrix-vector, and vector-Jacobian operations dominating transformer forward and backward passes. The paper treats BLAS throughput as the main determinant of end-to-end training time, then benchmarks it across FP16 and FP32 on Apple Silicon through both the MPS backend and the MLX framework, against CUDA on NVIDIA GPUs. The identity that carries the argument is the FP16 acceleration ratio on each platform: NVIDIA's Tensor Cores multiply FP16 throughput by 5x-6x on the backpropagation kernel, while Apple's FP16 path delivers near-zero speedup in MPS and only 20-30% in MLX on GEMMs, so the same model trains much faster on CUDA.
What would settle it
Replace Apple Silicon's BLAS library with one achieving CUDA-class FP16 throughput, or run the same training on an Apple chip with dedicated FP16 tensor hardware, and check whether the end-to-end LLM training gap closes proportionally. If the timing gap stays large while the BLAS gap closes, the paper's root-cause attribution fails.
Extended reading notes
Core claim
The paper claims that the end-to-end LLM training performance gap between Apple Silicon and NVIDIA GPUs can be explained by BLAS kernel throughput differences (the primary factor) together with system-level issues such as page faults and kernel launch overhead (secondary factors). In the three BLAS operation families that dominate transformer training — matrix-matrix product, matrix-vector product, and vector-Jacobian product — NVIDIA GPUs show strong FP16 acceleration (about 5x-6x for vector-Jacobian product), while Apple's MPS backend gets almost none and the MLX framework gets only about 20-30% on matrix-matrix product. Apple Silicon also shows memory consumption that grows gradually during training, page faults that rise with it, and kernel launches that are tens of times slower than CUDA's. The paper concludes that these observed factors, not memory capacity or thermal limits, are the root causes of the performance gap, and that Apple Silicon's advantage appears only in the scenario where the CUDA device must offload via ZeRO-Offload.
Load-bearing premise
The argument assumes that the measured page faults, kernel launch latencies, and BLAS throughput differences are the causes of the end-to-end timing gap and not merely correlated symptoms of a deeper bottleneck in Apple's driver, scheduler, or memory hierarchy.
Editorial extensions
If this is right
- If BLAS FP16 throughput is the primary bottleneck, Apple Silicon LLM training will keep trailing NVIDIA until the FP16 compute path is improved; FP32 training narrows the gap but does not close it.
- If page faults and rising RSS are secondary contributors, Apple Silicon is especially fragile for workloads that run near the unified-memory limit, as the Whisper-large near-capacity run showed large, unstable per-pass times.
- If the kernel launch overhead is significant, training workloads with many small operations will be disproportionately slower on Apple Silicon than workloads dominated by large GEMMs.
- If GPU VRAM is too small for a model, Apple Silicon becomes the faster choice: the ZeRO-Offload overhead can make an RTX 4090 slower than an M2 Ultra, so the large unified-memory pool is a real advantage.
- If the measured energy-per-iteration figures hold, Apple Silicon offers a meaningful energy-efficiency advantage over some NVIDIA GPUs, which matters for sustained and affordable training.
Reading between the lines
- The paper's own data would allow a further test: run the same end-to-end training on Apple Silicon with an alternative BLAS implementation whose FP16 throughput matches CUDA, and see whether the end-to-end gap closes proportionally; if it does not, the attribution is wrong.
- The measurements suggest an optimization roadmap the paper only sketches: porting MLX's kernel strategies into the MPS backend, or adding dedicated FP16 hardware to future Apple chips, would be the highest-leverage changes for LLM training.
- The observed page-fault and RSS growth pattern likely applies beyond training, to memory-hungry inference with very long contexts; that is a testable extension the paper did not run.
- For benchmark designers, the paper implies that FP16 throughput, not peak single-precision TFLOPS, is the metric that predicts real LLM training speed on Apple Silicon.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper empirically compares LLM training performance on Apple Silicon (M2 Pro, M2 Max, M2 Ultra) against several NVIDIA GPUs (RTX 4000, 2080Ti, A6000, RTX 4090) across three memory scenarios: sufficient memory on all devices, insufficient NVIDIA VRAM with ZeRO-Offload, and near-capacity Apple unified memory. It reports end-to-end forward/backward times, memory usage, page faults, energy consumption, GPU kernel launch times, and BLAS microbenchmarks for matrix-matrix, matrix-vector, and vector-Jacobian products in FP32 and FP16. The paper concludes that the end-to-end gap is primarily explained by BLAS kernel throughput differences, especially weak FP16 support on Apple Silicon, with secondary contributions from page faults, kernel launch overhead, and memory management behavior.
Significance. If the central attribution were established, the paper would be useful to ML practitioners choosing between Apple Silicon and NVIDIA hardware, and to Apple/software vendors prioritizing FP16 support and kernel optimizations. The study has concrete strengths: it includes side-by-side end-to-end training measurements on multiple platforms, covers a realistic range of memory-constrained scenarios, and reports the striking result that Apple devices can outperform NVIDIA GPUs when ZeRO-Offload is forced. The energy-per-iteration comparison is also a useful datapoint. However, the core causal claim that BLAS and system-level factors 'explain' the FP32 end-to-end gap is not supported by the data as presented, because the largest BLAS gap is in FP16 while the end-to-end runs are FP32. The system-level measurements are correlational and incomplete, and the paper lacks basic reproducibility details such as software versions and measurement repeats. The paper is likely to be a useful empirical contribution after substantial revision that either strengthens the causal analysis or reframes the claims as observations and hypotheses rather than identified root causes.
major comments (3)
- [§4.4 and §4.1] The central claim that BLAS performance is the 'primary factor' behind the end-to-end gap is contradicted by the paper's own precision settings. Section 4.1 states that all end-to-end runs use FP32, with AMP.FP16 and AMP.BF16 only on CUDA. Section 4.4 and Figure 5 report that for FP32 batched matrix-matrix product, MLX and MPS are 'slightly weaker but largely comparable' to CUDA, while the large BLAS gap appears in FP16 with Tensor Cores. An FP16 BLAS deficit therefore cannot explain an FP32 end-to-end training gap. To support the root-cause conclusion, the authors need either an end-to-end FP16 experiment on Apple Silicon or a kernel-level decomposition showing that FP32 BLAS kernels dominate iteration time and that their modest throughput gap quantitatively accounts for the reported 2-3x end-to-end gap. Neither is present.
- [§4.3] The system-level attributions in Section 4.3 are correlational and do not bear the causal load placed on them. Page faults are measured only on the M2 Max and never on an NVIDIA device, so there is no comparative baseline showing that page-fault counts differ in a way that explains the timing gap. The kernel launch time microbenchmark (Table 6) shows Apple kernels launch more slowly, but the paper does not report how many kernel launches occur per training iteration or estimate the total launch overhead relative to the measured iteration times. The gradual RSS increase on Apple devices is presented as a 'major reason' for underperformance, but it could be an allocator behavior or memory-prefetch policy rather than a bottleneck. The authors should either provide a quantitative model that converts observed page-fault counts and launch times into predicted training-time overhead, or explicitly weaken the language from 'identified the root causes' to 'observed system-level correlates.'
- [§3 and Table 4 / Figure 3] The experimental methodology is under-specified for a measurement paper, and this directly affects confidence in the headline comparisons. Table 4 and Figure 3 report single timings without error bars, number of repetitions, or variance. The paper does not state the software versions (PyTorch, MLX, macOS, CUDA, driver versions) on any testbed, nor the exact CPU/RAM configuration of the NVIDIA servers. The measurement procedures for page faults and energy consumption are not described (e.g., which tool was used, sampling rate, whether whole-system or only GPU energy). Without these details, the reported quantitative gaps cannot be independently checked or compared across configurations. I request that the authors add reproducibility information and repeated-measurement statistics for all timing and energy results.
minor comments (5)
- [§4.2, Table 5] The energy comparison in Table 5 reports 'GPU energy per iteration,' but the text then adds an estimated 70W for CPU/memory on NVIDIA platforms without measuring it. Please clarify whether the table is GPU-only energy and present the whole-system estimate as a separate, clearly labeled calculation.
- [Figure 3] Figure 3 is hard to read: subfigure captions are small, some axis labels are missing, and the FP32/FP16 annotation is ambiguous because the color groupings are not explained in the caption. Please enlarge and add a legend.
- [§4.3] The sentence 'our measurement shows the contract results' contains a typo; it should read 'contrary results.'
- [References] Reference [4] (MLX) lacks a complete citation with venue, year, and version; please provide a full archival reference or a stable URL with an access date.
- [§4.3, Figure 4] Figure 4 shows RSS and timing over iterations, but the text says RSS 'maintained at a lower level than the model size.' Please clarify whether this is a measured value or a theoretical estimate, and explain why RSS can remain below the model's theoretical memory footprint.
Circularity Check
No significant circularity: an empirical measurement study with independent kernel-level and end-to-end observations.
full rationale
The paper contains no mathematical derivation chain, fitted parameters, or self-citation used as load-bearing evidence; its claims are empirical comparisons. The end-to-end training times are measured independently from the BLAS kernel throughputs and system metrics (page faults, kernel launch latency, energy). The conclusion in Section 4.4 that BLAS performance is a primary factor and system issues are secondary is a post hoc causal attribution, not a reduction of the end-to-end gap to the BLAS measurements by construction; the quantities are distinct and each is directly measured. No equation equates input and output, and no fitted parameter is renamed as a prediction. The paper's own data do contain an internal tension that a reviewer should weigh as a correctness risk: the end-to-end FP32 comparisons are explained with an FP16 BLAS advantage that Apple never uses in those runs, and page faults are measured only on M2 Max without an NVIDIA comparison. But a mismatch between evidence and conclusion is an overclaim, not circularity, and the cited BLAS paper [10] is external rather than self-citational. Under the stated standards, the honest finding is no significant circularity.
Assumptions & free parameters
assumptions (3)
- domain assumption The selected workloads (Whisper, GPT-2) and their datasets are representative of typical LLM training workloads.
- domain assumption Page faults and kernel launch time are causal contributors to the end-to-end performance gap, not merely correlated symptoms.
- domain assumption The measured GPU energy consumption is comparable across platforms and reflects total training energy.
Cite this review
Pith. "Pith review of Profiling Apple Silicon Performance for ML Training." pith.science (2026). https://pith.science/paper/IRALRQN7
@misc{pith2026250114925,
author = {Pith},
title = {Pith review of: Profiling Apple Silicon Performance for ML Training},
year = {2026},
howpublished = {\url{https://pith.science/paper/IRALRQN7}},
note = {Machine review of arXiv:2501.14925}
}
read the original abstract
Apple Silicon has attracted much attention for its performance and role in machine learning (ML) training. Unlike NVIDIA GPUs, which have traditionally dominated ML training, Apple Silicon has a significant difference in memory architecture. It uses Unified Memory, which integrates CPU and GPU memory instead of separate CPU memory and GPU VRAM. However, it is difficult to tell whether Unified Memory means more performance benefits. This paper investigates the performance differences by training several large language model (LLM) workloads end-to-end under different memory scenarios. The results show a significant performance gap between Apple Silicon and NVIDIA GPUs. This paper attributes this gap to system-level factors such as page faults, power consumption, and kernel launch time. In addition, the performance difference of basic linear algebra subprograms (BLAS) on the NVIDIA GPUs and Apple Silicon chips is analyzed to further explain the observed gap.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
https://pytorch .org/get- started/previous-versions/, 2024
Previous PyTorch Versions — pytorch.org. https://pytorch .org/get- started/previous-versions/, 2024. [Accessed 19-12-2024]
work page 2024
-
[2]
Tyers, and Gregor Weber
Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M. Tyers, and Gregor Weber. Common voice: A massively-multilingual speech corpus, 2020
2020
-
[3]
Nvidia Corporation. Nvidia Marketplace. https:// marketplace.nvidia.com/en-us/enterprise/laptops-workstations/,
-
[4]
MLX: Efficient and flexible machine learning on apple silicon, 2023
Awni Hannun, Jagrit Digani, Angelos Katharopoulos, and Ronan Col- lobert. MLX: Efficient and flexible machine learning on apple silicon, 2023
work page 2023
-
[5]
Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adapta- tion of large language models, 2021
work page 2021
-
[6]
Apple Inc. Apple unleashes m1. https://www .apple.com/newsroom/ 2020/11/apple-unleashes-m1/, 2020. [Accessed 14-12-2024]
work page 2020
-
[7]
Apple Inc. Apple introduces m2 ultra. https://www .apple.com/ newsroom/2023/06/apple-introduces-m2-ultra/, 2023. [Accessed 14- 12-2024]
work page 2023
-
[8]
Apple unveils m2 pro and m2 max: next-generation chips for next-level workflows
Apple Inc. Apple unveils m2 pro and m2 max: next-generation chips for next-level workflows. https://www .apple.com/newsroom/2023/ 01/apple-unveils-m2-pro-and-m2-max-next-generation-chips-for- next-level-workflows/, 2023. [Accessed 14-12-2024]
work page 2023
Show all 20 references
-
[9]
Accelerated pytorch training on mac
Apple Inc. Accelerated pytorch training on mac. https:// developer.apple.com/metal/pytorch/, 2024. [Accessed 14-12-2024]
2024
-
[10]
Enabling massive deep neural networks with the graphblas
Jeremy Kepner, Manoj Kumar, José Moreira, Pratap Pattnaik, Mauricio Serrano, and Henry Tufo. Enabling massive deep neural networks with the graphblas. In 2017 IEEE High Performance Extreme Computing Conference (HPEC), pages 1–10. IEEE, 2017
2017
-
[11]
Adam Lewis, Soumik Ghosh, and N.-F. Tzeng. Run-time energy con- sumption estimation based on workload in server systems. In Proceed- ings of the 2008 Conference on Power A ware Computing and Systems , HotPower’08, page 4, USA, 2008. USENIX Association
2008
-
[12]
Maas, Raymond E
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, An- drew Y. Ng, and Christopher Potts. Learning word vectors for senti- ment analysis. In Dekang Lin, Yuji Matsumoto, and Rada Mihalcea, editors, Proceedings of the 49th Annual Meeting of the Association for Computatio...
2011
-
[13]
Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330, 1993
1993
-
[14]
Mixed precision training, 2018
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training, 2018
2018
-
[15]
Robust speech recognition via large- scale weak supervision
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large- scale weak supervision. arxiv 2022. arXiv preprint arXiv:2212.04356, 10, 2022
2022 arXiv
-
[16]
Language models are unsupervised multitask learners
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019
2019
-
[17]
{Zero-offload}: Democratizing {billion-scale} model training
Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. {Zero-offload}: Democratizing {billion-scale} model training. In 2021 USENIX Annual Technical Conference (USENIX ATC 21), pages 551–564, 2021
2021
-
[18]
Attention is all you need
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017
2017
-
[19]
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Syl- vain Gugger, Ma...
2020
-
[2024]
[Accessed 14-12-2024]
2024
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.