{"id":"fef39d03-8fe6-4ee9-ac7a-6d6e310868cd","arxiv_id":"2501.14925","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Apple Silicon trains LLMs 3 to 4 times slower than similarly priced NVIDIA GPUs when memory fits, but can beat NVIDIA when VRAM is exceeded and ZeRO-Offload is required.","lead":"This paper measures how Apple Silicon and NVIDIA systems compare when training large language models, finding Apple's chips are slower when all data fits in memory but can win when NVIDIA must offload data. It points to page faults, kernel launch time, and BLAS kernels as the main causes of the gap.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own BLAS data undercut the central claim: FP32 matmul throughput on Apple is 'largely comparable' to CUDA (§4.4), while the end-to-end gap is reported in FP32 (§4.1); the primary-factor explanation rests on an FP16 advantage that Apple never uses in those runs.","rationale":"The paper's observational findings—Apple loses in-memory training but wins under ZeRO-offload—are plausible, and the MLX-versus-MPS comparison is a useful internal control. The weakness is the inference from BLAS microbenchmarks to end-to-end causation. The reader flagged correlation-versus-causation; I find a sharper internal inconsistency. Section 4.1 fixes precision to FP32 for the end-to-end runs and adds AMP only for CUDA. Section 4.4 concludes BLAS is primary, but its own Figure 5 shows FP32 matmul throughput on Apple is 'largely comparable' to CUDA; the dramatic gap is in FP16. A claim that an FP16 deficit explains an FP32 gap requires an extra argument not supplied. This is the most load-bearing concern because it targets the paper's headline: 'we believe we have identified the root causes.' If it lands, the paper must retreat to a narrower claim about FP16/mixed-precision readiness or supply a kernel-level decomposition. A controlled profile experiment can settle the question, so conditional acceptance remains the right posture, but the accepted version should be required to include that decomposition.","tokens_in":9576,"tokens_out":7588,"duration_ms":68286,"concrete_test":"Instrument the Scenario-1 Whisper-medium FP32 run (batch size 4) on M2 Max and RTX 4090 with per-kernel tracing (Metal System Trace and nsys/ncu). For each training iteration, sum the GPU time attributed to BLAS-like kernels (gemm, matmul) versus all other kernels. If BLAS kernels consume a similar fraction and similar absolute time per iteration on both platforms while total iteration time differs by the reported 2-3x, the §4.4 primary-factor claim is contradicted; if BLAS time is dominant and disproportionate on Apple, the claim survives.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"In §4.1 the end-to-end comparisons are run in FP32 on all platforms, with mixed precision only on CUDA ('the torch AMP lacks support for Apple silicon SoCs'), so Apple is never shown in an FP16 end-to-end configuration. In §4.4, Figure 5 reports that in FP32, MLX and MPS are 'slightly weaker but largely comparable' to CUDA for batched matrix-matrix product, while the large BLAS gap is in FP16 (Tensor Cores). Since the end-to-end gap being explained is an FP32 gap, an FP16 BLAS deficit cannot be the 'primary factor' for it. The conclusion 'we believe we have identified the root causes' would need either an end-to-end experiment that actually uses FP16 on Apple, or a kernel-level decomposition showing that FP32 BLAS kernels dominate iteration time and that their modest throughput gap quantitatively accounts for the reported 2-3x end-to-end gap. Neither is present. The page-fault and kernel-launch measurements are correlational as well—page faults are measured only on M2 Max and never on NVIDIA—so they cannot carry the causal load either.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper empirically compares LLM training performance on Apple Silicon (M2 Pro, M2 Max, M2 Ultra) against several NVIDIA GPUs (RTX 4000, 2080Ti, A6000, RTX 4090) across three memory scenarios: sufficient memory on all devices, insufficient NVIDIA VRAM with ZeRO-Offload, and near-capacity Apple unified memory. It reports end-to-end forward/backward times, memory usage, page faults, energy consumption, GPU kernel launch times, and BLAS microbenchmarks for matrix-matrix, matrix-vector, and vector-Jacobian products in FP32 and FP16. The paper concludes that the end-to-end gap is primarily explained by BLAS kernel throughput differences, especially weak FP16 support on Apple Silicon, with secondary contributions from page faults, kernel launch overhead, and memory management behavior.","tokens_in":9797,"tokens_out":3170,"duration_ms":43966,"significance":"If the central attribution were established, the paper would be useful to ML practitioners choosing between Apple Silicon and NVIDIA hardware, and to Apple/software vendors prioritizing FP16 support and kernel optimizations. The study has concrete strengths: it includes side-by-side end-to-end training measurements on multiple platforms, covers a realistic range of memory-constrained scenarios, and reports the striking result that Apple devices can outperform NVIDIA GPUs when ZeRO-Offload is forced. The energy-per-iteration comparison is also a useful datapoint. However, the core causal claim that BLAS and system-level factors 'explain' the FP32 end-to-end gap is not supported by the data as presented, because the largest BLAS gap is in FP16 while the end-to-end runs are FP32. The system-level measurements are correlational and incomplete, and the paper lacks basic reproducibility details such as software versions and measurement repeats. The paper is likely to be a useful empirical contribution after substantial revision that either strengthens the causal analysis or reframes the claims as observations and hypotheses rather than identified root causes.","major_comments":[{"comment":"The central claim that BLAS performance is the 'primary factor' behind the end-to-end gap is contradicted by the paper's own precision settings. Section 4.1 states that all end-to-end runs use FP32, with AMP.FP16 and AMP.BF16 only on CUDA. Section 4.4 and Figure 5 report that for FP32 batched matrix-matrix product, MLX and MPS are 'slightly weaker but largely comparable' to CUDA, while the large BLAS gap appears in FP16 with Tensor Cores. An FP16 BLAS deficit therefore cannot explain an FP32 end-to-end training gap. To support the root-cause conclusion, the authors need either an end-to-end FP16 experiment on Apple Silicon or a kernel-level decomposition showing that FP32 BLAS kernels dominate iteration time and that their modest throughput gap quantitatively accounts for the reported 2-3x end-to-end gap. Neither is present.","section":"§4.4 and §4.1"},{"comment":"The system-level attributions in Section 4.3 are correlational and do not bear the causal load placed on them. Page faults are measured only on the M2 Max and never on an NVIDIA device, so there is no comparative baseline showing that page-fault counts differ in a way that explains the timing gap. The kernel launch time microbenchmark (Table 6) shows Apple kernels launch more slowly, but the paper does not report how many kernel launches occur per training iteration or estimate the total launch overhead relative to the measured iteration times. The gradual RSS increase on Apple devices is presented as a 'major reason' for underperformance, but it could be an allocator behavior or memory-prefetch policy rather than a bottleneck. The authors should either provide a quantitative model that converts observed page-fault counts and launch times into predicted training-time overhead, or explicitly weaken the language from 'identified the root causes' to 'observed system-level correlates.'","section":"§4.3"},{"comment":"The experimental methodology is under-specified for a measurement paper, and this directly affects confidence in the headline comparisons. Table 4 and Figure 3 report single timings without error bars, number of repetitions, or variance. The paper does not state the software versions (PyTorch, MLX, macOS, CUDA, driver versions) on any testbed, nor the exact CPU/RAM configuration of the NVIDIA servers. The measurement procedures for page faults and energy consumption are not described (e.g., which tool was used, sampling rate, whether whole-system or only GPU energy). Without these details, the reported quantitative gaps cannot be independently checked or compared across configurations. I request that the authors add reproducibility information and repeated-measurement statistics for all timing and energy results.","section":"§3 and Table 4 / Figure 3"}],"minor_comments":[{"comment":"The energy comparison in Table 5 reports 'GPU energy per iteration,' but the text then adds an estimated 70W for CPU/memory on NVIDIA platforms without measuring it. Please clarify whether the table is GPU-only energy and present the whole-system estimate as a separate, clearly labeled calculation.","section":"§4.2, Table 5"},{"comment":"Figure 3 is hard to read: subfigure captions are small, some axis labels are missing, and the FP32/FP16 annotation is ambiguous because the color groupings are not explained in the caption. Please enlarge and add a legend.","section":"Figure 3"},{"comment":"The sentence 'our measurement shows the contract results' contains a typo; it should read 'contrary results.'","section":"§4.3"},{"comment":"Reference [4] (MLX) lacks a complete citation with venue, year, and version; please provide a full archival reference or a stable URL with an access date.","section":"References"},{"comment":"Figure 4 shows RSS and timing over iterations, but the text says RSS 'maintained at a lower level than the model size.' Please clarify whether this is a measured value or a theoretical estimate, and explain why RSS can remain below the model's theoretical memory footprint.","section":"§4.3, Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is positioned as a broad profiling study, but the evidence provided supports only a subset of the conclusions. The most important issue is the precision mismatch between the FP32 end-to-end experiments and the FP16 BLAS results used for the primary explanation. This is fixable in principle by adding an FP16 end-to-end experiment on Apple Silicon or by substantially reframing the conclusion as a hypothesis about FP16 acceleration. I would also suggest the authors check whether the journal's scope and standards accommodate a purely empirical comparison with this level of experimental detail; a more rigorous version with reproducibility artifacts would be a stronger fit."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is a useful empirical dataset: it benchmarks M2-series Macs against several NVIDIA GPUs for LLM training under three memory scenarios, and the scenario-2 result (Apple beats a 4090 when the GPU has to ZeRO-offload) is practically interesting and probably robust. Second, the root-cause claim does not survive contact with the paper's own numbers. Section 4.1 runs all end-to-end training in FP32; only CUDA gets AMP.FP16/BF16. Section 4.4 shows that in FP32 the Apple BLAS kernels are 'largely comparable' to CUDA, while the dramatic gap is in FP16, where NVIDIA uses Tensor Cores. So an FP16 BLAS deficit cannot be the primary factor for an FP32 end-to-end gap. That is not a subtle issue; it's the central explanatory paragraph (end of §4.4) resting on a data-type mismatch.\n\nWhat the paper does well: the three-scenario framing is sensible, the page-fault growth pattern on Apple is a real observation worth reporting, the kernel-launch latency table is clean, and the MLX-vs-MPS comparison inside the Apple ecosystem is useful. The energy numbers, while single measurements, align with common experience. The paper is honest about the limitations of torch AMP on Apple.\n\nThe soft spots beyond the FP16/FP32 issue: no error bars or repeated measurements in the key figures, no software versions (macOS, PyTorch, MPS, MLX) given, and no artifacts. Page faults are measured only on M2 Max, never on NVIDIA, so the comparison is incomplete. Kernel launch overhead is tens of microseconds per launch; even with hundreds of launches per iteration that is milliseconds, not the seconds-scale gap, so calling it a 'secondary factor' is generous. The paper never provides a quantitative decomposition showing that the measured BLAS throughput differences actually account for the observed end-to-end time differences. Until that is done, the attribution to 'root causes' is a hypothesis.\n\nWho should read this: practitioners deciding between a Mac and an NVIDIA workstation for small-scale LLM training, and people working on Apple's ML software stack. It deserves a serious referee round: the data are new, the scenarios are well chosen, and the main flaw is fixable by re-scoping the conclusions and adding artifacts/repeated runs. But as submitted, I would not accept the causal narrative.","headline":"A genuinely useful Apple-vs-NVIDIA benchmark dataset with an overreach: the paper's own FP32 end-to-end runs can't be explained by its headline FP16 BLAS gap.","tokens_in":10320,"tokens_out":2370,"would_cite":true,"duration_ms":19975,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper explains Apple Silicon's LLM training gap as a BLAS kernel problem, dominated by weak FP16 acceleration and compounded by page faults and slow kernel launches.","keywords":["Apple Silicon","LLM training","unified memory","BLAS kernels","FP16 acceleration","page faults","kernel launch latency","ZeRO-Offload"],"falsifier":"Replace Apple Silicon's BLAS library with one achieving CUDA-class FP16 throughput, or run the same training on an Apple chip with dedicated FP16 tensor hardware, and check whether the end-to-end LLM training gap closes proportionally. If the timing gap stays large while the BLAS gap closes, the paper's root-cause attribution fails.","tokens_in":9382,"feed_emoji":"🍎","tokens_out":9284,"duration_ms":90791,"temperature":0.7,"pith_summary":"This paper sets out to explain why Apple Silicon Macs train large language models much more slowly than comparably priced NVIDIA GPUs, and to identify the layer of the system responsible. End-to-end pretraining and fine-tuning of Whisper and GPT-2 variants across three memory scenarios shows that M2-series chips consistently lag NVIDIA devices when model and data fit in memory, but beat them when GPU VRAM is insufficient and CUDA training must fall back on ZeRO-Offload. The paper attributes the gap to two causes: secondary system effects, including gradually rising memory usage with page faults and kernel launch latencies tens of times higher than CUDA, and a primary compute effect, namely that the BLAS kernels (the matrix operations at the heart of neural-network training) on Apple Silicon gain far less speedup from FP16 than NVIDIA Tensor Cores do. If this attribution is right, the bottleneck is not memory capacity or thermal behavior but the FP16 compute path, which points to a specific fix: better FP16 hardware and software support rather than more bandwidth.","feed_headline":"Matrix-math speed, not memory, explains Apple Silicon's LLM training gap","feed_subtitle":"Apple's weak FP16 matrix kernels and system overheads, not unified-memory capacity, set the speed limit.","key_machinery":"The load-bearing object is the BLAS kernel, the basic linear algebra subroutine that implements the matrix-matrix, matrix-vector, and vector-Jacobian operations dominating transformer forward and backward passes. The paper treats BLAS throughput as the main determinant of end-to-end training time, then benchmarks it across FP16 and FP32 on Apple Silicon through both the MPS backend and the MLX framework, against CUDA on NVIDIA GPUs. The identity that carries the argument is the FP16 acceleration ratio on each platform: NVIDIA's Tensor Cores multiply FP16 throughput by 5x-6x on the backpropagation kernel, while Apple's FP16 path delivers near-zero speedup in MPS and only 20-30% in MLX on GEMMs, so the same model trains much faster on CUDA.","core_discovery":"The paper claims that the end-to-end LLM training performance gap between Apple Silicon and NVIDIA GPUs can be explained by BLAS kernel throughput differences (the primary factor) together with system-level issues such as page faults and kernel launch overhead (secondary factors). In the three BLAS operation families that dominate transformer training — matrix-matrix product, matrix-vector product, and vector-Jacobian product — NVIDIA GPUs show strong FP16 acceleration (about 5x-6x for vector-Jacobian product), while Apple's MPS backend gets almost none and the MLX framework gets only about 20-30% on matrix-matrix product. Apple Silicon also shows memory consumption that grows gradually during training, page faults that rise with it, and kernel launches that are tens of times slower than CUDA's. The paper concludes that these observed factors, not memory capacity or thermal limits, are the root causes of the performance gap, and that Apple Silicon's advantage appears only in the scenario where the CUDA device must offload via ZeRO-Offload.","pith_inferences":["The paper's own data would allow a further test: run the same end-to-end training on Apple Silicon with an alternative BLAS implementation whose FP16 throughput matches CUDA, and see whether the end-to-end gap closes proportionally; if it does not, the attribution is wrong.","The measurements suggest an optimization roadmap the paper only sketches: porting MLX's kernel strategies into the MPS backend, or adding dedicated FP16 hardware to future Apple chips, would be the highest-leverage changes for LLM training.","The observed page-fault and RSS growth pattern likely applies beyond training, to memory-hungry inference with very long contexts; that is a testable extension the paper did not run.","For benchmark designers, the paper implies that FP16 throughput, not peak single-precision TFLOPS, is the metric that predicts real LLM training speed on Apple Silicon."],"forward_implications":["If BLAS FP16 throughput is the primary bottleneck, Apple Silicon LLM training will keep trailing NVIDIA until the FP16 compute path is improved; FP32 training narrows the gap but does not close it.","If page faults and rising RSS are secondary contributors, Apple Silicon is especially fragile for workloads that run near the unified-memory limit, as the Whisper-large near-capacity run showed large, unstable per-pass times.","If the kernel launch overhead is significant, training workloads with many small operations will be disproportionately slower on Apple Silicon than workloads dominated by large GEMMs.","If GPU VRAM is too small for a model, Apple Silicon becomes the faster choice: the ZeRO-Offload overhead can make an RTX 4090 slower than an M2 Ultra, so the large unified-memory pool is a real advantage.","If the measured energy-per-iteration figures hold, Apple Silicon offers a meaningful energy-efficiency advantage over some NVIDIA GPUs, which matters for sustained and affordable training."],"supporting_citations":[{"why":"establishes the PyTorch MPS backend as the software baseline for Apple Silicon training","marker":"[1]"},{"why":"supplies the MLX framework used for BLAS and end-to-end comparisons","marker":"[4]"},{"why":"defines the LoRA fine-tuning method used in the experiments","marker":"[5]"},{"why":"supplies the premise that BLAS performance largely determines end-to-end training time","marker":"[10]"},{"why":"defines mixed precision training and the FP16/AMP settings used in benchmarks","marker":"[14]"},{"why":"provides the Whisper speech model used as one end-to-end workload","marker":"[15]"},{"why":"provides the GPT-2 text model used for pretraining and fine-tuning","marker":"[16]"},{"why":"defines ZeRO-Offload, the offloading method that makes CUDA devices slower in the constrained-VRAM scenario","marker":"[17]"}],"fun_headline_variants":["Weak FP16 kernels, not unified memory, explain Apple LLM gap","Apple's LLM gap due to math kernels, not memory capacity","Apple Silicon LLM training lag hinges on math kernels, not RAM","FP16 math speed, not memory, limits Apple Silicon LLM training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes that the measured page faults, kernel launch latencies, and BLAS throughput differences are the causes of the end-to-end timing gap and not merely correlated symptoms of a deeper bottleneck in Apple's driver, scheduler, or memory hierarchy.","fun_headline_variants_meta":{"raw":{"variants":["Weak FP16 kernels, not unified memory, explain Apple LLM gap","Apple's LLM gap due to math kernels, not memory capacity","Apple Silicon LLM training lag hinges on math kernels, not RAM","FP16 math speed, not memory, limits Apple Silicon LLM training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000185,"raw_usage":{"total_tokens":1300,"prompt_tokens":900,"completion_tokens":400,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":516,"completion_tokens_details":{"reasoning_tokens":323}},"tokens_in":516,"tokens_out":400,"duration_ms":3762,"temperature":1.0,"reasoning_tokens":323,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:45:48.017464+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace Apple Silicon's BLAS library with one achieving CUDA-class FP16 throughput, or run the same training on an Apple chip with dedicated FP16 tensor hardware, and check whether the end-to-end LLM training gap closes proportionally. If the timing gap stays large while the BLAS gap closes, the paper's root-cause attribution fails.","supporting_citations":[{"cited_title":"https://pytorch .org/get- started/previous-versions/, 2024","cited_arxiv_id":null,"evidence_quote":"establishes the PyTorch MPS backend as the software baseline for Apple Silicon training"},{"cited_title":"MLX: Efficient and flexible machine learning on apple silicon, 2023","cited_arxiv_id":null,"evidence_quote":"supplies the MLX framework used for BLAS and end-to-end comparisons"},{"cited_title":"Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen","cited_arxiv_id":null,"evidence_quote":"defines the LoRA fine-tuning method used in the experiments"},{"cited_title":"Enabling massive deep neural networks with the graphblas","cited_arxiv_id":null,"evidence_quote":"supplies the premise that BLAS performance largely determines end-to-end training time"}],"review_version":1}