{"id":"a62d1dc7-62fb-4a44-9cc9-0be3aad92431","arxiv_id":"2505.06461","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"On an iPhone 15 Pro, CPU-only llama.cpp inference achieved 17 tokens/s with a 1B model, beating the 12.8 tokens/s GPU path, but only for small models and without a confirmed cause.","lead":"Benchmarking small language models on an iPhone 15 Pro, this paper reports that a CPU-only llama.cpp setup can generate text faster than GPU acceleration for a 1-billion-parameter model (17 vs 12.8 tokens/s). The result challenges GPU-first assumptions for mobile AI, but the paper provides no error bars, no code, and no verified explanation for the CPU advantage.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The CPU-vs-GPU claim hinges on an unverified 'GPU-Enabled' baseline; at the pinned llama.cpp commit the default n_gpu_layers is 0, so the 12.8 tk/s GPU figure may not have used the GPU at all.","rationale":"The reader's weakest assumption correctly identified that the 'GPU-Enabled (Default)' baseline is unverified, and the paper provides no GPU utilization, layer-offload count, or Metal kernel timing. This stress test agrees with that concern but sharpens it with a specific mechanism: at the pinned llama.cpp commit 8648c52, the CLI default for n_gpu_layers is 0, so a run labeled 'GPU-Enabled (Default)' could silently execute entirely on the CPU. That possibility is not merely speculative, because the reported GPU number (12.8 tk/s) is very close to the 1-thread CPU F16 number (12.0 tk/s), and because Section 8.2 admits the authors could not successfully profile the GPU path with the Metal Debugger. The paper's central claim is an empirical existence proof, so its soundness depends on configuration provenance rather than on a derivation or fitting procedure; there is no independent mathematical support that could rescue the comparison if the GPU baseline was not actually GPU. At the same time, the CPU-only result (17.0 tk/s with two threads) is internally consistent with the paper's other CPU datapoints and with prior work such as Li et al. [18] showing that mobile inference engines often gain little from GPU acceleration. Therefore the right disposition is not rejection but a firm condition: the authors must demonstrate, with backend logs or equivalent evidence, that the GPU baseline genuinely executed on the Metal backend at the reported 12.8 tk/s. Since the reader's CONDITIONAL verdict already makes such verification a condition of acceptance, this stress test does not change the verdict; it only makes the required check more specific and more urgent.","tokens_in":33465,"tokens_out":4506,"duration_ms":48875,"concrete_test":"Re-run the exact benchmark at commit 8648c52 with explicit backend placement instrumentation. Build with Metal enabled, invoke the same workload with n_gpu_layers=0 and n_gpu_layers=99 (or the equivalent iOS API setting), and record ggml_backend_sched node assignments plus Metal GPU time using MTLCaptureManager or Instruments. If the 'GPU-enabled' run's logs show that most GGML_OP_MUL_MAT nodes executed on CPU, or if n_gpu_layers=99 does not produce a materially different throughput from the reported 12.8 tk/s, the headline CPU-vs-GPU comparison is invalid. If the logs show genuine Metal execution of the GEMMs and n_gpu_layers=99 still yields roughly 12.8 tk/s while n_gpu_layers=0 yields 12.0 tk/s, the existence claim stands as a configuration-specific result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is an existence proof: CPU-only execution (two threads, F16) at 17.0 tk/s beats a GPU-accelerated baseline at 12.8 tk/s for Llama-3.2-1B (Figure 4b, Section 5.2). For that comparison to mean what the abstract says, the 'GPU-Enabled (Default)' configuration in Section 4.3 must actually offload substantial computation to the iPhone GPU. The paper never reports the value of n_gpu_layers / -ngl, never logs node-to-backend assignments, never reports GPU utilization, and never provides Metal kernel timings. This is not a cosmetic omission: llama.cpp's backend selection is config-driven, and at commit 8648c52 (the pinned commit) the common CLI default for --n-gpu-layers is 0, i.e., CPU-only, unless explicitly overridden. If the authors relied on that default, the 'GPU' baseline is itself a CPU run, and the reported 12.8 tk/s is not a GPU measurement at all. The proximity of 12.8 tk/s to the 1-thread CPU F16 value of 12.0 tk/s (Figure 4b) makes this concern concrete rather than hypothetical. The paper's own Section 8.2 states that Metal Debugger profiling was unstable and crashed repeatedly, and Section 5.4's Remark concedes that the CPU advantage 'remains without a detailed low-level explanation.' Thus the most load-bearing premise, that the GPU path was genuinely exercised, is both unverified and, given the default configuration risk, potentially false. If the GPU baseline was never executed, the headline result is an artifact of configuration provenance, not a hardware trade-off.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an empirical benchmark of on-device LLM inference on an iPhone 15 Pro using llama.cpp at a pinned commit, comparing a CPU-only mode (1-6 threads) against a \"GPU-Enabled (Default)\" mode across six models and three precisions, with throughput in tokens per second as the metric. The headline claim is that for Llama-3.2-1B at F16, a two-thread CPU configuration achieves 17.0 tk/s versus 12.8 tk/s for the GPU-enabled configuration. The paper also profiles operation-level time shares, reporting that matrix multiplication dominates prefill (87.6%) and decode (76.2%) for Llama-3.2-1B-F16, and experiments with graph-level and tensor-level parallelism to improve CPU throughput. The manuscript candidly acknowledges that the CPU advantage lacks a detailed low-level explanation and that GPU profiling was unstable.","tokens_in":33694,"tokens_out":5573,"duration_ms":50317,"significance":"If the CPU-vs-GPU result is substantiated, the paper would provide a useful existence proof that a carefully configured CPU path can outperform a default GPU path for small on-device LLMs, challenging a common GPU-first assumption. The GEMM dominance measurement for Llama-3.2-1B-F16 is a valuable data point for engine optimization, and the thread-oversubscription finding is practically relevant. I also credit the authors for pinning the exact commit, describing thermal mitigation, running five repetitions, and being explicit about profiling limitations. However, the significance is substantially muted because the central comparison rests on an unverified GPU baseline, the throughput figures are reported without any variance, and the paper's own remarks concede that the key causal explanations were not established.","major_comments":[{"comment":"The 'GPU-Enabled (Default)' configuration is not actually demonstrated to use the GPU. The paper never reports the value of n_gpu_layers (-ngl), never shows ggml backend-assignment logs, and never provides GPU utilization or Metal kernel timings. At the pinned commit 8648c52, the default value of n_gpu_layers is 0, meaning a run with the default CLI would execute on the CPU only. The reported GPU value of 12.8 tk/s for Llama-3.2-1B-F16 is close to the 1-thread CPU value of 12.0 tk/s, which makes this concern concrete rather than hypothetical. Since the headline 'CPU beats GPU' claim depends entirely on the 12.8 tk/s figure actually being produced by the GPU backend, please provide the exact command-line invocation (including any -ngl flag), the 'llm_load_tensors: offloading' output, and evidence that Metal kernels were executed. If the default configuration was used without an explicit -ngl override, the claim should be withdrawn or the experiment rerun with a verified GPU offload setting.","section":"§4.3, §5.2, Fig. 4(b)"},{"comment":"The paper reports only the average of five runs, with no error bars, standard deviation, or per-run values. Throughput measurements on a phone are strongly affected by thermal state and background activity; the ice-cooling procedure is described but no temperature is logged or controlled for, so the statement 'to ensure statistical reliability' is not supported by the data shown. In addition, the benchmark protocol is under-specified: the fixed prompt is 'The meaning of life is ' (7 tokens), but the number of generated tokens per run is not stated, and the mention of 128-token context windows in §5.1 is not sufficient to reproduce the measurement. Please report per-run values or variance, state the generation length, and clarify whether the reported tk/s is decode-only or includes prefill.","section":"§4.4, Fig. 4"},{"comment":"The profiling analysis claims that GGML_OP_MUL_MAT accounts for 87.6% of prefill and 76.2% of decode time for Llama-3.2-1B-F16, but the surrounding text says this analysis was performed in a CPU-only environment, while Figure 6 includes GPU bars. Given §8.2's statement that Metal Debugger profiling repeatedly crashed during data transfer, it is unclear whether any GPU-side timings were successfully captured and how they were obtained. Please specify which backend each sub-figure corresponds to and how the GPU measurements were made, or remove the GPU bars if they are not based on actual measurements.","section":"§6.1, Figs. 5-6"},{"comment":"There is a mismatch between the abstract's claim that 'GPU memory transfer overhead and CPU thread optimization play a critical role' and the paper's own admission in §5.4 that the CPU advantage 'remains without a detailed low-level explanation.' Similarly, §7.2 states that the graph-parallel version 'nearly matching default GPU-enabled execution,' which inherits the unverified GPU baseline from §4.3. Please either provide direct evidence for the asserted overhead mechanisms (e.g., Metal buffer allocation or memory-transfer measurements) or temper the abstract and conclusion to match the admitted lack of explanation.","section":"Abstract, §5.4 Remark, §7.2"}],"minor_comments":[{"comment":"In the opening paragraph, 'Section 6 discussed profiling results' should be 'Section 6 discusses profiling results.'","section":"§1"},{"comment":"The notation 'Q41' appears to refer to Q4_1 quantization; it is confusing and should be made consistent with the Q4/Q8 notation used elsewhere.","section":"Footnote 1"},{"comment":"The full compute-graph figure is extremely large and will likely be unreadable in a two-column page; consider replacing it with a trimmed version showing one decoder block.","section":"Fig. 1"},{"comment":"The description 'GPU-Enabled (Default): The model utilized the iPhone's integrated GPU' should be rephrased to 'was intended to utilize' to reflect the verification issue raised in Major Comment 1.","section":"§4.3"},{"comment":"The limitation that the study is device-specific is only stated in §8.3; consider adding a sentence to the abstract or conclusion noting that the findings are from a single device.","section":"§8.3"},{"comment":"The text says 'The full graph can be accessed on GitHub' but no URL is provided; please add the link or remove the pointer.","section":"§3"},{"comment":"The pseudocode uses undefined macros such as INIT and APPLY_WEIGHT; please define them or replace them with plain descriptive text.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The most serious concern, which I have also flagged to the authors as Major Comment 1, is that the 'GPU-Enabled (Default)' baseline may never have actually exercised the GPU: the paper is pinned to a llama.cpp commit whose default n_gpu_layers is 0, and the reported 12.8 tk/s is suspiciously close to the 1-thread CPU value. If the authors cannot produce logs showing that -ngl was set and that Metal kernels ran, the headline result should be treated as unsubstantiated and the paper should be reconsidered. I recommend the editor require this verification before any further consideration."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know up front. This paper claims CPU-only inference beats GPU acceleration for sub-1.5B LLMs on an iPhone 15 Pro, and it may well be true in practice. But the paper never verifies that its 'GPU-Enabled (Default)' configuration actually offloads computation to the Metal GPU, and at the pinned commit the llama.cpp default for n_gpu_layers is 0. The 12.8 tk/s GPU figure sits close to the 1-thread CPU F16 figure of 12.0, which is a red flag. Until the authors report -ngl, node-to-backend assignments, or Metal kernel timings, the headline result is not established. The stress-test note is on target.\n\nWhat's genuinely good: the measurement campaign is clearly described—device, iOS version, a fixed llama.cpp commit, six models across precisions, 1 to 6 CPU threads. The GEMM profiling showing MUL_MAT at 87.6% of prefill and 76.2% of decode is a solid, reproducible observation that matches conventional wisdom. The graph-level parallelism section is honest: they built it, it helped a bit on CPU, and Version 3 that combined CPU+GPU got worse. That they report a failed hybrid experiment without spinning it is to their credit.\n\nThe soft spots are mainly around evidence quality. Five-run averages with no error bars and no raw data make it hard to judge whether CPU vs GPU gaps are real or noise. The paper's own Remark admits the CPU advantage 'remains without a detailed low-level explanation.' That is fine, but it means the paper is a benchmark report, not a mechanistic study. And the qualitative claim is not new: the cited Li et al. already report that mobile GPU acceleration often gives little advantage. The newness here is the specific iPhone 15 Pro numbers and the graph-parallel exploration.\n\nOverall, I would send this to peer review rather than desk reject because the central issue is verifiable and fixable; a referee can ask for the GPU offload evidence and error bars. As it stands, though, the abstract overstates what the data show. If the GPU baseline was CPU, the paper's main contribution collapses to a re-run of known results. Worth engaging, but with skepticism.","headline":"The CPU-beats-GPU headline is plausible but unsupported: the 'GPU-enabled' baseline was likely never on the GPU at the pinned llama.cpp commit.","tokens_in":34318,"tokens_out":3465,"would_cite":false,"duration_ms":33362,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"On an iPhone, a CPU beats the GPU at running a 1B model","keywords":["on-device LLM inference","CPU vs GPU","llama.cpp","iPhone 15 Pro","GEMM bottleneck","thread scaling","quantization","mobile inference"],"falsifier":"Run the same LLaMA 3.2-1B-F16 model on the same iPhone 15 Pro build while recording GPU utilization through Instruments or by instrumenting Metal buffer copies; if the default GPU path spends most of its time waiting on the CPU or copies the model weights every token, or if an explicitly tuned Metal kernel pushes GPU throughput above 17 tokens per second, the claimed CPU-superiority result is an artifact of the default build rather than a hardware trade-off.","tokens_in":33158,"feed_emoji":"📱","tokens_out":3045,"duration_ms":24717,"temperature":0.7,"pith_summary":"This paper challenges the assumption that GPU acceleration is always faster than CPU execution for small language models on phones. Using llama.cpp on an iPhone 15 Pro, the authors report that running a 1-billion-parameter model on two CPU threads at F16 precision generates 17 tokens per second, while the GPU-enabled default configuration manages only 12.8. They also measure that matrix multiplication dominates both the prefill phase (87.6% of time) and the decode phase (76.2%), and they show that CPU performance peaks when thread count matches the two performance cores. The paper's broader point is that choosing a backend for on-device inference should be an empirical, workload-aware decision rather than a reflexive GPU-first one.","feed_headline":"CPU beats GPU running a 1B model on iPhone","feed_subtitle":"Llama-3.2-1B hits 17 tokens/s on two CPU threads but only 12.8 on the GPU default.","key_machinery":"The load-bearing object is the llama.cpp compute graph: the model is a directed graph of tensor operations (GEMMs, norms, attention), executed node by node through the ggml tensor library, with each node dispatched to a backend chosen at compile time (CPU via NEON, GPU via Metal). Because each decoder layer's operations run serially, the cost structure is set by the seven matrix multiplications per layer, and the paper identifies GGML_OP_MUL_MAT, the operation node for general matrix multiplication, as the dominant cost. This graph view supplies both the profiling target (which operations and which of the seven GEMMs take the time) and the optimization lever (scheduling independent nodes in parallel).","core_discovery":"The central claim is an existence proof: for a small on-device LLM, a carefully tuned CPU-only configuration can beat the phone's GPU. Specifically, LLaMA 3.2-1B in F16 on the iPhone 15 Pro with two CPU threads delivers 17.0 tokens per second versus 12.8 for the GPU-enabled default. The same pattern holds for Qwen2-0.5B, where CPU with several threads exceeds the GPU baseline, while for models of 1.5B parameters and larger the GPU generally wins. The authors attribute the CPU advantage to reduced kernel-launch and memory-transfer overhead on small workloads and to matching thread count to the chip's performance cores, while cautioning that the low-level mechanism 'remains without a detailed low-level explanation' because iOS profiling tools are limited.","pith_inferences":["If the fixed 7-token prompt is replaced with longer or batched inputs, the balance may shift: larger matrices give the GPU more work to amortize kernel launch overhead, so CPU's advantage may shrink or reverse on long-context prefill.","The authors' graph-parallel scheduling idea (running Q, K, V, and FFN GEMMs concurrently on the CPU) is a testable template for other engines and could be combined with quantization to widen the CPU margin further.","A direct comparison against Apple's own optimized runtime and against a GPU-tuned Metal implementation in a newer llama.cpp release, rather than the default build, would reveal how much of the 17-vs-12.8 gap is intrinsic hardware behavior and how much is the default Metal path being under-optimized.","The same measurement methodology could be applied to Android devices with different core topologies to see whether the 'performance-core count' rule generalizes across SoCs."],"forward_implications":["For sub-1B and 1B models on phones, deployment strategies should treat CPU-only execution as a first-class option rather than a fallback, since it can beat the default GPU path at equal precision.","Thread count should be tuned to the device's performance-core count; adding threads beyond that point stops helping and can degrade throughput.","Because GEMM dominates both prefill and decode, engineering effort aimed at faster small-batch matrix multiplication (SIMD tuning, cache-aware blocking) should transfer directly to end-to-end inference speed.","Quantization to Q4 gives a 1.5x to 2.5x speedup across most configurations, so precision choice is as important as backend choice for on-device latency.","Hybrid CPU+GPU execution is not automatically beneficial: the authors' attempt to split graph nodes across both backends dropped throughput to about 6 tokens per second, suggesting synchronization overhead can exceed the GPU's compute advantage on small batches."],"supporting_citations":[{"why":"llama.cpp is the inference engine under test; the paper's graphs, scheduler, and backend selection all come from it.","marker":"[9]"},{"why":"Prior measurement study reporting that mobile inference engines often show little benefit from GPU acceleration, which the paper's findings corroborate and refine.","marker":"[18]"},{"why":"Transformer-Lite, a GPU-focused mobile inference engine, supplies the optimization-mindset counterpoint the paper is arguing against.","marker":"[17]"},{"why":"K-quant quantization methods in llama.cpp define the Q4 and Q8 configurations whose speedups the paper measures.","marker":"[8]"},{"why":"Qwen2 is one of the six benchmarked model families, providing the 0.5B and 1.5B data points.","marker":"[31]"},{"why":"The LLaMA 3.2 family is the primary benchmarked model, including the 1B model that anchors the headline result.","marker":"[10]"}],"fun_headline_variants":["CPU beats GPU for 1B LLM on iPhone 15 Pro","Small LLM: CPU outruns GPU on mobile devices","On-device LLM: CPU wins over GPU for 1B model","CPU faster than GPU for iPhone LLM inference","1B LLM on iPhone: CPU beats GPU at 17 tok/s"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes the default 'GPU-enabled' llama.cpp configuration actually offloads the heavy computation to the iPhone GPU and is a reasonably optimized baseline; the paper does not report GPU utilization, layers offloaded, or Metal kernel timing, and it concedes the CPU advantage lacks a detailed low-level explanation.","fun_headline_variants_meta":{"raw":{"variants":["CPU beats GPU for 1B LLM on iPhone 15 Pro","Small LLM: CPU outruns GPU on mobile devices","On-device LLM: CPU wins over GPU for 1B model","CPU faster than GPU for iPhone LLM inference","1B LLM on iPhone: CPU beats GPU at 17 tok/s"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000279,"raw_usage":{"total_tokens":1645,"prompt_tokens":921,"completion_tokens":724,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":633}},"tokens_in":537,"tokens_out":724,"duration_ms":5748,"temperature":1.0,"reasoning_tokens":633,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:41:39.492837+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same LLaMA 3.2-1B-F16 model on the same iPhone 15 Pro build while recording GPU utilization through Instruments or by instrumenting Metal buffer copies; if the default GPU path spends most of its time waiting on the CPU or copies the model weights every token, or if an explicitly tuned Metal kernel pushes GPU throughput above 17 tokens per second, the claimed CPU-superiority result is an artifact of the default build rather than a hardware trade-off.","supporting_citations":[{"cited_title":"llama.cpp","cited_arxiv_id":null,"evidence_quote":"llama.cpp is the inference engine under test; the paper's graphs, scheduler, and backend selection all come from it."},{"cited_title":"Large Language Models on Mobile Devices: Measurements, Analysis, and Insights","cited_arxiv_id":null,"evidence_quote":"Prior measurement study reporting that mobile inference engines often show little benefit from GPU acceleration, which the paper's findings corroborate and refine."},{"cited_title":"K-Quants: Optimized Quanti- zation for Llama.cpp","cited_arxiv_id":null,"evidence_quote":"K-quant quantization methods in llama.cpp define the Q4 and Q8 configurations whose speedups the paper measures."},{"cited_title":"The Llama 3 Herd of Models","cited_arxiv_id":null,"evidence_quote":"The LLaMA 3.2 family is the primary benchmarked model, including the 1B model that anchors the headline result."}],"review_version":1}