{"id":"8e5a1762-e556-434e-8f99-ffa67ee374ac","arxiv_id":"2508.11269","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A benchmarking study of LLM inference on three edge devices with a proposed MBU metric that reduces to a standard throughput normalization.","lead":"This paper presents ELIB, a benchmarking framework for large language models on edge devices, along with a proposed metric called MBU for memory bandwidth use. The metric is a simple ratio of achieved to peak bandwidth, and the study's measurements are undermined by broken GPU inference results on two of the three test platforms.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GPU OpenCL runs on NanoPI/Xiaomi have ~10x worse perplexity than CPU, yet are used as valid LLM-inference benchmarks; this invalidates the empirical support for MBU.","rationale":"The paper's central contribution is MBU, a normalized memory-bandwidth utilization metric, and the empirical claim that it enables comparison across edge platforms and correlates with throughput and FLOPS. This claim rests on the validity of the benchmark measurements. The most load-bearing assumption is that all configurations reported in Table 6 perform actual LLM inference. Table 6 and Section 5.2 show that OpenCL GPU inference on NanoPI and Xiaomi produces perplexity scores of ~54-68, roughly 10x worse than CPU scores of ~4-6. The paper itself acknowledges this as 'significantly poorer inference accuracy' and attributes it to suboptimal parallelization and data precision issues, but still uses these GPU rows in the analysis and conclusions. If the GPU path is producing near-garbage outputs, its FLOPS/throughput/MBU are not representative of LLM inference; they are measurements of a malfunctioning stack. Therefore, the empirical validation of MBU is compromised for two of the three platforms. This is a correctness risk, not merely a novelty dispute. A concrete correctness gate (comparing GPU and CPU logits/tokens on fixed prompts) would definitively settle whether the GPU runs are valid. If they are not, those rows should be excluded, and many of the paper's quantitative comparisons (e.g., GPU speedups, MBU trends) would change or become unsupported. I agree with the reader's weakest assumption and verdict, so no adjustment is needed.","tokens_in":20321,"tokens_out":7690,"duration_ms":81011,"concrete_test":"Re-run the NanoPI and Xiaomi OpenCL-GPU benchmarks for q4_0 with a correctness gate: decode a fixed set of 20 prompts with greedy sampling and compare the generated tokens (or logits) against the CPU OpenBLAS output for the same model. If the GPU outputs diverge on more than a small fraction (e.g., top-1 agreement < 95%), classify the configuration as 'broken inference' and exclude those rows from Table 6 and from all throughput/MBU/FLOPS analysis in Section 5.2. Then recompute the claimed GPU-vs-CPU speedups and MBU correlations; if the conclusions in §5.2 change materially, the paper's central claims are unsupported for those platforms.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.2 (Accuracy) and Table 6 show that OpenCL GPU inference on NanoPI and Xiaomi yields perplexity scores ~54-68 versus 4-6 on CPU, a 10x degradation. The paper itself states 'the perplexity score was almost 10 times higher compared to the CPU inference, indicating significantly poorer inference accuracy' and attributes it to 'suboptimal parallelization design and data precision issues' (Section 5.2, RQ3). Nevertheless, all GPU rows from these two platforms are included in Table 6 and used in Section 5.2 to draw conclusions about throughput, FLOPS, MBU, and the trade-off between quantization and accuracy. If the GPU path is not performing correct LLM inference (as the perplexity figures strongly suggest), then the FLOPS, throughput, and MBU numbers for those configurations do not measure LLM inference performance; they measure a broken or fallback kernel path. The central claim that MBU is a useful normalized metric for comparing LLM inference across edge platforms depends on these measurements being valid. Since the two most constrained platforms have invalid GPU results, the empirical support for MBU's correlation with throughput and FLOPS, and the heuristics derived from it (e.g., 'GPU acceleration improves throughput'), is compromised. This is not a mere accuracy trade-off; a 10x perplexity increase indicates the model output is essentially unreliable, so those configurations should be excluded or clearly separated in any benchmarking claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ELIB, a benchmarking framework for LLM inference on edge devices, and proposes a new metric, Model Bandwidth Utilization (MBU), defined as the ratio of achieved to peak memory bandwidth. ELIB is deployed on three platforms (NanoPI RK3588, Xiaomi Snapdragon 778, MacBook Air M2) using llama.cpp and five GGML quantized LLaMA-7B variants. The paper reports FLOPS, throughput, latency, perplexity, and MBU, and derives heuristics about maximizing MBU and the constraints that limit it. The central empirical contribution is the claim that MBU is a normalized, comparable metric that correlates with throughput and FLOPS across edge platforms, and that GPU acceleration improves throughput and MBU on all devices.","tokens_in":1625,"tokens_out":1549,"duration_ms":43163,"significance":"If the framework and metric were validated, ELIB would address a real gap: comparing LLM inference across heterogeneous edge devices with different acceleration stacks. The open-source benchmarking tool, the integration of multiple backends (OpenBLAS, OpenCL, Metal), and the automatic quantization flow are practical contributions, and the paper explicitly promises reproducible documentation. However, the validity of the GPU results on two of the three platforms is explicitly contradicted by the paper's own accuracy data: OpenCL GPU inference on NanoPI and Xiaomi yields perplexity scores near 54-68 versus 4-7 on CPU, indicating that the GPU path is not performing correct LLM inference. Because these invalid GPU measurements are then used to support the paper's main throughput, FLOPS, and MBU conclusions, the empirical foundation of the claimed results is compromised. In addition, the claimed correlation between MBU and throughput is partly tautological given Equations (1)-(2).","major_comments":[{"comment":"The OpenCL GPU rows for NanoPI and Xiaomi report perplexity scores of roughly 54-68, while the CPU rows report 4-7 on the same models. The paper itself states that this is 'almost 10 times higher' and attributes it to 'suboptimal parallelization design and data precision issues.' Yet these same GPU rows are included in Table 6 and used in Section 5.2 to support conclusions about GPU FLOPS, throughput, MBU, and the trade-off between quantization and accuracy. A 10x perplexity degradation indicates that the GPU path is producing near-random outputs, so FLOPS/throughput/MBU values for those configurations do not measure LLM inference performance. These rows must be excluded or clearly separated, and all GPU-related heuristics must be re-derived without them. This is load-bearing, not a minor presentation issue.","section":"Section 5.2 (Accuracy/RQ3), Table 6"},{"comment":"MBU is defined as Achieved Memory Bandwidth / Peak Memory Bandwidth, and Achieved Memory Bandwidth is defined as (Total Model Parameter Size + KV Cache Size) / TPOT. Since TPOT is the reciprocal of throughput, for a fixed model, fixed quantization, and fixed batch/sequence configuration, MBU is proportional to throughput by construction: MBU = (Model+KV size) / (Peak Bandwidth) x Throughput. Therefore the statement in RQ1 that 'a higher MBU value consistently correlates with higher FLOPS and throughput' is not an empirical finding but a mathematical consequence of the metric definition. Similarly, the heuristic that increasing batch size increases MBU partly restates the standard effect of batch size on throughput and KV-cache size. The paper should either redefine MBU so that this correlation is non-trivial, or clearly present the throughput correlation as a definitional property rather","section":"Section 4.2, Eqs. (1)-(2), and RQ1 in Section 5.2"},{"comment":"All benchmark values are reported as single numbers with no error bars, no standard deviations, and no explicit number of repeated runs for throughput/latency/FLOPS. The text mentions '100 inference iterations' only in the accuracy discussion. On mobile and IoT devices, thermal throttling, DVFS, and background processes can cause significant run-to-run variation. Without repeated measurements and dispersion statistics, comparative claims such as 'GPU throughput is 1.18X/1.41X/1.53X higher than CPU' cannot be assessed for statistical significance. This is especially important because the 4-thread vs 8-thread FLOPS result (4 threads outperforming 8 threads) is itself counterintuitive and needs robustness checks.","section":"Table 6 and Section 5.2"}],"minor_comments":[{"comment":"The manuscript contains numerous typos and inconsistencies: 'rencent', 'quantitation' (should be 'quantization'), 'TTLM' versus 'TTLM/TTLM' terminology, 'Xiaomi Redmi Node12' (Table 1) versus 'Redmi Note12 Turbo' in text, and 'Lox' in Table 2 (likely 'Low'). These should be fixed.","section":"Throughout"},{"comment":"The figure captions and axis labels are duplicated and have unusual formatting (e.g., '0.0 0', '20 .00'). The subfigures are not clearly referenced in the text; please label subpanels (a), (b), etc., and refer to them explicitly.","section":"Figures 3-6"},{"comment":"The quantization naming convention is described (qX_Y), but the table uses 'q4 0' with a space instead of 'q4_0'. This makes it easy to misread the quantization types. Use the underscore notation consistently.","section":"Section 5.1.1, Table 5"},{"comment":"The FLOPS definition says 'floating-point operations per second,' but quantized GGML inference on CPU/GPU may use integer or mixed-precision kernels. How FLOPS is computed for quantized models (e.g., whether dequantization is counted) is not specified and should be clarified.","section":"Section 4.2"},{"comment":"The paper cites community leaderboards and frameworks through footnotes rather than formal references; some references are incomplete (e.g., no page numbers for DAWNBench). Please unify the citation style.","section":"References"}],"recommendation":"reject","confidential_remarks":"As submitted, the paper's central empirical claims rest on GPU benchmark results that the authors themselves show to be inaccurate by an order of magnitude. The MBU-throughput correlation is also definitionally guaranteed. These are load-bearing issues that would require re-running the GPU experiments and substantially reframing the contributions, so I recommend rejection rather than minor or major revision. The ELIB tool and the idea of a normalized bandwidth-utilization metric may have merit, but the current manuscript does not provide valid evidence for them."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper has a real benchmark dataset and an honest discussion of a perplexity problem, but the central metric is a standard bandwidth-utilization ratio and the GPU numbers on two of three platforms should not have been used for the throughput/MBU analysis. The novelty claim doesn't survive contact with the paper's own equations.\n\nWhat's good: They built a llama.cpp-based harness, ran five quantizations on three devices, and report throughput, latency, FLOPS, perplexity. The CPU results and MacBook Metal results look plausible. They openly say the OpenCL GPU path on NanoPI and Xiaomi produces ~10x worse perplexity (54-68 vs 4-6), which is real and useful to know. If you're deploying on those devices, that warning is the most valuable thing in the paper.\n\nWhere it falls down: First, MBU = (model bytes + KV bytes) / TPOT divided by datasheet peak bandwidth. That's just achieved-over-peak memory bandwidth utilization. The paper's claim that higher MBU correlates with higher throughput is partly tautological because MBU is computed from throughput. Second, the same GPU rows with near-random outputs are still reported and analyzed for throughput, FLOPS, and MBU. A configuration that generates garbage isn't a valid LLM-inference benchmark. The paper even flags the accuracy issue but keeps the numbers in the main analysis, which weakens every heuristic drawn from them. Third, the heuristic that minimizing KV cache size maximizes MBU conflicts with Equation 2, where KV cache is in the numerator. Fourth, there are no error bars, no repo link despite the \"open-source project\" claim, and no run counts.\n\nThe MacBook data is fine, and the CPU data is fine, so the paper could be reworked into a narrower empirical study. As it stands, the central contribution is mislabeled and the evidence for it is damaged.\n\nFor peer review: I'd send it out because the benchmark study is real and the perplexity finding is worth recording, but I'd expect major revision or a reframe as a deployment report, not a new metric.","headline":"Worth a desk read for the CPU/Metal benchmark data, but the MBU 'novel metric' is a standard roofline ratio and the OpenCL GPU rows on two platforms are effectively garbage-in.","tokens_in":21137,"tokens_out":4008,"would_cite":false,"duration_ms":39257,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new metric, MBU, measures how well edge devices use memory bandwidth when running quantized LLMs, and benchmarks reveal GPU-accelerated runs can be fast but badly inaccurate.","keywords":["LLM inference","edge computing","memory bandwidth utilization","benchmarking","model quantization","GPU acceleration","perplexity","KV cache"],"falsifier":"Re-run the GPU benchmarks on the board and phone with a correctness sanity check: decode a fixed prompt and compare the output tokens (or exact logits) to the CPU output. If the GPU outputs diverge or perplexity stays above ~20 on WikiText-2, the reported GPU FLOPS, throughput, and MBU values do not represent valid LLM inference.","tokens_in":20218,"feed_emoji":"⚡","tokens_out":8358,"duration_ms":79290,"temperature":0.7,"pith_summary":"The paper introduces ELIB, a benchmarking tool for large language model inference on edge devices, and a new metric, MBU (Model Bandwidth Utilization), defined as the ratio of achieved memory bandwidth to the hardware's peak memory bandwidth. The central argument is that LLM token generation on edge hardware is memory-bound, so MBU is a normalized, platform-independent measure of how efficiently a device uses its available memory bandwidth when running a given quantized model; higher MBU should track higher FLOPS and throughput. Using three edge platforms (an IoT board, an Android phone, and a laptop) and five quantized LLaMA models, the paper reports MBU values and derives heuristics for optimizing MBU through batch size, sequence length, and KV cache management, subject to memory capacity and latency constraints. The paper's own accuracy data show that GPU acceleration on the board and phone produced perplexity scores around 54-68, roughly ten times worse than CPU scores of 4-6, which means those GPU runs were not performing correct inference despite reporting high FLOPS, throughput, and MBU.","feed_headline":"Fast but wrong: edge GPU LLM runs fail accuracy on 2 of 3 devices","feed_subtitle":"A new MBU metric benchmarks memory-bandwidth efficiency, and its first run shows GPU paths silently break inference on two of three devices.","key_machinery":"The MBU metric (Model Bandwidth Utilization): MBU = achieved memory bandwidth / peak memory bandwidth, with achieved memory bandwidth = (total model parameter size + KV cache size) / time per output token. It is a normalized ratio meant to capture how efficiently an LLM uses the memory system during decoding, the presumed bottleneck. ELIB's Model-Graph-Kernel runtime, which separates model, graph, and kernel layers, is the supporting mechanism that lets the same benchmarking flow run across CPU and GPU backends on different operating systems.","core_discovery":"The paper's central claim is that MBU—the ratio of achieved memory bandwidth to peak memory bandwidth, where achieved bandwidth is (model parameter size + KV cache size) divided by time per output token—is a valid normalized metric for comparing LLM inference across heterogeneous edge hardware and quantization schemes. In a memory-bound decoding process, the authors argue, MBU is the quantity that actually determines token throughput, and optimizing it (by increasing batch size, tuning sequence length, and compressing the KV cache) is the right way to improve inference performance without raising raw FLOPS. The paper also claims that MBU optimization is constrained by memory capacity and tot","pith_inferences":["MBU as defined is essentially per-token bytes moved (model + KV size divided by time per token) normalized by peak bandwidth, so on a memory-bound workload it ranks configurations similarly to throughput normalized by peak bandwidth; the novelty is the normalization, not the raw measurement.","The near-random perplexity on the GPU paths suggests those MBU and FLOPS numbers should not be interpreted as LLM inference performance; a fair comparison would either exclude those configurations or fix the GPU path first.","A testable extension: decrease quantized model size enough or increase batch size enough to enter the compute-bound regime, where the MBU-throughput correlation should weaken; if it does not, the memory-bound assumption behind MBU is incomplete."],"forward_implications":["If MBU is correct, edge platforms and quantized models become comparable by a single normalized number, independent of raw hardware speed.","Optimizing MBU—larger batch sizes, tuned sequence lengths, compressed KV caches—should improve token throughput even when FLOPS stay constant.","The benchmark's accuracy data imply that high FLOPS, throughput, and MBU on a GPU path are not sufficient: perplexity must be checked, because two of the three devices' GPU runs were near-random.","The heuristics (batch size, sequence length, KV cache management) provide a practical starting point for tuning LLM deployment on new edge hardware."],"supporting_citations":[{"why":"Defines the LLaMA model family that all benchmarked models are derived from.","marker":"[6]"},{"why":"Provides the perplexity metric used to evaluate inference accuracy in the benchmark.","marker":"[37]"},{"why":"Describes the GPU API used on the board and phone whose inference accuracy degraded.","marker":"[28]"},{"why":"Describes the GPU API used on the laptop that preserved CPU-level accuracy.","marker":"[30]"},{"why":"Supplies the BLAS implementation used for GPU linear algebra in the benchmark runs.","marker":"[29]"},{"why":"Motivates the KV cache size equation and the optimization levers the paper derives.","marker":"[36]"},{"why":"Introduces the transformer architecture whose attention cache is the KV cache counted in MBU.","marker":"[1]"}],"fun_headline_variants":["Edge LLM speed? New MBU metric ties throughput to bandwidth","MBU benchmark: memory bandwidth is the real LLM bottleneck","ELIB framework: MBU metric finds GPU inference failures","How to benchmark edge LLMs: MBU metric beats raw FLOPS"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The load-bearing assumption is that the GPU benchmark results on the board and phone are valid LLM inference measurements; if those GPU runs are actually producing near-random text (perplexity 54-68 vs CPU 4-6), then their FLOPS, throughput, and MBU numbers are not describing LLM inference performance.","fun_headline_variants_meta":{"raw":{"variants":["Edge LLM speed? New MBU metric ties throughput to bandwidth","MBU benchmark: memory bandwidth is the real LLM bottleneck","ELIB framework: MBU metric finds GPU inference failures","How to benchmark edge LLMs: MBU metric beats raw FLOPS"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001202,"raw_usage":{"total_tokens":4769,"prompt_tokens":703,"completion_tokens":4066,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":447,"completion_tokens_details":{"reasoning_tokens":3992}},"tokens_in":447,"tokens_out":4066,"duration_ms":33327,"temperature":1.0,"reasoning_tokens":3992,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T20:00:32.201760+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the GPU benchmarks on the board and phone with a correctness sanity check: decode a fixed prompt and compare the output tokens (or exact logits) to the CPU output. If the GPU outputs diverge or perplexity stays above ~20 on WikiText-2, the reported GPU FLOPS, throughput, and MBU values do not represent valid LLM inference.","supporting_citations":[{"cited_title":"Huyen, Evaluation metrics for language modeling, The Gradient (2019)","cited_arxiv_id":null,"evidence_quote":"Provides the perplexity metric used to evaluate inference accuracy in the benchmark."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes the GPU API used on the board and phone whose inference accuracy degraded."},{"cited_title":"Gebraad, A","cited_arxiv_id":null,"evidence_quote":"Describes the GPU API used on the laptop that preserved CPU-level accuracy."},{"cited_title":"Vaswani, N","cited_arxiv_id":null,"evidence_quote":"Introduces the transformer architecture whose attention cache is the KV cache counted in MBU."}],"review_version":1}