REVIEW 5 major objections 5 minor 16 references
Generative AI on the Edge: Architecture and Performance Evaluation
T0 review · 5 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read A Raspberry Pi cluster running lightweight quantized LLMs can serve interactive conversations without a GPU, achieving 5-12 tokens per second.
desk verdict Useful testbed and measurements, but the abstract's headline throughput is not backed by the body's own numbers, and the latency table has arithmetic errors. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is a containerized edge cluster: four Raspberry Pi 5 nodes (quad-core ARM Cortex-A76, 8 GB RAM) joined by a Gigabit switch under K3s, a Kubernetes distribution stripped down for constrained devices. The PromptAI front-end and proxy are built as API services so model containers can be swapped without changing the orchestration. Model weights use 4-bit GGUF quantization, a format that trades a small accuracy loss for a large memory and computation reduction, and inference runs on CPU through an open-source GGUF runtime. The evaluation distinguishes the two inference phases—prefill, which processes the prompt and builds the key-value cache, and decode, which generates tokens autoregressively—because they have different resource demands; all throughput and latency numbers are reported separately for these two phases.
What would settle it
Run the same K3s cluster and the same eight models while two or more users send prompts concurrently, or while a lightweight RAN stack executes on the same nodes. If per-user generation throughput falls below roughly 5 tokens per second or CPU/RAM utilization exceeds about 50% under that contention, the paper's central feasibility claim is not supported for shared or multi-user edge deployments.
Extended reading notes
Core claim
On its own terms, the paper's central discovery is a quantitative feasibility result: a commodity single-board computer cluster, with no GPU and only CPU inference, can serve interactive conversational AI at human-readable speeds. Across eight quantized decoder-only models, the authors report decode-phase throughput between roughly 5 and 12 tokens per second for the lightweight models, with mean CPU utilization around 50% and memory footprints as low as 0.65 GB for the smallest model. They also map the design space: the 7B-class models (InternLM, Mistral, Llama2) run but take 228-252 seconds per dialogue, whereas Yi completes the same dialogues in about 47 seconds, at the cost of Winogrande accuracy near chance level (0.49). Context-length sensitivity is measured via coefficient of variation, showing Mistral and InternLM as the most stable and Zephyr as the least. The authors present this as evidence that edge GenAI is not only possible but practical for conversational workloads, provided model size and quantization are chosen to match the hardware.
Load-bearing premise
The evaluation assumes the LLM cluster runs in isolation, with the Open RAN workload on separate hardware and only one conversational request at a time; if a real small-cell deployment must share the same Raspberry Pis with RAN processing or serve multiple users simultaneously, the measured throughput and resource usage may not hold.
Editorial extensions
If this is right
- Lightweight quantized LLMs on ARM CPUs reach 5-12 tokens per second, which sits in the range humans can comfortably read, so an interactive voice or text assistant is feasible without a GPU or cloud round-trip.
- Because CPU and RAM usage stay below 50%, a future co-located small-cell deployment could in principle share the device with other functions, although the paper does not measure that co-location.
- The API-based PromptAI design means edge nodes can change models on demand, letting an operator trade accuracy (favoring InternLM, Gemma, Llama3) against latency (favoring Yi) per deployment site.
- Larger 7B-class models are technically runnable on this hardware but have end-to-end latencies of several minutes per dialogue, so they are not viable for interactive edge use in their current form.
- The results support the paper's conclusion that localized inference for remote or bandwidth-constrained 6G/Open RAN sites is achievable without reliance on cloud infrastructure.
Reading between the lines
- The reported numbers come from isolated, single-request tests; under concurrent multi-user traffic or with RAN processing sharing the same CPUs, throughput would likely drop below the 5-12 tokens/s range, so the practical deployment envelope is probably narrower than the headline claim.
- The architecture's API-based model pooling would map naturally onto Open RAN's xApp/rApp intelligence layer, but the latency budgets shown here do not include RAN control-plane deadlines, so a direct transfer would require re-measurement.
- Accuracy was measured on non-fine-tuned base models; fine-tuned domain variants would likely score higher on Winogrande, shifting the accuracy-latency trade-off and possibly making small models more competitive with large ones.
- Tensor-parallel inference across the four Pis (distributed decode) is a plausible next step the paper itself flags as future work; if it raises decode throughput substantially, medium models like Llama3 could become interactive.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper designs and evaluates a Raspberry Pi 5 cluster running K3s and Docker to serve quantized (GGUF) large language models for a conversational PromptAI assistant in an edge/ORAN scenario. It compares eight models of three size classes on latency, throughput, CPU/memory usage, accuracy (Winogrande), and stability. The authors claim that CPU-only lightweight models (Yi, Phi, Llama3) achieve 5–12 tokens/s with under 50% CPU and RAM usage, suggesting that GenAI can be locally served without cloud infrastructure.
Significance. If the quantitative claims are properly substantiated, the paper would provide a useful open testbed and comparative performance data for a relevant 6G/ORAN edge scenario. Strengths include a reproducible architecture (K3s/Docker, public GitHub/DockerHub repositories), direct measurements against external datasets (OASST1, Winogrande) without fitted parameters, and evaluation across model families. However, the headline throughput and resource claims are not currently verifiable from the reported data, and there are internal inconsistencies in the latency numbers.
major comments (5)
- [Abstract and §IV.A] The abstract's central claim—'generation throughput of 5 to 12 tokens per second'—is never reported as an absolute measurement in the evaluation. §IV.A and Figure 3 present only relative comparisons (e.g., 'Yi ... 170% greater than other small models'), and the only absolute timing for Yi is the 47.41 s end-to-end total in Figure 4d. Without a reported tokens/s value per model, and without the generated-token count L, the reader cannot verify the headline throughput range or even compute it from the given data (the 500-token cap bounds Yi's end-to-end throughput at ≈10.7 tokens/s including prefill, making the upper bound of 12 tokens/s unattainable under that cap). Please add a table of measured tokens/s for prefill, decode, and end-to-end, and state explicitly whether these are llama.cpp benchmark values or PromptAI end-to-end measurements.
- [§IV.A, Figure 4(a–c)] The reported latencies for Gemma and Zephyr are internally inconsistent. For Gemma, the stated prefill time (82.02 ms) plus decode time (238.93 ms) equals 320.95 ms, not the reported total of 251.98 ms; for Zephyr, 102.62 ms + 233.88 ms = 336.50 ms, not 243.63 ms. This suggests that 'total time per token' is defined differently from the sum of prefill and decode, or that one of the numbers is erroneous. The definition must be clarified and the numbers corrected, since these values are the basis for the model comparisons.
- [§IV.A, Figure 4d and §IV] The end-to-end latency comparison in Figure 4d conflates prefill and decode and is workload-dependent. The paper does not report the distribution of generated tokens (L) for the 50 OpenAssistant dialogues, nor the number of turns per dialogue per model, and the maximum generation is set to 500. Without these statistics, a total time of 47.41 s for Yi cannot be converted into a token/s figure, and the claimed 5–12 tokens/s cannot be checked. Please report the mean/median/maximum generated tokens per dialogue and give per-phase timing breakdowns.
- [§IV.B and Abstract] The abstract's second quantitative claim, 'less than 50% CPU and RAM usage,' is not directly supported by the data shown. Figure 5 reports memory usage in GB (0.65–3.14 GB) and CPU utilization textually as 'centering around 50%' with no exact percentages; no conversion to percentage of the 8 GB RAM is provided. Please report CPU utilization percentages and RAM utilization as percentages (or state the available RAM), so the 50% claim is verifiable.
- [§IV.C and Table I] The accuracy evaluation is described as using 'non-fine-tuned variants' and 'pre-trained models,' but Table I lists instruction-tuned or chat checkpoints (e.g., Yi-Coder-1.5B-Chat, stablelm-zephyr-3b, Phi-3.5-mini-instruct). These are fine-tuned models, not base models. Clarify which exact Hugging Face checkpoints were used; otherwise the accuracy results and the statement about base-model comparability are confusing.
minor comments (5)
- [Abstract and §III] The abstract says 'a single commodity Raspberry Pi,' but the testbed described in Section III and Figure 1 is a cluster of four Raspberry Pi 5s; please reconcile this wording.
- [§IV.A] The statement 'maximum token generation is set to 500' should clarify whether this cap applies per turn or per whole dialogue, since this materially affects the interpretation of the total-time values.
- [Figures 3 and 5] Figure 3's y-axis lacks units; it should be labeled tokens/s. Figure 5 mixes memory in GB and CPU in percent on what appears to be a single axis; separate axes or separate panels would improve readability.
- [§IV.C] Winogrande is a coreference-resolution benchmark, not an NLI task; the text should be corrected accordingly.
- [§V] The conclusion contains a typo: 'we will to leverage' should be 'we will leverage.'
Circularity Check
No circularity: the paper's numbers are direct measurements against external datasets and benchmarks, with no fitted-parameter or self-citation dependence.
full rationale
This is a measurement and benchmarking paper. The central claims about throughput, latency, CPU and memory usage, and accuracy are direct observations from the Raspberry Pi testbed, evaluated against external datasets (OASST1 for conversational prompts and Winogrande for accuracy). No quantity is defined in terms of another result derived in the same paper, and no parameter is fitted to data and then renamed as a prediction. The paper contains no self-citations that carry argumentative weight: reference [5] is an external mobile-LLM evaluation framework, and reference [16] is the external LM Evaluation Harness; neither is authored by the present authors, and neither is used to justify the paper's conclusions. The skeptical reader's concern that the abstract's '5 to 12 tokens per second' figure is not explicitly tabulated in Section IV is a reporting or reproducibility gap, not circularity: the claim is not derived by construction from any input, and it could in principle be verified by external measurement. Under the hard rules, a missing number or an unverified headline is a correctness or completeness concern, not a circular step. Therefore the circularity score is 0.
Assumptions & free parameters
assumptions (4)
- domain assumption The 50 OASST1 dialogues, each with at least five turns, are representative of conversational AI workloads on edge networks.
- domain assumption Winogrande accuracy is an adequate proxy for the quality of quantized models in edge tasks.
- domain assumption The performance of the LLM cluster is assessed in isolation from the ORAN cluster, so results reflect dedicated LLM inference.
- domain assumption 4-bit GGUF quantization preserves the models' capabilities sufficiently for the claims to generalize to their unquantized versions.
Cite this review
Pith. "Pith review of Generative AI on the Edge: Architecture and Performance Evaluation." pith.science (2026). https://pith.science/paper/CFZ4DWKZ
@misc{pith2026241117712,
author = {Pith},
title = {Pith review of: Generative AI on the Edge: Architecture and Performance Evaluation},
year = {2026},
howpublished = {\url{https://pith.science/paper/CFZ4DWKZ}},
note = {Machine review of arXiv:2411.17712}
}
read the original abstract
6G's AI native vision of embedding advance intelligence in the network while bringing it closer to the user requires a systematic evaluation of Generative AI (GenAI) models on edge devices. Rapidly emerging solutions based on Open RAN (ORAN) and Network-in-a-Box strongly advocate the use of low-cost, off-the-shelf components for simpler and efficient deployment, e.g., in provisioning rural connectivity. In this context, conceptual architecture, hardware testbeds and precise performance quantification of Large Language Models (LLMs) on off-the-shelf edge devices remains largely unexplored. This research investigates computationally demanding LLM inference on a single commodity Raspberry Pi serving as an edge testbed for ORAN. We investigate various LLMs, including small, medium and large models, on a Raspberry Pi 5 Cluster using a lightweight Kubernetes distribution (K3s) with modular prompting implementation. We study its feasibility and limitations by analyzing throughput, latency, accuracy and efficiency. Our findings indicate that CPU-only deployment of lightweight models, such as Yi, Phi, and Llama3, can effectively support edge applications, achieving a generation throughput of 5 to 12 tokens per second with less than 50\% CPU and RAM usage. We conclude that GenAI on the edge offers localized inference in remote or bandwidth-constrained environments in 6G networks without reliance on cloud infrastructure.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
5g nr; technical specification group radio access network; nr; release 18, 2023
3GPP. 5g nr; technical specification group radio access network; nr; release 18, 2023
work page 2023
-
[2]
Imagenet classification with deep convolutional neural networks
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM , 60(6):84–90, 2017
2017
-
[3]
Learning to for- get: Continual prediction with lstm
Felix A Gers, J ¨urgen Schmidhuber, and Fred Cummins. Learning to for- get: Continual prediction with lstm. Neural computation , 12(10):2451– 2471, 2000
work page 2000
-
[4]
Attention is all you need
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017
2017
-
[5]
Melting point: Mobile evaluation of language transformers
Stefanos Laskaridis, Kleomenis Katevas, Lorenzo Minto, and Hamed Haddadi. Melting point: Mobile evaluation of language transformers. ArXiv, abs/2403.12844, 2024
arXiv 2024
-
[6]
Inc. Gartner. Gartner hype cycle for edge computing, 2024, 2024
work page 2024
-
[7]
Mobile-llama: Instruction fine-tuning open-source llm for network anal- ysis in 5g networks
Khen Bo Kan, Hyunsu Mun, Guohong Cao, and Youngseok Lee. Mobile-llama: Instruction fine-tuning open-source llm for network anal- ysis in 5g networks. IEEE Network , 2024
work page 2024
-
[8]
Efficient memory management for large language model serving with pagedattention
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles , pages 611–626, 2023
2023
Show all 16 references
-
[9]
Ai benchmark: Running deep neural networks on android smartphones
Andrey Ignatov, Radu Timofte, William Chou, Ke Wang, Max Wu, Tim Hartley, and Luc Van Gool. Ai benchmark: Running deep neural networks on android smartphones. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops , pages 0–0, 2018
2018
-
[10]
Mlperf inference benchmark
Vijay Janapa Reddi, Christine Cheng, David Kanter, Peter Mattson, Guenther Schmuelling, Carole-Jean Wu, Brian Anderson, Maximilien Breughe, Mark Charlebois, William Chou, et al. Mlperf inference benchmark. In 2020 ACM/IEEE 47th Annual International Symposium on Computer Archit...
2020
-
[11]
A first look at deep learning apps on smartphones
Mengwei Xu, Jiawei Liu, Yuanqiang Liu, Felix Xiaozhu Lin, Yunxin Liu, and Xuanzhe Liu. A first look at deep learning apps on smartphones. In The World Wide Web Conference , pages 2125–2136, 2019
2019
-
[12]
A survey on mobile edge computing: The communication perspective
Yuyi Mao, Changsheng You, Jun Zhang, Kaibin Huang, and Khaled B Letaief. A survey on mobile edge computing: The communication perspective. IEEE communications surveys & tutorials , 19(4):2322– 2358, 2017
2017
-
[13]
Zero: Memory optimizations toward training trillion parameter models
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis , pages 1–16. IEEE, 2020
2020
-
[14]
Zeus: Understand- ing and optimizing GPU energy consumption of DNN training
Jie You, Jae-Won Chung, and Mosharaf Chowdhury. Zeus: Understand- ing and optimizing GPU energy consumption of DNN training. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23) , pages 119–139, 2023
2023
-
[15]
Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale
Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, et al. Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale. In SC22: International C...
2022
-
[16]
A framework for few-shot language model evalua- tion
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, et al. A framework for few-shot language model evalua- tion. V ersion v0. 0.1. Sept , 10:8–9, 2021
2021
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.