{"id":"7e55808c-1b50-43c9-b741-83b6d3f5b44b","arxiv_id":"2411.17712","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Small quantized language models (Yi, Phi, Llama3) achieve 5 to 12 tokens per second on a CPU-only Raspberry Pi 5 K3s cluster with under 50% CPU and RAM usage, supporting edge inference for 6G applications.","lead":"This paper measures how well small AI language models run on a cluster of Raspberry Pi computers with no graphics card, using a lightweight Kubernetes setup. It reports that CPU-only models can generate 5 to 12 tokens per second while using under half of the machine's resources, suggesting that edge devices could host AI assistants in remote 6G networks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's 5–12 tokens/s is never reported in the evaluation; Fig. 4d's 47.41 s total time for Yi is consistent with that range only if average generated length is 235–500 tokens, which the paper does not establish.","rationale":"In good faith, the paper's core claim is an empirical performance statement that is plausible: 1–4B Q4 models on a Raspberry Pi 5 can produce a few tokens per second. The open-source repository and K3s testbed are real contributions, and I am not questioning honesty. The load-bearing weakness is internal support: the single quantitative claim in the abstract cannot be reconstructed from the reported figures and text. Section IV gives percentages and relative comparisons but no absolute throughput numbers, and the one absolute latency figure implies a narrow and undisclosed output-length range for the headline to hold. This is a correctness/reporting risk rather than a disagreement with community consensus. The proposed test settles it: either inferred tokens/s matches 5–12, in which case a table of absolute values and output lengths should be added, or it does not, in which case the abstract overstates the result. I do not make ORAN co-location or concurrency the primary concern because the paper explicitly scopes to the LLM cluster and places ORAN in a separate cluster; that is a scope limitation rather than a contradiction. The reader's weakest_assumption is reasonable but not the most load-bearing issue, hence partial agreement. Verdict remains CONDITIONAL: the central claim may be true but needs a revision that reports per-model absolute throughput, generated-token counts, and a reconciliation of Fig. 4d with the abstract.","tokens_in":13534,"tokens_out":13437,"duration_ms":130520,"concrete_test":"Rerun the paper's 50 OpenAssistant dialogues (via the provided GitHub repository) on the same Raspberry Pi 5/K3s setup for Yi, Phi, and Llama3 Q4, recording per-dialogue generated-token counts and end-to-end wall-clock times; compute tokens/s as mean generated tokens divided by mean total time, compare with the abstract's 5–12 tokens/s, and report time-to-first-token.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is the abstract's '5 to 12 tokens per second with less than 50% CPU and RAM usage.' Section IV.A reports only relative throughput comparisons (e.g., Yi's Decode throughput is 170% greater than other small models); it never gives the measured tokens/s per model. The only absolute timing for the best small model is Fig. 4d: Yi's average total end-to-end time is 47.41 s. With Yi's reported prefill of 13.79 ms/token and an average prompt of roughly 33 tokens, prefill contributes about 0.5 s, so implied generation throughput is L/46.9 tokens/s, where L is the average generated tokens per dialogue. For the claimed 5–12 tokens/s to hold, L must be between about 235 and 500 tokens, the stated maximum. The paper never reports L, nor does it state whether 'throughput' is a llama.cpp decode benchmark or an end-to-end PromptAI measurement. If the selected OpenAssistant dialogues have typical assistant responses below 235 tokens, the abstract contradicts its own latency data; if L is near 500, the workload is unrepresentative of interactive conversation and roughly 47 s per response does not obviously constitute effective support. The headline number is therefore unverified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper designs and evaluates a Raspberry Pi 5 cluster running K3s and Docker to serve quantized (GGUF) large language models for a conversational PromptAI assistant in an edge/ORAN scenario. It compares eight models of three size classes on latency, throughput, CPU/memory usage, accuracy (Winogrande), and stability. The authors claim that CPU-only lightweight models (Yi, Phi, Llama3) achieve 5–12 tokens/s with under 50% CPU and RAM usage, suggesting that GenAI can be locally served without cloud infrastructure.","tokens_in":13779,"tokens_out":6158,"duration_ms":51472,"significance":"If the quantitative claims are properly substantiated, the paper would provide a useful open testbed and comparative performance data for a relevant 6G/ORAN edge scenario. Strengths include a reproducible architecture (K3s/Docker, public GitHub/DockerHub repositories), direct measurements against external datasets (OASST1, Winogrande) without fitted parameters, and evaluation across model families. However, the headline throughput and resource claims are not currently verifiable from the reported data, and there are internal inconsistencies in the latency numbers.","major_comments":[{"comment":"The abstract's central claim—'generation throughput of 5 to 12 tokens per second'—is never reported as an absolute measurement in the evaluation. §IV.A and Figure 3 present only relative comparisons (e.g., 'Yi ... 170% greater than other small models'), and the only absolute timing for Yi is the 47.41 s end-to-end total in Figure 4d. Without a reported tokens/s value per model, and without the generated-token count L, the reader cannot verify the headline throughput range or even compute it from the given data (the 500-token cap bounds Yi's end-to-end throughput at ≈10.7 tokens/s including prefill, making the upper bound of 12 tokens/s unattainable under that cap). Please add a table of measured tokens/s for prefill, decode, and end-to-end, and state explicitly whether these are llama.cpp benchmark values or PromptAI end-to-end measurements.","section":"Abstract and §IV.A"},{"comment":"The reported latencies for Gemma and Zephyr are internally inconsistent. For Gemma, the stated prefill time (82.02 ms) plus decode time (238.93 ms) equals 320.95 ms, not the reported total of 251.98 ms; for Zephyr, 102.62 ms + 233.88 ms = 336.50 ms, not 243.63 ms. This suggests that 'total time per token' is defined differently from the sum of prefill and decode, or that one of the numbers is erroneous. The definition must be clarified and the numbers corrected, since these values are the basis for the model comparisons.","section":"§IV.A, Figure 4(a–c)"},{"comment":"The end-to-end latency comparison in Figure 4d conflates prefill and decode and is workload-dependent. The paper does not report the distribution of generated tokens (L) for the 50 OpenAssistant dialogues, nor the number of turns per dialogue per model, and the maximum generation is set to 500. Without these statistics, a total time of 47.41 s for Yi cannot be converted into a token/s figure, and the claimed 5–12 tokens/s cannot be checked. Please report the mean/median/maximum generated tokens per dialogue and give per-phase timing breakdowns.","section":"§IV.A, Figure 4d and §IV"},{"comment":"The abstract's second quantitative claim, 'less than 50% CPU and RAM usage,' is not directly supported by the data shown. Figure 5 reports memory usage in GB (0.65–3.14 GB) and CPU utilization textually as 'centering around 50%' with no exact percentages; no conversion to percentage of the 8 GB RAM is provided. Please report CPU utilization percentages and RAM utilization as percentages (or state the available RAM), so the 50% claim is verifiable.","section":"§IV.B and Abstract"},{"comment":"The accuracy evaluation is described as using 'non-fine-tuned variants' and 'pre-trained models,' but Table I lists instruction-tuned or chat checkpoints (e.g., Yi-Coder-1.5B-Chat, stablelm-zephyr-3b, Phi-3.5-mini-instruct). These are fine-tuned models, not base models. Clarify which exact Hugging Face checkpoints were used; otherwise the accuracy results and the statement about base-model comparability are confusing.","section":"§IV.C and Table I"}],"minor_comments":[{"comment":"The abstract says 'a single commodity Raspberry Pi,' but the testbed described in Section III and Figure 1 is a cluster of four Raspberry Pi 5s; please reconcile this wording.","section":"Abstract and §III"},{"comment":"The statement 'maximum token generation is set to 500' should clarify whether this cap applies per turn or per whole dialogue, since this materially affects the interpretation of the total-time values.","section":"§IV.A"},{"comment":"Figure 3's y-axis lacks units; it should be labeled tokens/s. Figure 5 mixes memory in GB and CPU in percent on what appears to be a single axis; separate axes or separate panels would improve readability.","section":"Figures 3 and 5"},{"comment":"Winogrande is a coreference-resolution benchmark, not an NLI task; the text should be corrected accordingly.","section":"§IV.C"},{"comment":"The conclusion contains a typo: 'we will to leverage' should be 'we will leverage.'","section":"§V"}],"recommendation":"major_revision","confidential_remarks":"The paper reports a useful testbed and comparative measurements, but the reporting of the headline numbers is currently inadequate. The fixes needed—adding absolute tokens/s, reconciling the latency sums, and reporting generated-token statistics—are well within scope and should be achievable with the existing data. The paper would also benefit from an explicit statement that the LLM cluster was evaluated in isolation from ORAN workloads, as the current abstract wording slightly overreaches. With these changes, the contribution could be acceptable for a systems/measurement venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: the testbed and measurements are real, but the abstract's signature number (5–12 tokens/s) never appears in the evaluation, and two of the reported per-token latencies don't add up. The qualitative finding is almost certainly correct; the quantitative headline is not yet supported.\n\nWhat's new and good: the combination of a Raspberry Pi 5 cluster, K3s, llama.cpp, and GGUF models is a sensible testbed, and they ship code and container images. Using 50 real OASST1 dialogues, repeating each run three times, and checking accuracy on Winogrande is more than many papers at this level do. The stability observation (no OOM or restarts across models) is useful practical knowledge. The architecture description is clear enough that someone could reproduce the setup.\n\nSoft spots, in order of seriousness. First, the stress-test note is on target: the body reports only relative throughput comparisons (e.g., Yi's decode throughput is 170% higher than other small models), never an absolute tokens/s. The only absolute time for Yi is the 47.41 s end-to-end average in Fig. 4d. Working backward, that number only supports the abstract's 5–12 tokens/s if the average generated response is 235–500 tokens, a range the paper never reports. The ambiguity between llama.cpp's decode benchmark and the end-to-end PromptAI measurement makes this worse. Second, the arithmetic in Section IV.A is off: Gemma's total (251.98 ms) is less than the sum of its stated prefill and decode times (82.02 + 238.93 = 320.95 ms), and Zephyr's total (243.63 ms) similarly falls short of 102.62 + 233.88 = 336.5 ms. That suggests either a typo in the figure or a systematic reporting error; either way it erodes confidence in the other numbers. Third, there are no confidence intervals on accuracy, and no measurements under concurrent requests or with RAN traffic sharing the box. The separate-cluster assumption is a reasonable scoping decision, but it should be stated as a limitation rather than implied.\n\nNone of this sinks the paper. The central claim—that small quantized LLMs can serve interactive conversational workloads on a CPU-only Pi—matches community measurements and is the kind of engineering knowledge that is worth having. Who this is for: people planning low-cost edge AI deployments, especially in the ORAN/6G space, who want a reproducible baseline. It deserves a serious referee. I'd recommend major revision: put the absolute tokens/s in the body, report the distribution of generated lengths, fix the latency arithmetic, and add error bars and a concurrency caveat. After that, I'd happily cite it.","headline":"Useful testbed and measurements, but the abstract's headline throughput is not backed by the body's own numbers, and the latency table has arithmetic errors.","tokens_in":14344,"tokens_out":3076,"would_cite":false,"duration_ms":27457,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Raspberry Pi cluster running lightweight quantized LLMs can serve interactive conversations without a GPU, achieving 5-12 tokens per second.","keywords":["edge AI","large language model inference","Raspberry Pi cluster","K3s","Open RAN","GGUF quantization","throughput evaluation","6G networks"],"falsifier":"Run the same K3s cluster and the same eight models while two or more users send prompts concurrently, or while a lightweight RAN stack executes on the same nodes. If per-user generation throughput falls below roughly 5 tokens per second or CPU/RAM utilization exceeds about 50% under that contention, the paper's central feasibility claim is not supported for shared or multi-user edge deployments.","tokens_in":13336,"feed_emoji":"🤖","tokens_out":7378,"duration_ms":64125,"temperature":0.7,"pith_summary":"Sixth-generation networks promise AI built into the radio access edge, but most large language models assume cloud GPUs. This paper tests the opposite assumption: can a four-node Raspberry Pi 5 cluster, orchestrated by the lightweight Kubernetes distribution K3s, run conversational LLMs on CPUs alone? Using 4-bit quantized versions of eight models from 1.5B to 7.7B parameters, the measurements show that small and medium models—Yi, Phi, and Llama3—generate 5 to 12 tokens per second while using less than half of the available CPU and RAM. The authors read this as evidence that localized GenAI inference is feasible in remote or bandwidth-constrained 6G/Open RAN deployments without cloud offload, while larger models trade latency and memory for higher accuracy.","feed_headline":"CPU-only LLMs hit 5-12 tokens per second on a Pi 5 cluster","feed_subtitle":"Benchmark of eight quantized models: Yi, Phi, and Llama3 run interactive chat using under half the Pi's CPU and RAM.","key_machinery":"The carrying mechanism is a containerized edge cluster: four Raspberry Pi 5 nodes (quad-core ARM Cortex-A76, 8 GB RAM) joined by a Gigabit switch under K3s, a Kubernetes distribution stripped down for constrained devices. The PromptAI front-end and proxy are built as API services so model containers can be swapped without changing the orchestration. Model weights use 4-bit GGUF quantization, a format that trades a small accuracy loss for a large memory and computation reduction, and inference runs on CPU through an open-source GGUF runtime. The evaluation distinguishes the two inference phases—prefill, which processes the prompt and builds the key-value cache, and decode, which generates tokens autoregressively—because they have different resource demands; all throughput and latency numbers are reported separately for these two phases.","core_discovery":"On its own terms, the paper's central discovery is a quantitative feasibility result: a commodity single-board computer cluster, with no GPU and only CPU inference, can serve interactive conversational AI at human-readable speeds. Across eight quantized decoder-only models, the authors report decode-phase throughput between roughly 5 and 12 tokens per second for the lightweight models, with mean CPU utilization around 50% and memory footprints as low as 0.65 GB for the smallest model. They also map the design space: the 7B-class models (InternLM, Mistral, Llama2) run but take 228-252 seconds per dialogue, whereas Yi completes the same dialogues in about 47 seconds, at the cost of Winogrande accuracy near chance level (0.49). Context-length sensitivity is measured via coefficient of variation, showing Mistral and InternLM as the most stable and Zephyr as the least. The authors present this as evidence that edge GenAI is not only possible but practical for conversational workloads, provided model size and quantization are chosen to match the hardware.","pith_inferences":["The reported numbers come from isolated, single-request tests; under concurrent multi-user traffic or with RAN processing sharing the same CPUs, throughput would likely drop below the 5-12 tokens/s range, so the practical deployment envelope is probably narrower than the headline claim.","The architecture's API-based model pooling would map naturally onto Open RAN's xApp/rApp intelligence layer, but the latency budgets shown here do not include RAN control-plane deadlines, so a direct transfer would require re-measurement.","Accuracy was measured on non-fine-tuned base models; fine-tuned domain variants would likely score higher on Winogrande, shifting the accuracy-latency trade-off and possibly making small models more competitive with large ones.","Tensor-parallel inference across the four Pis (distributed decode) is a plausible next step the paper itself flags as future work; if it raises decode throughput substantially, medium models like Llama3 could become interactive."],"forward_implications":["Lightweight quantized LLMs on ARM CPUs reach 5-12 tokens per second, which sits in the range humans can comfortably read, so an interactive voice or text assistant is feasible without a GPU or cloud round-trip.","Because CPU and RAM usage stay below 50%, a future co-located small-cell deployment could in principle share the device with other functions, although the paper does not measure that co-location.","The API-based PromptAI design means edge nodes can change models on demand, letting an operator trade accuracy (favoring InternLM, Gemma, Llama3) against latency (favoring Yi) per deployment site.","Larger 7B-class models are technically runnable on this hardware but have end-to-end latencies of several minutes per dialogue, so they are not viable for interactive edge use in their current form.","The results support the paper's conclusion that localized inference for remote or bandwidth-constrained 6G/Open RAN sites is achievable without reliance on cloud infrastructure."],"supporting_citations":[{"why":"Supplies the mobile-LLM evaluation methodology and the memory/CPU constraint argument; the conversational prompt set used in the experiments comes from this line of work.","marker":"[5]"},{"why":"Shows that transformer serving is memory-bound, motivating the paper's focus on memory and quantization on constrained devices.","marker":"[8]"},{"why":"Context for standardized inference benchmarking that the on-device measurements are positioned against.","marker":"[10]"},{"why":"Evidence that on-device deep learning workloads are growing, motivating edge LLM evaluation.","marker":"[11]"},{"why":"Provides the evaluation harness used to measure common-sense reasoning accuracy of the quantized models.","marker":"[16]"}],"fun_headline_variants":["Pi 5 cluster runs LLMs at 5-12 tokens/s CPU-only","Edge GenAI feasible: Pi cluster serves chat without GPU","Raspberry Pi 5 handles interactive LLMs under 50% CPU load","Lightweight LLMs on Pi 5: practical edge inference for 6G","No-GPU Pi 5 cluster delivers usable conversational AI speeds"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes the LLM cluster runs in isolation, with the Open RAN workload on separate hardware and only one conversational request at a time; if a real small-cell deployment must share the same Raspberry Pis with RAN processing or serve multiple users simultaneously, the measured throughput and resource usage may not hold.","fun_headline_variants_meta":{"raw":{"variants":["Pi 5 cluster runs LLMs at 5-12 tokens/s CPU-only","Edge GenAI feasible: Pi cluster serves chat without GPU","Raspberry Pi 5 handles interactive LLMs under 50% CPU load","Lightweight LLMs on Pi 5: practical edge inference for 6G","No-GPU Pi 5 cluster delivers usable conversational AI speeds"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000614,"raw_usage":{"total_tokens":2889,"prompt_tokens":1013,"completion_tokens":1876,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":629,"completion_tokens_details":{"reasoning_tokens":1779}},"tokens_in":629,"tokens_out":1876,"duration_ms":13855,"temperature":1.0,"reasoning_tokens":1779,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:14:36.737641+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same K3s cluster and the same eight models while two or more users send prompts concurrently, or while a lightweight RAN stack executes on the same nodes. If per-user generation throughput falls below roughly 5 tokens per second or CPU/RAM utilization exceeds about 50% under that contention, the paper's central feasibility claim is not supported for shared or multi-user edge deployments.","supporting_citations":[{"cited_title":"Mlperf inference benchmark","cited_arxiv_id":null,"evidence_quote":"Context for standardized inference benchmarking that the on-device measurements are positioned against."},{"cited_title":"A first look at deep learning apps on smartphones","cited_arxiv_id":null,"evidence_quote":"Evidence that on-device deep learning workloads are growing, motivating edge LLM evaluation."},{"cited_title":"A framework for few-shot language model evalua- tion","cited_arxiv_id":null,"evidence_quote":"Provides the evaluation harness used to measure common-sense reasoning accuracy of the quantized models."}],"review_version":1}