{"id":"4e314401-fffc-418e-8804-f4b995fcd9a9","arxiv_id":"2506.19030","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"WiLLM is a new open-source platform plus dataset for running and measuring LLM services over live wireless networks, with evidence that LLM traffic is bidirectional, bursty, and coupled to network slicing.","lead":"This paper introduces WiLLM, an open-source wireless testbed built on OpenAirInterface that lets researchers run large language model services over real 5G connections, and it releases about 1.6 million synchronized measurements. It is useful because it lowers the barrier to studying how LLM traffic behaves on cellular networks, a topic that now mixes AI and telecom engineering.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LLM-specific 'fundamental differences' rest on LLM-only measurements; without a same-testbed DNN baseline the central motivation for WiLLM's new scheduling and slicing design is unproven.","rationale":"The reader's weakest assumption is the missing DNN baseline, and that is the load-bearing gap. The paper has real independent support: an open-source implementation, a large synchronized dataset, and a deployed testbed, so the platform claim is credible. But the headline framing depends on LLM-specific behavior; Section 2 only compares two LLM tasks (image-to-text vs. text-to-image) and offers no control condition. An additional internal inconsistency strengthens this concern: the paper says bidirectional transitions occur 'within a single session,' yet Figures 3 and 4 are separate experiment groups rather than interleaved session traces. The missing baseline is testable, so the paper should remain CONDITIONAL, the same verdict the reader reached.","tokens_in":28813,"tokens_out":5286,"duration_ms":56980,"concrete_test":"Run the same WiLLM testbed under identical radio configurations, PRB allocations, and slice policies while collecting the same 58-metric traces for a representative traditional DNN workload, such as ResNet-50 image classification with uplink images and small downlink labels, and a video-analytics pipeline with downlink streaming. Then compare three quantities against the LLM measurements: (a) uplink/downlink/inference latency composition per slice configuration, (b) inference-time coefficient of variation for identical inputs, and (c) byte-arrival burstiness and Markovianity tests. If the traditional DNN workload is statistically indistinguishable from the LLM workload on these axes, the 'fundamentally differ' assertion in Section 1 is not supported, and WiLLM's LLM-specific architecture must be rejustified as a platform contribution rather than a demonstrated necessity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that LLM services 'fundamentally differ' from traditional DNN applications, and this is the stated motivation for every WiLLM design choice (direction-aware scheduling, dynamic slicing, adaptive resource management). That claim is supported only by measurements of LLM workloads on a single testbed. Section 2.1 contrasts image-to-text (Fig. 3) with text-to-image (Fig. 4) to show asymmetric bottlenecks, but both are LLM tasks; no traditional DNN workload is run under the same radio and slice conditions. A ResNet-50 image-classification service sending large images uplink and small labels downlink, or a video-analytics pipeline, could plausibly show the same component-latency composition. Likewise, Section 2.2's 'identical prompts... varying by over 300%' and Section 2.3's non-Markovian token streams are asserted without distributional evidence or goodness-of-fit tests, and the figures show scheduled PRBs and bytes for LLM traffic only. Because the paper explicitly claims these insights 'reveal that LLM services violate core assumptions' (Section 1), the architecture's necessity depends on distinguishing LLM from DNN behavior; without that controlled comparison, the central scientific narrative is under-supported, even though the platform and dataset remain valuable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper argues that LLM services 'fundamentally differ' from traditional DNN applications when carried over wireless networks, and that this difference demands new wireless system design. The argument rests on three insights from measurements on an OpenAirInterface-based testbed (USRP B210 gNB, GPU-equipped core network, LLaVA and Llama 3.2 models): bidirectional heavy-load asymmetry with opposite bottlenecks for image-to-text versus text-to-image workloads; computation-communication coupling producing dynamic bottleneck migration as slice resources change; and token-stream burstiness with state dependencies that the paper claims defeat Markovian traffic models. The paper then presents WiLLM, an open-source platform extending OpenAirInterface with dynamic slice compatibility, application-layer tunneling for universal UE compatibility, multi-UE multi-slice scheduling, dual-mode resource allocation, and cross-layer APIs, validated through a smart-glasses case study that targets a stable 2-second response time. The paper also releases a dataset of 1,649,996 synchronized records across four operational scenarios and proposes two appendix benchmarking metrics (LAREI and LSEQ).","tokens_in":28974,"tokens_out":21130,"duration_ms":189359,"significance":"If the central claim is adequately supported, the contribution is significant on two fronts. The open platform and the roughly 1.6-million-record synchronized dataset are valuable community resources: the scenario record counts in Section 5.2 are internally consistent, the implementation claims are concrete (USRP B210, RTX 4090, specific OAI extensions, released code and videos), and the smart-glasses demonstration is a realistic use case. Second, the three insights, if they survive a controlled comparison against a conventional DNN workload in the same testbed, would provide a falsifiable motivation for LLM-aware scheduling and slicing. That comparison is currently absent, and two of the three insights rest on unsupported assertions (the '>300%' inference-time variation in Section 2.2 and the non-Markovian claim in Section 2.3). The platform and dataset stand regardless; the scientific narrative would be strengthened either by adding the DNN control or by re-scoping the claims to the studied LLM services.","major_comments":[{"comment":"The central thesis, stated in the abstract and Section 1, is that LLM services 'fundamentally differ' from traditional DNN applications, and every WiLLM design decision (direction-aware scheduling, dynamic slice optimization, adaptive resource management) is justified by that thesis. The evidence in Section 2, however, consists entirely of LLM workloads: Figures 3 and 4 compare two LLM tasks (image-to-text versus text-to-image), Figure 5 compares three slice configurations of the same LLM service, and Figures 6 and 7 show PRB allocation and byte transmission for LLM traffic only. The contrasts with DNNs are taken from the literature, not from the testbed: Section 2.1 describes image classification services that 'upload raw images and receive lightweight labels,' and Section 2.2 cites ResNet-50 as requiring 'approximately 4 GFLOPs per image,' but no such workload is run under the same USRP B210 radio and slice conditions. A control workload such as ResNet-50 classification or YOLO detection could plausibly exhibit the same latency-composition asymmetry and bursty byte transmission, which would undermine the claimed 'fundamental' distinction. Since Section 1 states that the measurements 'reveal that LLM services violate core assumptions' of current wireless design, I recommend running at least one representative DNN workload on the same testbed with the same decompositions, or explicitly re-scoping the three insights to the measured LLM services.","section":"Section 2 (Insights 1-3) and Abstract"},{"comment":"Section 2.2 asserts that 'identical prompts can result in inference times varying by over 300% based on these generation parameters,' but no supporting data appears anywhere in the manuscript. No distribution, box plot, CDF, or parameter sweep accompanies this claim, even though it is the direct empirical basis for the computation-communication coupling insight. Please either present the evidence (for example, the inference-time distribution for a fixed prompt under swept generation parameters such as temperature, beam width, or context length) or weaken the statement to match what is actually shown.","section":"Section 2.2"},{"comment":"Section 2.3 claims that LLM token streams 'defy traditional Markovian models,' yet no Markov model is fitted and no statistical test is reported. Establishing non-Markovianity would require quantifying temporal dependence (for example, lagged autocorrelation of the byte or token stream, or prediction-error comparison against a Markov baseline) and a goodness-of-fit assessment; as published, Figures 6 and 7 are illustrative time series of LLM traffic that do not support the claim. Please add the statistical analysis or soften the claim to one of strong temporal correlation and burstiness.","section":"Section 2.3"},{"comment":"The component-latency panels in Figures 3 and 4 are ambiguous. The legend on the right panels lists 'Uplink Downlink Inference,' but only two rows of percentages are printed (for example, Figure 3 shows 11.8-24.8% and 74.6-87.1%), and the text attributes the 74-87% share to inference for image-to-text while a naive legend-order reading would assign that share to downlink. Figure 5's right panel is similarly incomplete: it omits one component value for Slice 3 (54.7 + 43.1 = 97.8%). Since these figures are the primary evidence for Insights 1 and 2, please redraw the panels so each component's segment is explicitly labeled, show all components so the percentages sum to 100%, and verify that the mapping of segments to uplink, downlink, and inference matches the text.","section":"Section 2.1, Figures 3-5"}],"minor_comments":[{"comment":"Appendix F states that 'More importantly, we conducted a causal analysis of the results shown in Figure 3 to Figure 7 of the dataset,' but no causal analysis appears in the appendix or the main text; no causal method, intervention, or confounder analysis is presented. Please either add the analysis or delete the sentence.","section":"Appendix F"},{"comment":"LAREI and LSEQ depend on a normalization coefficient omega and a calibration parameter delta whose values and fitting procedures are never specified, so a third party cannot compute either metric as published. Both metrics are also derived from the same dataset used to support the qualitative insights; please describe the calibration protocol and validate on held-out data, or label the metrics as descriptive summaries rather than benchmarking tools.","section":"Appendix G"},{"comment":"The legend of Table 2 uses the identical glyph for all three support levels (full, partial, and not supported), which makes the matrix unreadable; please use distinct symbols for the three levels.","section":"Appendix C, Table 2"},{"comment":"The dataset metric count is inconsistent: Section 5.2 says '58 metrics' but then enumerates 22 UE-layer, 25 gNB-layer, and 18 core/edge-layer metrics (65 total), while the tables in Appendix H list 12 UE, 12 server, and 29 RAN metrics (53 total). Please reconcile the stated count with the tables and the enumeration.","section":"Section 5.2"},{"comment":"Figure 13 appears corrupted: it contains glyph sequences such as '/uni00000013 /uni00000015/...' instead of a learning curve, and the caption claims it demonstrates online learning progress. Please replace it with an actual plot of the UCB convergence.","section":"Section 5.3.2, Figure 13"},{"comment":"The middle panel of Figure 5 shows total latency increasing from 1.536 s (Slice 1) to 1.957 s (Slice 3) as slice resources increase, but the text does not discuss this counterintuitive trend; please explain what differs across the slices (for example, request sizes, concurrency, or service class).","section":"Section 2.2, Figure 5"},{"comment":"In Challenge 5, the text claims 'microsecond-level sampling precision' from an NTP-based synchronizer while also stating that synchronization errors are maintained within ±1.0 milliseconds; these statements are inconsistent, and microsecond-level precision is not credible for NTP. Please align the two statements.","section":"Section 4.2"},{"comment":"Please fix minor language errors, including 'A simple query may yield trigger a terse response' (Section 2.3), 'Figure 1 and 2 illustrates' (Section 1), 'Core Netowork' (Figure 10), 'Deplays' (Figure 11 caption), and 'WiLLM is an extend of OAI' (Appendix C note).","section":"Throughout"},{"comment":"The phrase 'unprecedented burstiness' is stronger than the evidence presented; consider 'pronounced' unless a quantitative comparison to non-LLM traffic is added.","section":"Section 2.3, wording"}],"recommendation":"major_revision","confidential_remarks":"Two points for the editor. First, the priority claim ('first open-source framework ... for LLM service research') rests on the comparison tables in Appendices B and C, and the relation of the Tree-Branch-Fruit slicing concept to the authors' earlier LLM-Slice work (reference [20]) should be clarified in revision; I could not fully verify external novelty from the manuscript alone. Second, the deciding factor for my verdict is the missing same-testbed DNN control: if the authors provide it in revision, I would expect to recommend acceptance; if they decline, the claims should be re-scoped and the paper judged on the platform and dataset alone, which may be a weaker but still valid contribution for a systems-oriented venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I'd send you to this paper for the artifact, not the thesis. WiLLM is a real engineering contribution: an OpenAirInterface-based platform with application-layer tunneling so commodity UEs can use slices, dynamic slice compatibility, dual-mode scheduling, cross-layer APIs, plus a 1.6M-record synchronized dataset and a smart glasses case study. The release of code and data makes it reproducible, and the latency decomposition in Figures 3-5 is plausible. For the wireless-LLM community, this is an enabling testbed that lowers the barrier for AI researchers who don't want to touch MAC schedulers. That part deserves credit.\n\nThe soft spot is the central claim that LLM services 'fundamentally differ' from traditional DNN applications. The evidence is LLM-only: both image-to-text and text-to-image are LLM tasks, and no ResNet-style or video-analytics baseline runs on the same testbed. Without that controlled comparison, the bidirectional asymmetry and bottleneck migration could be properties of the workload, not of LLMs specifically. The 300% inference-time variation and the non-Markovian token streams are asserted without distributional evidence or statistical tests. This matters because the architecture's motivation leans on the contrast; the platform stays useful either way, but the paper's scientific narrative is under-supported.\n\nThe appendix metrics LAREI and LSEQ are the weakest part: omega and delta are calibration constants fit to the same data, with no estimation procedure or validation. I'd cut them or reframe as preliminary observations. The citation pattern looks fine; the self-citation to LLM-slice is on-topic and not an issue.\n\nBottom line: this is a paper for researchers who want to experiment with LLM services over realistic cellular systems, and a serious referee should engage with it. The main revision needed is either a same-testbed DNN baseline or a more careful claim like 'LLM services exhibit these characteristics in our measurements' instead of 'fundamentally differ.' I'd accept conditional on that change.","headline":"A genuinely useful open wireless-LLM testbed and dataset, wrapped in an overclaimed 'fundamental differences' narrative that needs a same-testbed DNN baseline.","tokens_in":29603,"tokens_out":1433,"would_cite":true,"duration_ms":16755,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM services over wireless are not just heavier DNN traffic: WiLLM's 5G testbed and 1.6 million measurements make the differences visible.","keywords":["LLM services","wireless networks","network slicing","5G testbed","resource scheduling","token burstiness","multimodal inference","open-source platform"],"falsifier":"Run the same smart-glasses scenario on the same testbed with a conventional DNN task (for example, image classification or object detection) using matched image sizes and response sizes, then compare latency-component shares, slice-induced bottleneck movement, and byte-transmission burstiness. If the conventional workload reproduces the same asymmetry, migration, and burstiness, the paper's three fundamental distinctions are not LLM-specific.","tokens_in":28525,"feed_emoji":"📡","tokens_out":6958,"duration_ms":68200,"temperature":0.7,"pith_summary":"LLM services over cellular networks, the paper argues, are not simply heavier versions of traditional DNN traffic: they behave as a different workload class with three measurable properties. On its testbed, image-to-text sessions are dominated by inference time while text-to-image sessions are dominated by downlink transmission, so the heavy load moves between directions inside one session; increasing radio slice resources shifts the bottleneck between computation and communication; and token generation produces bursty, state-dependent byte streams that resist average-based provisioning. If these properties hold, existing wireless schedulers and network slices tuned for predictable, unidirectional traffic will under-serve LLM applications. To make that claim usable, the paper contributes WiLLM, an open-source end-to-end wireless platform, a 1.65-million-record synchronized multi-layer dataset, and a smart-glasses demonstration that stabilizes multimodal responses at two seconds.","feed_headline":"LLM traffic breaks wireless assumptions; open platform shows how","feed_subtitle":"A 5G testbed logs asymmetry, bottleneck shifts, and token bursts in 1.6 million synchronized records.","key_machinery":"The load-bearing object is WiLLM itself, a working end-to-end wireless testbed built by extending an open-source 5G protocol stack with GPU inference placed in the core network. Three named mechanisms carry the design: the Tree-Branch-Fruit slicing hierarchy (shared radio 'tree,' standard 5G slice 'branches,' and LLM-specific 'fruit' slices), application-layer tunneling that lets ordinary devices reach slice resources without protocol changes, and dual-mode scheduling that can act inside the scheduler or through an external optimization engine. The empirical case rests on an NTP-synchronized measurement pipeline that records 58 synchronized metrics across UE, radio access, and core/edge layers, yielding the 1,649,996-record dataset used to derive the three insights.","core_discovery":"The paper's central claim is that LLM services differ from traditional DNN applications in three fundamental ways: bidirectional heavy-load asymmetry, computation–communication coupling in slicing, and token-stream unpredictability. The evidence shows inference accounting for 74–87% of latency in uplink-heavy image requests and network transmission for 81–86% of latency in downlink-heavy image generation; slice-resource increases moving the dominant latency component from inference to transmission; and byte transmission varying non-linearly with controlled physical-resource-block allocation. These observations are presented as intrinsic to generative multimodal LLMs, not as artifacts of the testbed. On that basis, the paper proposes WiLLM, an open platform whose Tree-Branch-Fruit slicing, application-layer tunneling, dual-mode scheduling, and cross-layer APIs let non-experts test LLM-aware scheduling and slicing, backed by 1,649,996 synchronized records across 58 metrics.","pith_inferences":["If the three properties generalize across models and base stations, LLM wireless traffic should be modeled as a state-dependent generative process rather than by Markovian or fixed-bitrate models; the released dataset is directly usable to test which model class predicts observed scheduling failures.","The synchronized multi-layer records open a concrete machine-learning task the paper does not pursue: predicting next-interval PRB demand or detecting an imminent bottleneck migration from UE and RAN metrics.","The bidirectional asymmetry points to a design the paper motivates but only partially develops: slicing parameters that differ per direction, such as separate uplink and downlink resource guarantees for the same LLM slice.","A direct reproducibility check of the paper's central thesis would be to re-run its measurement protocol with a non-generative DNN workload on the same radios; the design claims stand or fall on that contrast, not on the platform itself."],"forward_implications":["Schedulers should become direction-aware, allocating resources according to which direction is currently bottleneck-bound rather than treating the service as a fixed uplink or downlink flow.","Network slice parameters need runtime reconfiguration, because the computation–communication bottleneck migrates as slice resources change.","Resource management must respond to token bursts in fine granularity, since average-based allocation misses deadline violations and peak provisioning wastes resources.","Researchers without wireless expertise can evaluate scheduling and slicing algorithms on realistic cellular hardware, and can compare them against the released dataset.","Placing LLM inference in the core network, rather than at the edge or in the cloud, gives operators coordinated control of communication and computation resources."],"supporting_citations":[{"why":"Supplies the open-source 5G protocol stack that WiLLM extends with LLM-aware slicing, scheduling, and cross-layer APIs.","marker":"[25]"},{"why":"Earlier work by the authors introducing dedicated LLM network slicing, the direct ancestor of the Tree-Branch-Fruit design.","marker":"[20]"},{"why":"Provides the multimodal LLaVA model used for the image-to-text workloads in the measurements.","marker":"[21]"},{"why":"Provides the Llama model family used for the text-generation workloads in the testbed.","marker":"[13]"},{"why":"Defines the static service-profile assumption of network slicing that the paper claims LLM traffic violates.","marker":"[2]"},{"why":"Supplies the ResNet-50 example of fixed-computation DNN inference used as the traditional baseline contrast.","marker":"[18]"},{"why":"Supplies the YOLO detector example of predictable DNN output used as the traditional baseline contrast.","marker":"[31]"}],"fun_headline_variants":["WiLLM: first open framework exposes LLM-wireless asymmetry","LLM token bursts shift bottlenecks; open 5G testbed shows","WiLLM: first open-source wireless testbed for LLM services","LLM traffic reality: bidirectional, bursty, bottleneck-shifting","Open WiLLM platform logs 1.6M records of LLM 5G traffic"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The three claimed LLM-specific properties are inferred from a single testbed configuration, with one base station, two model families, and no side-by-side measurement of a conventional DNN workload under the same conditions.","fun_headline_variants_meta":{"raw":{"variants":["WiLLM: first open framework exposes LLM-wireless asymmetry","LLM token bursts shift bottlenecks; open 5G testbed shows","WiLLM: first open-source wireless testbed for LLM services","LLM traffic reality: bidirectional, bursty, bottleneck-shifting","Open WiLLM platform logs 1.6M records of LLM 5G traffic"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001006,"raw_usage":{"total_tokens":4263,"prompt_tokens":963,"completion_tokens":3300,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":3200}},"tokens_in":579,"tokens_out":3300,"duration_ms":22858,"temperature":1.0,"reasoning_tokens":3200,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:38:20.428979+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same smart-glasses scenario on the same testbed with a conventional DNN task (for example, image classification or object detection) using matched image sizes and response sizes, then compare latency-component shares, slice-induced bottleneck movement, and byte-transmission burstiness. If the conventional workload reproduces the same asymmetry, migration, and burstiness, the paper's three fundamental distinctions are not LLM-specific.","supporting_citations":[{"cited_title":"Marina, Saravana Manickam, Alex Dawson, Raymond Knopp, and Christian Bonnet","cited_arxiv_id":null,"evidence_quote":"Supplies the open-source 5G protocol stack that WiLLM extends with LLM-aware slicing, scheduling, and cross-layer APIs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Llama model family used for the text-generation workloads in the testbed."},{"cited_title":"A Study on 5G Network Slice Isolation Based on Native Cloud and Edge Computing Tools","cited_arxiv_id":"2502.02842","evidence_quote":"Defines the static service-profile assumption of network slicing that the paper claims LLM traffic violates."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ResNet-50 example of fixed-computation DNN inference used as the traditional baseline contrast."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the YOLO detector example of predictable DNN output used as the traditional baseline contrast."}],"review_version":2}