{"id":"58113f9f-6d72-4aa2-91d3-0d196eff1942","arxiv_id":"2505.04174","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"An on-device LLM using context prompts and quantization can adjust Wi-Fi roaming thresholds to balance handover frequency and signal quality better than fixed-rule and PPO baselines in small indoor and outdoor tests.","lead":"This paper puts a small language model on a laptop to make Wi-Fi roaming decisions, using prompts with signal strength, location, and time to pick an access point or adjust when to switch. It works in the authors' indoor and outdoor tests, but the model is too slow for the most time-sensitive roaming step.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own latency measurements (Sec. 3.5, Fig. 5) contradict the 'real-time actions in the PHY/MAC stack' headline; only the 30-second threshold-adjustment task is actually demonstrated on-device.","rationale":"The paper intends to demonstrate an on-device LLM performing context-aware Wi-Fi roaming with real-time PHY/MAC actions. For the central claim to hold, the LLM must make decisions within the 10–100 ms near-real-time budget stated in the introduction. The paper's own Sec. 3.5 and Fig. 5 show inference times of seconds on an A100 GPU, which is worse on-device, so this condition clearly fails for Task 1. The surviving demonstration is Task 2, where the threshold is updated every 30 seconds; that cadence tolerates seconds-scale inference and is supported by the on-device MacBook demo in Appendix A. The reader identified exactly this latency contradiction as the weakest assumption, and I agree that it is load-bearing because it invalidates the headline 'real-time actions' rather than merely weakening a secondary metric. I credit the authors for explicitly disclosing the latency limitation and for providing a demo video and code repository for the threshold-adjustment task, which lends credibility to the narrower claim. The paper can be salvaged by reframing the abstract and contributions to separate the demonstrated 30-second threshold adaptation from the unrealized near-real-time AP-selection goal, and by releasing data and code for statistical validation. Therefore the reader's CONDITIONAL verdict remains appropriate; my read does not change it.","tokens_in":10733,"tokens_out":5129,"duration_ms":50800,"concrete_test":"Run the released llama.cpp Q2K Llama-3.1-8B model (Table 7, Sec. C) on the same MacBook M3 used in the demo and measure end-to-end latency for Task 1 BSSID-selection prompts sampled from the test set. If the median per-decision latency exceeds 100 ms, the abstract's 'real-time actions' claim for Task 1 is falsified. Report the latency distribution and compare it with Fig. 5.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Abstract; Sec. 1) is that an on-device LLM issues 'real-time actions executed in the PHY/MAC stack.' The paper itself falsifies this for Task 1: Sec. 3.5 states the STA must decide within 10–100 ms, but Fig. 5 reports inference times 'on the order of seconds' on an A100 GPU that 'far exceed' the requirement. Since the Sec. 3.4 evaluation uses an A100 (Fig. 5) and the on-device demo in Appendix A is explicitly only for Task 2, no evidence supports a 10–100 ms on-device BSSID-selection loop. Task 2 (Sec. 4) is a threshold update every 30 s, a slow supervisory loop, not a real-time lower-layer action. The abstract's 'real-time actions' therefore overstate what is demonstrated; the surviving claim is a context-driven 30-second threshold adjustment on a MacBook. This limitation is acknowledged in Sec. 3.5 but is not reconciled with the abstract and introduction.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes using a compressed on-device LLM (Llama-3.1-8B with Q2K quantization, 3.2 GB) as a Wi-Fi roaming controller. It formulates two tasks: (T1) selecting the best BSSID from structured context (RSSI, location, time) and (T2) periodically adjusting the scanRSSI roaming threshold every 30 seconds on a MacBook. The authors combine chain-of-thought prompting, LoRA fine-tuning, DPO/ORPO preference optimization, and quantization, and compare against legacy heuristics, opt-HO/opt-RSSI oracles, and PPO on self-collected indoor/outdoor data, reporting that the LLM achieves a better #HO/AvgRSSI trade-off than heuristics and PPO. The paper includes an honest limitations section acknowledging that LLM inference latency is on the order of seconds on an A100 GPU, far above the 10–100 ms roaming budget, but the abstract and introduction nevertheless claim 'real-time actions executed in the PHY/MAC stack.' The demonstrated, narrower contribution is a context-driven threshold-adjustment loop, with the on-device demo in Appendix A covering only Task 2.","tokens_in":10937,"tokens_out":5534,"duration_ms":53652,"significance":"If the empirical claims held, the contribution would be notable as an early demonstration of application-layer LLM control of a lower-layer wireless parameter, with a credible edge-deployment pipeline and a clear comparison framework. The authors deserve credit for releasing code and a demonstration video, for including an explicit limitations section, and for separating the slow threshold-adjustment task from the harder AP-selection task. However, the current evidence does not support the headline real-time cross-layer claim: the paper's own latency measurements contradict it, the on-device demonstration is qualitative rather than quantitative, and most quantitative conclusions rest on single runs without variance information. The contribution is better framed as a feasibility study of a 30-second supervisory threshold-adjustment loop, not as real-time PHY/MAC control.","major_comments":[{"comment":"The central claim that the on-device LLM issues 'real-time actions executed in the PHY/MAC stack' is contradicted by the paper's own latency data: §3.5 states that Task 1 decisions must be made within 10–100 ms, while Fig. 5 reports inference times on the order of seconds on an A100 GPU and says these 'far exceed' the requirement. Since no on-device latency measurement for Task 1 is provided anywhere in the paper, the load-bearing claim of real-time AP selection is unsupported and should be removed or explicitly replaced by a Task-2-only claim.","section":"Abstract; §1; §3.5; Fig. 5"},{"comment":"Table 2's 'accuracy' is the fraction of LLM-selected APs that match opt-HO labels, where opt-HO is generated by exhaustive search minimizing #HO over the same test sequence used for evaluation. Because the fine-tuning objective is likewise handover minimization, the reported 85% agreement partially measures the model's ability to reproduce the authors' own label-generation rule rather than an independent ground truth. The paper should also report agreement with opt-RSSI labels or with a separately labeled human-preference dataset, and should report accuracy with multiple seeds and confidence intervals.","section":"§3.4; Table 2"},{"comment":"All quantitative results in Figs. 4, 6, 7 and Tables 2–6 are single point estimates with no error bars, no repeated trials, and no seeds, despite the fact that LLM generation is stochastic and LoRA fine-tuning involves random initialization. The claimed improvements over PPO (e.g., 93 vs. 107 handovers in Fig. 4) are small relative to the variability expected from such runs and cannot be assessed without statistical support. The dataset is also self-collected and not released, which further limits reproducibility.","section":"§3.4; §4.3; Fig. 4; Fig. 7; Tables 3–6"},{"comment":"The on-device demonstration in Appendix A covers only Task 2 and reports no quantitative latency, throughput, memory, or power measurements; the statement that 'comprehensive performance metrics confirm the feasibility' is not supported by the presented evidence. Fig. 5's A100 measurements cannot serve as a proxy because the MacBook M3 has different performance characteristics, so the paper should provide a measured on-device inference-time distribution and resource-utilization numbers for the 30-second loop before claiming practical feasibility.","section":"Appendix A; §4.3; §5 Q3"}],"minor_comments":[{"comment":"The paragraph beginning 'Effect of Quantization' contains an incomplete/typo-laden opening: 'Q4 K MTable 4 compares...' should be 'Table 4 compares...'.","section":"§4.3"},{"comment":"The phrase 'opt-HO-global optimal (exahasutive) results' contains a typo ('exahasutive' should be 'exhaustive'), and 'textsfLegacy' appears as a format artifact in §3.4.","section":"§3.4; Table 2"},{"comment":"The caption should specify that the latency plot is for the A100 GPU only and should explicitly note that no on-device latency is reported elsewhere.","section":"Fig. 5 caption"},{"comment":"The statement that SFT+DPO 'lowers the error rate' should make the comparison base explicit: it lowers the error rate relative to No FT and SFT, but ORPO has a higher error rate (33.00%), which the text does acknowledge.","section":"Table 3"},{"comment":"The claim that the LLM's reasoning is 'validated through the subsequent successful handover' conflates a single successful handover with correctness of the reasoning; this should be framed as an anecdotal illustration rather than validation.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's advertised novelty ('first cross-layer use of an on-device LLM') is difficult to verify without a broader prior-art search, and the mismatch between the abstract and the limitations section will be a concern for readers. If the authors can reframe the contribution as a Task-2-only feasibility study and add statistical and on-device measurements, the paper could be viable for a systems-oriented venue. The paper's fit with ICML's core ML scope is borderline; the value lies in the systems demonstration rather than in a new ML method."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know two things about this one. The useful half is the on-device demo, not the real-time claim. The paper overreaches in its abstract and introduction; if you read only Section 4 and Appendix A, you get a plausible result: a compressed 8B LLM, Q2K-quantized, runs on a MacBook and adjusts the Wi-Fi roaming threshold every 30 seconds with reasonable effect. If you read Section 3, you get a BSSID-selection task that the authors themselves admit cannot meet the 10–100 ms budget—their Fig 5 shows seconds on an A100.\n\nWhat is genuinely new is the application: as far as I can tell from their references, this is the first on-device LLM pointed at PHY/MAC roaming. The engineering pipeline is real: CoT, LoRA, DPO, ORPO, quantization, and an actual on-device demonstration with code on GitHub. They are also honest about the latency problem in Section 3.5, which is more than many such papers do.\n\nThe soft spots are real and proportionate. All performance numbers are single-run point estimates; no error bars, no seeds. The dataset is self-collected and not public. Table 2's 'accuracy' is agreement with opt-HO labels that are generated by the same #HO objective used to train and evaluate the model, so high agreement is not an external validation. The 'first' novelty claim is plausible within their reference list but not against a broad literature search. Even within the paper, the context ablation (Table 6) shows that adding all context features slightly hurts performance—that is fine, but it undercuts the 'context-aware' spin.\n\nThe bottom line: the narrow result—a slow, on-device threshold adjustment loop that beats fixed thresholds across varied scenarios—is credible. The headline 'real-time actions in the PHY/MAC stack' is not. This is a legitimate application-paper idea that needs a major rewrite to match the evidence.\n\nFor peer review: send it out. A serious referee can sort the claims, demand error bars and a public dataset, and push the authors to reframe the latency story. The paper deserves that engagement, but the current version would need heavy revision before publication.","headline":"The on-device threshold-adjustment demo is a credible modest result, but the 'real-time actions in the PHY/MAC stack' headline is contradicted by the paper's own latency numbers.","tokens_in":11501,"tokens_out":2522,"would_cite":false,"duration_ms":25067,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims the first on-device LLM that reasons in the application layer and issues actions in the PHY/MAC stack, using a 3.2 GB quantized Llama-3.1-8B to balance Wi-Fi roaming stability and signal quality better than fixed…","keywords":["on-device LLM","Wi-Fi roaming","cross-layer wireless control","dynamic threshold adjustment","BSSID selection","quantization","LoRA fine-tuning","chain-of-thought prompting"],"falsifier":"Measure the end-to-end latency on the target edge device, not a server GPU, from RSSI scan to LLM output for Task 1; if the median latency exceeds 100 ms under realistic load, the claim of real-time PHY/MAC actions is unsupported. Additionally, run a no-context version of the same prompt on the same test data: if the context-augmented prompt does not consistently beat the no-context baseline on the handover/RSSI tradeoff across indoor and outdoor environments, the claimed benefit of contextual reasoning fails.","tokens_in":10503,"feed_emoji":"📶","tokens_out":5065,"duration_ms":48644,"temperature":0.7,"pith_summary":"The paper tries to establish that a compressed on-device large language model, running at the application layer, can manage Wi-Fi roaming decisions that normally live in the PHY/MAC stack, replacing fixed RSSI thresholds and heuristic handover rules. Concretely, it claims the LLM can select the best access point (BSSID) from structured context prompts (Task 1) and can periodically adjust the roaming trigger threshold on-device (Task 2), and that on indoor and outdoor logs this beats legacy heuristics and a PPO baseline on the roaming-stability/signal-quality tradeoff. The realistic core is the second task, because the paper itself reports LLM inference taking seconds versus the 10–100 ms budget for AP selection, so the near-real-time claim in Task 1 is presented as a future prospect rather than a demonstrated achievement. If the paper is right, edge devices can carry an LLM that reads context such as location and time and tunes lower-layer wireless parameters without retraining per environment.","feed_headline":"On-device LLM tunes Wi-Fi roaming better than fixed rules","feed_subtitle":"A 3 GB quantized model reads RSSI, location, and time to balance signal strength against handovers on a laptop.","key_machinery":"The load-bearing mechanism is the structured prompt as the interface between lower-layer Wi-Fi measurements and the LLM's application-layer reasoning. Chain-of-thought prompting with a few labeled examples turns raw RSSI-BSSID scans plus situational context into an interpretable decision trace; LoRA fine-tuning adapts the pretrained model to Wi-Fi-specific cues; DPO or ORPO aligns decisions with preference data; and Q2K quantization compresses the model to 2.56 bits per parameter, fitting the 16 GB unified memory of the test laptop. The control loop runs the LLM every 30 seconds to set the roaming threshold, so the fast per-scan decisions remain rule-based while the slow, context-aware reasoning loop adjusts the rules.","core_discovery":"In the paper's own terms, the discovery is that a quantized Llama-3.1-8B LLM, trimmed to 3.2 GB by Q2K quantization, can run on a consumer-class Apple M-series laptop and, given structured prompts of RSSI values, location, time, and battery state, make roaming decisions that balance average signal strength against handover count better than fixed-threshold legacy logic, a random heuristic, and a PPO agent. For Task 1 (choosing which BSSID to roam to), the LLM reduced handovers to 93 versus 100 for Legacy and 107 for PPO while keeping average RSSI close to Legacy, though it did not match the global optimum. For Task 2 (when to roam), the LLM adjusts the roaming threshold every 30 seconds and outperforms any single fixed threshold across indoor and outdoor scenarios, with Q2K quantization cutting model size by about 2.7 times relative to Q8 with negligible loss in decision quality. The paper explicitly acknowledges that near-real-time BSSID selection within 10–100 ms is not met by current inference times, and positions the demonstrated on-device use case as the slower threshold-adjustment task.","pith_inferences":["If future edge AI accelerators close the 10–100 ms latency gap, the same prompt-and-quantization pipeline could be applied to other PHY/MAC control loops with natural slow cadences, such as beam management in mmWave systems or energy-saving mode selection in IoT devices, which the paper does not discuss.","The 30-second threshold cadence suggests a broader design pattern: pairing a slow, context-aware LLM with a fast, deterministic feedback loop. A testable extension would be to vary the cadence with mobility speed and measure where the LLM's benefit over a fixed threshold disappears.","The paper's context ablation shows that adding all available context is not always optimal (146 handovers with all context versus 142 with Time plus Battery), which implies that prompt feature selection, not just model capacity, drives performance; an editor-level inference is that attentional or learned feature gating could further improve the results.","If the Q2K result holds at scale, 2-bit quantization on edge LLMs could make 'LLM as MAC configurator' a realistic product pattern even before LLM inference itself becomes millisecond-fast, because the slow control loop hides the inference latency."],"forward_implications":["If a 3.2 GB on-device LLM can tune roaming thresholds, then similar slow-cadence wireless control tasks, such as channel selection or power-save policy, become candidates for on-device LLM reasoning without cloud connectivity.","The 30-second threshold-adjustment cadence generalizes into a two-timescale architecture: a slow LLM reasoning loop that periodically reconfigures a fast rule-based control loop, which could extend to other protocol parameters beyond roaming.","Because the LLM can absorb new context signals by extending the prompt format rather than retraining, the approach promises easier adaptation to new buildings, mobility patterns, or user preferences compared with DRL agents that need environment-specific retraining.","The paper's own latency measurements imply that for near-real-time lower-layer decisions, on-device LLMs are not yet ready, so the realistic near-term impact is on control tasks with decision intervals of tens of seconds or more.","The interpretable chain-of-thought reasoning output could provide a diagnostic trace for network operators, showing why a particular handover or threshold change was made."],"supporting_citations":[{"why":"Supplies the Llama-3.1-8B base model used as the backbone in both tasks and in the quantization and model-comparison experiments.","marker":"(Grattafiori et al., 2024)"},{"why":"LoRA is the parameter-efficient fine-tuning method that reaches about 85% accuracy while using only about 20% of the VRAM and 30% of the GPU-hours of full training.","marker":"(Hu et al., 2022)"},{"why":"DPO is the preference-optimization step that lowers the invalid-selection error rate from 22.83% to 12.66% when combined with supervised fine-tuning.","marker":"(Rafailov et al., 2023)"},{"why":"Chain-of-thought prompting is the technique shown to reduce handover count and improve average RSSI in the few-shot evaluation.","marker":"(Wei et al., 2022)"},{"why":"PPO is the deep-reinforcement-learning baseline for AP selection that the LLM surpasses in both handover count and average RSSI.","marker":"(Schulman et al., 2017)"},{"why":"Phi-4-14B serves as the larger-model comparison point in the threshold-adjustment task, showing higher RSSI but more handovers and longer inference time.","marker":"(Abdin et al., 2024)"},{"why":"ORPO is the alternative preference-optimization method compared in Table 3, which cuts handovers further but raises the error rate to 33%.","marker":"(Hong et al., 2024)"}],"fun_headline_variants":["LLM on your laptop decides Wi-Fi roaming smarter than rules","Quantized LLM beats heuristics for Wi-Fi roaming on-device","Context-aware LLM Wi-Fi roaming: signal vs handoffs tuned","3GB LLM improves Wi-Fi roaming decisions on edge hardware","AI roaming: on-device LLM balances signal and handovers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's central 'real-time' claim rests on the premise that an on-device LLM can make a BSSID-selection decision within the 10–100 ms roaming budget, and the paper's own latency measurements (seconds on an A100 GPU) show that premise is not yet met.","fun_headline_variants_meta":{"raw":{"variants":["LLM on your laptop decides Wi-Fi roaming smarter than rules","Quantized LLM beats heuristics for Wi-Fi roaming on-device","Context-aware LLM Wi-Fi roaming: signal vs handoffs tuned","3GB LLM improves Wi-Fi roaming decisions on edge hardware","AI roaming: on-device LLM balances signal and handovers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000221,"raw_usage":{"total_tokens":1474,"prompt_tokens":992,"completion_tokens":482,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":608,"completion_tokens_details":{"reasoning_tokens":392}},"tokens_in":608,"tokens_out":482,"duration_ms":4642,"temperature":1.0,"reasoning_tokens":392,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:35:44.596147+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the end-to-end latency on the target edge device, not a server GPU, from RSSI scan to LLM output for Task 1; if the median latency exceeds 100 ms under realistic load, the claim of real-time PHY/MAC actions is unsupported. Additionally, run a no-context version of the same prompt on the same test data: if the context-augmented prompt does not consistently beat the no-context baseline on the handover/RSSI tradeoff across indoor and outdoor environments, the claimed benefit of contextual reasoning fails.","supporting_citations":[],"review_version":1}