{"id":"4264da76-dd88-42d6-8b7c-407c383203fc","arxiv_id":"2507.06499","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A DRL-based querying policy trained only on a single-parameter queue simulation transfers zero-shot to real WiFi (5-50 agents) and cellular networks and adapts its query rate to congestion.","lead":"The paper trains a reinforcement learning agent (QNet) to decide when to query an edge-cloud server over a shared wireless network, using only its own delay and age feedback, and shows the policy transfers from a one-parameter queue simulation to real WiFi and cellular networks without retraining.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'any number of agents' generalization claim is unsupported: QNet is trained in a single-agent queue with exogenous q, so the paper never tests the endogenous feedback created when many QNet agents share the network. A multi-agent simulation sweep would settle whether this gap is real.","rationale":"The reader's weakest_assumption identifies the same gap; my read agrees. The paper's empirical WiFi transfer is credible for the tested range, and I do not object to the sim-to-real approach itself. But the abstract and problem statement claim 'any number of other agents,' and the training simulator never contains more than one agent. The five WiFi operating points (5, 25, 30, 40, 50) provide only sparse interpolation; they cannot certify 'any number.' The multi-agent simulation test above would turn the extrapolation into a checkable claim. Other concerns (uncalibrated max-Q selection, missing significance tests, no released code) are real but secondary; they affect reproducibility and interpretation, whereas the multi-agent feedback gap directly targets the headline generalization claim. Verdict remains CONDITIONAL pending such a test.","tokens_in":21550,"tokens_out":7185,"duration_ms":84735,"concrete_test":"Build the paper's queue model with N agents, each executing the trained QNet policy, sharing one geometric(q) server, and sweep N over {5, 25, 50, 100, 200} and q over (0.01, 1). Measure per-agent mean estimation error, query rate, and age. If at N=100 or N=200 the per-agent error exceeds the single-agent baseline (or the N=50 result) by more than 20%, or if query rates oscillate or collapse, the 'any number' claim fails. This test isolates the endogenous feedback that the paper's single-agent training cannot capture.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim includes zero-shot generalization to 'any number of other agents' (Abstract; Section 3). The training simulator (Section 4, Figure 2) contains exactly one agent: the only arrivals are the agent's own queries, and all contention from other agents is collapsed into an exogenous per-episode service probability q. Thus the learned policy has never observed a scenario in which its own querying and that of other QNet agents jointly determine the congestion it experiences. In a real deployment where all agents run QNet, the aggregate query load is endogenous: if QNet queries less under perceived contention, contention falls, changing the very q the policy was trained against; if it queries more, contention rises. The WiFi experiments (Section 6) test N=5,25,30,40,50 and show good transfer, which is encouraging, but they do not test the feedback regime outside this range, and the abstract's 'any number' is an extrapolation. Moreover, q is defined inconsistently (service completion vs. successful transmission, Section 4), and because the simulator never drops packets, the policy is trained without permanent query loss, yet real PER reaches 60% (Figure 10b). The most load-bearing gap is therefore the missing multi-agent coupling: without it, the 'any number' claim has no mechanistic support and rests only on interpolation across the tested N values.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes QNet, a deep RL policy that decides, at each decision instant, whether an edge-device should query an edge-cloud over a shared wireless network whose state is unknown. QNet is trained exclusively on a low-fidelity single-server queue simulation with a single parameter q (geometric service time), using domain randomization over q in (0.05, 1). Three separate RL models are trained for disjoint q-ranges and are combined at deployment by taking the action with the maximum Q-value among the six Q-values output by the three critics. The resulting policy is evaluated zero-shot on real WiFi networks (ORBIT testbed, 5 to 50 agents) and on cellular traces replayed with Mahimahi, and it is compared against Always Query, threshold-based, and probabilistic querying baselines. The central empirical claim is that QNet transfers from simulation to real networks, adapts its query rate to unknown contention, and matches or outperforms the baselines except at very low contention (5 agents), where Always Query is better.","tokens_in":21781,"tokens_out":4460,"duration_ms":57498,"significance":"If the claims hold, the paper is a useful practical contribution: it demonstrates that a one-parameter queue model with domain randomization can produce a communication policy that transfers zero-shot to real WiFi and cellular networks, and it provides a head-to-head comparison with several baselines. The real-network experiments, especially the ORBIT WiFi experiments with 5 to 50 agents and packet error rates up to 60%, are a genuine strength, as is the explicit attempt to validate simulation-to-real transfer through overlapping scatter plots and age-binned error tables. The main limitations are that the abstract's 'any number of other agents' claim is extrapolated beyond the tested range, and the deployment-time combination of the three trained critics via an uncalibrated maximum over Q-values is not justified.","major_comments":[{"comment":"The deployment rule picks the action corresponding to the maximum of six Q-values produced by three critics trained on different q-ranges with different n-step returns (60, 20, 10) and different target entropies (0.09, 0.3, 0.6). Q-values from different SAC critics are not on a common scale; an arbitrary offset or scaling in one critic can dominate the maximum and make the selected action depend on training artifacts rather than on the current network state. The paper gives no justification for this max-over-critics rule and no ablation showing that it beats alternatives. Please either justify the calibration of the Q-values, replace the rule with a principled arbitration (for example, an estimate of q, a weighted average of the policies, or a hierarchical selector), or provide empirical evidence that the six-Q-value maximum is necessary and robust.","section":"Section 5.3, Figure 7"},{"comment":"The claim that the policy 'generalizes well to any number of other agents' is not supported by the training or the experiments. The simulation model of Section 4 contains exactly one agent: the only arrivals to the queue are the agent's own queries, and all contention from other agents is collapsed into the exogenous per-episode service parameter q. The WiFi experiments do run multiple QNet agents and show encouraging transfer for 5 to 50 agents, but that is interpolation over a tested range, not evidence for arbitrary N, and the training distribution never exposes the policy to the endogenous feedback in which its own querying and other QNet agents' querying jointly determine the congestion. A concrete test would be a multi-agent simulation sweep in which all agents use QNet and N is varied (for example, N = 100, 200), or training with N as part of the randomized domain. At minimum, the abstract and Section 3 should be rephrased to claim generalization over the tested range rather than 'any number'.","section":"Abstract, Section 3, Section 4"},{"comment":"The simulator never drops packets: every enqueued packet is eventually served, and the agent receives a response to every successful query. In the real WiFi experiments, however, the packet error rate reaches 60% (Figure 10b), meaning that a substantial fraction of queries or responses are permanently lost, an event class never experienced during training. Additionally, q is described both as the probability that a packet finishes service in a time slot and as the probability that a packet is successfully transmitted in an attempt (Section 4, paragraph beginning 'The probability q models...'), which are not the same quantity. The paper should clarify what q represents, state explicitly whether permanent query loss is part of the simulation model, and explain why the learned policy still performs well in a deployment regime with significant loss. Without this clarification, the mechanism behind the claimed sim-to-real transfer for high-contention WiFi scenarios is incomplete.","section":"Section 4, Figure 2, Section 6.2"},{"comment":"The cellular experiments use two agents, each with its own uplink and downlink traces replayed by Mahimahi. This setup models time-varying channel conditions, but the two agents do not actually contend for a shared network resource. Consequently, the cellular results validate transfer across a range of network conditions, but they do not test the shared-network aspect of the problem, and they should not be presented as evidence for multi-agent generalization. Please state this limitation explicitly and, if possible, add a cellular or emulated experiment in which agents share a bottleneck.","section":"Section 7.1"}],"minor_comments":[{"comment":"The text 'Ground Trurth' in Figure 2 is a typo and should read 'Ground Truth'.","section":"Figure 2"},{"comment":"The sentence 'Varying the number of agents has us observe close to no retires to a significant fraction of retries' contains a typo ('retires' should be 'retries') and would benefit from rewording.","section":"Section 6.2"},{"comment":"The table reports means and standard deviations but does not state the number of repeated experiments or runs behind each entry, and no statistical significance tests are provided. Since many reported differences are within one standard deviation (for example, the 25-agent row, where QNet and Always Query both report 1.3), the claim that QNet 'does at least as well or better' would be strengthened by confidence intervals or pairwise tests.","section":"Section 6.4, Table 2"},{"comment":"The claim that simulated and real scatter plots 'overlap quite well' is qualitative; please add a quantitative agreement metric (for example, root-mean-square difference or a correlation coefficient) and report the number of simulation episodes and real experiments used in each age bin.","section":"Figures 9 and 12, Tables 1 and 3"},{"comment":"The reward scaling parameters (r_s = 5, r = 8e4), the per-range n-step values, and the target entropy values are stated but no sensitivity analysis is provided. Since these are central to making training work, a brief ablation or a statement of how they were selected would help reproducibility.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a systems-oriented empirical study rather than a theoretical contribution, and its main value is in the real-network validation. The most important fix is the deployment rule in Section 5.3: as written, taking the maximum over six uncalibrated Q-values is not a defensible action-selection method and could undermine the reported results if the choice of critic is not robust. The 'any number of agents' claim in the abstract should also be softened unless a multi-agent simulation sweep is added. The cellular experiments do not actually test shared-network contention, so their role in the paper's narrative should be recalibrated. I do not see grounds for rejection; the empirical core is plausible and the requested changes are within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a useful empirical paper on training a single-agent DRL query policy in a one-parameter queue and transferring it zero-shot to real WiFi and cellular networks. It is the first demonstration I know of for this specific transfer, and the WiFi results with up to 50 agents are credible. It is not a method that changes how networks are designed, but it is a solid engineering contribution for edge-cloud robotics.\n\nWhat is new: the simulation model is a single FCFS queue with geometric service time parameter q, randomized over (0.05,1). The agent learns when to query a cloud based on its own age and estimate. Deployment uses three models trained for different q ranges, choosing action by max Q-value across all six outputs. The real-world validation is the strong part: ORBIT WiFi with 5 to 50 agents all running QNet, plus Mahimahi replay of 4G traces, and the simulated-vs-real scatter overlap looks believable. The paper also does a fair comparison to Always Query, threshold policies, and probabilistic policies, and honestly shows Always Query wins at 5 agents.\n\nSoft spots: the abstract's 'any number of agents' is an extrapolation beyond the tested range. Training has only one agent; contention is exogenous through q. The real experiments do include multiple QNet agents, so feedback is present up to 50 agents, but the claim of arbitrary numbers is not supported. The max-Q selection across three uncalibrated models is ad hoc; the q definition slips between service-completion probability and transmission-success probability; there are no significance tests and no code/data release. None of these sink the central empirical claim, but they should be fixed in revision. The missing-code issue matters for reproduction.\n\nBottom line: this deserves peer review. It is a genuine first result, the experiments are real, and the limitations are fixable. The authors should tone down the 'any number' claim, justify or replace the max-Q selection, and release code and data.\n\nFor your reading group: maybe, if you work on wireless MARL or sim-to-real. I would cite it.","headline":"Useful first sim-to-real result for DRL query control over wireless: credible WiFi/cellular transfer but the 'any number of agents' claim and uncalibrated max-Q selection need fixing.","tokens_in":22383,"tokens_out":2972,"would_cite":true,"duration_ms":35154,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Trained only on a one-parameter queue simulation, the query policy QNet transfers zero-shot to real WiFi and cellular networks and matches or beats fixed policies under contention.","keywords":["multi-agent reinforcement learning","deep reinforcement learning","query control","edge-cloud communication","age of information","sim-to-real transfer","domain randomization","wireless networks"],"falsifier":"Deploy QNet on a network with enough agents that the agents' own querying measurably slows the network, for example a WiFi testbed with 100 agents or with cross-traffic that reacts to query load, and compare the measured average-age versus estimation-error scatter with the simulation envelope. If the scatter leaves the envelope at a fixed apparent q, or if the policy's query rate fails to track rising contention, the single-parameter sufficiency assumption is falsified.","tokens_in":21301,"feed_emoji":"📶","tokens_out":11841,"duration_ms":124598,"temperature":0.7,"pith_summary":"The paper asks when an agent sharing a wireless network with an unknown number of other agents should query its edge-cloud for fresh information. Its answer is a deep reinforcement learning policy, Query Net (QNet), that decides whether to query from the age of its latest received measurement and its current estimate of the source. The policy is trained entirely on a deliberately simple simulation in which the network and edge-cloud form a single first-come-first-served queue whose service time is geometric with one parameter q, randomized over (0.05, 1) during training. The central empirical claim is that this simulation-only policy transfers zero-shot to real WiFi networks with 5 to 50 agents and to real cellular traces. Under contention it matches or beats Always Query, threshold, and probabilistic baselines while querying less; only in very lightly loaded networks does Always Query win.","feed_headline":"One-parameter queue trains query policy that transfers to real WiFi","feed_subtitle":"The policy queries less under congestion and matches baselines on real WiFi and cellular.","key_machinery":"The load-bearing object is the single-parameter service-facility simulator: the wireless network plus the edge-cloud is one first-come-first-served queue, and a packet in service finishes in a given slot with probability q, so service times are geometric(q). That one number q stands in for all other traffic, packet errors, and congestion, and randomizing it over (0.05, 1) exposes training to conditions from near-idle to heavily queued. Around this simulator sit three QNet instances, each with an estimator network that forms the agent's estimate from received measurements and an actor-critic pair that sees the estimate plus the age of the most recent measurement; deployment takes the maximum Q-value across the three critics. The age signal carries the argument across the simulation-to-real gap: it is a technology-agnostic summary of how the shared network treats the agent's own queries, and it is the only network information the policy is allowed to use.","core_discovery":"QNet is a three-network deep RL pipeline: an estimator reconstructs the source state from sparse delayed measurements, and actor-critic networks score Query versus Don't Query given the current estimate and the age of the newest measurement. Because a single policy trained over the full q range fails to vary its query rate with network speed, the paper trains one QNet per q range, (0.05, 0.1), (0.1, 0.3), and (0.3, 1.0), and at deployment selects the action with the largest Q-value among the three critics. The core finding is that this simulator-trained policy generalizes to real networks without fine-tuning: on a real WiFi testbed the average-age versus estimation-error scatter overlaps the simulation scatter, and as contention rises the policy lowers its query rate. Concretely, at 50 agents QNet reports mean estimation error 25.0 with standard deviation 2.3, versus 31.3 with standard deviation 9.2 for Always Query, while at 5 agents Always Query reports 0.6 versus QNet's 1.0. On cellular traces the same pattern holds: QNet is competitive in good conditions and has lower mean error than Always Query when baseline round-trip times exceed 0.2 seconds.","pith_inferences":["One extension the paper leaves implicit is to let q vary within an episode or depend on recent query load, modeling non-stationary contention and congestion rather than a fixed per-episode service rate.","Because the policy's input is an age and a learned estimate rather than raw measurements, the same decision layer may transfer across source processes with similar estimation-error dynamics if the estimator is retrained for the new source.","The three-model max-Q selection suggests a probabilistic generalization: instead of three discrete q ranges, one could maintain a belief over q from observed delays and compute query values by posterior-weighted averaging, making the same idea work continuously."],"forward_implications":["An agent can be deployed on an unfamiliar shared network with no per-network training and no knowledge of how many other agents are present; the same simulator-trained model is used without modification on WiFi and cellular.","At 25 WiFi agents, Always Query and QNet have similar mean estimation error but Always Query queries about 1.5 times as often, so QNet achieves equal tracking accuracy with a materially lower load on the network.","Fixed threshold and probabilistic policies require choosing a threshold or scaling per agent count, whereas QNet adapts its query rate automatically as contention rises.","The overlap between simulated and real age-versus-error scatter suggests the low-fidelity queue can serve as a cheap screening tool: configurations whose average age and error fall inside the simulated envelope are expected to transfer to real networks."],"supporting_citations":[{"why":"supplies the soft actor-critic algorithm used to train the policy.","marker":"[14]"},{"why":"supplies the automatic temperature-tuning rule used in training.","marker":"[15]"},{"why":"supplies the discrete-action formulation that lets the actor output a probability distribution over Query and Don't Query.","marker":"[8]"},{"why":"supplies the domain randomization method used to vary q during training and bridge simulation and reality.","marker":"[31]"},{"why":"supplies the vehicle-state traces used as the agents' sources and as ground truth in experiments.","marker":"[25]"},{"why":"supplies the network emulator used to replay real cellular traces in the evaluation.","marker":"[30]"},{"why":"supplies one set of real 4G cellular traces used in the cellular experiments.","marker":"[1]"},{"why":"supplies another set of real 4G cellular traces used in the cellular experiments.","marker":"[41]"},{"why":"supplies the model-based threshold communication policy the paper adapts as a baseline.","marker":"[16]"}],"fun_headline_variants":["Deep RL learns when to talk on shared networks, transfers to real WiFi","Simulation-trained query policy reduces communication under contention","One-parameter sim trains QNet to query less under real congestion","RL agent uses sparse feedback to decide when to communicate on shared nets","Policy trained on abstract queue generalizes to real WiFi and cellular"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole transfer claim rests on assuming that the effect of every other user and channel condition can be summarized by one number, the per-slot probability that a packet finishes service, and that this number does not change as a result of how often agents query.","fun_headline_variants_meta":{"raw":{"variants":["Deep RL learns when to talk on shared networks, transfers to real WiFi","Simulation-trained query policy reduces communication under contention","One-parameter sim trains QNet to query less under real congestion","RL agent uses sparse feedback to decide when to communicate on shared nets","Policy trained on abstract queue generalizes to real WiFi and cellular"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000283,"raw_usage":{"total_tokens":1761,"prompt_tokens":1123,"completion_tokens":638,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":739,"completion_tokens_details":{"reasoning_tokens":551}},"tokens_in":739,"tokens_out":638,"duration_ms":7013,"temperature":1.0,"reasoning_tokens":551,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:03:27.415520+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Deploy QNet on a network with enough agents that the agents' own querying measurably slows the network, for example a WiFi testbed with 100 agents or with cross-traffic that reacts to query load, and compare the measured average-age versus estimation-error scatter with the simulation envelope. If the scatter leaves the envelope at a fixed apparent q, or if the policy's query rate fails to track rising contention, the single-parameter sufficiency assumption is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the soft actor-critic algorithm used to train the policy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the domain randomization method used to vary q during training and bridge simulation and reality."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the vehicle-state traces used as the agents' sources and as ground truth in experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the network emulator used to replay real cellular traces in the evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies another set of real 4G cellular traces used in the cellular experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the model-based threshold communication policy the paper adapts as a baseline."}],"review_version":1}