{"id":"6a849b25-e933-438b-949a-226d3ef87748","arxiv_id":"1908.03984","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An online Q-learning scheme with channel-model-based Q-table initialization improves UAV sum-rate throughput and convergence speed for mobile NOMA users.","lead":"A drone learns in real time where to fly so that several ground users can upload data to it at the highest combined speed. The authors show that starting the drone's learning process with a rough radio-channel model makes it learn faster.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Location-only Q-learning has no fixed reward target in a non-stationary environment, so the claimed online tracking ability is not yet established.","rationale":"I agree with the reader's identified weakest assumption. The paper's central positive evidence is qualitative convergence and tracking plots; there is no theoretical support for Q-learning convergence on a non-stationary MDP, and the location-only state excludes exactly the information that changes as users move. This is the most load-bearing concern because it directly affects whether the method learns an online maneuver policy or merely an average reward map for a quasi-static environment. The concern is addressable: a restricted mobility class can be stated explicitly, or the state can be augmented with user-location estimates, and multi-seed statistics can bound the variability. I also considered the mismatched heuristic baseline as an alternative concern, but it affects the magnitude of the claimed improvement rather than whether learning occurs. Since the reader already made acceptance conditional on acknowledging non-stationarity and adding statistics, my pass does not change the verdict. No code or data are released, but that is a reproducibility limitation rather than a correctness flaw.","tokens_in":10756,"tokens_out":4236,"duration_ms":52440,"concrete_test":"Re-run the Section IV scenario with the original trajectory and with a random-walk mobility model, over at least 20 independent channel/fading seeds, keeping all RL hyperparameters fixed. For each seed, record the slot at which the moving-average throughput first stays within 2% of its N=10000 value, and report the mean and standard deviation of throughput at N=5000 and N=10000 for the RL, enhanced RL, and heuristic baselines. If under random-walk mobility the location-only RL design no longer beats the heuristic by a statistically significant margin, or if the convergence slot varies by more than a few thousand slots across seeds, the non-stationary state omission is load-bearing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Sec. III-A defines the state space as S={q[n]}, the action space as A=~A, and the reward as the instantaneous sum-rate R[n] from Eq. (1). That reward depends on h_k[n], which is driven by the users' time-varying positions and random fading, neither of which is included in the state. The Bellman update in Eq. (6) is therefore applied to a non-stationary reward process, not to the fixed MDP {S,A,P,R} introduced in the same section: P is deterministic on q alone, but R_a(s,s') is not a function of s alone. The paper offers no convergence argument for this case; the comment in Sec. III-A that learning parameters should be chosen properly is the standard stationary-MDP caveat. The 'convergence' in Fig. 2 and the tracking shown in Fig. 3 are empirical observations on one fixed, slow, closed user-trajectory set. The central claim that the UAV can learn an online maneuver policy for mobile users under causal CSI is therefore not yet supported: what may be learned is a time-averaged map from locations to rewards that works when user geometry repeats or drifts slowly. If user mobility is arbitrary, the location-only state cannot represent the dependence of the optimal action on user positions.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies a UAV-enabled uplink NOMA system with mobile ground users. The UAV only knows users' locations and CSI causally. The objective (P1) is to maximize the average sum-rate over a finite horizon by choosing the UAV's horizontal displacement at each slot from five options. The authors propose a Q-learning solution whose state is the UAV's grid location and whose reward is the instantaneous sum-rate from Eq. (1). To speed convergence, they add an offline pretraining stage that initializes the Q-table using rewards predicted from a probabilistic LoS average channel model. Section IV reports numerical comparisons against a heuristic average-channel follower, claiming significant throughput gains and faster convergence for the enhanced variant.","tokens_in":10968,"tokens_out":12438,"duration_ms":130805,"significance":"The paper's core idea—using a crude wireless channel model to seed Q-learning for UAV maneuver design—is interesting and potentially useful, and the sum-rate expression in Eq. (1) is correctly derived for the NOMA MAC. However, the central claim is not yet established. The state space omits the time-varying user positions and CSI that drive the reward, so the Q-learning is applied to a non-stationary process with no convergence guarantee; the numerical results are single-run curves that do not provide statistical support; and the comparison baseline is mismatched to the simulated channel. With a corrected formulation and stronger evaluation, the work could make a moderate contribution to online UAV trajectory design.","major_comments":[{"comment":"The MDP used for Q-learning is not well posed. The state space is S={q[n]}, containing only the UAV's location, while the reward r_n=\\tilde{R}[n] depends on the user positions {w_k[n]} and channel gains {h_k[n]} from Eq. (1), which are time-varying and absent from the state. Therefore R_a(s,s') is not a function of the state/action pair as required by the MDP {S,A,P,R}, and the Bellman update in Eq. (6) is applied to a non-stationary reward process. The statement in Section III-A that α and γ should be chosen properly is the standard stationary-MDP condition and does not cover this case. The empirical 'convergence' in Fig. 2 is measured on one fixed slow user-trajectory set, so the paper does not establish that a location-only Q-table can learn an online maneuver policy for arbitrary mobile users. Please either include the user geometry in the state (with a tractable discretization), formulate the problem as a contextual MDP, or provide a convergence/stability argument under explicit slow-mobility assumptions.","section":"Section III-A, Eq. (6)"},{"comment":"The description of the reward relative to the action is inconsistent. The text says that after choosing a_n=λ[n] and moving to q[n+1], the UAV receives r_n=\\tilde{R}[n], but \\tilde{R}[n] is determined by q[n] and the channel at slot n, before the action's effect takes place. As written, the reward does not depend on the action taken at step n, so the Q-update in Eq. (6) would reinforce an action with a reward that is not a consequence of that action. Please correct the time indexing (e.g., r_n=\\tilde{R}[n+1], or redefine the state/action timing) and confirm that the simulation uses the corrected indexing.","section":"Section III-A, reward timing"},{"comment":"The numerical evaluation is based on single runs with no multiple seeds or confidence intervals. Q-learning with ε-greedy exploration and random fading/shadowing is stochastic, so the reported curves do not establish statistical significance. Moreover, the implementation in Section III-A appears to be a single pass n=1,...,N over one flight, not repeated episodes; Q-learning convergence requires each state-action pair to be visited sufficiently often (or an appropriate online step-size schedule), and a one-trajectory run cannot substantiate the phrase 'almost converge at N>4000.' Please provide multi-seed statistics, define the plotted metric precisely, and clarify whether the Q-table is updated once per slot in a single episode or over repeated episodes.","section":"Section IV, Figs. 2 and 4"},{"comment":"The comparison baseline 'Probabilistic LoS channel-based heuristic design' assumes the average channel model \\bar{h}_k[n] from Eq. (7) and solves P2.n, while the simulated environment uses the segmented LoS/NLoS channel with Rician fading and NLoS parameters (Section IV). Since the heuristic's model is mismatched by construction, the throughput gains in Fig. 4 may be largely an artifact of model mismatch rather than of the RL-based or enhanced-RL-based algorithms. Please compare against a stronger baseline (e.g., an offline trajectory with full knowledge of user locations and channels, or an online policy with a state that includes the observed user locations) to support the claim of 'significant improvement.' In addition, the enhanced approach's pretraining depends on hand-selected parameters C=10, D=0.6 and C=11.9, D=0.13; the paper should justify these choices or provide a principled selection rule, since the benefit of the enhanced method is shown to depend on them.","section":"Section IV, heuristic baseline and parameter selection"}],"minor_comments":[{"comment":"The cross-reference 'Fig. 37' should be 'Fig. 3' (the footnote marker appears to have been inserted into the cross-reference).","section":"Section IV"},{"comment":"References [7] and [22] are duplicate entries for the same Al-Hourani paper; Reference [21] should list 'D. Tse' rather than 'David. T'.","section":"References"},{"comment":"The offline pretraining stage is not fully specified: please state the number of training episodes or steps, the initialization of the pretraining Q-table, and the stopping criterion used to decide when the pretrained Q-table is ready.","section":"Section III-B"},{"comment":"The y-axis 'instantaneous throughput' is not defined; clarify whether it is the per-slot throughput, a moving average, or the cumulative average over the run.","section":"Section IV, Fig. 2"},{"comment":"The transmit power is denoted \\tilde{P} in Section II but P in Section IV and in P2.n; please unify the notation.","section":"Section IV"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for a communications journal, but the theoretical formulation and the numerical evidence both need substantial work before the claims can be accepted. The location-only state space in a non-stationary environment is the main correctness risk, and the single-run curves do not yet support the abstract's claim of significant improvement. I see no indication of misconduct; the paper is candid about some limitations, but the central result is not yet established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a competent, incremental algorithmic study. It does not change the fundamentals of UAV-NOMA or RL, but it does one practical thing worth having: it applies Q-learning to UAV maneuver design for mobile ground users under causal CSI, and it speeds up convergence by initializing the Q-table with a rough channel model. The system model is standard and Eq. (1) is correct for the NOMA MAC. The new elements—segmented channels, online mobile users, and expert-knowledge initialization—are genuinely novel in combination, though each is modest on its own.\n\nWhat the paper does well: clear problem statement, sensible RL formulation with finite grid, and simulations that are consistent with the qualitative claims. The enhanced RL approach plausibly reduces exploration time, and Figs. 2–4 support that story. The citation pattern is fine; the prior work is properly acknowledged.\n\nWhere it is soft. The main issue, and the stress-test note gets this right: the state space is only the UAV's location, but the reward is the instantaneous sum-rate, which depends on user positions and fading that are neither in the state nor controlled. So the Bellman update in Eq. (6) is being applied to a non-stationary reward process, not the fixed MDP from Section III-A. There is no convergence guarantee, and the 'convergence' in Fig. 2 is an empirical observation on one slow, closed user-trajectory set. This does not falsify the engineering claim, but the claim should be scoped: what is learned is, at best, a time-averaged location-to-reward map that works when user geometry drifts slowly or repeats. The paper should say this.\n\nThe numerical section is weaker than the rest. Single-run curves, no error bars or multiple seeds; the baseline heuristic uses a probabilistic LoS model in an environment that is deliberately NLoS-heavy, so the comparison is loaded in the RL's favor. The pretraining parameters C, D, eta are hand-set, and Fig. 5 is a sensitivity check but not a systematic one. No code or data are released, which makes it hard to check the single-run behavior.\n\nNone of this is fatal. The core idea is sound, the implementation is careful, and the limitations are addressable. I would send it to review, with a strong request for multi-seed statistics and explicit discussion of the non-stationary MDP issue.","headline":"A competent, incremental RL-for-UAV-NOMA study whose main soft spot is that the Q-learning state omits the time-varying user geometry, making the claimed convergence empirical rather than theoretically grounded.","tokens_in":11534,"tokens_out":2226,"would_cite":true,"duration_ms":24133,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A UAV with only causal CSI can learn an online maneuver policy for NOMA users via Q-learning, and pretraining the Q-table on an average channel model makes it converge faster.","keywords":["UAV maneuver design","NOMA","reinforcement learning","Q-learning","online trajectory optimization","sum-rate throughput","causal CSI","LoS probability channel model"],"falsifier":"Run the same three-user 200 m by 200 m setup with user speeds comparable to or faster than the UAV's one-slot displacement, and check whether the RL-based and enhanced RL-based approaches still beat the mean-channel heuristic and whether the learned policy transfers to a different realization of the same user mobility process; if throughput falls back to the heuristic level or the Q-table fails to stabilize, the claim that location-only state suffices for online maneuver design is refuted.","tokens_in":10507,"feed_emoji":"📡","tokens_out":7823,"duration_ms":79687,"temperature":0.7,"pith_summary":"The paper considers a UAV acting as an uplink base station for ground users that transmit with non-orthogonal multiple access (NOMA), where multiple users share the same time-frequency resource. The UAV must decide its next horizontal move at each time slot before knowing where users will go, seeing only the current user locations and channel state. The authors propose a Q-learning controller whose state is the UAV's grid cell and whose action is one of five displacements, with the immediate reward equal to the instantaneous NOMA sum-rate. They then enhance it by pretraining the Q-table offline using a coarse probabilistic line-of-sight channel model, treating that model as expert knowledge. Their numerical results show that both learned controllers outperform a heuristic that greedily chases the position maximizing average-channel sum-rate, and that the pretrained table converges considerably faster.","feed_headline":"Q-learning lets a drone chase mobile users for better NOMA rates","feed_subtitle":"A rough channel model pretrains the drone's Q-table, cutting exploration time and beating average-channel heuristics.","key_machinery":"The central object is the Q-table, a lookup table over state-action pairs in which the state is the UAV's horizontal grid cell and the action is one of five displacement vectors: hover, left, right, forward, and backward. Updates follow the standard Bellman equation $Q(s,a)=Q(s,a)+\\alpha(r+\\gamma \\max_{a'} Q(s',a')-Q(s,a))$, with $\\epsilon$-greedy exploration. The load-bearing addition is the initialization procedure: before online flight, the UAV runs the same Q-learning loop offline using rewards computed from a probabilistic LoS average channel gain $\\bar{h}_k[n]=p_{k,\\mathrm{LoS}}[n]\\bar{\\beta} d_k[n]^{-\\bar{\\alpha}}+\\eta(1-p_{k,\\mathrm{LoS}}[n])\\bar{\\beta} d_k[n]^{-\\bar{\\alpha}}$, assuming users stay at their initial positions, and the converged table becomes the initial Q-table for the online phase. This carries the argument because it converts well-established wireless channel knowledge into a warm start, shortening the exploration period that otherwise dominates early throughput.","core_discovery":"The central discovery, stated on the paper's own terms, is that online maneuver design under causal information can be solved by table-based Q-learning without any a priori model of user mobility, and that the learning can be accelerated by initializing the Q-table with values trained on a rough analytical channel model. In the simulated 200 m by 200 m obstacle environment with three mobile users and a segmented LoS/NLoS channel, the RL-based approaches converge after roughly 4000 to 5000 time slots, track the moving users, and keep the UAV in LoS positions relative to most users, while achieving higher average throughput than the probabilistic-LoS heuristic design. The enhanced approach works because the pretrained table is a warm start: it already encodes rough reward information at each UAV location, so online Q-learning spends less time exploring unproductive cells. The choice of pretraining model matters, with a probabilistic LoS model giving faster convergence than a pure LoS model, indicating that the expert knowledge should match the actual propagation environment.","pith_inferences":["Beyond the paper, the warm-start idea should transfer to deep Q-networks when the grid is too fine or the action space becomes continuous, since the pretraining model can generate experience replay data offline.","A testable extension is to replace the analytic LoS model with an empirical radio map built from previous flights in the same area; this should preserve causal operation while giving even faster convergence.","An implicit limitation is that sum-rate maximization is indifferent to fairness among users; adding a per-user rate constraint or a fairness reward to the same Q-learning loop is a natural next step.","The convergence result depends on users moving slowly relative to the UAV; at higher user speeds, the location-only state may fail to track, so an explicit condition on user speed relative to UAV displacement would make the claim sharper."],"forward_implications":["A UAV can serve mobile NOMA users without knowing their future trajectories, removing a central assumption of offline trajectory optimization.","Rough channel knowledge, not an exact model, is enough to accelerate learning, so the method remains usable when only statistical information about an area is available.","The matching between the pretraining channel model and the real environment controls the gain: a mismatched simpler model (pure LoS) converges slower than a probabilistic LoS model.","The learned policy tends to position the UAV where most users are in LoS, aligning the machine-learned behavior with physical intuition about elevation-dependent blockage.","Because the reward is the instantaneous NOMA sum-rate, the approach inherits the property that the sum-rate is independent of the successive interference cancellation decoding order, which simplifies implementation."],"supporting_citations":[{"why":"This work establishes UAV-enabled wireless communications as the motivating application for the paper.","marker":"[1]"},{"why":"This work introduces UAV communication combined with non-orthogonal multiple access, motivating the NOMA uplink setting.","marker":"[5]"},{"why":"This work provides the zero-initialized Q-table trajectory optimization precedent that the RL-based approach follows.","marker":"[14]"},{"why":"This work supplies the segmented LoS/NLoS channel model used to simulate the practical propagation environment.","marker":"[18]"},{"why":"This work gives the Q-learning and Bellman update machinery that the maneuver design algorithm is built on.","marker":"[19]"},{"why":"This work provides the multiple-access channel capacity result used to express the NOMA sum-rate reward.","marker":"[21]"},{"why":"This work gives the probabilistic LoS average channel gain formula used to pretrain the Q-table in the enhanced approach.","marker":"[22]"}],"fun_headline_variants":["Q-learning with pretrained table boosts UAV NOMA throughput","Drone RL maneuver design improves NOMA sum-rate online","Pretrained Q-table speeds up drone learning for NOMA","Table-based RL beats LoS heuristic for UAV NOMA rates","Rough channel model warm start accelerates drone path RL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a Q-table whose state is only the UAV's location can learn a useful policy even though the reward depends on user positions and fading that change over time and are not part of the state.","fun_headline_variants_meta":{"raw":{"variants":["Q-learning with pretrained table boosts UAV NOMA throughput","Drone RL maneuver design improves NOMA sum-rate online","Pretrained Q-table speeds up drone learning for NOMA","Table-based RL beats LoS heuristic for UAV NOMA rates","Rough channel model warm start accelerates drone path RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000317,"raw_usage":{"total_tokens":1718,"prompt_tokens":794,"completion_tokens":924,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":410,"completion_tokens_details":{"reasoning_tokens":843}},"tokens_in":410,"tokens_out":924,"duration_ms":10401,"temperature":1.0,"reasoning_tokens":843,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:56:04.627061+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same three-user 200 m by 200 m setup with user speeds comparable to or faster than the UAV's one-slot displacement, and check whether the RL-based and enhanced RL-based approaches still beat the mean-channel heuristic and whether the learned policy transfers to a different realization of the same user mobility process; if throughput falls back to the heuristic level or the Q-table fails to stabilize, the claim that location-only state suffices for online maneuver design is refuted.","supporting_citations":[{"cited_title":"Wireless communication s with unmanned aerial vehicles: Opportunities and challenges,","cited_arxiv_id":null,"evidence_quote":"This work establishes UAV-enabled wireless communications as the motivating application for the paper."},{"cited_title":"UA V communication based on non-orthogonal multiple access,","cited_arxiv_id":null,"evidence_quote":"This work introduces UAV communication combined with non-orthogonal multiple access, motivating the NOMA uplink setting."},{"cited_title":"Trajectory optimization for autonomous flying base station via reinforcement learni ng,","cited_arxiv_id":null,"evidence_quote":"This work provides the zero-initialized Q-table trajectory optimization precedent that the RL-based approach follows."},{"cited_title":"Optimal positioning of flying re lays for wireless networks: A LOS map approach,","cited_arxiv_id":null,"evidence_quote":"This work supplies the segmented LoS/NLoS channel model used to simulate the practical propagation environment."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"This work gives the Q-learning and Bellman update machinery that the maneuver design algorithm is built on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"This work provides the multiple-access channel capacity result used to express the NOMA sum-rate reward."}],"review_version":1}