REVIEW 4 major objections 5 minor 22 references
Online Maneuver Design for UAV-Enabled NOMA Systems via Reinforcement Learning
T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A UAV with only causal CSI can learn an online maneuver policy for NOMA users via Q-learning, and pretraining the Q-table on an average channel model makes it converge faster.
desk verdict A competent, incremental RL-for-UAV-NOMA study whose main soft spot is that the Q-learning state omits the time-varying user geometry, making the claimed convergence empirical rather than theoretically grounded. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the Q-table, a lookup table over state-action pairs in which the state is the UAV's horizontal grid cell and the action is one of five displacement vectors: hover, left, right, forward, and backward. Updates follow the standard Bellman equation $Q(s,a)=Q(s,a)+\alpha(r+\gamma \max_{a'} Q(s',a')-Q(s,a))$, with $\epsilon$-greedy exploration. The load-bearing addition is the initialization procedure: before online flight, the UAV runs the same Q-learning loop offline using rewards computed from a probabilistic LoS average channel gain $\bar{h}_k[n]=p_{k,\mathrm{LoS}}[n]\bar{\beta} d_k[n]^{-\bar{\alpha}}+\eta(1-p_{k,\mathrm{LoS}}[n])\bar{\beta} d_k[n]^{-\bar{\alpha}}$, assuming users stay at their initial positions, and the converged table becomes the initial Q-table for the online phase. This carries the argument because it converts well-established wireless channel knowledge into a warm start, shortening the exploration period that otherwise dominates early throughput.
What would settle it
Run the same three-user 200 m by 200 m setup with user speeds comparable to or faster than the UAV's one-slot displacement, and check whether the RL-based and enhanced RL-based approaches still beat the mean-channel heuristic and whether the learned policy transfers to a different realization of the same user mobility process; if throughput falls back to the heuristic level or the Q-table fails to stabilize, the claim that location-only state suffices for online maneuver design is refuted.
Extended reading notes
Core claim
The central discovery, stated on the paper's own terms, is that online maneuver design under causal information can be solved by table-based Q-learning without any a priori model of user mobility, and that the learning can be accelerated by initializing the Q-table with values trained on a rough analytical channel model. In the simulated 200 m by 200 m obstacle environment with three mobile users and a segmented LoS/NLoS channel, the RL-based approaches converge after roughly 4000 to 5000 time slots, track the moving users, and keep the UAV in LoS positions relative to most users, while achieving higher average throughput than the probabilistic-LoS heuristic design. The enhanced approach works because the pretrained table is a warm start: it already encodes rough reward information at each UAV location, so online Q-learning spends less time exploring unproductive cells. The choice of pretraining model matters, with a probabilistic LoS model giving faster convergence than a pure LoS model, indicating that the expert knowledge should match the actual propagation environment.
Load-bearing premise
The load-bearing premise is that a Q-table whose state is only the UAV's location can learn a useful policy even though the reward depends on user positions and fading that change over time and are not part of the state.
Editorial extensions
If this is right
- A UAV can serve mobile NOMA users without knowing their future trajectories, removing a central assumption of offline trajectory optimization.
- Rough channel knowledge, not an exact model, is enough to accelerate learning, so the method remains usable when only statistical information about an area is available.
- The matching between the pretraining channel model and the real environment controls the gain: a mismatched simpler model (pure LoS) converges slower than a probabilistic LoS model.
- The learned policy tends to position the UAV where most users are in LoS, aligning the machine-learned behavior with physical intuition about elevation-dependent blockage.
- Because the reward is the instantaneous NOMA sum-rate, the approach inherits the property that the sum-rate is independent of the successive interference cancellation decoding order, which simplifies implementation.
Reading between the lines
- Beyond the paper, the warm-start idea should transfer to deep Q-networks when the grid is too fine or the action space becomes continuous, since the pretraining model can generate experience replay data offline.
- A testable extension is to replace the analytic LoS model with an empirical radio map built from previous flights in the same area; this should preserve causal operation while giving even faster convergence.
- An implicit limitation is that sum-rate maximization is indifferent to fairness among users; adding a per-user rate constraint or a fairness reward to the same Q-learning loop is a natural next step.
- The convergence result depends on users moving slowly relative to the UAV; at higher user speeds, the location-only state may fail to track, so an explicit condition on user speed relative to UAV displacement would make the claim sharper.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies a UAV-enabled uplink NOMA system with mobile ground users. The UAV only knows users' locations and CSI causally. The objective (P1) is to maximize the average sum-rate over a finite horizon by choosing the UAV's horizontal displacement at each slot from five options. The authors propose a Q-learning solution whose state is the UAV's grid location and whose reward is the instantaneous sum-rate from Eq. (1). To speed convergence, they add an offline pretraining stage that initializes the Q-table using rewards predicted from a probabilistic LoS average channel model. Section IV reports numerical comparisons against a heuristic average-channel follower, claiming significant throughput gains and faster convergence for the enhanced variant.
Significance. The paper's core idea—using a crude wireless channel model to seed Q-learning for UAV maneuver design—is interesting and potentially useful, and the sum-rate expression in Eq. (1) is correctly derived for the NOMA MAC. However, the central claim is not yet established. The state space omits the time-varying user positions and CSI that drive the reward, so the Q-learning is applied to a non-stationary process with no convergence guarantee; the numerical results are single-run curves that do not provide statistical support; and the comparison baseline is mismatched to the simulated channel. With a corrected formulation and stronger evaluation, the work could make a moderate contribution to online UAV trajectory design.
major comments (4)
- [Section III-A, Eq. (6)] The MDP used for Q-learning is not well posed. The state space is S={q[n]}, containing only the UAV's location, while the reward r_n=\tilde{R}[n] depends on the user positions {w_k[n]} and channel gains {h_k[n]} from Eq. (1), which are time-varying and absent from the state. Therefore R_a(s,s') is not a function of the state/action pair as required by the MDP {S,A,P,R}, and the Bellman update in Eq. (6) is applied to a non-stationary reward process. The statement in Section III-A that α and γ should be chosen properly is the standard stationary-MDP condition and does not cover this case. The empirical 'convergence' in Fig. 2 is measured on one fixed slow user-trajectory set, so the paper does not establish that a location-only Q-table can learn an online maneuver policy for arbitrary mobile users. Please either include the user geometry in the state (with a tractable discretization), formulate the problem as a contextual MDP, or provide a convergence/stability argument under explicit slow-mobility assumptions.
- [Section III-A, reward timing] The description of the reward relative to the action is inconsistent. The text says that after choosing a_n=λ[n] and moving to q[n+1], the UAV receives r_n=\tilde{R}[n], but \tilde{R}[n] is determined by q[n] and the channel at slot n, before the action's effect takes place. As written, the reward does not depend on the action taken at step n, so the Q-update in Eq. (6) would reinforce an action with a reward that is not a consequence of that action. Please correct the time indexing (e.g., r_n=\tilde{R}[n+1], or redefine the state/action timing) and confirm that the simulation uses the corrected indexing.
- [Section IV, Figs. 2 and 4] The numerical evaluation is based on single runs with no multiple seeds or confidence intervals. Q-learning with ε-greedy exploration and random fading/shadowing is stochastic, so the reported curves do not establish statistical significance. Moreover, the implementation in Section III-A appears to be a single pass n=1,...,N over one flight, not repeated episodes; Q-learning convergence requires each state-action pair to be visited sufficiently often (or an appropriate online step-size schedule), and a one-trajectory run cannot substantiate the phrase 'almost converge at N>4000.' Please provide multi-seed statistics, define the plotted metric precisely, and clarify whether the Q-table is updated once per slot in a single episode or over repeated episodes.
- [Section IV, heuristic baseline and parameter selection] The comparison baseline 'Probabilistic LoS channel-based heuristic design' assumes the average channel model \bar{h}_k[n] from Eq. (7) and solves P2.n, while the simulated environment uses the segmented LoS/NLoS channel with Rician fading and NLoS parameters (Section IV). Since the heuristic's model is mismatched by construction, the throughput gains in Fig. 4 may be largely an artifact of model mismatch rather than of the RL-based or enhanced-RL-based algorithms. Please compare against a stronger baseline (e.g., an offline trajectory with full knowledge of user locations and channels, or an online policy with a state that includes the observed user locations) to support the claim of 'significant improvement.' In addition, the enhanced approach's pretraining depends on hand-selected parameters C=10, D=0.6 and C=11.9, D=0.13; the paper should justify these choices or provide a principled selection rule, since the benefit of the enhanced method is shown to depend on them.
minor comments (5)
- [Section IV] The cross-reference 'Fig. 37' should be 'Fig. 3' (the footnote marker appears to have been inserted into the cross-reference).
- [References] References [7] and [22] are duplicate entries for the same Al-Hourani paper; Reference [21] should list 'D. Tse' rather than 'David. T'.
- [Section III-B] The offline pretraining stage is not fully specified: please state the number of training episodes or steps, the initialization of the pretraining Q-table, and the stopping criterion used to decide when the pretrained Q-table is ready.
- [Section IV, Fig. 2] The y-axis 'instantaneous throughput' is not defined; clarify whether it is the per-slot throughput, a moving average, or the cumulative average over the run.
- [Section IV] The transmit power is denoted \tilde{P} in Section II but P in Section IV and in P2.n; please unify the notation.
Circularity Check
No circularity found: the RL and enhanced-RL gains are empirical comparisons against an independent heuristic, with a warm-start prior rather than a conclusion forced by the inputs.
full rationale
The paper contains no derivation chain that reduces a claimed result to its own inputs. The central claims are numerical: the RL-based designs outperform a probabilistic-LoS heuristic, and the enhanced RL design converges faster because of Q-table initialization. The reward in Eq. (6) is the realized instantaneous sum-rate from Eq. (1), while the baseline (P2.n) is a separate heuristic; no RL parameter is fitted to force the comparison. For the enhanced method, the paper states that during the training procedure 'the UAV computes the rewards by using the above rough channel estimations in (7)', but after that warm start the same Q-learning update in Eq. (6) is driven by the actual segmented-channel rewards, so the pretrained Q-table is a prior rather than the predicted conclusion. Eq. (7) is an external empirical LoS-probability model cited to [22], and the pretraining parameters are explicitly approximate ('we properly choose these parameters for getting rough channel predictions'), not values inferred from the test environment. Self-citations such as [2], [4], [12], [15], and [17] are background references and are not load-bearing for the numerical comparisons. The most serious concern is non-circular: Section III-A defines S={q[n]} and rn=R[n], while R[n] depends on mobile users' positions and fading that are absent from the state, so the Bellman update in Eq. (6) is applied to a non-stationary reward process and the convergence in Fig. 2 is an empirical observation rather than a consequence of the stated MDP. That is a validity limitation, not a circularity.
Assumptions & free parameters
free parameters (4)
- Probabilistic LoS parameters C and D for Q-table pretraining =
C=10, D=0.6; also C=11.9, D=0.13 in Fig. 4
- Average channel parameters for pretraining: beta_bar, alpha_bar, eta =
beta_bar=-30 dB, alpha_bar=2.3, eta=0.1
- Learning hyperparameters alpha, gamma, epsilon =
alpha=0.3, gamma=0.9, epsilon=0.9
- Grid resolution and UAV step lambda =
20x20 grid, lambda=10 m
assumptions (4)
- standard math Uplink MAC capacity region with SIC and Gaussian signaling yields the sum-rate log2(1 + P sum_k h_k / sigma^2).
- domain assumption The mobile-user scenario can be treated as a stationary MDP with state S={q[n]}, so standard Q-learning convergence conditions apply.
- domain assumption A probabilistic LoS average model with parameters chosen within empirical ranges provides useful reward estimates for Q-table initialization.
- domain assumption The simulated scenario with K=3 users, predetermined trajectories, and specified segmented channel parameters is representative enough to support generalized claims.
Cite this review
Pith. "Pith review of Online Maneuver Design for UAV-Enabled NOMA Systems via Reinforcement Learning." pith.science (2026). https://pith.science/paper/PD45KYQF
@misc{pith2026190803984,
author = {Pith},
title = {Pith review of: Online Maneuver Design for UAV-Enabled NOMA Systems via Reinforcement Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/PD45KYQF}},
note = {Machine review of arXiv:1908.03984}
}
read the original abstract
This paper considers an unmanned aerial vehicle enabled-up link non-orthogonal multiple-access system, where multiple mobile users on the ground send independent messages to a unmanned aerial vehicle in the sky via non-orthogonal multiple-access transmission. Our objective is to design the unmanned aerial vehicle dynamic maneuver for maximizing the sum-rate throughput of all mobile ground users over a finite time horizon.
Figures
Reference graph
Works this paper leans on
-
[1]
Wireless communication s with unmanned aerial vehicles: Opportunities and challenges,
Y . Zeng, R. Zhang, and T. J. Lim, “Wireless communication s with unmanned aerial vehicles: Opportunities and challenges,” IEEE Commun. Mag., vol. 54, no. 5, pp. 36–42, May 2016
work page 2016
-
[2]
Throughput maximizatio n for UA V - enabled mobile relaying systems,
Y . Zeng, R. Zhang, and T. J. Lim, “Throughput maximizatio n for UA V - enabled mobile relaying systems,” IEEE Trans. Commun. , vol. 64, no. 12, pp. 4983–4996, Dec. 2016
work page 2016
-
[3]
Capacity characterization of UA V -enabled two-user broadcast channel,
Q. Wu, J. Xu, and R. Zhang, “Capacity characterization of UA V -enabled two-user broadcast channel,” IEEE J. Sel. Areas Commun. , vol. 36, no. 9, pp. 1955–1971, Sep. 2018
work page 1955
-
[4]
Joint trajectory and commun ication design for multi-UA V enabled wireless networks,
Q. Wu, Y . Zeng, and R. Zhang, “Joint trajectory and commun ication design for multi-UA V enabled wireless networks,” IEEE Trans. Wireless Commun., vol. 17, no. 3, pp. 2109–2121, Jan. 2018
work page 2018
-
[5]
UA V communication based on non-orthogonal multiple access,
Y . Liu, Z. Qin, Y . Cai, Y . Gao, G. Y e. Li, and A. Nallanathan , “UA V communication based on non-orthogonal multiple access,” IEEE Wireless Commun., vol. 26, no. 1, pp. 52–57, Feb. 2019
work page 2019
-
[6]
Fundamental rate limits of UA V -enabled m ultiple access channel with trajectory optimization
P . Li and J. Xu, “Fundamental rate limits of UA V -enabled m ultiple access channel with trajectory optimization.” IEEE Trans. Wireless Commun., vol.19, no. 1, pp. 458–474, Jan. 2020
work page 2020
-
[7]
Optimal LA P altitude for maximum coverage,
A. Al-Hourani, S. Kandeepan, and S. Lardner, “Optimal LA P altitude for maximum coverage,” IEEE Wireless Commun. Lett., vol. 3, no. 6, pp. 569–572, Dec. 2014
work page 2014
-
[8]
Placement optimi zation of UA V -mounted mobile base stations,
J. Lyu, Y . Zeng, R. Zhang, and T. J. Lim, “Placement optimi zation of UA V -mounted mobile base stations,” IEEE Commun. Lett. , vol. 21, no. 3, pp. 604–607, Mar. 2017
work page 2017
Show all 22 references
-
[9]
Placement optimization for UA V -enabled wireless networks with multi-hop backhauls,
P . Li and J. Xu, “Placement optimization for UA V -enabled wireless networks with multi-hop backhauls,” J. Commun. Inf. Netw., vol. 3, no. 4, pp. 64–73, Dec. 2018. 7
2018
-
[10]
Mobile unmanned aerial vehicles (UA Vs) for energy-efficient Internet of thi ngs communi- cation,
M. Mozaffari, W. Saad, M. Bennis, and M. Debbah, “Mobile unmanned aerial vehicles (UA Vs) for energy-efficient Internet of thi ngs communi- cation,” IEEE Trans. Wireless Commun. , vol. 16, no. 11, pp. 7574–7589, Nov. 2017
2017
-
[11]
Throughput maximization fo r UA V - enabled wireless powered communication networks,
L. Xie, J. Xu, and R. Zhang, “Throughput maximization fo r UA V - enabled wireless powered communication networks,” IEEE Internet Things J. , vol. 6, no. 2, pp. 1690–1703, Apr. 2019
2019
-
[12]
Cellular-enabled UA V c ommu- nication: A connectivity-constrained trajectory optimiz ation perspec- tive,
S. Zhang, Y . Zeng, and R. Zhang, “Cellular-enabled UA V c ommu- nication: A connectivity-constrained trajectory optimiz ation perspec- tive,”IEEE Trans. Commun. , vol. 67, no. 3, pp. 2580–2604, Mar. 2019
2019
-
[13]
Deep learning in physical layer communications,
Z. Qin, H. Y e, G. Y . Li, and B.-H. F. Juang, “Deep learning in physical layer communications,” IEEE Wireless Commun. , vol. 26, no. 2, pp. 93– 99, Apr. 2019
2019
-
[14]
Trajectory optimization for autonomous flying base station via reinforcement learni ng,
H. Bayerlein, P . D. Kerret, and D. Gesbert, “Trajectory optimization for autonomous flying base station via reinforcement learni ng,” in Proc. IEEE SPAWC, Jun. 2018
2018
-
[15]
Energy-ef ficient UA V control for effective and fair communication coverage: A deep reinforcement learning approach
C. H. Liu, Z. Chen, J. Tang, J. Xu, and C. Piao, “Energy-ef ficient UA V control for effective and fair communication coverage: A deep reinforcement learning approach”, IEEE J. Sel. Areas Commun. , vol. 36, no. 9, pp. 2059–2070, Sep. 2018
2018
-
[16]
Interferenc e management for cellular-connected UA Vs: A deep reinforcement learning approach,
U. Challita, W. Saad, and C. Bettstetter, “Interferenc e management for cellular-connected UA Vs: A deep reinforcement learning approach,” IEEE Trans. Wireless Commun. , vol. 18, no. 4, pp. 2125–2140, Apr. 2019
2019
-
[17]
Path design for cellular-connected U A V with reinforcement learning,
Y . Zeng and X. Xu, “Path design for cellular-connected U A V with reinforcement learning,” in IEEE Globecom , pp. 1–6, Dec. 2019
2019
-
[18]
Optimal positioning of flying re lays for wireless networks: A LOS map approach,
J. Chen and D. Gesbert, “Optimal positioning of flying re lays for wireless networks: A LOS map approach,” in Proc. IEEE ICC , Jul. 2017
2017
-
[19]
R. S. Sutton and A. G. Barto, Introduction to Reinforcement Learning, 2nd ed. Cambridge, Massachusetts: MIT Press, 2017
2017
-
[20]
A review of deep learning methods and applications for unmann ed aerial vehicles,
A. Carrio, C. Sampedro, A. Rodriguez-Ramos, and P . Camp oy, “A review of deep learning methods and applications for unmann ed aerial vehicles,” J. Sensors , vol. 2017, no. 3296874, 2017
2017
-
[21]
David. T and P . Viswanath, Fundamentals of Wireless Communication , Cambridge University press, 2005
2005
-
[22]
Optimal L AP altitude for maximum coverage,
A. Al-Hourani, S. Kandeepan, and S. Lardner, “Optimal L AP altitude for maximum coverage,” IEEE Wireless Commun. Lett., vol. 3, no. 6, pp. 569–572, Dec. 2014
2014
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.