Pith. sign in

REVIEW 4 major objections 5 minor 22 references

Online Maneuver Design for UAV-Enabled NOMA Systems via Reinforcement Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read A UAV with only causal CSI can learn an online maneuver policy for NOMA users via Q-learning, and pretraining the Q-table on an average channel model makes it converge faster.

desk verdict A competent, incremental RL-for-UAV-NOMA study whose main soft spot is that the Q-learning state omits the time-varying user geometry, making the claimed convergence empirical rather than theoretically grounded. read the letter →

arxiv 1908.03984 v3 pith:PD45KYQF submitted 2019-08-12 cs.IT eess.SPmath.IT

classification cs.ITeess.SPmath.IT
keywords UAVmaneuverdesignNOMAreinforcementlearningQ-learningonlinetrajectoryoptimizationsum-ratethroughputcausalCSILoSprobabilitychannelmodel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper considers a UAV acting as an uplink base station for ground users that transmit with non-orthogonal multiple access (NOMA), where multiple users share the same time-frequency resource. The UAV must decide its next horizontal move at each time slot before knowing where users will go, seeing only the current user locations and channel state. The authors propose a Q-learning controller whose state is the UAV's grid cell and whose action is one of five displacements, with the immediate reward equal to the instantaneous NOMA sum-rate. They then enhance it by pretraining the Q-table offline using a coarse probabilistic line-of-sight channel model, treating that model as expert knowledge. Their numerical results show that both learned controllers outperform a heuristic that greedily chases the position maximizing average-channel sum-rate, and that the pretrained table converges considerably faster.

What carries the argument

The central object is the Q-table, a lookup table over state-action pairs in which the state is the UAV's horizontal grid cell and the action is one of five displacement vectors: hover, left, right, forward, and backward. Updates follow the standard Bellman equation $Q(s,a)=Q(s,a)+\alpha(r+\gamma \max_{a'} Q(s',a')-Q(s,a))$, with $\epsilon$-greedy exploration. The load-bearing addition is the initialization procedure: before online flight, the UAV runs the same Q-learning loop offline using rewards computed from a probabilistic LoS average channel gain $\bar{h}_k[n]=p_{k,\mathrm{LoS}}[n]\bar{\beta} d_k[n]^{-\bar{\alpha}}+\eta(1-p_{k,\mathrm{LoS}}[n])\bar{\beta} d_k[n]^{-\bar{\alpha}}$, assuming users stay at their initial positions, and the converged table becomes the initial Q-table for the online phase. This carries the argument because it converts well-established wireless channel knowledge into a warm start, shortening the exploration period that otherwise dominates early throughput.

What would settle it

Run the same three-user 200 m by 200 m setup with user speeds comparable to or faster than the UAV's one-slot displacement, and check whether the RL-based and enhanced RL-based approaches still beat the mean-channel heuristic and whether the learned policy transfers to a different realization of the same user mobility process; if throughput falls back to the heuristic level or the Q-table fails to stabilize, the claim that location-only state suffices for online maneuver design is refuted.

Watch

Extended reading notes

Core claim

The central discovery, stated on the paper's own terms, is that online maneuver design under causal information can be solved by table-based Q-learning without any a priori model of user mobility, and that the learning can be accelerated by initializing the Q-table with values trained on a rough analytical channel model. In the simulated 200 m by 200 m obstacle environment with three mobile users and a segmented LoS/NLoS channel, the RL-based approaches converge after roughly 4000 to 5000 time slots, track the moving users, and keep the UAV in LoS positions relative to most users, while achieving higher average throughput than the probabilistic-LoS heuristic design. The enhanced approach works because the pretrained table is a warm start: it already encodes rough reward information at each UAV location, so online Q-learning spends less time exploring unproductive cells. The choice of pretraining model matters, with a probabilistic LoS model giving faster convergence than a pure LoS model, indicating that the expert knowledge should match the actual propagation environment.

Load-bearing premise

The load-bearing premise is that a Q-table whose state is only the UAV's location can learn a useful policy even though the reward depends on user positions and fading that change over time and are not part of the state.

Editorial extensions

If this is right

  • A UAV can serve mobile NOMA users without knowing their future trajectories, removing a central assumption of offline trajectory optimization.
  • Rough channel knowledge, not an exact model, is enough to accelerate learning, so the method remains usable when only statistical information about an area is available.
  • The matching between the pretraining channel model and the real environment controls the gain: a mismatched simpler model (pure LoS) converges slower than a probabilistic LoS model.
  • The learned policy tends to position the UAV where most users are in LoS, aligning the machine-learned behavior with physical intuition about elevation-dependent blockage.
  • Because the reward is the instantaneous NOMA sum-rate, the approach inherits the property that the sum-rate is independent of the successive interference cancellation decoding order, which simplifies implementation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the warm-start idea should transfer to deep Q-networks when the grid is too fine or the action space becomes continuous, since the pretraining model can generate experience replay data offline.
  • A testable extension is to replace the analytic LoS model with an empirical radio map built from previous flights in the same area; this should preserve causal operation while giving even faster convergence.
  • An implicit limitation is that sum-rate maximization is indifferent to fairness among users; adding a per-user rate constraint or a fairness reward to the same Q-learning loop is a natural next step.
  • The convergence result depends on users moving slowly relative to the UAV; at higher user speeds, the location-only state may fail to track, so an explicit condition on user speed relative to UAV displacement would make the claim sharper.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies a UAV-enabled uplink NOMA system with mobile ground users. The UAV only knows users' locations and CSI causally. The objective (P1) is to maximize the average sum-rate over a finite horizon by choosing the UAV's horizontal displacement at each slot from five options. The authors propose a Q-learning solution whose state is the UAV's grid location and whose reward is the instantaneous sum-rate from Eq. (1). To speed convergence, they add an offline pretraining stage that initializes the Q-table using rewards predicted from a probabilistic LoS average channel model. Section IV reports numerical comparisons against a heuristic average-channel follower, claiming significant throughput gains and faster convergence for the enhanced variant.

Significance. The paper's core idea—using a crude wireless channel model to seed Q-learning for UAV maneuver design—is interesting and potentially useful, and the sum-rate expression in Eq. (1) is correctly derived for the NOMA MAC. However, the central claim is not yet established. The state space omits the time-varying user positions and CSI that drive the reward, so the Q-learning is applied to a non-stationary process with no convergence guarantee; the numerical results are single-run curves that do not provide statistical support; and the comparison baseline is mismatched to the simulated channel. With a corrected formulation and stronger evaluation, the work could make a moderate contribution to online UAV trajectory design.

major comments (4)
  1. [Section III-A, Eq. (6)] The MDP used for Q-learning is not well posed. The state space is S={q[n]}, containing only the UAV's location, while the reward r_n=\tilde{R}[n] depends on the user positions {w_k[n]} and channel gains {h_k[n]} from Eq. (1), which are time-varying and absent from the state. Therefore R_a(s,s') is not a function of the state/action pair as required by the MDP {S,A,P,R}, and the Bellman update in Eq. (6) is applied to a non-stationary reward process. The statement in Section III-A that α and γ should be chosen properly is the standard stationary-MDP condition and does not cover this case. The empirical 'convergence' in Fig. 2 is measured on one fixed slow user-trajectory set, so the paper does not establish that a location-only Q-table can learn an online maneuver policy for arbitrary mobile users. Please either include the user geometry in the state (with a tractable discretization), formulate the problem as a contextual MDP, or provide a convergence/stability argument under explicit slow-mobility assumptions.
  2. [Section III-A, reward timing] The description of the reward relative to the action is inconsistent. The text says that after choosing a_n=λ[n] and moving to q[n+1], the UAV receives r_n=\tilde{R}[n], but \tilde{R}[n] is determined by q[n] and the channel at slot n, before the action's effect takes place. As written, the reward does not depend on the action taken at step n, so the Q-update in Eq. (6) would reinforce an action with a reward that is not a consequence of that action. Please correct the time indexing (e.g., r_n=\tilde{R}[n+1], or redefine the state/action timing) and confirm that the simulation uses the corrected indexing.
  3. [Section IV, Figs. 2 and 4] The numerical evaluation is based on single runs with no multiple seeds or confidence intervals. Q-learning with ε-greedy exploration and random fading/shadowing is stochastic, so the reported curves do not establish statistical significance. Moreover, the implementation in Section III-A appears to be a single pass n=1,...,N over one flight, not repeated episodes; Q-learning convergence requires each state-action pair to be visited sufficiently often (or an appropriate online step-size schedule), and a one-trajectory run cannot substantiate the phrase 'almost converge at N>4000.' Please provide multi-seed statistics, define the plotted metric precisely, and clarify whether the Q-table is updated once per slot in a single episode or over repeated episodes.
  4. [Section IV, heuristic baseline and parameter selection] The comparison baseline 'Probabilistic LoS channel-based heuristic design' assumes the average channel model \bar{h}_k[n] from Eq. (7) and solves P2.n, while the simulated environment uses the segmented LoS/NLoS channel with Rician fading and NLoS parameters (Section IV). Since the heuristic's model is mismatched by construction, the throughput gains in Fig. 4 may be largely an artifact of model mismatch rather than of the RL-based or enhanced-RL-based algorithms. Please compare against a stronger baseline (e.g., an offline trajectory with full knowledge of user locations and channels, or an online policy with a state that includes the observed user locations) to support the claim of 'significant improvement.' In addition, the enhanced approach's pretraining depends on hand-selected parameters C=10, D=0.6 and C=11.9, D=0.13; the paper should justify these choices or provide a principled selection rule, since the benefit of the enhanced method is shown to depend on them.
minor comments (5)
  1. [Section IV] The cross-reference 'Fig. 37' should be 'Fig. 3' (the footnote marker appears to have been inserted into the cross-reference).
  2. [References] References [7] and [22] are duplicate entries for the same Al-Hourani paper; Reference [21] should list 'D. Tse' rather than 'David. T'.
  3. [Section III-B] The offline pretraining stage is not fully specified: please state the number of training episodes or steps, the initialization of the pretraining Q-table, and the stopping criterion used to decide when the pretrained Q-table is ready.
  4. [Section IV, Fig. 2] The y-axis 'instantaneous throughput' is not defined; clarify whether it is the per-slot throughput, a moving average, or the cumulative average over the run.
  5. [Section IV] The transmit power is denoted \tilde{P} in Section II but P in Section IV and in P2.n; please unify the notation.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the RL and enhanced-RL gains are empirical comparisons against an independent heuristic, with a warm-start prior rather than a conclusion forced by the inputs.

full rationale

The paper contains no derivation chain that reduces a claimed result to its own inputs. The central claims are numerical: the RL-based designs outperform a probabilistic-LoS heuristic, and the enhanced RL design converges faster because of Q-table initialization. The reward in Eq. (6) is the realized instantaneous sum-rate from Eq. (1), while the baseline (P2.n) is a separate heuristic; no RL parameter is fitted to force the comparison. For the enhanced method, the paper states that during the training procedure 'the UAV computes the rewards by using the above rough channel estimations in (7)', but after that warm start the same Q-learning update in Eq. (6) is driven by the actual segmented-channel rewards, so the pretrained Q-table is a prior rather than the predicted conclusion. Eq. (7) is an external empirical LoS-probability model cited to [22], and the pretraining parameters are explicitly approximate ('we properly choose these parameters for getting rough channel predictions'), not values inferred from the test environment. Self-citations such as [2], [4], [12], [15], and [17] are background references and are not load-bearing for the numerical comparisons. The most serious concern is non-circular: Section III-A defines S={q[n]} and rn=R[n], while R[n] depends on mobile users' positions and fading that are absent from the state, so the Bellman update in Eq. (6) is applied to a non-stationary reward process and the convergence in Fig. 2 is an empirical observation rather than a consequence of the stated MDP. That is a validity limitation, not a circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central algorithm does not introduce new physical entities. However, its claimed advantage depends on several hand-selected numerical choices: the pretraining channel model parameters (C, D, beta_bar, alpha_bar, eta), the learning hyperparameters (alpha, gamma, epsilon), and the grid resolution (grid count, lambda). These are not fitted to data, but they are chosen for the simulations and would need retuning in new environments. The main domain assumptions are the applicability of tabular Q-learning to a non-stationary mobile-user environment and the usefulness of a rough average channel model as a prior.

free parameters (4)
  • Probabilistic LoS parameters C and D for Q-table pretraining = C=10, D=0.6; also C=11.9, D=0.13 in Fig. 4
    Chosen within empirical ranges in Section IV to construct the approximate channel model in Eq. (7); the enhanced RL result depends on these values.
  • Average channel parameters for pretraining: beta_bar, alpha_bar, eta = beta_bar=-30 dB, alpha_bar=2.3, eta=0.1
    Used in Eq. (7) to compute rewards during offline Q-table initialization; chosen by hand in Section IV, not measured or learned.
  • Learning hyperparameters alpha, gamma, epsilon = alpha=0.3, gamma=0.9, epsilon=0.9
    Selected by comparing settings in Fig. 5; the claimed convergence speed and throughput of the enhanced RL approach are evaluated with these values.
  • Grid resolution and UAV step lambda = 20x20 grid, lambda=10 m
    The finite state and action spaces are defined by these choices in Section IV; different grids would change the achievable policy and learned Q-table.
assumptions (4)
  • standard math Uplink MAC capacity region with SIC and Gaussian signaling yields the sum-rate log2(1 + P sum_k h_k / sigma^2).
    Invoked as Eq. (1) and used as the reward in both the RL design and the baseline; this is a standard information-theoretic result from [21].
  • domain assumption The mobile-user scenario can be treated as a stationary MDP with state S={q[n]}, so standard Q-learning convergence conditions apply.
    Section III-A defines S={q[n]} and applies the Bellman update in Eq. (6) without justifying the Markov property in a non-stationary environment with moving users and time-varying CSI.
  • domain assumption A probabilistic LoS average model with parameters chosen within empirical ranges provides useful reward estimates for Q-table initialization.
    Section III-B relies on Eq. (7) as expert knowledge; the usefulness of the enhanced method depends on this model being informative for the real environment.
  • domain assumption The simulated scenario with K=3 users, predetermined trajectories, and specified segmented channel parameters is representative enough to support generalized claims.
    Numerical results in Section IV use one 200m x 200m area, one set of trajectories, and chosen LoS/NLoS parameters; generalization beyond this setup is asserted without proof.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Online Maneuver Design for UAV-Enabled NOMA Systems via Reinforcement Learning." pith.science (2026). https://pith.science/paper/PD45KYQF

@misc{pith2026190803984,
  author       = {Pith},
  title        = {Pith review of: Online Maneuver Design for UAV-Enabled NOMA Systems via Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PD45KYQF}},
  note         = {Machine review of arXiv:1908.03984}
}
read the original abstract

This paper considers an unmanned aerial vehicle enabled-up link non-orthogonal multiple-access system, where multiple mobile users on the ground send independent messages to a unmanned aerial vehicle in the sky via non-orthogonal multiple-access transmission. Our objective is to design the unmanned aerial vehicle dynamic maneuver for maximizing the sum-rate throughput of all mobile ground users over a finite time horizon.

Figures

Figures reproduced from arXiv: 1908.03984 by the authors.

Figure 1
Figure 1. Simulation setup with blue cubes denoting obstacles [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 3
Figure 3. UAV’s locations after convergence at different part [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Throughput comparison among RL-based, enhanced RL- [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Throughput comparison under different learning par [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 21 canonical work pages

  1. [1]

    Wireless communication s with unmanned aerial vehicles: Opportunities and challenges,

    Y . Zeng, R. Zhang, and T. J. Lim, “Wireless communication s with unmanned aerial vehicles: Opportunities and challenges,” IEEE Commun. Mag., vol. 54, no. 5, pp. 36–42, May 2016

  2. [2]

    Throughput maximizatio n for UA V - enabled mobile relaying systems,

    Y . Zeng, R. Zhang, and T. J. Lim, “Throughput maximizatio n for UA V - enabled mobile relaying systems,” IEEE Trans. Commun. , vol. 64, no. 12, pp. 4983–4996, Dec. 2016

  3. [3]

    Capacity characterization of UA V -enabled two-user broadcast channel,

    Q. Wu, J. Xu, and R. Zhang, “Capacity characterization of UA V -enabled two-user broadcast channel,” IEEE J. Sel. Areas Commun. , vol. 36, no. 9, pp. 1955–1971, Sep. 2018

  4. [4]

    Joint trajectory and commun ication design for multi-UA V enabled wireless networks,

    Q. Wu, Y . Zeng, and R. Zhang, “Joint trajectory and commun ication design for multi-UA V enabled wireless networks,” IEEE Trans. Wireless Commun., vol. 17, no. 3, pp. 2109–2121, Jan. 2018

  5. [5]

    UA V communication based on non-orthogonal multiple access,

    Y . Liu, Z. Qin, Y . Cai, Y . Gao, G. Y e. Li, and A. Nallanathan , “UA V communication based on non-orthogonal multiple access,” IEEE Wireless Commun., vol. 26, no. 1, pp. 52–57, Feb. 2019

  6. [6]

    Fundamental rate limits of UA V -enabled m ultiple access channel with trajectory optimization

    P . Li and J. Xu, “Fundamental rate limits of UA V -enabled m ultiple access channel with trajectory optimization.” IEEE Trans. Wireless Commun., vol.19, no. 1, pp. 458–474, Jan. 2020

  7. [7]

    Optimal LA P altitude for maximum coverage,

    A. Al-Hourani, S. Kandeepan, and S. Lardner, “Optimal LA P altitude for maximum coverage,” IEEE Wireless Commun. Lett., vol. 3, no. 6, pp. 569–572, Dec. 2014

  8. [8]

    Placement optimi zation of UA V -mounted mobile base stations,

    J. Lyu, Y . Zeng, R. Zhang, and T. J. Lim, “Placement optimi zation of UA V -mounted mobile base stations,” IEEE Commun. Lett. , vol. 21, no. 3, pp. 604–607, Mar. 2017

Show all 22 references
  1. [9]

    Placement optimization for UA V -enabled wireless networks with multi-hop backhauls,

    P . Li and J. Xu, “Placement optimization for UA V -enabled wireless networks with multi-hop backhauls,” J. Commun. Inf. Netw., vol. 3, no. 4, pp. 64–73, Dec. 2018. 7

  2. [10]

    Mobile unmanned aerial vehicles (UA Vs) for energy-efficient Internet of thi ngs communi- cation,

    M. Mozaffari, W. Saad, M. Bennis, and M. Debbah, “Mobile unmanned aerial vehicles (UA Vs) for energy-efficient Internet of thi ngs communi- cation,” IEEE Trans. Wireless Commun. , vol. 16, no. 11, pp. 7574–7589, Nov. 2017

  3. [11]

    Throughput maximization fo r UA V - enabled wireless powered communication networks,

    L. Xie, J. Xu, and R. Zhang, “Throughput maximization fo r UA V - enabled wireless powered communication networks,” IEEE Internet Things J. , vol. 6, no. 2, pp. 1690–1703, Apr. 2019

  4. [12]

    Cellular-enabled UA V c ommu- nication: A connectivity-constrained trajectory optimiz ation perspec- tive,

    S. Zhang, Y . Zeng, and R. Zhang, “Cellular-enabled UA V c ommu- nication: A connectivity-constrained trajectory optimiz ation perspec- tive,”IEEE Trans. Commun. , vol. 67, no. 3, pp. 2580–2604, Mar. 2019

  5. [13]

    Deep learning in physical layer communications,

    Z. Qin, H. Y e, G. Y . Li, and B.-H. F. Juang, “Deep learning in physical layer communications,” IEEE Wireless Commun. , vol. 26, no. 2, pp. 93– 99, Apr. 2019

  6. [14]

    Trajectory optimization for autonomous flying base station via reinforcement learni ng,

    H. Bayerlein, P . D. Kerret, and D. Gesbert, “Trajectory optimization for autonomous flying base station via reinforcement learni ng,” in Proc. IEEE SPAWC, Jun. 2018

  7. [15]

    Energy-ef ficient UA V control for effective and fair communication coverage: A deep reinforcement learning approach

    C. H. Liu, Z. Chen, J. Tang, J. Xu, and C. Piao, “Energy-ef ficient UA V control for effective and fair communication coverage: A deep reinforcement learning approach”, IEEE J. Sel. Areas Commun. , vol. 36, no. 9, pp. 2059–2070, Sep. 2018

  8. [16]

    Interferenc e management for cellular-connected UA Vs: A deep reinforcement learning approach,

    U. Challita, W. Saad, and C. Bettstetter, “Interferenc e management for cellular-connected UA Vs: A deep reinforcement learning approach,” IEEE Trans. Wireless Commun. , vol. 18, no. 4, pp. 2125–2140, Apr. 2019

  9. [17]

    Path design for cellular-connected U A V with reinforcement learning,

    Y . Zeng and X. Xu, “Path design for cellular-connected U A V with reinforcement learning,” in IEEE Globecom , pp. 1–6, Dec. 2019

  10. [18]

    Optimal positioning of flying re lays for wireless networks: A LOS map approach,

    J. Chen and D. Gesbert, “Optimal positioning of flying re lays for wireless networks: A LOS map approach,” in Proc. IEEE ICC , Jul. 2017

  11. [19]

    R. S. Sutton and A. G. Barto, Introduction to Reinforcement Learning, 2nd ed. Cambridge, Massachusetts: MIT Press, 2017

  12. [20]

    A review of deep learning methods and applications for unmann ed aerial vehicles,

    A. Carrio, C. Sampedro, A. Rodriguez-Ramos, and P . Camp oy, “A review of deep learning methods and applications for unmann ed aerial vehicles,” J. Sensors , vol. 2017, no. 3296874, 2017

  13. [21]

    David. T and P . Viswanath, Fundamentals of Wireless Communication , Cambridge University press, 2005

  14. [22]

    Optimal L AP altitude for maximum coverage,

    A. Al-Hourani, S. Kandeepan, and S. Lardner, “Optimal L AP altitude for maximum coverage,” IEEE Wireless Commun. Lett., vol. 3, no. 6, pp. 569–572, Dec. 2014

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.