{"id":"55b25ea0-6e1c-49a6-8fc8-ad13a7cd9e09","arxiv_id":"2411.14052","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A maximum-entropy mean-field deep Q-network is proposed for joint trajectory, user association, and power control in dense UAV networks.","lead":"This paper uses mean-field game theory and deep reinforcement learning to let hundreds of drones acting as base stations jointly choose where to hover, which ground user to serve, and how much power to transmit as user demand changes over time. The simulations suggest a gain in energy efficiency over simpler baselines, but the paper's proof of a unique equilibrium has a serious gap.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1 is unproven: the proof stops at the bound (ς1ς2+ς3)dM(L,L̂) without ever establishing ς1ς2+ς3 < 1, and the imported Lemmas 1–2 are not verified for the indicator-based reward (20).","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing concern: the proof of Theorem 1 fails to establish the contraction factor ς1ς2+ς3 < 1 and imports lemmas without verifying their conditions for this specific reward and transition structure. My independent reading confirms this is the central defect. The reward (20) contains threshold indicators that are not obviously Lipschitz in the mean-field, and the energy queue dynamics (18) add further nonlinearities. Moreover, the paper never even defines what ς1, ς2, ς3 are, nor does it relate them to the discount factor γ or the penalty factors σ and ξ. The proof is therefore internally incomplete, and Theorem 1 cannot be accepted as proved. Since the reader's verdict of REJECT is based on this exact issue, no adjustment is needed. I am not raising a novel objection; I am affirming that the identified gap is decisive and that the simulated results, however plausible, do not compensate for the missing theoretical support because the claimed equilibrium is the justification for the algorithm's behaviour.","tokens_in":23372,"tokens_out":4664,"duration_ms":51492,"concrete_test":"Independently compute or bound ς1, ς2, ς3 for the specific system with the simulation parameters (γ=0.9, σ=240, η=0 dB, finite state/action spaces with Le energy levels and Lp power levels). Since the state and action sets are finite after discretization, one can formulate Υ1 and Υ2 as finite-dimensional operators and numerically estimate their Lipschitz constants over all pairs of distributions (e.g., by exhaustive or random sampling of the probability simplex). If any computed ratio dM(Υ(L),Υ(L̂))/dM(L,L̂) exceeds or reaches 1, or if ς1ς2+ς3 ≥ 1 is found, Theorem 1 is false as stated. Alternatively, verify the hypotheses of Lemma 1 [46] and Lemma 2 [35] for reward (20) and transition (18); if the reward is not Lipschitz in the mean-field due to the SINR threshold, the imported lemmas cannot be applied.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is Theorem 1: the fixed-point iteration L' = Υ2(Υ1(L),L) converges to a unique stationary mean-field equilibrium. Appendix A attempts a Banach fixed-point argument, but the final bound in Eq. (48) is dM(Υ(L),Υ(L̂)) ≤ (ς1ς2+ς3)dM(L,L̂), and the paper provides no argument, computation, or bound showing ς1ς2+ς3 < 1. Without strict contraction, neither existence nor uniqueness follows from the Banach theorem. The lemmas themselves are imported: Lemma 1 from [46] and Lemma 2 from [35] are invoked with 'proof omitted', and their hypotheses (Lipschitz reward and transition in the measure, compactness, etc.) are never checked for this model. This is not merely a missing constant: the reward in Eq. (20) includes the rate term (9) built from indicators 1(SINR ≥ η), and the energy queue (18) contains min/max nonlinearities; whether these satisfy the Lipschitz conditions of [35,46] is not established. If ς1ς2+ς3 ≥ 1, the contraction argument collapses and the claimed uniqueness of the mean-field equilibrium is unsupported. The theorem is load-bearing because the ME-MFDQN algorithm and the reported energy-efficiency gains are justified as approximations to a well-defined equilibrium; without a proved equilibrium, the theoretical foundation of the paper fails.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies ultra-dense UAV downlink networks with time-varying user demands. Each UAV jointly optimizes its hovering location, user association, and transmit power to maximize its own cumulative energy efficiency under interference and energy constraints. The authors formulate this as a stochastic game, then approximate it as a mean-field game (MFG) in which a representative UAV interacts with a state-action distribution mean-field. They state Theorem 1, claiming that the fixed-point iteration L' = Y2(Y1(L), L) converges to a unique stationary mean-field equilibrium. They then propose a maximum-entropy mean-field deep Q-network (ME-MFDQN) for fully and partially observable scenarios, and present simulations showing energy-efficiency gains over several baselines, plus qualitative trends relating demand predictability and observation radius to flying probability and transmit power.","tokens_in":1406,"tokens_out":1536,"duration_ms":38848,"significance":"The application domain is timely, and the paper includes a fairly complete system model with an energy-harvesting queue, a fly-hover-communicate protocol, and a clear simulation comparison. If the mean-field equilibrium result were fully established, the paper would provide a scalable learning-theoretic foundation for decentralized UAV resource allocation under non-stationary demand. The empirical work is internally consistent, the parameters are listed in detail, and the qualitative conclusions (less flying and lower power when demand is more predictable or observation is wider) are plausible and clearly explained. However, the central theoretical contribution, Theorem 1, is not actually proved in the manuscript; the proof in Appendix A stops at a bound that is not shown to be a contraction. This is a load-bearing gap because the algorithm and the reported gains are justified as approximations to a well-defined equilibrium.","major_comments":[{"comment":"The proof of Theorem 1 ends with dM(Y(L), Y(Lhat)) <= (c1*c2 + c3) dM(L, Lhat). The Banach fixed-point theorem is then invoked, but the authors never establish the strict contraction condition c1*c2 + c3 < 1. Without this inequality, neither existence nor uniqueness of the mean-field equilibrium follows from the argument presented. This is not a purely cosmetic omission: the constants c1, c2, c3 are introduced only as abstract existence constants in Lemmas 1 and 2, and no bound or parameter condition is given for the specific reward (20), transition structure, or state and action spaces of this UAV model.","section":"Appendix A, Eq. (48)"},{"comment":"Lemma 1 is imported from [46] and Lemma 2 from [35] with proofs omitted, but the hypotheses of those lemmas are not verified for the present model. In particular, the reward in Eq. (20) contains the energy-efficiency term built from the indicator-based rate expression in Eq. (9) and the min/max nonlinearity in the energy queue update (18), and the state includes the two-state Markov demand process. No argument shows that the reward and transition kernels satisfy the Lipschitz conditions in the measure required by the cited lemmas, so the existence of c1, c2, c3 is not established for this system.","section":"Appendix A, Lemmas 1 and 2"},{"comment":"The theorem is advertised as proving existence and uniqueness of the mean-field equilibrium, and the ME-MFDQN algorithm is described as approximating that equilibrium by iterating between policy optimization (lines 5-15 of Algorithm 1) and mean-field update (line 16). Because the contraction proof is incomplete, the link between the algorithm's empirical convergence curves (Figs. 4-12) and a formal equilibrium guarantee is unsupported. The paper should either provide a complete proof of the contraction constant, or substantially reformulate the theoretical claims to reflect what is actually proven.","section":"Section III-B, Theorem 1 and Section IV-A"}],"minor_comments":[{"comment":"The last paragraph of the introduction says 'Finally, Section VII concludes the paper', but the conclusion is Section VI. Please correct the cross-reference.","section":"Section I"},{"comment":"The text says 'In Section V-B, we will extend the discussions to the partially observable scenario', but the partially observable extension appears in Section IV-B, not V-B.","section":"Section II"},{"comment":"The y-axis label is spelled 'Averge reward' in Figures 4(a), 5, and 10; it should be 'Average reward'.","section":"Figures 4, 5, 10"},{"comment":"Line 16 computes L_{k+1} from the current policy, but the stopping criterion is only described verbally as 'until the new mean-field L matches L'. A formal convergence criterion or tolerance would improve reproducibility.","section":"Algorithm 1"},{"comment":"The observation notation x_o(u,t) in {0,1,2} is introduced, but the mapping from these values to the belief state in the history compression z(t) is not fully specified; please clarify how the history is constructed from the raw observations.","section":"Section IV-B"}],"recommendation":"major_revision","confidential_remarks":"The main issue is a missing load-bearing proof: the contraction constant in Appendix A is never shown to be less than one, and the imported lemmas are not verified for this model. This is fixable only if the authors can provide a concrete argument (or a well-justified additional assumption) that yields c1*c2 + c3 < 1 and that verifies the Lipschitz conditions for the specific reward and transition structure. If they cannot, the theoretical contribution of the paper would be substantially weakened, and the practical results would stand only as an algorithmic study. I recommend sending the paper back with a request to complete or qualify the proof."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a competent, well-written application of mean-field RL to a genuinely new problem combination—joint trajectory, user association, and power control for dense UAVs serving users with Markov-modulated time-varying demands. The algorithm (max-entropy mean-field DQN) is a sensible extension, and the simulations, including partial observability, show internally consistent trends. The problem formulation alone is worth having.\n\nThe paper does not, however, prove its main theorem. In Appendix A, the proof of Theorem 1 (existence and uniqueness of the mean-field equilibrium) imports Lemma 1 from [46] and Lemma 2 from [35] without checking their hypotheses for this model. The reward (20) contains the indicator-based rate (9) and min/max energy queue dynamics (18); whether those maps are Lipschitz in the measure is not established. Then the final inequality (48) is dM(Υ(L),Υ(L̂)) ≤ (ς1ς2+ς3)dM(L,L̂), and no argument shows ς1ς2+ς3 < 1. Without that, the Banach fixed-point theorem does not apply. This is not a cosmetic omission; the theorem is load-bearing because the ME-MFDQN algorithm and the reported energy-efficiency gains are framed as approximating that equilibrium. The fix is feasible—either prove the contraction with the specific reward/transition or weaken the claim to 'empirically convergent' and present the theory as heuristic.\n\nOther soft spots are minor by comparison. No code or data is provided, and several hyperparameters (φ, ξ, emin, battery capacity, etc.) are not listed, so the simulations are hard to reproduce independently. There's also a small typo in Section VI saying 'Section VII concludes'.\n\nOn the credit side: the empirical study is honest, the parameter sweeps over q, σ, and SINR threshold give useful qualitative insight (less flying and lower power when demand is more predictable), and the partial-observability extension with the observation-radius tradeoff is a reasonable addition. The citation pattern is normal; the imported lemmas come from other authors' work, so there's no circularity issue.\n\nWho should read this: researchers in UAV communications or mean-field RL who want a clean statement of a time-varying-demand MFG formulation and a DQN-based baseline. It deserves a serious referee, but the referee should ask for the theorem to be fixed or downgraded before publication. I would not desk-reject it; I'd send it out with major revision.","headline":"Useful new MFG formulation for dense UAV resource allocation, but the central equilibrium theorem is not proven—the appendix misses the contraction bound.","tokens_in":24212,"tokens_out":2662,"would_cite":true,"duration_ms":25591,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a dense swarm of self-interested UAVs, each choosing where to fly and how strongly to transmit, has a unique mean-field equilibrium, and that a maximum-entropy deep reinforcement learning algorithm can find it…","keywords":["mean-field game","UAV networks","trajectory design","power control","deep reinforcement learning","maximum entropy","energy efficiency","mean-field equilibrium"],"falsifier":"In the paper's own simulated network, run the two-step iteration from two very different initial mean-field distributions and check whether they converge to the same limit; also estimate the Lipschitz constants in Lemmas 1 and 2 for the reward in (20). If the product $s_1 s_2 + s_3$ is at least 1, or the two runs settle at different distributions, the uniqueness claim of Theorem 1 fails.","tokens_in":23194,"feed_emoji":"🚁","tokens_out":8213,"duration_ms":73853,"temperature":0.7,"pith_summary":"The paper tries to establish that the non-cooperative resource-allocation problem in ultra-dense UAV networks is tractable: each UAV selfishly chooses its trajectory, user association, and downlink transmit power, and the paper shows this can be formulated as a mean-field game with a unique equilibrium. The motivation is that as UAV densities grow, the standard stochastic-game formulation becomes computationally hopeless, while a mean-field game reduces the interactions to a two-player game between a representative UAV and an aggregate distribution. A model-free maximum-entropy deep Q-network algorithm, ME-MFDQN, is then proposed to approximate the equilibrium, and simulations show it improves energy efficiency over baselines. The practical stake is a scalable, decentralized way to control dense UAV fleets serving ground users with unknown, time-varying demand.","feed_headline":"UAV swarms can settle on a unique flight-and-power equilibrium","feed_subtitle":"A mean-field game plus entropy-regularized deep Q-learning cuts interference and raises energy efficiency in simulations.","key_machinery":"The carrying object is the mean-field equilibrium, a policy-distribution pair $(\\pi^*, L^*)$ in which $\\pi^*$ is the representative UAV's best response to the fixed aggregate behavior $L^*$, and $L^*$ is the joint state-action distribution that arises when every UAV follows $\\pi^*$. The machinery is the two-step iteration $\\Upsilon(L) = \\Upsilon_2(\\Upsilon_1(L), L)$: fix the mean-field distribution, solve a single-agent reinforcement-learning problem for the best response, then update the distribution through the transition kernel. To make the first step model-free, the paper uses a maximum-entropy soft Q-function, so the policy is the Boltzmann softmax of the Q-values and exploration is controlled by a temperature parameter while the mean-field distribution is updated between training phases.","core_discovery":"The central claim is Theorem 1: with any initial mean-field distribution $L$, the fixed-point iteration $L' = \\Upsilon_2(\\Upsilon_1(L), L)$ converges to the unique stationary mean-field equilibrium. In the paper's two-step construction, $\\Upsilon_1$ maps a fixed mean-field distribution to the representative UAV's optimal policy, and $\\Upsilon_2$ maps that policy back to an updated joint state-action distribution. The proof reasons through the Banach fixed-point theorem, using two Lipschitz lemmas imported from prior work, and the algorithm ME-MFDQN approximates the same equilibrium by maximizing a soft Q-function with a policy-entropy bonus. The paper also extends the formulation to a partially observable scenario where each UAV records a compressed history trajectory, and reports that wider observation helps energy efficiency.","pith_inferences":["A testable extension is to compute the contraction coefficient numerically for the reward in (20) and the transition structure; if it is strictly below 1, the uniqueness proof would be confirmed for this model rather than inherited from other settings.","The mean-field assumption of homogeneous, exchangeable UAVs is the price of the simplification; heterogeneous fleets with different propulsion, battery, or cell sizes would likely need a multi-class mean-field formulation, which the paper does not address.","The observed robustness to a 10-UAV drop suggests mean-field RL policies could serve as fault-tolerant controllers for UAV networks, but the paper only demonstrates this for one small population change.","Comparing ME-MFDQN's learned flying probabilities to an analytic benchmark of optimal user tracking could reveal how close the model-free equilibrium is to the true best response."],"forward_implications":["If Theorem 1 is right, the two-step iteration converges to the same equilibrium no matter which initial mean-field distribution is chosen, so dense-UAV resource allocation has a well-defined single prediction.","Because ME-MFDQN is model-free, UAVs can learn the equilibrium from accumulated experience alone, without knowing the distribution of ground users' service demands.","The simulations indicate that as service demands become more temporally correlated, the learned policy flies less and transmits at lower power, increasing energy efficiency, a behavioral prediction of the mean-field model.","The algorithm degrades gracefully with partial observability: 75% observation nearly matches full observation, while 25% observation significantly reduces energy efficiency.","The robustness experiment shows that a policy trained with 361 UAVs continues to perform well with 351 UAVs, meaning the mean-field solution tolerates small population changes without retraining."],"supporting_citations":[{"why":"Supplies Lemma 1, the Lipschitz continuity of the best-response map $\\Upsilon_1$ used in the contraction proof of Theorem 1.","marker":"[46]"},{"why":"Supplies Lemma 2, the Lipschitz bounds on the distribution-update map $\\Upsilon_2$, and the Banach fixed-point framework for the equilibrium iteration.","marker":"[35]"},{"why":"Provides the entropy-regularized mean-field reinforcement learning framework and the convergence result that ME-MFDQN adapts.","marker":"[34]"},{"why":"Contributes the two-state Markov model of time-varying ground-user demand that drives the UAV's dynamic deployment problem.","marker":"[38]"},{"why":"Contributes the rotary-wing UAV propulsion energy model and the fly-hover-communicate protocol used in the energy-efficiency objective.","marker":"[39]"},{"why":"Provides the soft Q-function and Boltzmann policy form used to add maximum entropy to the deep Q-network.","marker":"[42]"},{"why":"Earlier mean-field reinforcement learning for downlink power control in ultra-dense UAV networks, which this paper extends to joint trajectory, association, and power decisions with dynamic demands.","marker":"[36]"}],"fun_headline_variants":["Unique UAV equilibrium via mean-field reinforcement learning","Mean-field game proves unique flight-and-power solution","Entropy-regularized DQN solves UAV swarm coordination","UAV swarms find unique strategy via mean-field game","Mean-field RL boosts energy efficiency in dense UAV nets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole equilibrium proof rides on the assumption that the iteration updating the crowd's behavior always pulls any two possible crowd behaviors closer together; the paper takes this shrinking property from other works and does not verify it for its specific reward and movement model.","fun_headline_variants_meta":{"raw":{"variants":["Unique UAV equilibrium via mean-field reinforcement learning","Mean-field game proves unique flight-and-power solution","Entropy-regularized DQN solves UAV swarm coordination","UAV swarms find unique strategy via mean-field game","Mean-field RL boosts energy efficiency in dense UAV nets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000408,"raw_usage":{"total_tokens":2135,"prompt_tokens":977,"completion_tokens":1158,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":1082}},"tokens_in":593,"tokens_out":1158,"duration_ms":9913,"temperature":1.0,"reasoning_tokens":1082,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:35:15.339134+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"In the paper's own simulated network, run the two-step iteration from two very different initial mean-field distributions and check whether they converge to the same limit; also estimate the Lipschitz constants in Lemmas 1 and 2 for the reward in (20). If the product $s_1 s_2 + s_3$ is at least 1, or the two runs settle at different distributions, the uniqueness claim of Theorem 1 fails.","supporting_citations":[{"cited_title":"Large popula tion stochastic dynamic games: Closed-loop McKean-Vlasov syst ems and the Nash certainty equivalence principle,","cited_arxiv_id":null,"evidence_quote":"Supplies Lemma 1, the Lipschitz continuity of the best-response map $\\Upsilon_1$ used in the contraction proof of Theorem 1."},{"cited_title":"Learning mean-field ga mes,","cited_arxiv_id":null,"evidence_quote":"Supplies Lemma 2, the Lipschitz bounds on the distribution-update map $\\Upsilon_2$, and the Banach fixed-point framework for the equilibrium iteration."},{"cited_title":"Approximately solving mean field g ames via entropy-regularized deep reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Provides the entropy-regularized mean-field reinforcement learning framework and the convergence result that ME-MFDQN adapts."},{"cited_title":"Chase or wait: Dynamic UA V deployme nt to learn and catch time-varying user activities,","cited_arxiv_id":null,"evidence_quote":"Contributes the two-state Markov model of time-varying ground-user demand that drives the UAV's dynamic deployment problem."},{"cited_title":"Energy minimization for wi reless communication with rotary-wing UA V,","cited_arxiv_id":null,"evidence_quote":"Contributes the rotary-wing UAV propulsion energy model and the fly-hover-communicate protocol used in the energy-efficiency objective."},{"cited_title":"Reinfor cement learning with deep energy-based policies,","cited_arxiv_id":null,"evidence_quote":"Provides the soft Q-function and Boltzmann policy form used to add maximum entropy to the deep Q-network."},{"cited_title":"Downlink tra nsmit power control in ultra-dense UA V network based on mean field g ame and deep reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Earlier mean-field reinforcement learning for downlink power control in ultra-dense UAV networks, which this paper extends to joint trajectory, association, and power decisions with dynamic demands."}],"review_version":1}