{"id":"aaf9cf06-1d69-483a-960c-8d19e71c63bd","arxiv_id":"2502.01268","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A hybrid meta-offline reinforcement learning algorithm trains a UAV to minimize data age and transmission power from static datasets and adapts to new tasks in under 40 epochs.","lead":"This paper combines two existing machine learning techniques, conservative Q-learning and model-agnostic meta-learning, to train a drone's flight plan from a small static dataset instead of online trial and error. The authors report the hybrid method adapts faster to new tasks and weather disruptions than baseline reinforcement learning methods in simulation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported results rest on a reward that is non-causal as printed: Eq. (6) assigns every time step the full-episode cost, making TD targets in Eqs. (8)-(9) unusable unless the implementation silently used a per-step reward.","rationale":"The reader's conditional verdict is well-founded. I agree with the weakest_assumption: Eq. (6) is the load-bearing flaw. In good faith, I do not see the non-causal reward as evidence of misconduct; it is most plausibly a typo where the author copied the objective into the reward box. But a typo in the central reward equation is precisely the kind of error that must be fixed before acceptance because the entire empirical section is downstream of it. I considered whether the absence of statistical error bars or the 'only algorithm/optimal' overclaim is more serious; those are real but secondary. Even with error bars and a better baseline, the results would be meaningless if the reward used in training is not the reward described. The proposed remedy—correct Eq. (6), specify the exact reward, and reproduce one headline number with a causal reward—is enough to make the paper conditionally acceptable. I would not escalate to REJECT because the overall design (CQL plus MAML inner/outer updates) is coherent and a simple per-step reward correction is plausible. Hence verdict_should_be remains UNCHANGED (CONDITIONAL).","tokens_in":14628,"tokens_out":5504,"duration_ms":52616,"concrete_test":"Ask the authors for the reward-function implementation, or reconstruct it from the description. Run the meta-testing experiment (Fig. 8) twice: (a) with the literal Eq. (6) reward computed from the completed episode at every step; (b) with the per-step reward r(t)=-Σ_k[δ_k A_k(t)+(λ/K)P_k(t)]. Report convergence epochs, AoI, power, and outage counts for both. If (a) cannot reproduce the claimed AoI=6/power=1.6 mW/under-40-epoch results while (b) can, Eq. (6) is a typo and the claims are conditional on the corrected reward; if neither reproduces, the numerical results are unsupported. A minimal additional check is to verify in the code that at update time the reward tensor contains no components from t'>t.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that CQL-MAML attains near-optimal AoI and power with few offline shots and fast adaptation. For that claim to be true, the reward used in training must be a causal function of the current state and action. As printed, Eq. (6) is r(t)=-(1/T)Σ_{t'=1}^T Σ_k [δ_k A_k(t')+(λ/K)P_k(t')]. This is the full episode cost repeated at every time step. Plugging it into the DQN loss (8) or the CQL loss (9) makes the TD target r+γ max Q(s',a') contain future AoI and power values that are not available at decision time t, so the Bellman updates are not a valid bootstrap and the learned Q-function cannot represent the causal objective in P1. The only way the simulations in Figs. 5-9 could have been run is if the code actually used a per-step reward such as r(t)=-Σ_k[δ_k A_k(t)+(λ/K)P_k(t)], which is not what the paper says. This is not a cosmetic typo: every numerical claim—convergence under 40 epochs, AoI=6/power=1.6 mW, outage avoidance—depends on which reward was coded. The paper must state the reward that was used, correct Eq. (6), and provide evidence (code or a precise implementation description) that the TD targets were causal.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CQL-MAML, a combination of conservative Q-learning (CQL) with model-agnostic meta-learning (MAML), for offline few-shot training of a UAV trajectory and scheduling policy that minimizes the age-of-information (AoI) and transmission power of IoT devices. The authors formulate a joint optimization problem P1, define an MDP with state/action/reward, and present Algorithms 1-4 for DQN, CQL, MAML, and the proposed meta-offline method. Numerical experiments in Section V compare CQL-MAML against CQL, DQN, and DQN-MAML in adaptive-objective and resilience (rain outage) scenarios, reporting faster convergence, lower AoI and power, and outage avoidance.","tokens_in":14953,"tokens_out":3512,"duration_ms":33687,"significance":"If the reported results are reproducible, the combination of offline RL with MAML is a practical contribution to wireless UAV trajectory optimization, because it addresses two real concerns: avoiding online interaction and adapting to new tasks or channel disruptions with limited data. The paper also evaluates a concrete resilience scenario (changing rain-outage areas), which is valuable for deployments in smart agriculture. The work is empirical; there is no theoretical optimality proof, and the 'optimal' claims are not backed by certificates. Nonetheless, the proposed architecture and the resilience experiment are of interest to the community.","major_comments":[{"comment":"The reward in Eq. (6) is non-causal as written: r(t) is defined as the negative of (1/T) times the sum over all time steps t' of the AoI and power costs, so the same full-episode cost is assigned to every time step. Substituting this r(t) into the DQN loss (8) or the CQL loss (9) makes the TD target r + γ max_a' Q(s',a') depend on future states' AoI and power values that are unavailable at decision time t. Thus the Bellman bootstrap is invalid and the simulations in Figs. 5-9 could not have been produced by the described algorithm unless the implementation used a per-step reward such as r(t) = -Σ_k [δ_k A_k(t) + (λ/K) P_k(t)]. The authors must correct Eq. (6), state precisely which reward was used in the code, and provide code or a sufficiently detailed implementation description to verify causality of the TD updates.","section":"§III-A, Eq. (6)"},{"comment":"Constraint (5c) is written |D| ≤ Dconst, while the sentence immediately below it says the constraint bounds the offline dataset size to be 'greater than or equal to a certain threshold size Dconst'. These directions are contradictory. Moreover, (5c) does not constrain the optimization variables v(t) and w(t), so it is not a constraint of P1 in the usual sense. Please correct the inequality direction and clarify whether this is a problem constraint or a data-availability assumption.","section":"§II-A, Eq. (5c)"},{"comment":"The central quantitative claims—CQL-MAML converges in fewer than 40 epochs, achieves AoI=6 and power=1.6 mW versus AoI=28 and power=6 mW for CQL, and avoids outages—are presented without error bars, multiple seeds, or confidence intervals. DQN/CQL training is stochastic, and the figures appear to show single runs (or a single aggregation). At least 5-10 independent training runs with mean and standard deviation should be reported before these comparative claims can be supported.","section":"§V-B, Figs. 5-8"},{"comment":"The abstract and conclusion claim that the algorithm 'achieves optimal joint AoI and transmission power' and is 'the only algorithm' that does so. No optimality certificate, lower bound, exhaustive search, or comparison against a known optimal policy is provided; the experiments only compare against DQN, DQN-MAML, and CQL baselines. These statements should be softened to claims of superiority over the considered baselines unless an optimal benchmark is added.","section":"§V-B and §VI"},{"comment":"The contribution list in Section I-D describes a 'meta-offline MARL framework,' and the abstract uses 'few-shot meta-offline RL,' but Algorithm 4 and the experiments describe a single-agent setup with one UAV and no multi-agent interactions. If MARL is intended, the formulation in Section II and the algorithm in Section IV are missing the multi-agent components (agent index, joint actions, shared/private rewards, or communication). The terminology should be corrected to avoid overclaiming novelty relative to prior multi-agent offline RL works.","section":"§I-D and §IV"},{"comment":"The 'few-shot' aspect is never made precise: Algorithm 4 samples 'k shots' in the support set, but the value of k is not specified in Table I or the experiments, and Section V uses dataset sizes of 100, 300, and 500 experiences, which are not the typical 'few-shot' regime in meta-learning benchmarks. The authors should state the exact k for support and query sets and discuss why the reported dataset sizes constitute 'few-shot' offline learning.","section":"§IV, Algorithm 4"}],"minor_comments":[{"comment":"The term 'UA V' is inconsistently spaced throughout the manuscript; it should be 'UAV' for uniformity.","section":"Throughout"},{"comment":"In the introduction, 'receives an award' should be 'receives a reward'; the same typo appears in the abstract and in Algorithm 1 comments.","section":"§I-A"},{"comment":"Algorithm 4 line 12 says 'Return model converged initial weights w0'; this should read 'Return the converged initial weights w0.'","section":"§IV, Algorithm 4"},{"comment":"The name 'meta-CQL' is used interchangeably with 'CQL-MAML' in Section V; consistency in naming would improve clarity.","section":"§V"},{"comment":"The neural network architecture is described only as '2 hidden layers'; the number of neurons per layer, activation functions, batch size, and exploration schedule (epsilon decay) are not given, making reproduction difficult.","section":"§V.A"},{"comment":"The noise power is denoted σ2 in Eq. (3) and in the text, but Table I gives σ2 = -100 dBm; the units and the symbol should be made consistent (e.g., sigma^2 vs sigma).","section":"Eq. (3)"},{"comment":"Reference [3] is incomplete: the standard citation for Sutton and Barto's book lacks the publisher and year; please complete all bibliographic entries.","section":"References"},{"comment":"'monte-Carlo loops' should be capitalized as 'Monte Carlo loops' and the sentence should be completed with a period.","section":"§V.C"}],"recommendation":"major_revision","confidential_remarks":"The paper is clearly positioned in the authors' own line of work, with several self-references, but the central empirical comparison is an independent measurement. The main blockage is the non-causal reward in Eq. (6), which must be corrected and verified with code before the results can be trusted. The lack of multiple seeds and error bars is a serious but fixable gap. The 'optimal' claims should be toned down. The paper may be suitable for a journal after these revisions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: the paper is a straightforward integration of CQL and MAML, applied to UAV trajectory/scheduling for AoI and power minimization. The results are plausibly real, but the manuscript has one genuinely load-bearing presentation error—the printed reward in Eq. (6) is non-causal—and a few smaller sloppiness items. I'd send it to review, but only after the authors fix the reward definition and release code or a detailed implementation description.\n\nWhat's new: CQL-MAML for offline UAV trajectory optimization with few-shot adaptation. That is new in this specific wireless setting, though not conceptually surprising: MAML finds initial weights, CQL trains offline. The value is in the demonstration. The paper does a few things well: it defines tasks via the λ trade-off, tests dataset size and number of meta-tasks, and includes a resilience experiment with a shifting outage area. The reported convergence advantage (under 40 epochs vs. more than 100 for CQL) is consistent with the expected effect of a good initialization, and the outage-avoidance result is a sensible empirical claim.\n\nNow the soft spots. Eq. (6) defines r(t) as the full-episode average cost repeated at every time step. If taken literally, the TD target in Eqs. (8)–(9) contains future AoI and power values, so the Bellman update is invalid and the learned Q-function cannot represent the causal objective. The simulations in Figs. 5–9 could only have been produced by a per-step reward, probably r(t) = -Σ_k[δ_k A_k(t) + (λ/K)P_k(t)], which is not what the paper says. That is not cosmetic; every number depends on which reward was coded. The dataset constraint (5c) is printed as |D| ≤ Dconst while the text says the dataset should be at least Dconst; one of them is wrong. There are no error bars or multiple seeds, so I cannot tell how robust the 40-epoch claim is. And “optimal” is used loosely—there is no optimality certificate or comparison to an exhaustive or optimal baseline.\n\nThe self-citation pattern does not bother me much; the authors' prior offline-RL and meta-RL papers are the natural building blocks, and the new content is the comparison itself, not a derivation from thin air.\n\nBottom line: this is a plausible incremental contribution for the wireless-RL community. With Eq. (6) fixed, the constraint clarified, and code/data released, it could be a solid workshop or journal paper. As posted, it should not be accepted without correction. I would still send it to peer review—a good referee will catch exactly these issues, and the direction is worth engaging.","headline":"A plausible CQL+MAML integration for offline UAV trajectory optimization, but the printed reward is non-causal and the optimality claims outrun the evidence.","tokens_in":15466,"tokens_out":2321,"would_cite":false,"duration_ms":22093,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A meta-offline RL algorithm adapts UAV trajectory planning to new environments in under 40 training epochs, using only offline data.","keywords":["age-of-information","meta-learning","offline reinforcement learning","precise agriculture","resilience","unmanned aerial vehicles","conservative Q-learning","few-shot learning"],"falsifier":"Run Algorithm 4 exactly as written with the reward in Eq. (6) and check whether the temporal-difference loss can even be computed at each step; if the algorithm does not converge to the reported AoI of 6 and power of 1.6 mW at $\\lambda = 300$, the claimed results cannot come from the described reward. Alternatively, with the intended per-step reward, compare CQL-MAML against CQL over many random seeds and verify that the meta-learned initialization converges in under 40 epochs while CQL needs more than 100; a single seed or a different initialization should not flip the gap.","tokens_in":14387,"feed_emoji":"🚁","tokens_out":8629,"duration_ms":64979,"temperature":0.7,"pith_summary":"This paper proposes a few-shot meta-offline reinforcement learning algorithm that combines conservative Q-learning (CQL) with model-agnostic meta-learning (MAML) to plan a UAV's trajectory and scheduling policy from a static offline dataset, without any online interaction with the environment. The authors aim to show that meta-learning provides an initialization that lets CQL adapt to a new environment, defined by a new trade-off weight $\\lambda$ between age-of-information (AoI) and transmission power, in under 40 training epochs, whereas plain CQL needs more than 100 epochs. They report that the method is the only offline scheme that reaches a joint optimum in their tests, achieving AoI of 6 and transmission power of 1.6 mW versus AoI of 28 and 6 mW for CQL, and that it also makes the UAV avoid a sudden heavy-rain outage area with near-zero outages after about 100 epochs. If correct, this would make offline-trained RL practical for UAV networks in smart agriculture and other settings where online training is costly, unsafe, or impossible.","feed_headline":"Meta-offline RL cuts UAV adaptation to under 40 epochs","feed_subtitle":"Combining CQL with MAML learns from static data, beats online DQN on age and power, and dodges rain outages.","key_machinery":"The load-bearing mechanism is the meta-learned initial weight vector $w_0$ of the Q-network, produced by an outer loop that minimizes the sum of CQL losses over meta-training tasks after each task's inner-loop CQL updates on support samples. Inside each task, CQL's conservative regularization $\\alpha \\hat{E}[\\log \\sum_{\\tilde a} \\exp(Q(s,\\tilde a)) - Q(s,a)]$ prevents out-of-distribution actions from being overestimated, which is what makes offline training possible at all; the meta-loop then turns that offline learner into a fast adapter by finding an initialization from which a few gradient steps on a new task's data suffice. The same mechanism is what the paper credits for resilience: the initialization encodes how to balance AoI and power and where safe regions are, so a new task with an outage zone can be learned quickly.","core_discovery":"The central claim is that CQL-MAML, an algorithm that pretrains a Q-network's initial weights with MAML over several tasks (each with a different $\\lambda$) and then runs a few CQL updates on a new task's offline support set, converges to the optimal policy where standard offline and online baselines do not. In the authors' experiments, the meta-learned initialization converges in fewer than 40 epochs on a new unseen task, while CQL with random initialization requires more than 100 epochs, and DQN and DQN-MAML fail entirely because of distributional shift. On the $\\lambda = 300$ test, the converged policy achieves an average AoI of 6 and transmission power of 1.6 mW, compared with AoI of 28 and power of 6 mW for CQL and AoI of 7 and 2.2 mW for an online DQN agent. The authors further claim that after meta-training on outage-free tasks, the same initialization lets the UAV adapt to five unseen tasks with unpredictable outage regions, avoiding the outage zone almost completely after about 100 training epochs, whereas CQL continues to incur roughly 20 outages. The paper presents this as the first combination of meta-learning and offline RL in the wireless domain.","pith_inferences":["If the printed reward in Eq. (6) is a typo and the implementation actually uses a per-step reward, then the paper's headline result is really about that causal variant, and an explicit per-step reward would strengthen reproducibility.","The meta-initialization appears to encode a general notion of moving toward devices with stale AoI while avoiding high transmission power, which could transfer to different numbers of devices or different channel models than those seen in meta-training—an empirical question the paper does not test directly.","The resilience result suggests the Q-network learns a spatial risk map of outage areas during meta-training; a testable extension would be to place the outage region at an entirely new location and measure how quickly the meta-initialized policy avoids it."],"forward_implications":["Offline datasets from prior UAV operations can replace online interaction for trajectory and scheduling optimization, removing safety and cost barriers.","When the service objective changes (a new $\\lambda$), the policy can be adapted from a few hundred offline data points in well under 100 epochs, rather than retraining from scratch.","The meta-initialized policy can learn to avoid sudden outage regions (e.g., heavy rain) without explicit online exploration, converging to near-zero outages after about 100 epochs.","Larger offline datasets and more meta-training tasks both speed convergence, and even two meta-tasks beat random-initialization CQL.","The same framework is positioned to extend beyond agriculture to other UAV verticals such as disaster response, where online training is often infeasible."],"supporting_citations":[{"why":"Supplies the conservative Q-learning loss that makes offline training viable and is the base learner in both the inner and outer meta-loops.","marker":"[21]"},{"why":"Provides the MAML procedure for meta-learning the initial Q-network weights across tasks with different objectives.","marker":"[23]"},{"why":"Defines the joint AoI and transmission power minimization problem with UAV path and scheduling that this paper benchmarks against.","marker":"[6]"},{"why":"Motivates offline RL by the cost, safety, and scalability problems of online interaction, and introduces the distributional shift problem.","marker":"[17]"},{"why":"Supplies the heavy-rain attenuation model used to create the outage regions in the resilience experiments.","marker":"[41]"},{"why":"Provides the fitting parameters for the rain attenuation model used in the outage scenarios.","marker":"[42]"}],"fun_headline_variants":["Meta-offline RL: UAVs adapt to new tasks in under 40 epochs","CQL+MAML beats online DQN on UAV AoI and power with offline data","Few-shot meta-offline RL: resilient UAV paths in 40 epochs, no online risk","Offline RL with meta-learning: UAVs learn to avoid outages in 40 epochs","Meta-offline RL: optimal UAV AoI and power from static datasets in <40 epochs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reward in Eq. (6) is a typo: as printed, it assigns the full episode cost to every time step, which is non-causal and would make the temporal-difference targets in Eqs. (8) and (9) ill-defined, so the reported results depend on the unstated assumption that the implementation used a per-step reward depending only on the current state and action.","fun_headline_variants_meta":{"raw":{"variants":["Meta-offline RL: UAVs adapt to new tasks in under 40 epochs","CQL+MAML beats online DQN on UAV AoI and power with offline data","Few-shot meta-offline RL: resilient UAV paths in 40 epochs, no online risk","Offline RL with meta-learning: UAVs learn to avoid outages in 40 epochs","Meta-offline RL: optimal UAV AoI and power from static datasets in <40 epochs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000385,"raw_usage":{"total_tokens":2096,"prompt_tokens":1068,"completion_tokens":1028,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":684,"completion_tokens_details":{"reasoning_tokens":915}},"tokens_in":684,"tokens_out":1028,"duration_ms":9158,"temperature":1.0,"reasoning_tokens":915,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T15:51:21.576588+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Algorithm 4 exactly as written with the reward in Eq. (6) and check whether the temporal-difference loss can even be computed at each step; if the algorithm does not converge to the reported AoI of 6 and power of 1.6 mW at $\\lambda = 300$, the claimed results cannot come from the described reward. Alternatively, with the intended per-step reward, compare CQL-MAML against CQL over many random seeds and verify that the meta-learned initialization converges in under 40 epochs while CQL needs more than 100; a single seed or a different initialization should not flip the gap.","supporting_citations":[{"cited_title":"Conservative Q-learning for offline reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the conservative Q-learning loss that makes offline training viable and is the base learner in both the inner and outer meta-loops."},{"cited_title":"Model-agnostic meta-learning for fast adaptation of deep networks,","cited_arxiv_id":null,"evidence_quote":"Provides the MAML procedure for meta-learning the initial Q-network weights across tasks with different objectives."},{"cited_title":"Multi-UA V path learning for age and power optimization in IoT with UA V battery recharge,","cited_arxiv_id":null,"evidence_quote":"Defines the joint AoI and transmission power minimization problem with UAV path and scheduling that this paper benchmarks against."},{"cited_title":"Specific attenuation model for rain for use in prediction methods,","cited_arxiv_id":null,"evidence_quote":"Supplies the heavy-rain attenuation model used to create the outage regions in the resilience experiments."},{"cited_title":"Meteorologically introduced impacts on aerial channels and UA V communications,","cited_arxiv_id":null,"evidence_quote":"Provides the fitting parameters for the rain attenuation model used in the outage scenarios."}],"review_version":1}