{"id":"0a2d2841-82a5-4c12-911e-26070fca4496","arxiv_id":"1908.06367","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"For RF-powered multi-source monitoring systems, the age-optimal sampling policy is threshold-based in each process's age, and a deep Q-network can learn it efficiently.","lead":"This paper studies sensor networks where battery-free sensors are charged by radio waves from the base station, and must decide when to send status updates to keep the information fresh. It shows the optimal freshness policy has a simple threshold structure and can be learned by a deep reinforcement learning algorithm.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The structural proof relies on unnormalized value iteration that does not converge for average-cost MDPs, so Theorems 1-2 are not rigorously established as written.","rationale":"The reader's weakest_assumption was the iid fading assumption, which is explicitly stated and standard for this class of MDP models; it limits applicability but is not an internal flaw. The more serious problem is that the proof of the central structural theorems uses a VIA convergence claim that is false for average-cost MDPs: without relative value normalization, the iterates in Eq. (18) do not converge to the Bellman solution. This directly affects Lemma 2 and therefore Theorems 1 and 2. The issue is likely repairable by switching to RVIA, since the state-independent normalization preserves the induction inequalities, so the verdict remains CONDITIONAL rather than REJECT. The reader's concerns about DRL validation and reproducibility are secondary and already reflected in the conditional verdict.","tokens_in":21828,"tokens_out":16527,"duration_ms":179810,"concrete_test":"Re-derive Lemma 2 using the relative value iteration update \\tilde V^{(m+1)}(s)=min_a[c(s,a)+Σ P(s'|s,a)\\tilde V^{(m)}(s')]-\\tilde V^{(m)}(s_ref), with \\tilde V^{(0)}=0, and check whether the inequalities in (21)-(24) hold at every m. If they hold, the threshold theorems are salvageable but Eq. (20) must be corrected to a normalized form. If they fail, run the RVIA for the N=1 grid of Fig. 3 (bmax=9, Amax=10, G=H=10) and verify whether the computed optimal policy satisfies Theorem 2; a violation would falsify the structural claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing problem is internal to the proof of the threshold results. Lemma 2 is proved by induction on VIA iterates, and Eq. (20) explicitly asserts that the iterates defined by Eq. (18) converge to V(s) satisfying Eq. (12). For an average-cost finite MDP, the unnormalized iteration V^{(m+1)}(s)=min_a [c(s,a)+Σ_{s'} P(s'|s,a) V^{(m)}(s')] either diverges or has a state-independent drift; it does not converge to the relative value function. A simple one-state, one-action example with cost 1 gives V^{(m)}=m, so Eq. (20) is false as written. Because every monotonicity comparison in Lemma 2 and every threshold implication in Theorems 1 and 2 uses V(s)=lim V^{(m)}(s), the structural claims are not rigorously established by the supplied argument. The repair is standard: use relative value iteration and subtract a reference-state value at each iteration; since the subtraction is state-independent, the induction inequalities in (21)-(24) would be preserved. But that repair is not what the paper does. The explicit iid fading assumption is a stated modeling restriction rather than an internal inconsistency; the VIA error is the stronger concern.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies a discrete-time monitoring system in which N source nodes are powered exclusively by RF energy transfer from a destination node, and each source observes a physical process whose freshness at the destination is measured by an Age-of-Information (AoI) metric. The authors formulate the joint design of wireless energy transfer and update scheduling as a finite-state, finite-action average-cost MDP, and they propose a deep Q-learning algorithm (Algorithm 1) to approximate the age-optimal policy. The main analytical contributions are structural: Theorem 1 claims that the age-optimal policy is threshold-based in each AoI component for general N; Theorem 2 claims that for N=1 the optimal policy is threshold-based in all state variables on a battery-sufficient subset of states; Theorem 3 gives an analogous result for the throughput-optimal policy. Numerical experiments illustrate the threshold structure, compare age-optimal and throughput-optimal policies, and evaluate the proposed DRL algorithm against RVIA for N=1.","tokens_in":22035,"tokens_out":11378,"duration_ms":124572,"significance":"If established, the threshold characterization is a valuable result: it gives a compact description of an otherwise high-dimensional optimal policy and permits a concrete comparison between age-optimal and throughput-optimal behavior. The paper is also to be credited for making its modeling assumptions explicit, particularly the i.i.d. quasi-static fading assumption and the finite-state discretization, and for constructing the MDP formulation and Bellman equations in a self-contained way. However, the central analytical proof as written relies on an invalid convergence assertion for unnormalized value iteration, and the numerical verification for N>1 does not clearly state whether the plotted policy is the true optimal policy or a learned approximation. The structural results appear plausible and are likely repairable by switching to relative value iteration, but the proof and the numerical verification need correction before the claims can be accepted.","major_comments":[{"comment":"The proof of Lemma 2 and the threshold theorems is based on unnormalized value iteration, and Eq. (20) states that the iterates V^(m)(s) converge to a solution V(s) of the average-cost Bellman equation (12). For an average-cost finite MDP this is false in general: for a single-state, single-action MDP with cost 1, the iteration gives V^(m)=m, so the limit is not finite. Since Lemma 2, Theorem 1, Theorem 2, and Theorem 3 in Section V all use V(s)=lim_m V^(m)(s), the structural results are not rigorously established as written. The repair is standard: replace the VIA step with relative value iteration, e.g., W^(m+1)(s) = T W^(m)(s) - T W^(m)(s_ref) for a fixed reference state s_ref, and initialize W^(0) as a constant so that the induction base holds. Because the subtracted term is independent of s, the monotonicity comparisons in (21)-(24) are preserved. This is a required fix, not merely a presentational change.","section":"Section IV, Eq. (18)-(20)"},{"comment":"The text describes these figures as presenting \"the structure of the age-optimal policy,\" but it does not state how the plotted actions were computed. For the N=2 case, the parameters in the caption give about 1.68 million states (per source: 6 AoI values, 6 battery levels, 6 downlink channel levels, 6 uplink channel levels), so it is not clear whether the figure comes from an exact RVIA computation or from the deep Q-network. If the actions are generated by the learned DRL policy, the figure demonstrates only that the learned policy has threshold-like behavior; it does not verify Theorem 1 for the true optimal policy. The authors should state the computation method explicitly, and for any N>1 verification they should either use the exact RVIA solution on a tractable instance or clearly label the plot as a property of the learned policy.","section":"Section VI-A, Figs. 2-3"},{"comment":"The paper claims that the proposed DRL algorithm learns the age-optimal policy in a computationally efficient manner, but no convergence guarantee is provided for the average-cost deep Q-learning update when the Q-function is approximated by a neural network. The target in (16) includes min_{a in A(\\bar s)} Q_k(\\bar s,a|\\beta_k), which is a moving target that depends on the current network weights; with function approximation, the tabular Q-learning convergence conditions cited in the paper do not apply, and the stopping criterion \"until convergence\" in Algorithm 1 is not specified. The empirical evidence in Fig. 5 is limited to a single N=1 instance and shows only a small gap, not proof of convergence to the optimal policy. The authors should either add formal conditions under which convergence to pi* is guaranteed or explicitly state that the convergence of the DRL algorithm is an empirical observation rather than a proven property.","section":"Section III-C, Algorithm 1 and Eq. (15)-(16)"}],"minor_comments":[{"comment":"The monotonicity of V with respect to g_j and h_j is asserted in a single sentence rather than proved by the same induction used for b_j; please expand the argument, including the effect of the changing action feasibility when h_j changes, or state that the proof follows by repeating the b_j argument verbatim.","section":"Section IV, proof of Lemma 2"},{"comment":"In the sentence beginning \"according to Lemma 3, we note that pi*(s)=H,\" the reference to Lemma 3 appears to be a typo; the conclusion follows from the definition of \\bar A_th,1 and the threshold property of the age-optimal policy, not from the throughput-optimal value-function lemma.","section":"Section V-B, Remark 4"},{"comment":"The assertion that the weak accessibility condition holds, and hence that a solution to Bellman's equations exists, is made without proof or a specific citation; a short justification or a precise reference to the corresponding condition in [45] would strengthen the formulation.","section":"Section III-B, Lemma 1"},{"comment":"There is a recurring typo \"throughout-optimal policies\" that should read \"throughput-optimal policies.\"","section":"Page 3, Section I-A"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth a serious look. The model is new and well-chosen: multiple RF-powered sources with finite batteries, i.i.d. fading, and a jointly optimized WET/scheduling policy. The main analytical results—Theorem 1's threshold structure in each AoI for N sources, Theorem 2's full threshold structure for N=1, and the comparison with the throughput-optimal policy—are the kind of clean structural claims that actually help designers. The paper also does a good job positioning itself against the prior EH/AoI literature, and the citation pattern looks fair. Credit where due: Theorem 1 is a real extension to the multi-source RF-powered case, and the throughput comparison in Section V is a nice addition.\n\nThe soft spots are real, though. The stress-test note about the VIA is correct. Equation (20) claims the unnormalized value iteration iterates converge to the V that satisfies the average-cost Bellman equation. For an average-cost MDP, unnormalized VIA typically diverges or drifts; it does not converge to the relative value function. Since the proofs of Lemma 2 and Theorems 1-2 use V(s)=lim V^{(m)}(s), the structural results are not rigorously established as written. The repair is standard (relative value iteration, subtracting a reference-state value at each step), and the authors even mention RVIA in passing, so the fix is likely straightforward. But the supplied proof has a genuine gap, and a referee should insist it be fixed.\n\nThe DRL claims are also softer than the paper suggests. The algorithm is a standard DQN variant for average-cost MDPs. For N=1 it is compared against RVIA and matches. For N>1, the learned policy is used to \"verify\" the threshold structure and to produce design curves, but there is no optimal baseline, so calling it age-optimal is not justified by the experiments. The paper also omits code, data, hyperparameters, and any error bars. That would matter less if the analytical results were fully rigorous, but as is, the empirical verification is doing more work than it can support.\n\nMinor: the weak accessibility condition used in Lemma 1 is asserted without proof. It is probably true for this finite MDP, but it should be shown or cited precisely.\n\nBottom line: the core ideas are sound and the structural results are likely correct, but the paper needs a revision—fix the VIA/RVIA proof, and either provide reproducible code or temper the DRL optimality claims. This deserves peer review, not a desk reject. I would cite it for Theorem 1 once the proof is cleaned up.","headline":"A genuinely useful multi-source RF-powered AoI paper whose structural results are likely right but whose proof as written leans on a false VIA convergence claim and whose DRL optimality is only validated for N=1.","tokens_in":22590,"tokens_out":1310,"would_cite":true,"duration_ms":16499,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["90C40"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proves that the age-optimal joint wireless-energy-transfer and scheduling policy for RF-powered monitors is threshold-based in each process's age, and provides a deep Q-learning algorithm that learns it.","keywords":["Age-of-Information","RF energy harvesting","Markov decision process","reinforcement learning","threshold policy","wireless powered communication","deep Q-network","status update scheduling"],"falsifier":"Run exhaustive value iteration on a small instance, such as two sources with six age levels, five battery levels, and quantized channel gains, and check every pair of states satisfying Theorem 1's conditions; if any pair has $T_j$ optimal at the lower-age state but not at the higher-age state, the threshold theorem is false. The same exhaustive check on a single-source instance would test the all-state-variable threshold claims of Theorem 2.","tokens_in":21611,"feed_emoji":"📡","tokens_out":8527,"duration_ms":79969,"temperature":0.7,"pith_summary":"The paper asks how a destination node that charges battery-equipped sensors over the radio should spend each time slot: broadcast energy to recharge them, or let one sensor transmit a status update, so that the weighted average age of the monitored processes stays smallest. It models this as an average-cost Markov decision process whose state includes each sensor's battery level, the age of each process at the destination, and both downlink and uplink channel gains, then proves the age-optimal policy is threshold-based. The main theorem states that for each process, once its age crosses a threshold while everything else is fixed, transmitting an update for that process becomes optimal and remains optimal for all larger ages. For a single source, the paper proves the optimal action is also threshold-based in battery level and channel gains. This matters because threshold rules are simple to describe and to implement on energy-starved devices, and because the same analysis shows the throughput-optimal policy has a different, non-age-aware structure.","feed_headline":"RF-powered updates follow one age threshold per source","feed_subtitle":"For sensors charged over the air, the best scheduling rule switches once as freshness drops.","key_machinery":"The load-bearing object is the Q-function of the average-cost MDP, together with the monotonicity of its value function. Lemma 2 proves the value function is non-increasing in each battery level and channel power gain and non-decreasing in each age. Theorem 1 is carried by the comparison inequality $Q(s_2,\\bar a)-Q(s_2,a') \\le Q(s_1,\\bar a)-Q(s_1,a')$, which transfers optimality from a low-age state to a high-age state. The proof relies on the transition-probability factorization $P(s'|s,a)=P(b'|b,g,h,a)P(A'|A,a)P(g')P(h')$, which decouples battery evolution, age evolution, and channel memorylessness. On the computational side, a deep Q-network approximates the Q-function with experience replay and $\\epsilon$-greedy exploration, making the large finite state space tractable.","core_discovery":"The central discovery is that the jointly optimal wireless-energy-transfer and scheduling policy for minimizing long-term average weighted sum-age has a threshold structure in age. Theorem 1 shows: for two system states that are identical except that process $j$'s age is at least as large in the second state, if transmitting process $j$'s update is optimal in the first state, it is also optimal in the second. Thus, for each process, fixing all other state variables, the optimal decision switches at most once as that process's age grows. For a single source-destination pair, Theorem 2 extends the monotonicity to all state variables: on states where the battery is sufficiently charged, the optimal action is threshold-based in the battery level, the age, and the downlink and uplink channel gains. The paper also analyzes the policy that maximizes average throughput and shows that, although it is threshold-based as well, it differs from the age-optimal policy exactly when the monitored age is large: the age-optimal rule prioritizes updating stale information even at the cost of throughput.","pith_inferences":["Because the policy is threshold-based, the entire decision rule can in principle be stored as at most one age threshold per process per coarse state configuration, which suggests a low-memory implementation even without the neural network.","The same monotonicity proof should transfer to other freshness metrics with the same age evolution, such as peak age or value-of-information variants, but the paper does not make that claim.","A natural test of the model's limits is to run the same learning algorithm under temporally correlated fading; if the threshold structure persists, the i.i.d. assumption is not essential, and if it breaks, the conclusion is tied to memoryless channels.","The structural difference between age- and throughput-optimal policies suggests a practical heuristic: switch from throughput-based to age-based scheduling only when a process's age is stale, with the crossover point identified by simulation."],"forward_implications":["A multi-source scheduler can be implemented by maintaining one age threshold per process and transmitting for any process whose age exceeds its threshold, instead of solving the MDP online.","For a single source, the optimal decision rule over the high-battery region is fully described by thresholds in battery level, age, and channel gains.","Age-optimal and throughput-optimal policies coincide when the monitored age is small but diverge when the age is large, so throughput-oriented design is safe only for fresh-state regimes.","The proposed deep Q-learning algorithm approaches the optimal average age in numerical experiments, with the remaining gap caused by exploration.","The achievable average weighted sum-age falls as battery capacity grows and rises as update packet size grows."],"supporting_citations":[{"why":"introduces the age-of-information metric and the average-age analysis that the paper's objective builds on.","marker":"[3]"},{"why":"supplies the generate-at-will update model used when a scheduled source creates a fresh packet.","marker":"[12]"},{"why":"earlier RF-powered single-source age-of-information policy that this work generalizes by including battery and channel dynamics.","marker":"[43]"},{"why":"earlier wireless-powered age-of-information analysis with infinite battery that this work extends to finite batteries and multiple sources.","marker":"[44]"},{"why":"provides the dynamic-programming optimality equations, value-iteration, and optimal-stationary-policy facts on which the MDP analysis rests.","marker":"[45]"},{"why":"contributes the deep Q-network with experience replay that the proposed algorithm adapts.","marker":"[46]"},{"why":"supplies the convergence conditions for average-cost Q-learning used to justify the learning algorithm.","marker":"[47]"}],"fun_headline_variants":["RF-powered sensors: one age threshold to keep updates fresh","Deep RL discovers threshold rule for age-optimal charging","Stale data triggers send: age-optimal RF policy","Freshness beats throughput when age crosses threshold"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes the downlink and uplink fading gains are independent and identically distributed from slot to slot, so the next channel state does not depend on the current one and the Markov state needs no channel memory.","fun_headline_variants_meta":{"raw":{"variants":["RF-powered sensors: one age threshold to keep updates fresh","Deep RL discovers threshold rule for age-optimal charging","Stale data triggers send: age-optimal RF policy","Freshness beats throughput when age crosses threshold"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000381,"raw_usage":{"total_tokens":2087,"prompt_tokens":1073,"completion_tokens":1014,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":689,"completion_tokens_details":{"reasoning_tokens":951}},"tokens_in":689,"tokens_out":1014,"duration_ms":10696,"temperature":1.0,"reasoning_tokens":951,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:47:57.381011+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run exhaustive value iteration on a small instance, such as two sources with six age levels, five battery levels, and quantized channel gains, and check every pair of states satisfying Theorem 1's conditions; if any pair has $T_j$ optimal at the lower-age state but not at the higher-age state, the threshold theorem is false. The same exhaustive check on a single-source instance would test the all-state-variable threshold claims of Theorem 2.","supporting_citations":[{"cited_title":"Real-time status: How often should one update?","cited_arxiv_id":null,"evidence_quote":"introduces the age-of-information metric and the average-age analysis that the paper's objective builds on."},{"cited_title":"Update or wait: How to keep your data fresh,","cited_arxiv_id":null,"evidence_quote":"supplies the generate-at-will update model used when a scheduled source creates a fresh packet."},{"cited_title":"Online transmission policy in wireless powered networks with urgency-aware age of information,","cited_arxiv_id":null,"evidence_quote":"earlier RF-powered single-source age-of-information policy that this work generalizes by including battery and channel dynamics."},{"cited_title":"Average age of information in wireless powered sensor networks,","cited_arxiv_id":null,"evidence_quote":"earlier wireless-powered age-of-information analysis with infinite battery that this work extends to finite batteries and multiple sources."},{"cited_title":"Dynamic programming and optimal control 3rd edition, volume ii,","cited_arxiv_id":null,"evidence_quote":"provides the dynamic-programming optimality equations, value-iteration, and optimal-stationary-policy facts on which the MDP analysis rests."},{"cited_title":"Human-level control through deep reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"contributes the deep Q-network with experience replay that the proposed algorithm adapts."}],"review_version":1}