{"id":"fad9b18a-726f-4d6d-bb91-6983f477ebf1","arxiv_id":"2501.06700","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Average reward RL, implemented as ARO-SAC, is reported to outperform discounted SAC by 15% in a RAN slicing radio resource management task.","lead":"This paper applies average reward reinforcement learning to radio resource management, adapting Soft Actor Critic into an average reward version called ARO-SAC. It reports a 15% performance gain over the standard discounted reward SAC in a RAN slicing simulator, arguing that the discounted objective was misaligned with wireless network goals.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 1 is not SAC: it deletes entropy regularization and uses an unjustified ρ update, so the 15% gain over vanilla SAC confounds the average-reward objective with a different policy-gradient algorithm.","rationale":"The paper's motivating observation is legitimate: Eq. (1) is an undiscounted long-term average objective, and Section IV's experiments show that, with a fixed horizon, increasing γ improves performance, and with fixed γ, extending the horizon improves average reward—consistent with the literature on discounting mismatch. However, the defining claim—that ARO-SAC achieves a 15% gain over the best discounted SAC because it adopts the average-reward objective—rests entirely on Algorithm 1, and Algorithm 1 is not a faithful average-reward version of SAC. SAC's distinctive entropy-regularized policy improvement is replaced by a plain log-π Q actor-critic update (line 17), and the entropy coefficient α disappears without discussion. The comparison in Fig. 3 therefore moves two levers at once, making the 15% figure uninterpretable as evidence for average-reward RL. The ρ update is independently suspect: the differential TD error defined in Eq. (6) contains V(s′), but the ε used for ρ in Eq. (8) is r − ρ − min Q(s,a), and the paper never shows that this drives ρ to the long-run average reward. The paper even acknowledges theoretical guarantees are future work. Thus the rejection is warranted. An ablation isolating ρ from the policy-update change is the minimal step that would make the claim testable, and releasing code would allow the community to verify the 15% figure, which currently rests on a single scenario with no error bars on the headline comparison.","tokens_in":8597,"tokens_out":5048,"duration_ms":50228,"concrete_test":"Run the same netgymenv RAN-slicing scenario with three algorithms and report mean ± CI over at least 10 seeds each: (A) ARO-SAC exactly as in Algorithm 1; (B) the same Algorithm 1 but with ρ fixed to 0 and target y = r + 0.99·min Q(s′, ã′) (discounted, no entropy); (C) vanilla SAC with γ = 0.99. If A ≈ B, the 15% gain comes from the non-SAC policy update rather than the average-reward objective; if B ≈ C, the gain comes from the ρ mechanism. Additionally, re-derive the fixed point of the ρ update on a small tabular MDP; if ρ* ≠ average reward, Algorithm 1 is not a valid average-reward RL algorithm.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section V-B and Algorithm 1 do not implement the claimed \"Average Reward Soft Actor-Critic.\" Line 17 is the plain policy gradient ∇θ Σ log πθ(a|s) Qφ(s,a); SAC's entropy term −α log πθ(a|s) and temperature parameter are absent. Line 12 uses the differential Q-learning target y = r − ρ + min_i Q_targ,i(s′, ã′), not SAC's target r + γ(min Q − α log π). Therefore ARO-SAC changes two independent design choices relative to the vanilla SAC baseline in Fig. 3: the return criterion (average vs. discounted) and the entire policy-update rule (maximum-entropy actor-critic vs. standard log-π Q policy gradient). The reported 15% gain therefore cannot be attributed to alignment with the average-reward objective; it may come from the removal of entropy regularization or from the altered update altogether. The ρ estimator is also not grounded. Eq. (6) correctly gives the differential TD error δ_t = R_{t+1} − ρ + V(S_{t+1}) − V(S_t), but Algorithm 1 updates ρ by gradient descent on ε_t = r − ρ − min Q(s,a) (Eq. 8, line 19), which omits the successor value. The fixed point of ρ ← ρ + β(r − ρ − Q(s,a)) is not generally the average reward; it is a state-action-dependent residual. No convergence or consistency argument is provided, and the paper itself lists theoretical guarantees as future work (Section VI). Absent a valid average-reward SAC, the central empirical claim is underdetermined.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that the standard discounted-reward RL objective is misaligned with the long-term average QoS goals of radio resource management (RRM), and uses a RAN slicing task to illustrate this. It proposes ARO-SAC, an adaptation of Soft Actor-Critic to the average-reward setting, and reports a 15% performance gain over vanilla SAC with gamma=0.99. The paper also studies the effect of the discount factor and horizon on the discounted formulation, showing that larger gamma and longer horizons help but gamma=1 is unstable.","tokens_in":8918,"tokens_out":3316,"duration_ms":29712,"significance":"If the central claim were supported, the paper would address a real and under-appreciated mismatch between the RL objective and wireless network optimization goals. The use of an industry-grade simulator (NetworkGym/NS-3) and the systematic study of discount factor and horizon effects are useful and credit-worthy. However, the proposed ARO-SAC algorithm is not a valid average-reward SAC: it removes SAC's entropy regularization and uses an unjustified update for the average-reward estimate. The empirical headline result is therefore confounded by multiple algorithmic changes and lacks reported uncertainty. As presented, the paper does not support its main contribution.","major_comments":[{"comment":"The average-reward estimator is updated by minimizing eps_t = r - rho - min Q(s,a), whose fixed point is rho = r - Q(s,a), a state-action-dependent residual rather than the average reward. This omits the successor value that appears in the differential TD error of Eq. (6), delta_t = R_{t+1} - rho + V(S_{t+1}) - V(S_t). No convergence or consistency argument is provided; the paper itself defers theoretical guarantees to Section VI. This invalidates the claim that Algorithm 1 implements an average-reward RL algorithm.","section":"Section V-B, Eq. (8), Algorithm 1 line 19"},{"comment":"Algorithm 1 is not Soft Actor-Critic. The policy update in line 17 is the plain policy gradient sum log pi(a|s) Q(s,a) without SAC's entropy regularization term -alpha log pi(a|s), and the target in line 12 is y = r - rho + min Q_targ(s', a') rather than SAC's r + gamma(min Q - alpha log pi). ARO-SAC therefore differs from the vanilla SAC baseline in both the return criterion and the policy-update rule. Consequently, the 15% gain in Fig. 3 cannot be attributed to alignment with the average-reward objective; it may stem from removing entropy regularization or from the altered update altogether.","section":"Algorithm 1, lines 12 and 17"},{"comment":"The headline comparison lacks error bars or confidence intervals, even though Fig. 2 reports them. The text states the result is averaged over 5 independent runs and 5 user-number combinations, but no variance is reported. Given that the central claim is an empirical performance gain, the absence of uncertainty quantification makes the 15% figure difficult to assess.","section":"Fig. 3"},{"comment":"The motivating example states that with gamma = 0.95, rewards beyond 50 time steps contribute only about 0.01 of their original value. This is quantitatively wrong: 0.95^50 is approximately 0.077, not 0.01. While illustrative, the error weakens the numerical motivation for the paper's central mismatch argument.","section":"Section III-B"}],"minor_comments":[{"comment":"There is a grammatical error: 'called the Average Reward Off policy Soft Actor Critic (ARO SAC) is an adaptation' should be 'called the Average Reward Off-policy Soft Actor-Critic (ARO-SAC), which is an adaptation.'","section":"Abstract"},{"comment":"There are missing citation placeholders '[?]' in the paragraph on previous DRL studies of RAN slicing; these should be filled before publication.","section":"Section III-B"},{"comment":"Equation (3) uses r(pi) for the average reward, while later equations use rho. The notation should be unified.","section":"Section V-A, Eq. (3)"},{"comment":"The line 'Update average reward estimator rho: nabla_rho 1/|B| sum (eps_t)^2' is not a complete update rule; it should specify the loss and the learning rate or optimizer used.","section":"Algorithm 1, line 19"},{"comment":"The paper states that the learning rate for rho needs careful selection and was set to 1e-5, but it does not report how this value was chosen or how sensitive the result is to it, which is relevant given the new hyperparameter.","section":"Section V-C"}],"recommendation":"reject","confidential_remarks":"The stress-test concern lands: the algorithm's average-reward estimator is not grounded in the differential TD error, and the policy update is not SAC's. These are load-bearing technical errors that cannot be fixed by local revision; the algorithm and experiments would need to be redesigned. The paper also overclaims novelty given that discounting mismatch has been studied in the cited literature ([30]). I would not consider this a borderline case."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper's real contribution is small but real: it demonstrates in a RAN slicing simulator that the discount factor and episode horizon move performance in the direction you'd expect if the wireless objective is long-term average, and that gamma=1 is unstable in practice. That is worth knowing. The proposed ARO-SAC is where the trouble starts.\n\nAlgorithm 1 is not SAC. The entropy term is gone; the policy update is plain log-pi Q. So the 15% gain over vanilla SAC confounds the average-reward objective with a different policy-gradient algorithm. The rho update is also off: minimizing (r - rho - min Q)^2 drives rho toward a state-action-dependent residual, not the average reward, because the successor value is missing. The paper itself lists theoretical guarantees as future work, which is honest but leaves the central mechanism unsupported.\n\nThe experiments are thin for the headline claim: one scenario, no error bars in Fig. 3, no code, incomplete hyperparameters. Tables II and III do have error bars and show the monotone trend, so the mismatch story is on firmer ground. Minor issues: the gamma=0.95 decay example is miscalculated (0.95^50 is about 0.077, not 0.01), there are unresolved '[?]' citation placeholders in Sections III-B and III-C, and the 'first to identify' claim is weakened by their own reference [30] which already discusses discounting mismatch.\n\nIf the authors redo the experiments with a correctly specified average-reward SAC (or rename the algorithm and add ARO-DDPG as a baseline), report error bars and hyperparameters, and release code, the underlying idea is salvageable. As written, the 15% number is not attributable to the average-reward objective.\n\nWho should read it: anyone working on RL for RRM, as a cautionary example of how easy it is to confound an algorithmic change with an objective change. I'd give it a serious referee because the question is relevant and the empirical mismatch demonstration is useful, but I'd expect major revision.\n\nRecommendation: engage with it, but require the corrections above.","headline":"The mismatch story is real, but the proposed ARO-SAC is not SAC, so the 15% gain is confounded.","tokens_in":9463,"tokens_out":2098,"would_cite":false,"duration_ms":19202,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that the standard discounted-reward RL formulation is misaligned with wireless RRM objectives and shows that an average-reward version of SAC closes the gap with a 15% gain.","keywords":["average reward reinforcement learning","radio resource management","RAN slicing","Soft Actor-Critic","discounted reward mismatch","wireless network optimization"],"falsifier":"Train ARO-SAC and discounted SAC on a synthetic MDP whose optimal average reward per step is known analytically; if ARO-SAC's learned $\\rho$ does not track the true average reward or its policy converges to a lower average reward than the optimal policy, the claim that it solves the average-reward objective fails. A simpler check is to run ARO-SAC on the RAN slicing task with the $\\rho$ update disabled and a fixed $\\rho=0$; if performance keeps the 15% gain, the gain would be attributable to the algorithm change rather than to the average reward estimate.","tokens_in":8360,"feed_emoji":"📶","tokens_out":5712,"duration_ms":52477,"temperature":0.7,"pith_summary":"The paper argues that standard reinforcement learning for wireless radio resource management maximizes discounted cumulative reward, while the actual network goal is undiscounted long-term average quality of service, and this mismatch costs performance. It shows empirically that increasing the discount factor or the episode horizon helps, but setting the discount factor to one causes unstable training. To close the gap, the paper introduces ARO-SAC, an average-reward version of Soft Actor-Critic that replaces the discount factor with a learned estimate of the average reward rate. In a RAN slicing simulator, ARO-SAC improves system performance by 15% over the best tuned discounted SAC with a discount factor of 0.99, while avoiding the instability of a discount factor of one. The paper's point is that average-reward RL is a better objective match for wireless network optimization and is practically implementable by modifying an off-policy actor-critic algorithm.","feed_headline":"Average-reward RL beats discounted RL by 15% in RAN slicing","feed_subtitle":"Replacing the discount factor with a learned average reward aligns RL with wireless long-term QoS goals.","key_machinery":"The central object is the average-reward TD error and the trainable average reward estimator $\\rho$. In discounted SAC the TD target is $r + \\gamma \\min_i Q_{\\phi_{targ,i}}(s', \\tilde a')$; in ARO-SAC it becomes $r - \\rho + \\min_i Q_{\\phi_{targ,i}}(s', \\tilde a')$. The estimator $\\rho$ is trained by gradient descent on $\\varepsilon_t = r(s_t,a_t) - \\rho_t - \\min(Q_1(s_t,a_t), Q_2(s_t,a_t))$, following the off-policy estimation strategy from ARO-DDPG. This machinery replaces geometric discounting of future rewards with a constant subtraction of the estimated average reward, making the value function measure deviations from the long-term average rather than a shrinking tail. It is what lets the policy optimize the undiscounted objective without setting the discount factor to one.","core_discovery":"The paper's central claim is that the widely used discounted reward RL formulation is misaligned with wireless RRM objectives, which are undiscounted long-term average QoS, and that switching to average reward RL closes the gap. The discovery is an algorithm, ARO-SAC: take SAC's actor-critic updates, drop the discount factor, and instead maintain a trainable average reward estimate, updated by gradient descent on the temporal-difference residual. The TD target becomes the reward minus the estimated average reward plus the target value of the next state, so the critic learns a differential value function rather than a discounted one. In the RAN slicing testbed with fixed horizon, ARO-SAC achieves a 15% gain over the best discounted SAC and avoids the divergence that appears with a discount factor of one. The author's intended conclusion is that average reward RL should replace discounted reward RL in RRM because it matches the true network objective.","pith_inferences":["Beyond the paper's claims: the same TD-target swap should transfer to other off-policy actor-critic algorithms, suggesting the 15% gain reflects objective alignment rather than a SAC-specific quirk.","Because the mismatch is about the objective, the benefit should grow in tasks with long horizons and non-stationary traffic; a testable prediction is that ARO-SAC's margin over discounted SAC increases as the episode period grows beyond the tested value.","The paper's update for the average reward estimate resembles a differential baseline, so convergence may depend on the ratio of learning rates between the estimator and the critics; the reported choice of a smaller learning rate for the estimator is a plausible practical rule.","Under the average reward objective, reward shaping choices such as the delay-violation penalty weight should be re-tuned, so the reported 15% gain may shift with different QoS trade-offs."],"forward_implications":["In the tested RAN slicing setup, a larger discount factor with fixed horizon increases cumulative reward, showing that discounted agents leave performance on the table.","Extending the environment horizon at a fixed discount factor improves average reward per step, confirming that horizon truncation compounds the objective mismatch.","Setting the discount factor to one is not a workable fix because training becomes unstable, motivating the average-reward reformulation.","ARO-SAC with a properly tuned learning rate for the average reward estimate delivers a 15% gain over vanilla SAC with a discount factor of 0.99 in the simulator.","The average reward RL framework adds one trainable parameter and one extra hyperparameter to SAC, so the gain comes with a modest tuning cost."],"supporting_citations":[{"why":"Defines the average reward RL objective, the differential return, and the Bellman equation that the paper re-formulates.","marker":"[1]"},{"why":"Supplies the off-policy average reward estimation strategy, training the average reward as a parameter via gradient descent, which ARO-SAC adapts.","marker":"[3]"},{"why":"The Soft Actor-Critic algorithm that ARO-SAC modifies to the average reward setting.","marker":"[29]"},{"why":"The network simulator used for the RAN slicing experiments.","marker":"[28]"},{"why":"The LTE traffic model that defines the simulator's traffic behavior.","marker":"[26]"},{"why":"Prior work noticing discounting mismatch in actor-critic methods, framing the paper's empirical investigation.","marker":"[30]"}],"fun_headline_variants":["Average reward RL boosts RAN slicing by 15%","ARO-SAC: align RL with wireless goals for 15% gain","Discounted RL misfit: average reward RL wins by 15%","Ditch discounting: average reward RL improves RRM 15%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the modified Bellman updates in Algorithm 1 define a valid and convergent average-reward SAC: the paper assumes the trainable average reward estimate $\\rho$ converges and that the altered TD target remains a sound learning signal, but it provides no proof.","fun_headline_variants_meta":{"raw":{"variants":["Average reward RL boosts RAN slicing by 15%","ARO-SAC: align RL with wireless goals for 15% gain","Discounted RL misfit: average reward RL wins by 15%","Ditch discounting: average reward RL improves RRM 15%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000566,"raw_usage":{"total_tokens":2656,"prompt_tokens":896,"completion_tokens":1760,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":512,"completion_tokens_details":{"reasoning_tokens":1683}},"tokens_in":512,"tokens_out":1760,"duration_ms":11585,"temperature":1.0,"reasoning_tokens":1683,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:54:48.196155+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train ARO-SAC and discounted SAC on a synthetic MDP whose optimal average reward per step is known analytically; if ARO-SAC's learned $\\rho$ does not track the true average reward or its policy converges to a lower average reward than the optimal policy, the claim that it solves the average-reward objective fails. A simpler check is to run ARO-SAC on the RAN slicing task with the $\\rho$ update disabled and a fixed $\\rho=0$; if performance keeps the 15% gain, the gain would be attributable to the algorithm change rather than to the average reward estimate.","supporting_citations":[{"cited_title":"Reinforcement learning: An introduction by richards’ sutton,","cited_arxiv_id":null,"evidence_quote":"Defines the average reward RL objective, the differential return, and the Bellman equation that the paper re-formulates."},{"cited_title":"Off-policy average reward actor-critic with deterministic policy search,","cited_arxiv_id":null,"evidence_quote":"Supplies the off-policy average reward estimation strategy, training the average reward as a parameter via gradient descent, which ARO-SAC adapts."},{"cited_title":"NetworkGym: Democratizing Network AI via Sim-aaS,","cited_arxiv_id":null,"evidence_quote":"The network simulator used for the RAN slicing experiments."},{"cited_title":"The NS-3 network simulator,","cited_arxiv_id":null,"evidence_quote":"The LTE traffic model that defines the simulator's traffic behavior."},{"cited_title":"A Deeper Look at Discounting Mismatch in Actor-Critic Algorithms","cited_arxiv_id":"2010.01069","evidence_quote":"Prior work noticing discounting mismatch in actor-critic methods, framing the paper's empirical investigation."}],"review_version":1}