{"id":"ca7e7dab-3c93-4b8d-aa31-f9aa4abadec5","arxiv_id":"2506.03586","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A hybrid PPO scheduler for RIS phase shifts and subcarriers reduces average queueing delay in simulated RIS-assisted OFDM systems.","lead":"This paper applies a hybrid deep reinforcement learning (PPO) approach to jointly tune RIS phase shifts and subcarrier assignments, aiming to minimize the average packet delay in RIS-assisted OFDM downlink systems. It reports simulation results showing lower delay than throughput-oriented and random baselines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reward in Eq. (12), r_t = -Σ_k q_k[t], is not equivalent to the average-delay objective (11) when user arrival rates differ, and Section IV-G's robustness claim rests on this unverified equivalence.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the reward in Eq. (12) is claimed to be a faithful proxy for the average-delay objective via Little's law, but the equivalence holds only under equal arrival rates. I agree with this reading and would emphasize that the issue is not limited to the specific unequal-arrival test; it is a structural mismatch in the reward design. Little's law requires weighting each user's queue length by 1/λ_k, and the unweighted sum used in Eq. (12) minimizes a different functional whenever λ_k differ. This directly affects the paper's robustness and fairness claims in Section IV-G, where the policy is trained under equal arrivals and then evaluated under unequal arrivals. The equal-arrival delay comparisons in Sections IV-D and IV-E are less sensitive to this mismatch, so the main delay-reduction claim for the nominal scenario is not overturned. The concern is concrete, correctable, and testable: retraining with an arrival-weighted reward would settle whether the mismatch substantively changes the reported results. I therefore recommend keeping the reader's CONDITIONAL verdict, since the issue is specific and fixable rather than fatal to the entire paper. No ad hominem or theatrical language is warranted; the paper is a competent simulation study with a genuine but localized weakness in the reward-objective alignment.","tokens_in":17300,"tokens_out":13125,"duration_ms":145805,"concrete_test":"Retrain the proposed hybrid DRL algorithm with an arrival-weighted reward r_t = -Σ_k q_k[t]/λ_k (equivalently, normalize each user's queue length by its own arrival rate) and evaluate it in the Section IV-G unequal-arrival scenario for the same gap values as Fig. 11, using the same hyperparameters and multiple random seeds. Compare average delay, backlog balance, and a delay-fairness metric (e.g., variance of per-user average delay) against the unweighted-reward policy. If the weighted reward yields a statistically significant improvement in average delay or delay fairness, the mismatch is material and the claim needs qualification; if the two are indistinguishable, the unweighted reward is an adequate proxy and the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-B justifies r_t = -Σ_k q_k[t] as a proxy for the average-delay objective (11) via Little's law. Little's law gives E[W_k] = E[q_k]/λ_k for each user, so per-user average delay is proportional to q_k/λ_k, not q_k. The objective (11) averages these per-user delays with equal weight 1/K, so the correct reward proxy is -Σ_k q_k[t]/λ_k, not the unweighted sum. The unweighted summation overweights users with large arrival rates. This gap matters directly in Section IV-G, where λ1=λ+gap, λ2=λ, λ3=λ-gap: the policy trained under equal arrival rates with the unweighted reward optimizes a different functional than the stated average delay. Consequently, the reported 'robustness and fairness' under different traffic flows (Figs. 9 and 11) is not backed by a reward function aligned with the objective. The equal-arrival experiments in Section IV-D are less affected, so the central delay-reduction claim for the nominal scenario is not overturned; the weakness specifically undermines the unequal-arrival robustness and fairness portion of the abstract's claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript considers a downlink RIS-assisted MISO-OFDM system with stochastic packet arrivals and queueing at the BS. The authors formulate a joint optimization of RIS phase shifts, subcarrier allocation, and active beamforming to minimize the average delay per user. They propose a hybrid DRL algorithm: PPO-Θ for the continuous RIS phases, a multi-agent PPO-N for the discrete subcarrier allocation, and MRT with water-filling for the active beamforming. The state space includes CSI, queue lengths, and current packet arrivals, and a transfer-learning pretraining stage (max-min rate) is used to accelerate convergence. Simulations compare the proposed method against max-sum-rate, max-min-rate, random, and without-RIS baselines across arrival rates, bursty traffic, varying numbers of RIS elements, unequal per-user arrival rates, jitter, and several robustness scenarios.","tokens_in":17517,"tokens_out":7188,"duration_ms":69011,"significance":"The paper addresses an important and timely problem, and the proposed framework is technically sound in its nominal equal-arrival-rate setting. The description of the DRL components is detailed, and the simulation study is fairly extensive, covering bursty arrivals, varying RIS sizes, unequal traffic, jitter, and robustness checks. The paper does not provide theoretical guarantees for the learned policy, but for a simulation-driven systems paper this is acceptable. The main weakness is a reward-objective mismatch under unequal arrival rates, which undermines the robustness and fairness claims for that scenario. If the reward is reweighted and the experiments re-run (or the claims narrowed), the paper would be a solid contribution to delay-aware RIS-assisted resource allocation.","major_comments":[{"comment":"The reward r_t = -Σ_k q_k[t] is claimed to be equivalent to the average-delay objective (11). The packet-level argument that cumulative queue lengths reflect aggregate delay is valid only when all users have equal arrival rates. With unequal rates, as in Section IV-G (λ1=λ+gap, λ2=λ, λ3=λ-gap), Little's law gives per-user average delay as E[q_k]/λ_k, so the correct per-slot proxy for the equally weighted per-user average in (11) is -Σ_k q_k[t]/λ_k (up to constants). The unweighted sum overweights users with high arrival rates. Consequently, the results in Fig. 9 and Fig. 11 do not demonstrate that the learned policy minimizes the stated objective; they show queue-balancing behavior under a mismatched reward. The abstract's claim of 'superior system robustness and fairness' is therefore not supported by a reward aligned with the objective in the unequal-rate setting. Please either reweight the reward by 1/λ_k and re-run the unequal-rate experiments, or explicitly reframe the Section IV-G results as empirical queue-balancing rather than delay-minimization under the stated objective.","section":"Section III-C, Eq. (21)"},{"comment":"The PPO-N critic loss in Eq. (21) is written as E[(r_t + V(s_{t+1}) - V(s_t))^2], omitting the discount factor γ that appears both in the TD error (20) and in the corresponding PPO-Θ critic loss (16). This is internally inconsistent; the TD target should be r_t + γ V(s_{t+1}). Please correct Eq. (21).","section":"Section III-C, Eq. (21)"},{"comment":"The main text states that user positions are fixed throughout each episode, but footnote 1 says 'user positions are randomly generated at each step within an episode during the training process.' These statements are contradictory and directly affect the interpretation of the simulation results. Please clarify whether user positions change per time step, per episode, or only across episodes, and align the footnote with the main text.","section":"Section IV-A, footnote 1"}],"minor_comments":[{"comment":"There are several typos in the introduction and throughout: 'propoosed' should be 'proposed', 'trainning' should be 'training', 'indicaters' in Section II-C should be 'indicators', 'brustiness' in Section IV-E should be 'burstiness', 'influcence' in Section III-B should be 'influence', 'necessiates' in Section I should be 'necessitates', 'illstrated' in Section III-B should be 'illustrated', and 'assised' in Section V should be 'assisted'.","section":"Section I-B"},{"comment":"In the paragraph describing the actor network input, 'the transmission power level ℓ_k[t]' should read 'the packet arrival level ℓ_k[t]', since ℓ_k[t] denotes the number of arriving packets, not a power level.","section":"Section III-B, actor network design"},{"comment":"The notation 'L3' appears in the 'Delay taps' column of Table III and in the text of Section IV-I, but the system model only defines L0, L1, and L2. This is presumably a typo for 'L2'. Please correct the notation for consistency.","section":"Section IV-I and Table III"},{"comment":"For the 'Max sum rate' baseline, the text says 'The algorithm proposed in [37] and [38] is respectively employed for subcarrier allocation and RIS reflection shift optimization.' Please specify which reference corresponds to which subproblem, as both references are IRS-enhanced OFDM papers and the division of labor is unclear.","section":"Section IV-B"},{"comment":"The reward in Eq. (12) penalizes only queueing delay, while the objective (11) explicitly includes both queueing and transmission delay. Even in the equal-arrival case, the reward is an indirect proxy that ignores the service-time component. This is acceptable for the simulation study if transmission delays are small relative to queueing delays, but the paper should state this approximation explicitly when motivating the reward.","section":"Section III-B, Eq. (12)"}],"recommendation":"major_revision","confidential_remarks":"The reward-objective mismatch under unequal arrival rates is the main technical issue; it is fixable by reweighting the reward or narrowing the claims, but it requires re-running the Section IV-G experiments and possibly retraining. The missing discount factor in Eq. (21) is a clear and easily correctable internal inconsistency. The paper is otherwise within the scope of a wireless communications journal and the simulation study is reasonably thorough. I recommend major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a solid engineering paper on hybrid PPO for delay-aware resource allocation in RIS-assisted OFDM. The algorithmic combination—continuous PPO for RIS phases, discrete multi-agent PPO for subcarrier allocation, and transfer learning from a max-min-rate reward—is genuinely new as a package, even if each piece is known. The system model is clearly described, the training setup is detailed, and the simulations show large delay reductions against sensible baselines. The complexity analysis is a plus. For the equal-arrival-rate scenario, which is the nominal setting, the central claim holds up: the reward in Eq. (12) is a valid proxy for average delay through Little's law when all λ_k are equal.\n\nThe main soft spot is the unequal-arrival robustness claim in Section IV-G. There, λ1=λ+gap, λ2=λ, λ3=λ−gap, but the reward remains r=−Σ q_k[t]. Little's law gives per-user average delay proportional to q_k/λ_k, so the equal-weight sum of queues is not aligned with the objective unless arrival rates are identical. The correct proxy would be −Σ q_k[t]/λ_k. The paper neither reweights the reward nor acknowledges this mismatch, so the reported robustness and fairness under different traffic flows (Figs. 9 and 11) is not actually backed by the optimization objective. This is a real flaw, though it only undermines the unequal-arrival portion of the abstract's claim, not the equal-arrival results.\n\nTwo smaller issues: Eq. (21) is missing the discount factor γ before V(s_{t+1}) in the critic loss, and the paper provides no error bars or released code/data, so the simulation results can't be independently checked or reproduced. The missing γ is likely a typo, but it should be fixed. The lack of artifacts is a common weakness, not a fatal one.\n\nWho should read this: anyone working on DRL for wireless scheduling or RIS-assisted systems will find the framework and simulation setup useful. It is not a theoretical contribution, but the empirical comparison is honest and the equal-arrival results are credible. This paper deserves a serious referee. I would send it to peer review with a request for the unequal-arrival reward fix, the Eq. (21) correction, and ideally some variance over random seeds.","headline":"Competent hybrid-PPO delay-minimization paper with a real reward-objective mismatch under unequal arrival rates; the equal-arrival results stand but the robustness claim needs work.","tokens_in":18038,"tokens_out":1660,"would_cite":false,"duration_ms":20337,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a hybrid deep reinforcement learning controller, jointly learning RIS phase shifts and subcarrier assignments from queue lengths, minimizes average packet delay in RIS-assisted OFDM downlinks.","keywords":["deep reinforcement learning","reconfigurable intelligent surface","OFDM","delay minimization","resource allocation","queue-aware scheduling","multi-agent PPO","transfer learning"],"falsifier":"On a small system ($K=3$, $N=8$, $M=36$) with unequal Poisson arrival rates, train the proposed method once with $r_t=-\\sum_k q_k[t]$ and once with $r_t=-\\sum_k \\lambda_k q_k[t]$; if the $\\lambda_k$-weighted reward does not lower average or per-user delay, the reward-proxy equivalence is the point of failure. On a fixed channel snapshot, exhaustively search all $K^N$ subcarrier assignments with MRT-water-filling beamforming and compare the best achievable delay against the learned policy; if exhaustive search does not outperform the policy, the simulation evidence would not demonstrate that the learning is near-optimal.","tokens_in":126,"feed_emoji":"📡","tokens_out":8118,"duration_ms":125824,"temperature":0.7,"pith_summary":"This paper tries to establish that the two coupled decisions that dominate delay in a reconfigurable-intelligent-surface (RIS) assisted OFDM downlink—setting the RIS phase shifts and assigning subcarriers to users—can be learned together by a hybrid deep reinforcement learning controller rather than solved by alternating optimization. The authors model each time slot as a Markov decision process and split the mixed action space: one continuous-action PPO network chooses the RIS phase vector, while a multi-agent discrete-action PPO assigns each subcarrier to one user, with active beamforming obtained in closed form by maximum-ratio transmission and water filling. The state includes channel state, current packet arrivals, and buffer backlogs, and the reward is the negative total backlog, which the paper argues is a proxy for average delay. Simulations with three users, 64 RIS elements, and random Poisson arrivals show lower average delay and jitter than random allocation, a no-RIS setup, max-sum-rate, and max-min-rate baselines, including under bursty traffic and unequal arrival rates. If correct, the paper offers a delay-aware design template for RIS systems that avoids per-slot mixed-integer optimization.","feed_headline":"Hybrid DRL policy cuts delay in RIS-assisted OFDM","feed_subtitle":"A queue-aware PPO agent tunes smart-surface phases and subcarrier assignments to outrun throughput-only baselines.","key_machinery":"The load-bearing mechanism is a hybrid proximal-policy-optimization controller split by action type. PPO-$\\Theta$ is a continuous-action actor-critic that maps a state containing direct and cascaded CSI, buffer lengths $q_k[t]$, and packet arrivals $\\ell_k[t]$ to the RIS phase vector $\\theta[t]$, using a convolutional feature extractor for the channel tensors. PPO-$N$ is a multi-agent discrete-action version in which each subcarrier has its own actor choosing one of $K$ users, while a centralized critic evaluates the global state; the two modules share the reward $r_t=-\\sum_k q_k[t]$. The active beamforming at the base station is computed, not learned, by maximum-ratio transmission followed by water-filling over the allocated subcarriers. The design work that carries the argument is the decomposition of the mixed action space and the inclusion of backlog and arrivals in the state, which makes the policy react to delay pressure rather than only to instantaneous rate.","core_discovery":"The central claim is that average packet delay in a RIS-assisted OFDM system can be minimized by a queue-aware learned policy over the hybrid action space, and that this beats conventional throughput-oriented or rate-fair baselines. The paper formulates slot-by-slot decisions as an MDP and trains two proximal-policy-optimization modules on the shared reward $r_t=-\\sum_k q_k[t]$: PPO-$\\Theta$ outputs continuous RIS phase shifts, and PPO-$N$ uses one agent per subcarrier to output discrete user assignments, thereby reducing the subcarrier action space from $K^N$ to $N$ parallel choices of size $K$. Active beamforming is not learned: after the two PPO modules fix phases and subcarrier-user pairings, the base station uses maximum-ratio transmission with water-filling power allocation. The evidence is simulation-based; across arrival rates, burst injections, unequal traffic flows, and three robustness scenarios, the proposed method keeps the lowest average delay and jitter, clears buffer bursts by shifting rate to heavily backlogged users, and maintains balanced backlogs. No theoretical guarantee is claimed for the learned policy, and the comparison rests on the simulated scenarios defined in the paper.","pith_inferences":["A direct extension the paper does not explore is reweighting the reward by arrival rate; a natural test is $r_t=-\\sum_k \\lambda_k q_k[t]$ under unequal traffic, since the backlog sum is only an exact delay proxy for equal arrival rates.","The two-module PPO decomposition is not tied to OFDM subcarriers and could be transferred to other mixed-action wireless scheduling problems, such as joint power and resource-block allocation or RIS phase design with user pairing.","The reported scale is three users; the claimed scalability of the multi-agent subcarrier design would need tests at larger $N$ and $K$, since the centralized critic still observes global state and its training cost is the likely bottleneck."],"forward_implications":["The hybrid split removes the need to alternate between phase optimization and subcarrier assignment in each time slot, since both decisions come from one trained controller.","The multi-agent subcarrier decomposition cuts the discrete action space from $K^N$ to $N$ agents each choosing among $K$ users, making the method applicable to systems with many subcarriers.","State features tied to delay—buffer backlog and current arrivals—let the policy prioritize heavily loaded users during bursts, which the paper shows clears backlog faster than a rate-maximizing or max-min-rate policy.","Pre-training the phase network on a max-min-rate reward and then switching to the backlog reward accelerates convergence, with the return stabilizing around 150 episodes in the reported setting.","The simulation comparisons imply that RIS assistance is necessary for low delay in this system: the no-RIS baseline has the highest average delay across the tested arrival rates."],"supporting_citations":[{"why":"Supplies the 3GPP Poisson packet arrival model used to generate traffic in the buffers.","marker":"[35]"},{"why":"Backs the use of negative total backlog as the reward proxy for average delay.","marker":"[36]"},{"why":"Supplies the max-sum-rate benchmark's subcarrier allocation scheme, a key baseline.","marker":"[37]"},{"why":"Supplies the max-sum-rate benchmark's RIS phase-shift optimization, a key baseline.","marker":"[38]"},{"why":"Motivates the multi-agent DRL decomposition used for subcarrier allocation.","marker":"[27]"}],"fun_headline_variants":["Queue-aware DRL tweaks RIS phases and subcarriers to slash delay","Hybrid PPO agents cut latency in smart-surface OFDM systems","DRL tunes RIS and subcarriers for lower packet delay","Delay-minimizing beamforming and allocation via hybrid RL","Multi-agent PPO reduces delay in RIS-assisted OFDM"],"cache_read_input_tokens":20224,"weakest_assumption_plain":"The reward $r_t=-\\sum_k q_k[t]$ is treated as a faithful proxy for average delay, but that equivalence is exact only when user arrival rates are equal; the unequal-arrival experiments keep the unweighted reward, so the policy may not be directly optimizing the stated delay objective in that regime.","fun_headline_variants_meta":{"raw":{"variants":["Queue-aware DRL tweaks RIS phases and subcarriers to slash delay","Hybrid PPO agents cut latency in smart-surface OFDM systems","DRL tunes RIS and subcarriers for lower packet delay","Delay-minimizing beamforming and allocation via hybrid RL","Multi-agent PPO reduces delay in RIS-assisted OFDM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000423,"raw_usage":{"total_tokens":2227,"prompt_tokens":1056,"completion_tokens":1171,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":672,"completion_tokens_details":{"reasoning_tokens":1083}},"tokens_in":672,"tokens_out":1171,"duration_ms":9146,"temperature":1.0,"reasoning_tokens":1083,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:58:45.135527+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a small system ($K=3$, $N=8$, $M=36$) with unequal Poisson arrival rates, train the proposed method once with $r_t=-\\sum_k q_k[t]$ and once with $r_t=-\\sum_k \\lambda_k q_k[t]$; if the $\\lambda_k$-weighted reward does not lower average or per-user delay, the reward-proxy equivalence is the point of failure. On a fixed channel snapshot, exhaustively search all $K^N$ subcarrier assignments with MRT-water-filling beamforming and compare the best achievable delay against the learned policy; if exhaustive search does not outperform the policy, the simulation evidence would not demonstrate that the learning is near-optimal.","supporting_citations":[{"cited_title":"Evolved universal terrestrial radio access network (EUTRAN),","cited_arxiv_id":null,"evidence_quote":"Supplies the 3GPP Poisson packet arrival model used to generate traffic in the buffers."},{"cited_title":"Power allocation for delay optimization in device- to-device networks: A graph reinforcement learning approach,","cited_arxiv_id":null,"evidence_quote":"Backs the use of negative total backlog as the reward proxy for average delay."},{"cited_title":"Intelligent reflecting surface meets OFDM: Protocol design and rate maximization,","cited_arxiv_id":null,"evidence_quote":"Supplies the max-sum-rate benchmark's RIS phase-shift optimization, a key baseline."},{"cited_title":"Spectrum sharing in vehicular networks based on multi-agent reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Motivates the multi-agent DRL decomposition used for subcarrier allocation."}],"review_version":1}