{"id":"60e4d1ab-7131-4672-87f9-470c11d1ed72","arxiv_id":"1908.08401","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"An actor-critic reinforcement learning framework for dynamic multichannel access matches or beats DQN in simulations, scales to 64 channels, and supports decentralized multi-user decisions without information exchange.","lead":"This paper applies a deep actor-critic reinforcement learning algorithm to the problem of choosing which wireless channels to access, for both one user and multiple competing users. It reports higher average rewards and faster decisions than a deep Q-network baseline when the number of channels is large, which matters for dynamic spectrum sharing.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The DQN comparison may rest on a tuning artifact: the baseline in Sec. VI-A is taken from [7] without retuning or target-network stabilization, and Fig. 4 shows single curves, so the reported DQN collapse at N=64 is not yet evidence that actor-critic is more scalable.","rationale":"The reader's weakest assumption is that the DQN baseline was not configured comparably and that single-run curves are unstable. My stress-test identifies the same issue as the single most load-bearing concern. The paper is a competent empirical RL study: the POMDP formulation is clear, the actor-critic algorithms are standard, and the additional multi-user and time-varying experiments provide useful breadth. The strongest claim, however, rests on a comparative evaluation against a DQN baseline that is imported from [7] without adaptation to N=32/64 and without the target-network stabilization that is standard for DQN. Because the reported DQN degradation with channel count is precisely the signature of Q-learning instability, the comparison does not establish that actor-critic is fundamentally more scalable rather than that the particular DQN configuration was not tuned. The absence of error bars or seed counts compounds this: even a well-configured RL comparison needs multiple runs to support a quantitative advantage. I do not see an internal inconsistency in the actor-critic derivation itself, and the runtime advantage is plausibly explained by the lack of minibatch replay. I therefore do not move the verdict: conditional acceptance remains appropriate, with the condition that the baseline be properly stabilized, tuned, and run over multiple seeds. This matches the reader's conditional verdict, so the verdict should remain unchanged.","tokens_in":19644,"tokens_out":5587,"duration_ms":90549,"concrete_test":"Re-run the DQN baseline of Sec. VI-A for N=64, p=0.9 with a standard target network updated every C steps, a learning-rate sweep over {1e-5, 3e-5, 1e-4, 3e-4}, an epsilon-decay schedule tuned for 64 channels, and 10 random seeds. Report mean and standard deviation of average reward across seeds. If the tuned DQN reaches or exceeds the actor-critic reward within one standard deviation, the claim that 'DQN cannot handle the case of 64 channels' is a tuning artifact and the central scalability claim fails as stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that actor-critic outperforms DQN for 32 and 64 channels, with DQN unable to handle 64 channels. The evidence for this is the average-reward comparison in Fig. 4. Two load-bearing conditions must hold for that evidence to support the claim. First, the DQN baseline must represent DQN's actual capability, not an accidentally poor configuration. The paper fixes DQN hyperparameters from [7] (replay memory 1,000,000, minibatch 32, two hidden layers) with no retuning for N=32/64 and, as described, does not mention a target network. Plain Q-learning with replay is known to be unstable with function approximation, especially as the action space grows; the collapse at N=64 is exactly the failure mode that target networks were introduced to prevent. Second, the comparison must be statistically stable: every reported curve is a single trajectory with no error bars, seed counts, or variance information. For a stochastic RL comparison, a single run is not enough to distinguish a real scalability advantage from training noise or an unlucky DQN initialization. The paper's other DQN comparisons in the multi-user and multi-channel-access cases share the same limitation, so the headline advantage is not independently corroborated. The time-efficiency comparison in Sec. VI-E is more robust because it follows from the minibatch-free actor-critic update, but that does not rescue the reward comparison. The central claim could still be true, but it is currently supported by an unfair or unrepresentative baseline and unquantified randomness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a deep actor-critic reinforcement learning framework for dynamic multichannel access, covering both a single-user scenario and a decentralized multi-user scenario. The system model treats channels as correlated Markovian sources with good/bad states, gives users only observations of the channels they select, and defines average-reward objectives for both the single-user and multi-user problems. The authors develop actor-critic algorithms (Algorithms 1 and 2) with standard TD-error-based critic updates and policy-gradient actor updates. The experimental section compares average reward and runtime against DQN, random access, the Whittle index heuristic, and an optimal policy with known dynamics, across 16, 32, and 64 channels. The central claim is that the proposed actor-critic framework achieves competitive performance at 16 channels and better performance than DQN at 32 and 64 channels, and that it handles large action spaces and time-varying environments effectively.","tokens_in":19985,"tokens_out":5915,"duration_ms":268446,"significance":"If the central claim holds, the paper provides a useful scalability result: an actor-critic approach that remains effective when the number of channels and the action space grow, in a setting where DQN degrades sharply. The paper's strengths include a clear problem formulation, standard and correctly stated actor-critic update equations, a decentralized multi-agent extension that requires no information exchange, and a comparison against several independent baselines including an optimal policy. The time-efficiency argument in Section VI-E is also reasonable and follows from avoiding minibatch replay. However, the comparative empirical claims currently rest on single-run curves and a DQN baseline that is not retuned for larger channel counts, so the significance is conditional on additional statistical evidence.","major_comments":[{"comment":"The DQN baseline is taken from [7] with a replay memory of 1,000,000 and minibatch of 32, with no retuning for N=32 and N=64, and the paper does not state whether a target network is used. The headline claim that actor-critic outperforms DQN at 32 and 64 channels depends entirely on the DQN curves in Fig. 4(b)-(c). Without either retuning the DQN for each problem size or explicitly justifying the fixed configuration, the comparison may reflect a poorly configured baseline rather than a scalability advantage of actor-critic. Please retune the DQN baseline for each N, state clearly whether a target network is employed, and report the corresponding DQN learning curves.","section":"Section VI-A, Fig. 4"},{"comment":"All reported learning curves and distribution tables appear to come from a single run, or a single 500-time-slot window for Tables I-IV, with no error bars, seed counts, or variance information. Because deep RL training is stochastic, a single trajectory does not support strong comparative statements such as \"the DQN framework cannot handle 64 channels\" in Section VI-B1. Please run multiple independent seeds for each configuration and report means with confidence intervals or standard deviations, and where feasible a pairwise significance test. Without this, the empirical advantage of the proposed framework is not statistically established.","section":"Sections VI-B, VI-C, VI-D; Figs. 4, 8, 9, 14; Tables I-IV"},{"comment":"The action space is defined as D = C(N,k) discrete actions in Section III-B3, but Section VI-A states that the actor's output layer has N neurons with softmax. The algorithm text also says the agent scores all actions in the action space and selects the highest score, which requires D output units. If in fact the actor outputs per-channel scores and the k largest are selected, that decision rule should be stated explicitly and the policy-gradient expressions in (25)-(26) should be reconciled with it. This matters because the multiple-good-channel experiments in Fig. 8 are used to support the claim that the framework handles large action spaces.","section":"Sections V-A and VI-A, Fig. 8"}],"minor_comments":[{"comment":"In the actor update line, the quantity \"δπ θj,t\" is undefined; it should presumably be the TD error δj,t.","section":"Algorithm 2"},{"comment":"There is a typo: \"when the nunber of channels is increased\" should read \"number.\"","section":"Section VI-B1"},{"comment":"The axis labels in the submitted version are garbled (e.g., \"P obability\" and \"Ave age Rewa d\"); these should be replaced with clean labels.","section":"Fig. 4"},{"comment":"The symbol R is used both for the long-term average reward and for the per-slot reward; please distinguish these, for instance by using R_t for the per-slot reward.","section":"Equations (11), (15), (19), (20)"},{"comment":"The Whittle index baseline is allowed to observe channels separately for 10,000 iterations to estimate transition probabilities; this gives it privileged information relative to the online learners, and the comparison would benefit from stating this explicitly when interpreting its low performance.","section":"Section VI-A, Whittle index heuristic"}],"recommendation":"major_revision","confidential_remarks":"The paper's central scalability claim is interesting and the algorithmic framework is standard, but the empirical evidence as presented is not yet sufficient to support the headline comparisons. The requested revisions -- multi-seed statistics and a fairer or better-documented DQN baseline -- are within the scope of a revision, so I would not reject at this stage. If the additional experiments show the DQN collapse persists under retuning and with error bars, the paper could be suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nThis paper applies actor-critic deep RL to dynamic multichannel access and claims it scales to 64 channels where DQN fails. The core claim is plausible but not yet evidenced. The algorithm is textbook actor-critic; the contribution is the application, the decentralized multi-user extension, and a broad set of simulations—round-robin, arbitrary switching, multiple good channels, multi-user with and without priorities, time-varying environments. That variety is a real strength. The reward design and the SINR-based justification for the collision discount are sensible, and the per-decision runtime comparison is genuinely useful: actor-critic skips minibatch replay, which explains the speedup and is backed by a reasonable complexity estimate. This part of the paper holds up.\n\nThe soft spot is the DQN baseline. The paper takes DQN hyperparameters directly from [7], with no retuning for N=32/64, and no target network is mentioned. Plain DQN with replay is known to become unstable as action space grows, so the reported collapse at 64 channels is exactly the failure mode that target networks were introduced to prevent. In other words, the headline 'better than DQN at 32 and 64 channels' may be a tuning artifact rather than a fundamental advantage. On top of that, all reward curves are single runs with no error bars, seed counts, or variance, so the comparisons are statistically unsupported. No code or data are released, which makes auditing harder. These issues affect all the reward-based comparisons, not just the single-user case.\n\nThe time-efficiency result is more robust, and the decentralized multi-user framework is interesting. But the main scaling claim needs work. To make it stick, the authors should provide multi-seed averages with variance, retune DQN (or at least add a target network), and consider releasing code or data. Those are reasonable requests for a revision, not reasons to reject outright.\n\nWho should read this: people doing RL for wireless access who want a clear, complete demonstration of actor-critic on a POMDP channel selection problem. It deserves serious refereeing. I would send it out, but with the expectation that the baseline comparison and statistical reporting be fixed before acceptance. The runtime story and the decentralized multi-agent part are worth keeping even if the scaling claim is weakened.","headline":"Competent applied RL paper whose scaling claim over DQN is plausible but not yet evidenced, because the DQN baseline is poorly configured and no variance is reported; the runtime advantage is the most solid result.","tokens_in":657,"tokens_out":1657,"would_cite":false,"duration_ms":572798,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that an actor-critic deep reinforcement learning framework outperforms a deep Q-network in dynamic multichannel access when the number of channels is 32 or 64, while staying competitive at 16 channels.","keywords":["dynamic multichannel access","deep reinforcement learning","actor-critic","POMDP","deep Q-network","multi-agent reinforcement learning","wireless spectrum access","temporal-difference learning"],"falsifier":"Rerun the 32- and 64-channel single-good-channel experiments with DQN hyperparameters retuned (for instance, sweep minibatch size and network width, and average over multiple random seeds with error bars). If the tuned DQN earns average rewards at or above the actor-critic agent's reported values, the paper's central scalability claim is refuted; if the tuned DQN still collapses at 64 channels, the claim survives.","tokens_in":19423,"feed_emoji":"📶","tokens_out":6193,"duration_ms":54933,"temperature":0.7,"pith_summary":"This paper tries to establish that an actor-critic deep reinforcement learning agent is a more scalable channel-access learner than a deep Q-network: it stays competitive at 16 channels and outperforms DQN at 32 and 64 channels, with roughly 87 to 93 percent less runtime per decision. The authors model each user's problem as a partially observable Markov decision process in which the user sees only the channels it selects, and they give single-agent and decentralized multi-agent algorithms. A sympathetic reader should care because spectrum access decisions must be made quickly and at scale, and the paper points to a standard RL architecture that avoids the replay-buffer bottleneck of DQN while learning both channel switching patterns and, in the multi-user case, collision avoidance.","feed_headline":"Actor-critic RL beats DQN at 32 and 64 channels","feed_subtitle":"No replay buffer plus policy gradients keeps rewards high as channel counts grow, in 87-93% less time.","key_machinery":"The load-bearing mechanism is a two-network actor-critic agent whose input is the last $\\Omega$ observations, stored as a sparse sliding window. The actor network ends in a softmax layer and assigns a score to every valid $k$-channel action; the critic estimates the state value and produces the temporal-difference error $\\delta_t = R_t + \\gamma V_\\mu(O_{t+1}) - V_\\mu(O_t)$. The actor is updated by the policy gradient $\\alpha \\nabla_\\theta \\log \\pi_\\theta(O_t,a_t)\\delta_t$, and the critic by minimizing $(\\delta_t)^2$. The absence of experience replay is what makes the argument work: the authors argue that the DQN baseline's minibatch replay is the dominant computational cost, so an actor-critic update with no replay buffer explains both the runtime savings and the ability to keep learning as the action space grows.","core_discovery":"The central claim is that replacing DQN's value-function-plus-replay design with an actor-critic design removes the main scalability bottleneck for learned multichannel access. In the authors' experiments, DQN is slightly better when there are 16 channels, but at 32 channels DQN earns negative average reward for switching probabilities at or below 0.85, and at 64 channels DQN cannot learn the task at all; the actor-critic agent produces positive, stable average rewards in both regimes. The same pattern appears when a user can access several channels at once: DQN's reward diminishes as the combinatorial action space grows, while the actor-critic reward keeps increasing. In the decentralized multi-user setting, actor-critic agents also outperform DQN agents in average reward and are better at avoiding collisions on excellent and good channels. The runtime claim is that per-decision computation is roughly $3/M$ times DQN's, with $M$ the DQN minibatch size, because the actor-critic agent updates from the critic's temporal-difference error without replaying stored transitions.","pith_inferences":["If the scalability claim holds, the actor-critic architecture may be the safer default for any spectrum-access task with action spaces large enough that replay becomes the bottleneck, even outside the exact correlated-Markov model tested here.","The paper does not tune the DQN baseline for large $N$, so the boundary at which DQN genuinely breaks is unknown; a fair benchmark might place it between 32 and 64 channels rather than at 32 or 64.","A natural extension is to vary the observation window $\\Omega$ and the actor/critic learning-rate ratio: the re-training experiments suggest the learning-rate decay schedule controls the speed-versus-stability tradeoff in time-varying environments.","The multi-agent results suggest a testable prediction: if user priorities are set by asymmetric rewards, independent actor-critic agents should converge to a division of excellent channels, which could be verified by longer runs with more than three users."],"forward_implications":["Actor-critic learning remains usable when the number of channels grows to 64 with a single good channel and switching probability as low as 0.75, where DQN fails or earns negative reward.","When a user can select $k$ channels per slot, the actor-critic agent's average sum reward continues to rise as $k$ grows, whereas DQN's reward drops, so the framework tolerates combinatorial action spaces better.","In a decentralized multi-user network with no information exchange, independent actor-critic agents learn to avoid each other's channels, producing higher average sum reward and fewer collisions than decentralized DQN agents.","Per-decision runtime is reduced by about 87 to 93 percent because the agent omits minibatch replay, which matters for real-time spectrum access.","After an unknown change point in the channel switching pattern, actor-critic agents recover to their previous reward level, while DQN recovers to a lower level."],"supporting_citations":[{"why":"Supplies the DQN baseline that the proposed framework must beat, and the known-dynamics optimal access policy used as a performance ceiling.","marker":"[7]"},{"why":"Supplies the Whittle index heuristic baseline and the indexability theory behind it.","marker":"[35]"},{"why":"Provides the reinforcement-learning foundations and policy-update framework the actor-critic algorithm builds on.","marker":"[6]"},{"why":"Provides natural actor-critic background for the policy-gradient update rule.","marker":"[5]"}],"fun_headline_variants":["Actor-critic RL scales to 64 channels where DQN fails","RL channel access: actor-critic beats DQN at scale","No replay buffer: actor-critic wins multichannel access","Actor-critic RL outperforms DQN in multichannel access","Multichannel access: actor-critic RL outlearns DQN"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's comparison claims rest on the assumption that the DQN baseline, configured with fixed replay memory and minibatch sizes from the prior work and not retuned for 32 or 64 channels, is the right benchmark; if DQN were properly tuned for the large-channel cases, the reported gap could shrink or disappear.","fun_headline_variants_meta":{"raw":{"variants":["Actor-critic RL scales to 64 channels where DQN fails","RL channel access: actor-critic beats DQN at scale","No replay buffer: actor-critic wins multichannel access","Actor-critic RL outperforms DQN in multichannel access","Multichannel access: actor-critic RL outlearns DQN"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000632,"raw_usage":{"total_tokens":2939,"prompt_tokens":984,"completion_tokens":1955,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":1861}},"tokens_in":600,"tokens_out":1955,"duration_ms":12556,"temperature":1.0,"reasoning_tokens":1861,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:01:41.646514+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the 32- and 64-channel single-good-channel experiments with DQN hyperparameters retuned (for instance, sweep minibatch size and network width, and average over multiple random seeds with error bars). If the tuned DQN earns average rewards at or above the actor-critic agent's reported values, the paper's central scalability claim is refuted; if the tuned DQN still collapses at 64 channels, the claim survives.","supporting_citations":[{"cited_title":"Deep reinforcement learning for dynamic multichannel access in wireless networks,","cited_arxiv_id":null,"evidence_quote":"Supplies the DQN baseline that the proposed framework must beat, and the known-dynamics optimal access policy used as a performance ceiling."},{"cited_title":"Indexability of restless bandit pro blems and optimality of Whittle index for dynamic multichan nel access,","cited_arxiv_id":null,"evidence_quote":"Supplies the Whittle index heuristic baseline and the indexability theory behind it."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the reinforcement-learning foundations and policy-update framework the actor-critic algorithm builds on."},{"cited_title":"Natural actor -critic,","cited_arxiv_id":null,"evidence_quote":"Provides natural actor-critic background for the policy-gradient update rule."}],"review_version":1}