Pith. sign in

REVIEW 3 major objections 5 minor 32 references

Fused-CPRO mixes learnable, source, and expert wireless policies so a constrained learner can meet long-term limits with far fewer online interactions.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 02:35 UTC pith:QT2IBJU3

load-bearing objection Solid constrained-RL methods paper that cleanly fuses the authors’ CSSCA actor with joint policy reuse and offline critic data, plus a bias-aware KKT proof; the “substantially faster” claim is only single-curve evidence so far. the 3 major comments →

arxiv 2607.09498 v1 pith:QT2IBJU3 submitted 2026-07-10 eess.SP

Fused Constrained Policy Reuse Optimization for Wireless Resource Allocation

An Liu , Zheyuan Zhou , Kexuan Wang This is my paper
classification eess.SP
keywords constrained reinforcement learningpolicy reusetransfer learningdomain knowledgewireless resource allocationCSSCAMU-MIMOMIMO-ISAC
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Wireless resource allocators that learn by trial and error waste energy and signaling while they explore, and randomly started policies often break delay or sensing limits before they improve. This paper claims that the way out is to treat the allocation policy as an explicit mixture of three ingredients: a new learnable policy, frozen source policies from related scenarios, and domain-knowledge rules such as queue-aware power splits or zero-forcing heuristics. Reuse probabilities over those ingredients are optimized jointly with the new policy under a constrained Markov decision process, using a successive-convex actor and a critic that is warm-started from offline trajectories of the old policies. Under standard regularity conditions the algorithm converges almost surely to a KKT point of the original constrained problem. On delay-constrained MU-MIMO power control and CRB-constrained MIMO-ISAC beamforming the same method reaches feasible high-performance regions substantially faster than pure online constrained RL baselines.

Core claim

Fused-CPRO constructs a mixed allocation policy as a convex combination of a learnable target policy, source policies, and domain-knowledge policies, jointly optimizes the target parameters and the reuse probabilities with a CSSCA actor under average-cost CMDP constraints, trains the critic from mixed offline-online data, and proves almost-sure convergence of every limiting point to an ϵ-KKT point whose residual vanishes as critic width grows.

What carries the argument

The knowledge-fused mixed policy π_θ(a|s)=∑ ρ_n π_n(a|s) whose reuse vector ρ is co-optimized with the target DNN parameters by CSSCA surrogates, while a critic is trained on a decaying mixture of offline old-policy data and online target samples.

Load-bearing premise

The critic’s local linearization must be able to represent the true surrogate Q-functions and the TD feature matrix must stay uniformly positive-definite over the whole compact parameter set; if either fails, the tracking lemmas that underwrite KKT convergence collapse.

What would settle it

On the same MU-MIMO or MIMO-ISAC benchmarks, run Fused-CPRO with the offline weight held fixed (never decaying) or with the critic width deliberately under-parameterized; if the policy still converges to a feasible KKT point and matches the reported sample-efficiency gains, the claimed necessity of the summability and representability conditions is false.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Fused-CPRO, a constrained actor-critic method for wireless resource allocation that mixes a learnable target policy with frozen source policies and domain-knowledge (DK) policies under jointly optimized reuse probabilities ρ. The actor uses constrained stochastic successive convex approximation (CSSCA) on non-convex average-cost CMDP objectives/constraints; the critic is trained from mixed offline-online data with a decaying offline weight ξ_t. The authors prove almost-sure convergence of the iterates to an ϵ_mQ-KKT point of the knowledge-fused CMDP (Theorem 1), with ϵ_mQ vanishing as critic width grows, under ergodicity, multi-time-scale step sizes, and neural TD representability/stability assumptions. Simulations on delay-constrained MU-MIMO power control and CRB-constrained MIMO-ISAC beamforming report lower power / higher sum-rate and faster feasible convergence than PPO-Lag, CPO, SCAOPO, SLDAC, and HRL.

Significance. If the theory and empirical claims hold, the work is a meaningful step for sample-efficient constrained RL in wireless systems: it jointly optimizes reuse weights over heterogeneous priors (source + DK), reuses offline trajectories in the critic, and supplies a bias-aware almost-sure KKT guarantee that accounts for mixture-ratio and offline-distribution mismatch—going beyond single-policy CSSCA methods (SCAOPO/SLDAC) and actor-only policy reuse (HRL). The two continuous-control wireless CMDPs are relevant stress tests. The main theoretical contribution is carefully structured (critic tracking, surrogate consistency, CSSCA KKT transfer). The practical significance of the “substantially faster / better” claim, however, depends on whether the reported learning curves are statistically reliable; that part is not yet solidly established in the manuscript.

major comments (3)
  1. The abstract, §V, and conclusion assert that Fused-CPRO “converges substantially faster” and improves empirical performance over PPO-Lag, CPO, SCAOPO, SLDAC, and HRL. Figs. 2–5 present single learning/reuse trajectories (or unreported averages) with no multi-seed means, standard-error bands, or statistical tests. In continuous-control CMDPs with non-convex queue/CRB costs, actor-critic variance is typically large enough that early-stage gaps can shrink or reverse under re-seeding. Without multi-seed tables (or at least error bars over ≥5–10 seeds) for power/delay and sum-rate/CRB, the empirical half of the central claim remains provisional and should be strengthened before acceptance.
  2. §IV-A, Assumption 2 and the admissible region (29) require multi-time-scale exponents satisfying, among other conditions, 1 > 2κ2−1 > κ1 > 0.43. In §V-A the authors set β^(ρ)_t = 1/t^{0.2} (κ2≈0.2), for which 2κ2−1 = −0.6 is not greater than κ1; β^(ψ)_t = 1/t^{0.7} and the MU-MIMO/ISAC schedules also leave η_t unspecified relative to (28)–(29). Theorem 1’s almost-sure ϵ_mQ-KKT guarantee is stated under these summability conditions. The manuscript should either (i) report experimental schedules that lie in the proven region, or (ii) clearly separate “theory-compliant” runs from practical schedules and discuss what is proven versus what is only empirically observed.
  3. Theorem 1 and Remark 1 require feasible initialization (or rely on the feasibility-restoration QP (25) not trapping at an undesired stationary point of constraint-violation minimization). The simulations do not report whether θ0 is feasible, how often restoration is active, or constraint-violation trajectories during early learning. Given that a main motivation is avoiding long-term constraint violations under costly online exploration, the paper should document initial feasibility, early-stage violation statistics, and the practical behavior of (24)–(25) on both benchmarks.
minor comments (5)
  1. DNN architectures (depth, width m_Q, activation), action projection details, and exact values of ζ_i, ϵ (DK Gaussian smoothing), and critic projection radius R_ω are not fully specified in §V; these affect reproducibility of both theory (Lemma 4) and experiments.
  2. Figs. 2–5 would benefit from shared axis scales across methods, explicit constraint-limit annotations on all constraint plots, and a short table of final mean±std metrics once multi-seed runs are added.
  3. Notation: C′_i vs C_i, and the dual use of “cost” for both objective and constraints, could be clarified once in §II-C; the mixture-ratio boundedness used in §IV is stated informally and could be listed as an explicit assumption.
  4. Related-work placement of HRL [21],[32] and the precise algorithmic differences from SLDAC (critic offline reuse + joint ρ update) are clear in §III-E; a short ablation removing only offline data (keeping ρ) would further isolate contributions if space allows.
  5. Minor presentation: arXiv id and some equation line breaks; ensure consistent use of “almost-sure” vs “with high probability” when invoking Lemma 4’s probabilistic DNN bounds inside a.s. statements.

Circularity Check

0 steps flagged

No significant circularity: KKT claim is bias-aware analysis of a new mixture/offline estimator, not a fit or definitional loop; self-citations supply CSSCA building blocks only.

full rationale

Walk of the derivation chain shows a standard constrained-RL construction, not a circular reduction. The CMDP costs C_i and mixed policy π_θ = Σ_n ρ_n π_n are defined first (Sec. II-C, Eq. 6); the policy-reuse gradients (Eqs. 8–11), mixed offline-online critic (Eqs. 16–20), and CSSCA surrogates (Eqs. 21–26) are then derived from that definition. Lemmas 1–3 explicitly bound the new bias terms from mixture ratios π_n/π_θ and the offline weight ξ_t rather than assuming them away; Theorem 1 then applies the known CSSCA KKT argument under those controlled biases. Self-citations to SCAOPO [9], SLDAC [10], HRL [21], and CSSCA [20] supply algorithmic scaffolding and the final KKT transfer step, but the target statement (ϵ_mQ-KKT of the fused problem under policy reuse and offline data) is not equivalent to those citations by construction, nor is any empirical curve a fitted parameter renamed as a prediction. No uniqueness theorem is imported to forbid alternatives; no ansatz is smuggled as a first-principles derivation; no quantity is defined in terms of the quantity it is said to predict. Score 1 reflects only routine same-group citation of the CSSCA backbone, which is not load-bearing circularity under the stated rules.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 1 invented entities

The central KKT claim rests on standard CMDP ergodicity, a battery of multi-time-scale step-size and offline-weight summability conditions, neural TD representability/stability, and the availability of useful frozen source and domain-knowledge policies. Free parameters include all step-size exponents, quadratic regularizers, offline weight schedule, and DNN widths/radii. No new physical entities are postulated; the ‘mixed policy’ and ‘reuse vector ρ’ are algorithmic constructs.

free parameters (5)
  • step-size exponents (κ1…κ5) / concrete schedules αt, βt, ηt, γt, ξt
    Chosen to satisfy the nonempty region (29); different exponents appear in the two simulation sections and directly control the multi-time-scale separation required by the proof.
  • quadratic surrogate regularizers ζi
    Control strong convexity of the CSSCA surrogates; values not reported, yet required for the QP subproblems.
  • critic DNN width mQ and projection radius Rω
    Enter the local-linearization error ϵmQ and the NTK-style bounds of Lemma 4; asymptotic vanishing of ϵmQ is claimed only as mQ→∞.
  • offline weight schedule ξt and buffer lengths Tt, Ttoff
    Trade warm-start bias against asymptotic consistency; must satisfy summable αtξt and the series in Assumption 2.6.
  • Gaussian smoothing variance ϵ for deterministic DK policies
    Ad-hoc conversion of rule-based actions into differentiable densities; affects mixture ratios and gradient estimators.
axioms (5)
  • domain assumption For every feasible θ the induced Markov chain is ergodic with geometric mixing in total variation (Assumption 1).
    Standard for average-cost RL analysis; required for stationary distributions and bias bounds in Lemmas 1–3.
  • ad hoc to paper Surrogate Q-functions lie in the local linearization class of the critic DNN and the TD matrix Aθ is uniformly positive definite over Θ (Assumption 3).
    Enables the critic-tracking bound; not automatically true for arbitrary wireless CMDPs or network initializations.
  • domain assumption Mixture ratios πn(a|s)/πθ(a|s) remain uniformly bounded and costs/DNN outputs are uniformly bounded on compact sets.
    Needed for stable policy-gradient estimators under policy reuse (Section IV-A).
  • standard math Slater condition (or feasible initialization) for the CSSCA KKT argument (Theorem 1 / Remark 1).
    Classic constraint qualification; without it the algorithm may converge only to a stationary point of the feasibility-restoration problem.
  • domain assumption Source and DK policies are frozen, available a priori, and generate useful offline datasets for the target scenario.
    Load-bearing for the claimed sample-efficiency gain; if priors are poor the reuse probabilities simply collapse to the target policy.
invented entities (1)
  • knowledge-fused mixed policy πθ = Σ ρn πn with jointly optimized reuse vector ρ independent evidence
    purpose: Unifies target, source and domain-knowledge policies inside a single CMDP so that reuse probabilities become optimizable decision variables.
    Algorithmic construct rather than a physical entity; independent evidence is the empirical reuse-probability trajectories and the faster learning curves.

pith-pipeline@v1.1.0-grok45 · 35710 in / 3555 out tokens · 41634 ms · 2026-07-13T02:35:22.578992+00:00 · methodology

0 comments
read the original abstract

Deep reinforcement learning (DRL) has been widely adopted for wireless resource allocation due to its model-free adaptability. However, online exploration is costly, as randomly initialized policies may violate long-term constraints before sufficient data are collected. Future wireless systems must cope with increasingly dynamic traffic, fluctuating channel conditions, and stringent energy efficiency requirements, demanding algorithms that can learn quickly with minimal environment interactions to reduce both energy consumption and signaling overhead. We develop Fused-CPRO, a knowledge-fused constrained policy reuse optimization method addressing these challenges. Fused-CPRO constructs the allocation policy as a mixture of a learnable target policy, source policies from related scenarios, and domain-knowledge (DK) policies from expert rules, jointly optimizing the target policy and reuse probabilities under a constrained Markov decision process (CMDP). This fusion of heterogeneous priors accelerates convergence and enhances robustness. Constrained stochastic successive convex approximation (CSSCA) handles non-convex objectives and constraints, while a critic trained from mixed offline-online data improves sample efficiency by reusing pre-collected experience. We prove almost-sure convergence to a Karush-Kuhn-Tucker (KKT) point. Simulations on delay-constrained multi-user multiple-input multiple-output (MU-MIMO) power control and Cramer-Rao bound (CRB)-constrained multiple-input multiple-output integrated sensing and communication (MIMO-ISAC) beamforming demonstrate that Fused-CPRO improves empirical performance and converges substantially faster than representative baselines.

Figures

Figures reproduced from arXiv: 2607.09498 by An Liu, Kexuan Wang, Zheyuan Zhou.

Figure 1
Figure 1. Figure 1: The framework of the proposed Fused-CPRO algorithm [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Simulation Results of the delay-constrained MU-MIMO task [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Evolution of the reuse probabilities in the MU-MIMO task. [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Performance of Fused-CPRO and baseline algorithms in the MIMO [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Evolution of the reuse probabilities in the MIMO-ISAC task. [PITH_FULL_IMAGE:figures/full_fig_p013_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

32 extracted references · 1 linked inside Pith

  1. [1]

    Stability properties of constrained queueing systems and scheduling policies for maximum throughput in multihop radio networks,

    L. Tassiulas and A. Ephremides, “Stability properties of constrained queueing systems and scheduling policies for maximum throughput in multihop radio networks,”IEEE Transactions on Automatic Control, vol. 37, no. 12, pp. 1936–1948, 1992

    1936

  2. [2]

    Largest weighted delay first scheduling: Large deviations and optimality,

    A. L. Stolyar and K. Ramanan, “Largest weighted delay first scheduling: Large deviations and optimality,”Annals of Applied Probability, pp. 1– 48, 2001

    2001

  3. [3]

    M. J. Neely,Stochastic Network Optimization with Application to Com- munication and Queueing Systems, ser. Synthesis Lectures on Learning, Networks, and Algorithms. Springer, Cham, 2010

    2010

  4. [4]

    An iteratively weighted MMSE approach to distributed sum-utility maximization for a MIMO interfering broadcast channel,

    Q. Shi, M. Razaviyayn, Z.-Q. Luo, and C. He, “An iteratively weighted MMSE approach to distributed sum-utility maximization for a MIMO interfering broadcast channel,”IEEE Transactions on Signal Processing, vol. 59, no. 9, pp. 4331–4340, 2011

    2011

  5. [5]

    A vector-perturbation technique for near-capacity multiantenna multiuser communication-part I: channel inversion and regularization,

    C. B. Peel, B. M. Hochwald, and A. L. Swindlehurst, “A vector-perturbation technique for near-capacity multiantenna multiuser communication-part I: channel inversion and regularization,”IEEE Trans. Commun., vol. 53, no. 1, pp. 195–202, 2005

    2005

  6. [6]

    Age of information aware radio resource management in vehicular networks: A proactive deep reinforcement learning perspective,

    X. Chen, C. Wu, T. Chen, H. Zhang, Z. Liu, Y . Zhang, and M. Bennis, “Age of information aware radio resource management in vehicular networks: A proactive deep reinforcement learning perspective,”IEEE Transactions on wireless communications, vol. 19, no. 4, pp. 2268–2281, 2020

    2020

  7. [7]

    Reinforcement learning for radio resource management in ran slicing: A survey,

    M. Zangooei, N. Saha, M. Golkarifard, and R. Boutaba, “Reinforcement learning for radio resource management in ran slicing: A survey,”IEEE Communications Magazine, vol. 61, no. 2, pp. 118–124, 2023

    2023

  8. [8]

    Constrained policy optimization,

    J. Achiam, D. Held, A. Tamar, and P. Abbeel, “Constrained policy optimization,” inICML. PMLR, 2017, pp. 22–31

    2017

  9. [9]

    Successive convex approx- imation based off-policy optimization for constrained reinforcement learning,

    C. Tian, A. Liu, G. Huang, and W. Luo, “Successive convex approx- imation based off-policy optimization for constrained reinforcement learning,”IEEE Transactions on Signal Processing, vol. 70, pp. 1609– 1624, 2022

    2022

  10. [10]

    Single-loop deep actor-critic for constrained reinforcement learning with provable convergence,

    K. Wang, A. Liu, and B. Lin, “Single-loop deep actor-critic for constrained reinforcement learning with provable convergence,”IEEE Transactions on Signal Processing, vol. 72, pp. 4871–4887, 2024

    2024

  11. [11]

    Transfer learning for reinforcement learning domains: A survey

    M. E. Taylor and P. Stone, “Transfer learning for reinforcement learning domains: A survey.”Journal of Machine Learning Research, vol. 10, no. 7, 2009

    2009

  12. [12]

    Probabilistic policy reuse in a reinforce- ment learning agent,

    F. Fern ´andez and M. Veloso, “Probabilistic policy reuse in a reinforce- ment learning agent,” inProceedings of the fifth international joint conference on Autonomous agents and multiagent systems, 2006, pp. 720–727

    2006

  13. [13]

    Probabilistic policy reuse for inter-task transfer learning,

    F. Fern ´andez, J. Garc ´ıa, and M. Veloso, “Probabilistic policy reuse for inter-task transfer learning,”Robotics and Autonomous Systems, vol. 58, no. 7, pp. 866–871, 2010

    2010

  14. [14]

    Beamforming optimization for integrated sensing and communication systems: A deep reinforcement learning approach,

    Z. Huang and A. Liu, “Beamforming optimization for integrated sensing and communication systems: A deep reinforcement learning approach,” Mobile Communications, vol. 48, no. 10, pp. 41–48, 2024

    2024

  15. [15]

    Altman,Constrained Markov decision processes

    E. Altman,Constrained Markov decision processes. CRC press, 1999, vol. 7

    1999

  16. [16]

    R. S. Sutton and A. G. Barto,Reinforcement learning: An introduction. MIT press, 2018

    2018

  17. [17]

    A finite-time analysis of two time-scale actor-critic methods,

    Y . F. Wu, W. Zhang, P. Xu, and Q. Gu, “A finite-time analysis of two time-scale actor-critic methods,”Advances in Neural Information Processing Systems, vol. 33, pp. 17 617–17 628, 2020

    2020

  18. [18]

    Model- free reinforcement learning in infinite-horizon average-reward markov decision processes,

    C.-Y . Wei, M. J. Jahromi, H. Luo, H. Sharma, and R. Jain, “Model- free reinforcement learning in infinite-horizon average-reward markov decision processes,” inInternational conference on machine learning. PMLR, 2020, pp. 10 170–10 180

    2020

  19. [19]

    On-policy deep reinforcement learning for the average-reward criterion,

    Y . Zhang and K. W. Ross, “On-policy deep reinforcement learning for the average-reward criterion,” inInternational Conference on Machine Learning. PMLR, 2021, pp. 12 535–12 545

    2021

  20. [20]

    Stochastic successive con- vex approximation for non-convex constrained stochastic optimization,

    A. Liu, V . K. N. Lau, and B. Kananian, “Stochastic successive con- vex approximation for non-convex constrained stochastic optimization,” IEEE Trans. Signal Process., vol. 67, no. 16, pp. 4189–4203, 2019

    2019

  21. [21]

    A hybrid reinforcement learning framework for hard latency constrained resource scheduling,

    L. Zhang, A. Liu, and K. Wang, “A hybrid reinforcement learning framework for hard latency constrained resource scheduling,”IEEE Internet of Things Journal, 2025

    2025

  22. [22]

    Average cost temporal-difference learning,

    J. N. Tsitsiklis and B. Van Roy, “Average cost temporal-difference learning,”Automatica, vol. 35, no. 11, pp. 1799–1808, 1999

    1999

  23. [23]

    A convergence theory for deep learning via over-parameterization,

    Z. Allen-Zhu, Y . Li, and Z. Song, “A convergence theory for deep learning via over-parameterization,” inICML. PMLR, 2019, pp. 242– 252

    2019

  24. [24]

    Generalization bounds of stochastic gradient descent for wide and deep neural networks,

    Y . Cao and Q. Gu, “Generalization bounds of stochastic gradient descent for wide and deep neural networks,”Proc. Adv. Neural Inf. Process. Syst., vol. 32, pp. 10 835–10 845, 2019

    2019

  25. [25]

    A finite-time analysis of Q-learning with neural network function approximation,

    P. Xu and Q. Gu, “A finite-time analysis of Q-learning with neural network function approximation,” inICML. PMLR, 2020, pp. 10 555– 10 565

    2020

  26. [26]

    A theoretical analysis of deep Q-learning,

    J. Fan, Z. Wang, Y . Xie, and Z. Yang, “A theoretical analysis of deep Q-learning,” inProc. Learn. Dyn. Control. PMLR, 2020, pp. 486–489

    2020

  27. [27]

    Boosted fitted Q-iteration,

    S. Tosatto, M. Pirotta, C. d’Eramo, and M. Restelli, “Boosted fitted Q-iteration,” inICML. PMLR, 2017, pp. 3434–3443

    2017

  28. [28]

    On finite-time convergence of actor-critic algorithm,

    S. Qiu, Z. Yang, J. Ye, and Z. Wang, “On finite-time convergence of actor-critic algorithm,”IEEE J. Sel. Areas Inf. Theory, vol. 2, no. 2, pp. 652–664, 2021

    2021

  29. [29]

    Feasible direction methods for stochastic programming problems,

    A. Ruszczy ´nski, “Feasible direction methods for stochastic programming problems,”Mathematical Programming, vol. 19, no. 1, pp. 220–229, 1980

    1980

  30. [30]

    Successive convex approximation: Analysis and appli- cations,

    M. Razaviyayn, “Successive convex approximation: Analysis and appli- cations,” Ph.D. dissertation, University of Minnesota, 2014

    2014

  31. [31]

    Benchmarking safe exploration in deep reinforcement learning,

    A. Ray, J. Achiam, and D. Amodei, “Benchmarking safe exploration in deep reinforcement learning,”arXiv preprint arXiv:1910.01708, vol. 7, no. 1, p. 2, 2019

    Pith/arXiv arXiv 1910

  32. [32]

    A policy reuse reinforcement learning framework for hard latency constrained resource scheduling,

    L. Zhang and A. Liu, “A policy reuse reinforcement learning framework for hard latency constrained resource scheduling,” in2025 IEEE Wireless Communications and Networking Conference (WCNC). IEEE, 2025, pp. 1–6

    2025