REVIEW 3 major objections 5 minor 30 references
Leveraging Constraint Violation Signals For Action-Constrained Reinforcement Learning
T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read Constraint-violation signals can replace feasible action samples for training RL action maps.
desk verdict CV-Flows is a practical and novel flow-training idea, but its SAC entropy approximation rests on an unverified coverage assumption that is weakest precisely on the non-convex tasks where the method claims its biggest win. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the CV-Flow: a conditional normalizing flow trained by reverse KL against the target $p(a|s) \propto e^{-\lambda\,CV(a,s)}$, where $CV$ is the sum of positive parts of inequality violations and equality violations. Its training loss $J^f(\psi) = \mathbb{E}[\lambda\,CV(f_\psi(\hat{a},s),s) - \log|\det J_{f_\psi}|]$ avoids sampling feasible actions entirely. The second piece is Proposition 1, which uses the trained flow's closeness to the uniform feasible target to replace $\log \pi(a|s)$ with $\log \mu_\phi(\hat{a}|s) + \|\hat{a}\|^2/2 + K(s)$, so SAC's entropy regularization is computed in latent space and the critic never backpropagates through the flow.
What would settle it
Take a constraint with two disjoint feasible regions, train the CV-Flow, then compare the policy log-probability computed with and without the Jacobian term over a grid of feasible actions; if the approximation error is large, or if the reverse KL between the flow and the uniform target remains high, Proposition 1's step fails.
Extended reading notes
Core claim
The central discovery is that the hard part of flow-based action-constrained RL—collecting feasible action samples from a constrained space—is unnecessary. The authors define the target $p(a|s)$ as a uniform distribution over feasible actions, realized as $e^{-\lambda\,CV(a,s)}$ normalized by a state-dependent constant, and train the flow by reverse KL divergence, which reduces to maximizing $\lambda\,CV(f_\psi(\hat{a},s),s)$ minus the log-determinant of the flow Jacobian. Because the trained flow approximates the uniform target, the log-probability of the combined SAC policy can be written in latent coordinates as $\log \mu_\phi(\hat{a}|s) + \|\hat{a}\|^2/2$ plus a state-only constant, dropping the Jacobian term. This yields an SAC variant whose actor and critic train on latent actions, with a projection step kept as a safety net. The same CV signal works for state-wise constraints, where the violation function is learned from environment interaction via a linear cost model.
Load-bearing premise
The method assumes the trained flow's distribution over feasible actions is close enough to the uniform feasible target that the log-density of the combined policy can drop the flow's Jacobian term, which fails if the flow misses part of a disconnected or multi-modal feasible region.
Editorial extensions
If this is right
- Pretraining a flow no longer requires HMC, rejection sampling, or decision diagrams for complex constraints; sampling from a Gaussian base distribution suffices.
- The number of QP solver calls during RL training drops by an order of magnitude on several benchmarks, and runtime on non-convex constraints improves by an order of magnitude over projection-based baselines.
- State-wise constraints without analytical action forms can be handled by learning the violation function from environment data, then training the same flow.
- The latent-space entropy term $\|\hat{a}\|^2/2$ is not cosmetic; ablations show removing it increases constraint violations and hurts learning.
- CV-Flow also combines with DDPG, not just SAC, and retains lower violation counts in that setting.
Reading between the lines
- A direct test of the method's core approximation: on a constraint with two disjoint feasible islands, the reverse-KL-trained flow may cover only one island, and then the dropped Jacobian term would misprice actions in the other; measuring $\log q$ versus $\log p$ over both islands would expose this.
- The same violation-signal training could be applied when the constraint is defined by a learned cost from high-dimensional observations, turning CV-Flows into a general safe-exploration prior; the paper only tests analytic and linearly approximated constraints.
- Since the flow is pretrained once and frozen, adapting it during RL as the state distribution shifts is a natural extension that the paper leaves open.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CV-Flows, a normalizing-flow mapping from a simple base distribution to the feasible action space of an action-constrained MDP. The flow is trained by minimizing reverse KL divergence between the flow distribution and a soft uniform target p(a|s) ∝ exp(-λ CV(a,s)) built from constraint-violation signals, avoiding the need to pre-sample feasible actions. The flow is then integrated with SAC: the latent action is stored in the replay buffer, the combined policy's log-probability is approximated as log μ(â|s)+||â||²/2+K(s) (Proposition 1), and the critic/policy updates use this approximation. Experiments on eight MuJoCo action-constrained tasks and four state-constrained tasks report an order-of-magnitude reduction in constraint violations (e.g., R+D: 0.01% vs 24.79-98.15% for baselines) with comparable or better returns, and faster training on non-convex constraints.
Significance. If the results hold, the paper makes a practical contribution: it removes the expensive pre-sampling of feasible actions that plagued flow-based ACRL methods, and it demonstrates that a flow can be trained directly from constraint-violation signals. The empirical violation reductions are consistent across tasks and are reported with standard deviations (Table 4), and the authors provide code. However, the core theoretical justification for the SAC integration, Proposition 1, rests on an unverified coverage assumption, and the return-comparison plots lack error bars, so the strength of the 'similar or better quality' claim is currently overstated.
major comments (3)
- [Section 3.3, Proposition 1, Eqs. (13)-(16)] The proof of Proposition 1 replaces the Jacobian log-determinant by assuming log p(a|s) ≈ log q(a|s), i.e., that the trained flow matches the uniform target. The training loss in Eq. (9) is a reverse-KL minimization; reverse KL is mode-seeking and does not penalize under-coverage of regions where p is large. For a non-convex feasible set such as the annulus constraints R+D and H+D in Table 2, a Gaussian-base flow could concentrate on a subset of the feasible region with little or no reverse-KL penalty, making log p(a|s) − log q(a|s) arbitrarily large on the uncovered part. Since Algorithm 2 uses a Gaussian base, and Section D.1 states that coverage cannot be exactly measured with a Gaussian base, the paper provides no evidence that the approximation holds in the non-convex cases. Consequently, the entropy term ||â||²/2 in Eqs. (17)-(18) is not a justified estimate of the combined policy's entropy. The authors should either prove coverage for their flow architecture, provide empirical coverage/recall measurements for the Gaussian-base flow (especially on R+D and H+D), or derive the SAC objective without this approximation.
- [Section 4, Figures 3 and 4] The paper states that each algorithm is trained with 10 random seeds, but the return learning curves in Figures 3 and 4 are shown as single lines without confidence bands, standard errors, or any statistical test. The abstract's claim of 'similar or better quality' and Section 4.1's claim of 'comparable results' are therefore not quantitatively supported for the return dimension of the central claim. Please report mean ± standard error or interquartile ranges across seeds, and ideally final-return summary statistics analogous to Table 4.
- [Section D.1, Eq. (21) and Figure 7] The recall/coverage evaluation presented in Figure 7 is performed with a uniform base distribution, and the text explicitly says this choice is because a Gaussian base does not allow exact measurement of coverage. However, the flow used in all RL experiments (Algorithm 2) is trained with a Gaussian base. Thus the property assumed in Proposition 1, that q(a|s) covers the feasible region, is never measured in the configuration actually deployed. Please provide an approximate coverage estimate for the Gaussian-base flow (e.g., by inverse-mapping rejection-sampled feasible points with a tolerance or using a thresholded latent-box criterion) and report it separately for the non-convex tasks.
minor comments (5)
- [Eq. (4) and Eq. (5)] The change-of-variables formula is misprinted: log q(a|s) should equal log ˆq(ˆa) − log|det J_fψ(ˆa;s)|, and the same minus sign is missing in the corresponding term inside the expectation in Eq. (5). The subsequent Eq. (9) is correct, so this appears to be a typesetting error, but it should be fixed for clarity.
- [Section 2.4 and Eq. (6)] The text describes the target p(a|s) as a uniform distribution over feasible actions with zero probability for infeasible actions, but Eq. (6) defines a strictly positive soft exponential density over the entire action space. Please clarify that this is a soft relaxation and state that the support is all of R^d; the 'zero probability' statement is only an approximation as λ → ∞.
- [Algorithm 2, line 7] The latent action is clipped to [−3,3] before being passed through the flow, but Proposition 1 and Eq. (16) assume an unclipped standard Gaussian base. This clipping is not reflected in the policy log-probability or the entropy term. Please quantify how often the clipping is active or modify the derivation to account for the bounded support.
- [Section 4.4 and Figure 7] The comparison between CV-Flow and standard flow is presented only as learning curves; please include the numerical accuracy, recall, and F1 values (with standard errors) implied by Figure 7, so the claimed superiority of CV-Flow on these metrics can be verified.
- [Section 4.1] Please state whether the 48-hour per-run cap was ever reached for any algorithm; if so, report the number of affected runs, since an inactive time cap can bias runtime comparisons and final-performance measurements.
Circularity Check
No load-bearing circularity: Proposition 1 is an explicit approximation and the reported violation reduction is an empirical result, not a fitted prediction.
full rationale
The central derivation chain is not circular. The flow model is trained by minimizing KL(q||p) with p(a|s) proportional to exp(-lambda * CV(a,s)) (Eq. 9), a target defined directly from the constraint functions and independent of the RL objective; the reported violation reduction is an empirical consequence of this training, measured during RL interaction against external baselines (DPre+, SPre+, NFW, FlowPG), not a fitted parameter renamed as a prediction. Proposition 1 (Eqs. 13-16) is an explicit approximation: it assumes the trained flow satisfies log q(a|s) approximately equals log p(a|s), i.e., that reverse-KL training has produced a uniform pushforward over the feasible set. That assumption is stated, not obtained by definition, and it is a potential correctness/coverage limitation, especially for non-convex constraints where reverse-KL can under-cover and where the deployed Gaussian-base flow's recall is not measured (the paper admits in Section D.1 that a uniform base is used for coverage measurement because a Gaussian base does not allow exact coverage measurement). This is a validity concern, not a circular reduction. Self-citations to the authors' FlowPG work provide background and a baseline (Sections 1 and 4), but they are not load-bearing for the main claim. No step reduces by construction to its own inputs.
Assumptions & free parameters
free parameters (3)
- lambda (penalty steepness) =
1000
- epsilon (equality constraint margin) =
not specified
- state sampling distribution pS(s) for flow pretraining =
uniform over bounded state components; Gaussian N(0,15) for HC+O
assumptions (5)
- standard math The flow f_psi is a bijective, differentiable mapping from a d-dimensional latent space to the action space, so the change-of-variables formula holds.
- domain assumption The reverse KL loss in Eq. (9) can be optimized by gradient descent despite the non-smooth max and absolute value terms in CV.
- ad hoc to paper The trained flow q(a|s) closely approximates the uniform target p(a|s) over feasible actions, so log p(a|s) is approximately log q(a|s) for states visited during RL.
- domain assumption For state-wise constraints, the next-state cost is linearly approximated as c_i(s_{t+1}) approximately equal to c_i(s_t) plus w_i(s_t)^T a_t.
- domain assumption The environment terminates when an infeasible action is executed, so a projection step is needed as a safety net.
Cite this review
Pith. "Pith review of Leveraging Constraint Violation Signals For Action-Constrained Reinforcement Learning." pith.science (2026). https://pith.science/paper/FOE2SUE2
@misc{pith2026250210431,
author = {Pith},
title = {Pith review of: Leveraging Constraint Violation Signals For Action-Constrained Reinforcement Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/FOE2SUE2}},
note = {Machine review of arXiv:2502.10431}
}
read the original abstract
In many RL applications, ensuring an agent's actions adhere to constraints is crucial for safety. Most previous methods in Action-Constrained Reinforcement Learning (ACRL) employ a projection layer after the policy network to correct the action. However projection-based methods suffer from issues like the zero gradient problem and higher runtime due to the usage of optimization solvers. Recently methods were proposed to train generative models to learn a differentiable mapping between latent variables and feasible actions to address this issue. However, generative models require training using samples from the constrained action space, which itself is challenging. To address such limitations, first, we define a target distribution for feasible actions based on constraint violation signals, and train normalizing flows by minimizing the KL divergence between an approximated distribution over feasible actions and the target. This eliminates the need to generate feasible action samples, greatly simplifying the flow model learning. Second, we integrate the learned flow model with existing deep RL methods, which restrict it to exploring only the feasible action space. Third, we extend our approach beyond ACRL to handle state-wise constraints by learning the constraint violation signal from the environment. Empirically, our approach has significantly fewer constraint violations while achieving similar or better quality in several control tasks than previous best methods.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
Adawadkar, A. M. K.; and Kulkarni, N. 2022. Cyber-security and reinforcement learning — A brief survey. Engineering Applications of Artificial Intelligence, 114: 105116
work page 2022
-
[2]
Amos, B.; and Kolter, J. Z. 2017. Optnet: Differentiable optimization as a layer in neural networks. In International Conference on Machine Learning, 136--145
work page 2017
-
[3]
Bhatia, A.; Varakantham, P.; and Kumar, A. 2019. Resource Constrained Deep Reinforcement Learning. In Proceedings of the Twenty-Ninth International Conference on Automated Planning and Scheduling, 610--620
work page 2019
-
[4]
Brahmanage, J. C.; Ling, J.; and Kumar, A. 2023. FlowPG: Action-constrained Policy Gradient with Normalizing Flows. In Thirty-seventh Conference on Neural Information Processing Systems
work page 2023
-
[5]
Brubaker, M.; Salzmann, M.; and Urtasun, R. 2012. A Family of MCMC Methods on Implicitly Defined Manifolds. In International Conference on Artificial Intelligence and Statistics, 161--172
work page 2012
-
[6]
H.; Sinha, A.; and Varakantham, P
Changyu, C.; Karunasena, R.; Nguyen, T. H.; Sinha, A.; and Varakantham, P. 2023. Generative Modelling of Stochastic Actions with Arbitrary Constraints in Reinforcement Learning . In Thirty-seventh Conference on Neural Information Processing Systems
work page 2023
-
[7]
Dalal, G.; Dvijotham, K.; Vecerik, M.; Hester, T.; Paduraru, C.; and Tassa, Y. 2018. Safe exploration in continuous action spaces. arXiv preprint arXiv:1801.08757
arXiv 2018
-
[8]
Dinh, L.; Sohl-Dickstein, J.; and Bengio, S. 2016. Density estimation using real nvp. In International Conference on Learning Representations
work page 2016
Show all 30 references
-
[9]
Fujimoto, S.; Meger, D.; and Precup, D. 2019. Off-Policy Deep Reinforcement Learning without Exploration
2019
-
[10]
Haarnoja, T.; Zhou, A.; Abbeel, P.; and Levine, S. 2018. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, 1861--1870. PMLR
2018
-
[11]
Kasaura, K.; Miura, S.; Kozuno, T.; Yonetani, R.; Hoshino, K.; and Hosoe, Y. 2023. Benchmarking Actor-Critic Deep Reinforcement Learning Algorithms for Robotics Control With Action Constraints. IEEE Robotics and Automation Letters, 8(8): 4449--4456
2023
-
[12]
Khoury, J.; and Nassar, M. 2020. A Hybrid Game Theory and Reinforcement Learning Approach for Cyber - Physical Systems Security . In NOMS 2020 - 2020 IEEE / IFIP Network Operations and Management Symposium , 1--9. ISSN: 2374-9709
2020
-
[13]
P.; and Welling, M
Kingma, D. P.; and Welling, M. 2022. Auto- Encoding Variational Bayes . ArXiv:1312.6114 [cs, stat]
2022 arXiv
-
[14]
P.; Hunt, J
Lillicrap, T. P.; Hunt, J. J.; Pritzel, A.; Heess, N.; Erez, T.; Tassa, Y.; Silver, D.; and Wierstra, D. 2016. Continuous control with deep reinforcement learning. In International Conference on Learning Representations
2016
-
[15]
Lin, J.-L.; Hung, W.; Yang, S.-H.; Hsieh, P.-C.; and Liu, X. 2021. Escaping from zero gradient: Revisiting action-constrained reinforcement learning via Frank-Wolfe policy optimization. In Uncertainty in Artificial Intelligence, 397--407
2021
-
[16]
Malik, S.; Anwar, U.; Aghasi, A.; and Ahmed, A. 2021. Inverse Constrained Reinforcement Learning . In Proceedings of the 38th International Conference on Machine Learning , 7390--7399. PMLR. ISSN: 2640-3498
2021
-
[17]
Mazoure, B.; Doan, T.; Durand, A.; Pineau, J.; and Hjelm, R. D. 2020. Leveraging exploration in off-policy algorithms via normalizing flows. In Conference on Robot Learning, 430--444. PMLR
2020
-
[18]
A.; Veness, J.; Bellemare, M
Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A. A.; Veness, J.; Bellemare, M. G.; Graves, A.; Riedmiller, M.; Fidjeland, A. K.; Ostrovski, G.; et al. 2015. Human-level control through deep reinforcement learning. nature, 518(7540): 529--533
2015
-
[19]
Nielsen, D.; Jaini, P.; Hoogeboom, E.; Winther, O.; and Welling, M. 2020. Survae flows: Surjections to bridge the gap between vaes and flows. Advances in Neural Information Processing Systems, 33: 12685--12696
2020
-
[20]
J.; Mohamed, S.; and Lakshminarayanan, B
Papamakarios, G.; Nalisnick, E.; Rezende, D. J.; Mohamed, S.; and Lakshminarayanan, B. 2021. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22(57): 1--64
2021
-
[21]
Pham, T.-H.; De Magistris, G.; and Tachibana, R. 2018. Optlayer-practical constrained optimization for deep reinforcement learning in the real world. In International Conference on Robotics and Automation, 6236--6243
2018
-
[22]
Rezende, D.; and Mohamed, S. 2015. Variational inference with normalizing flows. In International conference on machine learning, 1530--1538. PMLR
2015
-
[23]
E.; Ibarz, J.; Finn, C.; and Goldberg, K
Thananjeyan, B.; Balakrishna, A.; Nair, S.; Luo, M.; Srinivasan, K.; Hwang, M.; Gonzalez, J. E.; Ibarz, J.; Finn, C.; and Goldberg, K. 2021. Recovery rl: Safe reinforcement learning with learned recovery zones. IEEE Robotics and Automation Letters, 6(3): 4915--4922
2021
-
[24]
Todorov, E.; Erez, T.; and Tassa, Y. 2012. MuJoCo : A physics engine for model-based control. In 2012 IEEE / RSJ International Conference on Intelligent Robots and Systems , 5026--5033. ISSN: 2153-0866
2012
-
[25]
Van Hasselt, H.; Guez, A.; and Silver, D. 2016. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30
2016
-
[26]
N.; Smofsky, A.; and Bose, A
Ward, P. N.; Smofsky, A.; and Bose, A. J. 2019. Improving exploration in soft-actor-critic with normalizing flows policies. arXiv preprint arXiv:1906.02771
2019 arXiv
-
[27]
Zhang, J.; Zhang, C.; Wang, W.; and Jing, B.-Y. 2023. APAC: Authorized Probability-controlled Actor-Critic For Offline Reinforcement Learning
2023
-
[28]
Zhao, W.; He, T.; Chen, R.; Wei, T.; and Liu, C. 2023. State-wise Safe Reinforcement Learning : A Survey . ArXiv:2302.03122 [cs]
2023 arXiv
-
[29]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...
-
[30]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.