{"id":"90117550-0db4-446b-a369-4f69f08aadc6","arxiv_id":"2502.10431","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new method trains normalizing flows using constraint violation signals to map latent actions to feasible actions, reducing constraint violations by over 10x in several RL control benchmarks.","lead":"This paper trains a normalizing flow (a type of generative model) to output only actions that satisfy safety constraints, using constraint violation signals instead of pre-generated feasible action samples. Combined with the SAC reinforcement learning algorithm, it reaches similar rewards while cutting constraint violations by over 10x in several control benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Proposition 1 approximation is unsupported: reverse-KL training permits under-coverage of the feasible region, and the deployed Gaussian-base flow's coverage is never measured.","rationale":"The reader's conditional verdict already identifies the flow-target closeness assumption as the weak point; my stress test agrees and sharpens it. Reverse-KL training is mode-seeking, so the trained flow may cover only part of a non-convex feasible region while still achieving near-zero violation rate by staying safely inside one component. The paper's coverage evaluation in Figure 7 is run with a uniform base distribution, not the Gaussian base used in SAC, so it does not establish that the deployed flow satisfies log q ≈ log p on the states and actions encountered during RL. This is a genuine correctness risk for the theoretical framing of Proposition 1. It is not necessarily fatal to the empirical method: the released code, consistent violation reductions across benchmarks, and the ablation showing the entropy term's benefit give the practical claims independent support. The proposed concrete test would determine whether the approximation is actually load-bearing for performance or merely a flawed derivation with benign numerical consequences. Therefore the conditional verdict should stand rather than move to acceptance or rejection.","tokens_in":16474,"tokens_out":16484,"duration_ms":183470,"concrete_test":"In the released code, for the Gaussian-base CV-Flow used in Algorithm 2, compute empirical recall on R+D and H+D by: (i) sampling feasible actions from the annulus/shell via rejection sampling; (ii) mapping them back through the inverse flow f^{-1}; and (iii) checking whether the inverse lands within the 3σ latent box used for clipping, as in Eq. (21). If recall is below about 0.9 while accuracy is near 1, the flow is under-covering and Proposition 1's premise fails. Additionally, rerun R+D with the exact Jacobian-corrected log-probability log π(a|s)=log μ(â|s)-log|det J_f(â;s)|+const in the SAC objective and compare returns and violation counts; a material change would show that the approximation is load-bearing for the reported results.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is Proposition 1's replacement of log|det J| with a state-dependent constant via the assumption log q(a|s) ≈ log p(a|s). The training objective in Eq. (9) is a reverse-KL minimization. Reverse KL is mode-seeking: minimizing KL(q||p) with p uniform on the feasible set forces q to have negligible mass where p is zero (outside C(s)), but it does not force q to cover all of C(s). A flow that collapses onto one connected component of a non-convex or disconnected feasible region incurs no reverse-KL penalty for the uncovered components. Thus q can be very different from the uniform target p, and the approximation in Eqs. (14)-(16) can fail by an arbitrarily large amount on the missing region. The paper's own Figure 7 measures recall, i.e., coverage, but explicitly uses a uniform base distribution 'because a Gaussian base distribution ... does not allow exact measurement of coverage' (Section D.1). The Gaussian-base flow actually deployed in SAC (Algorithm 2) is never checked for coverage. If its recall on the non-convex R+D and H+D tasks is materially below 1, then the entropy term α||â||²/2 in Eqs. (17)-(18) is not the entropy of the combined policy, and the theoretical argument for the SAC+CVFlow objective is unsupported in exactly the non-convex cases emphasized in the paper.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CV-Flows, a normalizing-flow mapping from a simple base distribution to the feasible action space of an action-constrained MDP. The flow is trained by minimizing reverse KL divergence between the flow distribution and a soft uniform target p(a|s) ∝ exp(-λ CV(a,s)) built from constraint-violation signals, avoiding the need to pre-sample feasible actions. The flow is then integrated with SAC: the latent action is stored in the replay buffer, the combined policy's log-probability is approximated as log μ(â|s)+||â||²/2+K(s) (Proposition 1), and the critic/policy updates use this approximation. Experiments on eight MuJoCo action-constrained tasks and four state-constrained tasks report an order-of-magnitude reduction in constraint violations (e.g., R+D: 0.01% vs 24.79-98.15% for baselines) with comparable or better returns, and faster training on non-convex constraints.","tokens_in":16831,"tokens_out":8380,"duration_ms":75424,"significance":"If the results hold, the paper makes a practical contribution: it removes the expensive pre-sampling of feasible actions that plagued flow-based ACRL methods, and it demonstrates that a flow can be trained directly from constraint-violation signals. The empirical violation reductions are consistent across tasks and are reported with standard deviations (Table 4), and the authors provide code. However, the core theoretical justification for the SAC integration, Proposition 1, rests on an unverified coverage assumption, and the return-comparison plots lack error bars, so the strength of the 'similar or better quality' claim is currently overstated.","major_comments":[{"comment":"The proof of Proposition 1 replaces the Jacobian log-determinant by assuming log p(a|s) ≈ log q(a|s), i.e., that the trained flow matches the uniform target. The training loss in Eq. (9) is a reverse-KL minimization; reverse KL is mode-seeking and does not penalize under-coverage of regions where p is large. For a non-convex feasible set such as the annulus constraints R+D and H+D in Table 2, a Gaussian-base flow could concentrate on a subset of the feasible region with little or no reverse-KL penalty, making log p(a|s) − log q(a|s) arbitrarily large on the uncovered part. Since Algorithm 2 uses a Gaussian base, and Section D.1 states that coverage cannot be exactly measured with a Gaussian base, the paper provides no evidence that the approximation holds in the non-convex cases. Consequently, the entropy term ||â||²/2 in Eqs. (17)-(18) is not a justified estimate of the combined policy's entropy. The authors should either prove coverage for their flow architecture, provide empirical coverage/recall measurements for the Gaussian-base flow (especially on R+D and H+D), or derive the SAC objective without this approximation.","section":"Section 3.3, Proposition 1, Eqs. (13)-(16)"},{"comment":"The paper states that each algorithm is trained with 10 random seeds, but the return learning curves in Figures 3 and 4 are shown as single lines without confidence bands, standard errors, or any statistical test. The abstract's claim of 'similar or better quality' and Section 4.1's claim of 'comparable results' are therefore not quantitatively supported for the return dimension of the central claim. Please report mean ± standard error or interquartile ranges across seeds, and ideally final-return summary statistics analogous to Table 4.","section":"Section 4, Figures 3 and 4"},{"comment":"The recall/coverage evaluation presented in Figure 7 is performed with a uniform base distribution, and the text explicitly says this choice is because a Gaussian base does not allow exact measurement of coverage. However, the flow used in all RL experiments (Algorithm 2) is trained with a Gaussian base. Thus the property assumed in Proposition 1, that q(a|s) covers the feasible region, is never measured in the configuration actually deployed. Please provide an approximate coverage estimate for the Gaussian-base flow (e.g., by inverse-mapping rejection-sampled feasible points with a tolerance or using a thresholded latent-box criterion) and report it separately for the non-convex tasks.","section":"Section D.1, Eq. (21) and Figure 7"}],"minor_comments":[{"comment":"The change-of-variables formula is misprinted: log q(a|s) should equal log ˆq(ˆa) − log|det J_fψ(ˆa;s)|, and the same minus sign is missing in the corresponding term inside the expectation in Eq. (5). The subsequent Eq. (9) is correct, so this appears to be a typesetting error, but it should be fixed for clarity.","section":"Eq. (4) and Eq. (5)"},{"comment":"The text describes the target p(a|s) as a uniform distribution over feasible actions with zero probability for infeasible actions, but Eq. (6) defines a strictly positive soft exponential density over the entire action space. Please clarify that this is a soft relaxation and state that the support is all of R^d; the 'zero probability' statement is only an approximation as λ → ∞.","section":"Section 2.4 and Eq. (6)"},{"comment":"The latent action is clipped to [−3,3] before being passed through the flow, but Proposition 1 and Eq. (16) assume an unclipped standard Gaussian base. This clipping is not reflected in the policy log-probability or the entropy term. Please quantify how often the clipping is active or modify the derivation to account for the bounded support.","section":"Algorithm 2, line 7"},{"comment":"The comparison between CV-Flow and standard flow is presented only as learning curves; please include the numerical accuracy, recall, and F1 values (with standard errors) implied by Figure 7, so the claimed superiority of CV-Flow on these metrics can be verified.","section":"Section 4.4 and Figure 7"},{"comment":"Please state whether the 48-hour per-run cap was ever reached for any algorithm; if so, report the number of affected runs, since an inactive time cap can bias runtime comparisons and final-performance measurements.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about Proposition 1 is the right one to focus on; I would not accept the paper without either a proof of the coverage property or direct empirical measurement of coverage for the Gaussian-base flow. The empirical violation numbers are strong and the method is potentially valuable, so I see this as a major-revision issue rather than a rejection. Also, the paper draws heavily on the authors' prior FlowPG work; the novelty of CV-Flows relative to FlowPG is clear in principle, but the write-up could make that delineation even more explicit."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper is worth engaging with. The core idea—train a normalizing flow over feasible actions by minimizing reverse KL against a target defined by the constraint violation signal, avoiding feasible-sample generation entirely—is new and genuinely practical. The SAC integration is also well designed: store the latent action in the replay buffer, add the ||â||^2/2 surrogate entropy, and you skip backprop through the flow. The empirical evidence is consistent and substantial: across eight MuJoCo tasks and four state-constrained tasks, CV-Flow produces far fewer constraint violations than DPre+, SPre+, NFW, and FlowPG, with matching or better returns, and a large runtime advantage on the non-convex R+D and H+D tasks. Ten seeds, code released, standard deviations reported. That is real work.\n\nThe soft spot is Proposition 1, the load-bearing step of the SAC derivation. It replaces log p(a|s) with log q(a|s) because q was trained to approximate the uniform target p. Reverse KL is mode-seeking: minimizing KL(q||p) forces q to zero where p is zero, but does not force q to cover all of C(s). For a non-convex or disconnected feasible region, q can collapse onto one component without incurring any loss, and the approximation in Eqs. (14)–(16) can be off by an arbitrarily large amount over the missed region. The paper's own recall evaluation in Figure 7 uses a uniform base distribution because 'a Gaussian base distribution does not allow exact measurement of coverage.' But the flow actually used in SAC uses a Gaussian base. So the coverage of the deployed flow is never measured, and the entropy term in Eqs. (17)–(18) is not verified to be the true entropy on the non-convex tasks. That is a genuine hole, and it is most serious exactly where the paper claims its biggest advantage.\n\nMinor issues: the return curves in Figures 3 and 4 have no error bars, and the single λ=1000 is never varied. These are addressable. Also note that Algorithm 2 includes a projection step after the flow, so the violation numbers are really a measure of how often the flow leaves the feasible set, not end-to-end safety.\n\nNone of this sinks the paper. The empirical case is strong enough that the method works whether or not Proposition 1 is airtight, and the training objective is novel on its own. A serious referee should ask for a coverage check of the actual Gaussian-base flow, or at least a discussion of why mode-seeking is benign in these benchmarks. That is a standard revision request, not a rejection.\n\nBottom line: send it to review. It is a real contribution with a fixable theoretical gap.","headline":"CV-Flows is a practical and novel flow-training idea, but its SAC entropy approximation rests on an unverified coverage assumption that is weakest precisely on the non-convex tasks where the method claims its biggest win.","tokens_in":17255,"tokens_out":3552,"would_cite":true,"duration_ms":30975,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Constraint-violation signals can replace feasible action samples for training RL action maps.","keywords":["action-constrained reinforcement learning","normalizing flows","constraint violation signals","soft actor-critic","feasible action space","safe exploration","state-wise constraints","maximum entropy reinforcement learning"],"falsifier":"Take a constraint with two disjoint feasible regions, train the CV-Flow, then compare the policy log-probability computed with and without the Jacobian term over a grid of feasible actions; if the approximation error is large, or if the reverse KL between the flow and the uniform target remains high, Proposition 1's step fails.","tokens_in":16312,"feed_emoji":"🛡️","tokens_out":7685,"duration_ms":63711,"temperature":0.7,"pith_summary":"The paper tries to show that an RL agent can learn to produce only feasible actions without ever being given a dataset of feasible actions. It trains a normalizing flow by minimizing the KL divergence between the flow's output distribution and a target density built from the constraint-violation signal, $e^{-\\lambda\\,CV(a,s)}$, so the flow learns a smooth, invertible map from a Gaussian latent space into the feasible action space. Integrating this map with soft actor-critic lets the agent explore only feasible actions while storing latent actions in the replay buffer, avoiding backpropagation through the flow. On MuJoCo and state-constrained control benchmarks the resulting agent posts far fewer constraint violations than projection-based and prior flow-based baselines, often an order of magnitude fewer, with comparable or better returns.","feed_headline":"Constraint-violation signals replace feasible action samples in RL","feed_subtitle":"Normalizing flows trained on violation magnitude cut unsafe actions by ~10x in continuous control.","key_machinery":"The load-bearing object is the CV-Flow: a conditional normalizing flow trained by reverse KL against the target $p(a|s) \\propto e^{-\\lambda\\,CV(a,s)}$, where $CV$ is the sum of positive parts of inequality violations and equality violations. Its training loss $J^f(\\psi) = \\mathbb{E}[\\lambda\\,CV(f_\\psi(\\hat{a},s),s) - \\log|\\det J_{f_\\psi}|]$ avoids sampling feasible actions entirely. The second piece is Proposition 1, which uses the trained flow's closeness to the uniform feasible target to replace $\\log \\pi(a|s)$ with $\\log \\mu_\\phi(\\hat{a}|s) + \\|\\hat{a}\\|^2/2 + K(s)$, so SAC's entropy regularization is computed in latent space and the critic never backpropagates through the flow.","core_discovery":"The central discovery is that the hard part of flow-based action-constrained RL—collecting feasible action samples from a constrained space—is unnecessary. The authors define the target $p(a|s)$ as a uniform distribution over feasible actions, realized as $e^{-\\lambda\\,CV(a,s)}$ normalized by a state-dependent constant, and train the flow by reverse KL divergence, which reduces to maximizing $\\lambda\\,CV(f_\\psi(\\hat{a},s),s)$ minus the log-determinant of the flow Jacobian. Because the trained flow approximates the uniform target, the log-probability of the combined SAC policy can be written in latent coordinates as $\\log \\mu_\\phi(\\hat{a}|s) + \\|\\hat{a}\\|^2/2$ plus a state-only constant, dropping the Jacobian term. This yields an SAC variant whose actor and critic train on latent actions, with a projection step kept as a safety net. The same CV signal works for state-wise constraints, where the violation function is learned from environment interaction via a linear cost model.","pith_inferences":["A direct test of the method's core approximation: on a constraint with two disjoint feasible islands, the reverse-KL-trained flow may cover only one island, and then the dropped Jacobian term would misprice actions in the other; measuring $\\log q$ versus $\\log p$ over both islands would expose this.","The same violation-signal training could be applied when the constraint is defined by a learned cost from high-dimensional observations, turning CV-Flows into a general safe-exploration prior; the paper only tests analytic and linearly approximated constraints.","Since the flow is pretrained once and frozen, adapting it during RL as the state distribution shifts is a natural extension that the paper leaves open."],"forward_implications":["Pretraining a flow no longer requires HMC, rejection sampling, or decision diagrams for complex constraints; sampling from a Gaussian base distribution suffices.","The number of QP solver calls during RL training drops by an order of magnitude on several benchmarks, and runtime on non-convex constraints improves by an order of magnitude over projection-based baselines.","State-wise constraints without analytical action forms can be handled by learning the violation function from environment data, then training the same flow.","The latent-space entropy term $\\|\\hat{a}\\|^2/2$ is not cosmetic; ablations show removing it increases constraint violations and hurts learning.","CV-Flow also combines with DDPG, not just SAC, and retains lower violation counts in that setting."],"supporting_citations":[{"why":"Establishes the flow-mapping approach for ACRL and the need for feasible action samples (HMC/decision diagrams) that CV-Flows removes; also provides the FlowPG baseline.","marker":"Brahmanage, Ling, and Kumar 2023"},{"why":"Supplies the eight constrained MuJoCo benchmarks and hyperparameter settings used for evaluation.","marker":"Kasaura et al. 2023"},{"why":"Defines the zero-gradient problem in projection-based ACRL and provides the Frank-Wolfe baseline NFW.","marker":"Lin et al. 2021"},{"why":"Provides the SAC maximum-entropy objective that the paper modifies to latent-action form.","marker":"Haarnoja et al. 2018"},{"why":"Provides the real NVP normalizing flow architecture and change-of-variables log-density formula used by the flow model.","marker":"Dinh, Sohl-Dickstein, and Bengio 2016"},{"why":"Supplies the linear approximation of state-wise constraints as action constraints, reused for the state-constrained tasks.","marker":"Dalal et al. 2018"}],"fun_headline_variants":["Constraint violations, not samples, guide safe RL","Violation signals replace feasible action sampling in RL","Flow-based RL cuts violations 10x using CV signals","Learn RL actions from constraint violations, no samples","Action-constrained RL without collecting feasible actions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes the trained flow's distribution over feasible actions is close enough to the uniform feasible target that the log-density of the combined policy can drop the flow's Jacobian term, which fails if the flow misses part of a disconnected or multi-modal feasible region.","fun_headline_variants_meta":{"raw":{"variants":["Constraint violations, not samples, guide safe RL","Violation signals replace feasible action sampling in RL","Flow-based RL cuts violations 10x using CV signals","Learn RL actions from constraint violations, no samples","Action-constrained RL without collecting feasible actions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000236,"raw_usage":{"total_tokens":1519,"prompt_tokens":973,"completion_tokens":546,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":474}},"tokens_in":589,"tokens_out":546,"duration_ms":5744,"temperature":1.0,"reasoning_tokens":474,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T18:52:25.521852+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a constraint with two disjoint feasible regions, train the CV-Flow, then compare the policy log-probability computed with and without the Jacobian term over a grid of feasible actions; if the approximation error is large, or if the reverse KL between the flow and the uniform target remains high, Proposition 1's step fails.","supporting_citations":[{"cited_title":"C.; Ling, J.; and Kumar, A","cited_arxiv_id":null,"evidence_quote":"Establishes the flow-mapping approach for ACRL and the need for feasible action samples (HMC/decision diagrams) that CV-Flows removes; also provides the FlowPG baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the eight constrained MuJoCo benchmarks and hyperparameter settings used for evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the zero-gradient problem in projection-based ACRL and provides the Frank-Wolfe baseline NFW."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the real NVP normalizing flow architecture and change-of-variables log-density formula used by the flow model."}],"review_version":1}