{"id":"9a4d7524-872d-430d-8bd4-ef56e5fee37b","arxiv_id":"2412.04327","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Decoupling feasibility from objective optimization by training the RL policy over latent actions that map to feasible actions improves sample efficiency and constraint satisfaction in continuous constrained RL.","lead":"Action mapping trains a separate feasibility policy that maps a latent code to constraint-satisfying actions, then trains the main reinforcement learning policy in that latent space. In two simulated environments, this approach learns faster and violates constraints less often than projection, resampling, and Lagrangian baselines, especially with approximate feasibility models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Main comparison is confounded: the projection baseline used a stricter feasibility model than AM-SAC in the path-planning experiments, so the claimed advantage over projection is not controlled.","rationale":"I identify the baseline fairness issue as the most load-bearing because the manuscript's contribution is an empirical performance claim, and the one experiment that supports the 'especially with imperfect feasibility models' part of the abstract is the path-planning comparison. The paper's own appendix reveals a deliberate asymmetry in the feasibility model given to projection. This is not a circularity or an external-consensus dispute; it is an internal experimental-design confound. The reader's stated weakest assumption (KDE-based feasibility-policy coverage) is a legitimate theoretical risk, but there is at least direct visualization evidence (Fig. 5) and the method's behavior in the robot-arm case. The projection confound, by contrast, is documented in the paper and directly explains the main reported advantage. A single controlled rerun can settle it. Because the existing verdict is CONDITIONAL and this concern is addressable but unresolved, I would keep CONDITIONAL and require the controlled comparison before acceptance.","tokens_in":16247,"tokens_out":6531,"duration_ms":63552,"concrete_test":"Re-run the path-planning tasks with one shared feasibility model for both AM-SAC and SAC+Projection: same number S of spline-evaluation points (S=64), same clearance margin, same curvature bound, and no environment-specific inflation for projection. Also run the reverse control, AM-SAC with the conservative margins currently given to projection, and report return, task-solved rate, and violation curves over 25M steps with at least 5 seeds. If SAC+Projection with the shared model matches or beats AM-SAC, the central claim as stated is not supported; if AM-SAC still wins under the identical model, the confound is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline empirical claim—that action mapping significantly improves training performance and constraint satisfaction compared with projection, especially with imperfect feasibility models—is carried primarily by the path-planning experiment (Section 6.2, Figs. 4c/4d). That comparison is not between equal methods. Appendix C.1 states that in the path planning environment, the projection baseline required 'a higher distance to obstacles and a tighter curvature bound had to be enforced.' Thus SAC+Projection was given a feasibility model with extra conservative margins that AM-SAC did not use. A more conservative feasibility model changes the feasible action set, reducing the agent's ability to collect targets and increasing unavoidable constraint violations. The observed gap in Fig. 4 may therefore be an artifact of the unequal treatment, not a property of action mapping. The appendix's own sensitivity analysis (Fig. A.3) shows AM-SAC performance is sensitive to feasibility-model accuracy, making a controlled comparison essential. The internal claim that a perfect feasibility policy transforms the SCMDP into an unconstrained MDP (Section 4) is also only approximate—the paper concedes the learned policy is not surjective and can still generate infeasible actions—but the decisive missing evidence is a same-model comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes 'action mapping,' a training strategy for constrained reinforcement learning in continuous action spaces. In a pretraining phase (Section 4.1, Algorithm 2), a feasibility policy pi_f is trained, via a KDE-based approximation of the Jensen-Shannon divergence (Eq. 16), to map a latent space Z onto the state-dependent feasible action set A+_s using a given feasibility model g. In a second phase (Section 4.2, Algorithm 1), an objective policy pi_o is trained with PPO or SAC entirely in the latent space, so that the composed policy pi = pi_f composed with pi_o selects only feasible actions; the authors argue that this 'effectively transforms' the SCMDP into an unconstrained MDP. Experiments compare AM-PPO and AM-SAC against plain PPO/SAC, Lagrangian PPO/SAC, and feasibility-model-based replacement, resampling, and projection baselines in two environments: a 7-DOF robot arm pose task with a perfect feasibility model and a non-holonomic spline-based path-planning task with an approximate feasibility model. The paper reports that action mapping trains faster, reaches higher return, and incurs fewer constraint violations, with the largest advantages in the approximate-model setting, and that it allows a single latent Gaussian to express multi-modal action distributions (Fig. 5).","tokens_in":16489,"tokens_out":15794,"duration_ms":148642,"significance":"If the reported results withstand scrutiny, the contribution is useful: it offers a simple recipe (pretrain a feasible-action generator, then run an off-the-shelf SAC/PPO in its latent space), and the compute-overhead measurements (Appendix F) make a concrete case that the approach is cheaper than projection-based integration of feasibility models. The paper is honest about its limitations: Section 7 explicitly concedes that the learned feasibility policy is not surjective, can still emit infeasible actions, and only relaxes rather than removes the constraints, and Appendix G provides a sensitivity analysis showing that performance depends on feasibility-model fidelity. The central empirical claim, however, rests on a confounded comparison in the approximate-model environment (see major comments), and no code is released, so the reported gains are not yet independently reproducible; the significance of the paper is therefore conditional on a controlled re-evaluation of the projection baseline.","major_comments":[{"comment":"The claim in Section 6.2 that action mapping outperforms action projection with an approximate feasibility model is carried by a comparison that is not controlled. Appendix C.1 states that in the path-planning environment 'a higher distance to obstacles and a tighter curvature bound had to be enforced' for the projection baseline, so SAC+Projection was effectively given a smaller feasible action set than AM-SAC was. Because Appendix G (Fig. A.3) shows that AM-SAC performance depends sensitively on the feasibility model's definition of the feasible set, the gap in Figs. 4c and 4d cannot be attributed to the action-mapping mechanism rather than to the unequal treatment of the two methods. Please either rerun SAC+Projection with exactly the same feasibility model as AM-SAC (same obstacle margins, curvature bound, and number of spline evaluation points) or report both model settings for both methods; without this, the Section 6.2 conclusion and the abstract's 'especially with imperfect feasibility models' claim are not supported.","section":"Section 6.2, Appendix C.1, Figs. 4c-4d"},{"comment":"The abstract and Section 4 present the 'effectively transformed into an unconstrained MDP' statement as the paper's central insight, but Section 7 concedes that the learned feasibility policy is not surjective, can still generate infeasible actions, and therefore 'does not remove all constraints from the SCMDP, but still significantly relaxes the constraints'; Section 4.2 similarly notes that pi_f is 'neither perfect nor strictly bijective.' The empirical contributions do not depend on the unqualified idealization, and the Section 7 caveat is appropriate, but the framing should be consistent: please qualify the transformation claim at the point of assertion (for example, 'approximately transforms' or 'relaxes') rather than leaving the unqualified version in the abstract and Section 4.","section":"Section 4 vs. Section 7"},{"comment":"The method's load-bearing premise is that the pretrained feasibility policy provides near-uniform coverage of the feasible set, including disconnected feasible sets, so that the objective policy can in principle reach every feasible action. The paper offers a qualitative illustration for a single state (Fig. 5a) and reports the empirical success of AM-SAC, but it provides no quantitative evaluation of the feasibility policy's recall: Appendix A advises monitoring precision and the average distance between feasible actions, yet no such numbers are reported for either environment. Since Eq. (16) is an approximate KDE-based gradient with fixed bandwidth sigma = 0.1, proposal noise sigma-prime, and N = 1024 samples, the paper should report the feasibility policy's precision and recall over a representative set of partial states, ideally together with an oracle-sampler upper bound (e.g., AM-SAC with exact rejection sampling over a finely discretized feasible set) to quantify the loss from imperfect coverage.","section":"Section 4.1, Eq. (16), Appendix A"}],"minor_comments":[{"comment":"The abstract and Section 6 use 'significantly' for the reported improvements, but each configuration uses only three seeds with min-max bands and no statistical tests; please either add more seeds with standard errors or a statistical comparison, or reword to 'consistently.' The description of a distinctive 'second jump' for AM-SAC (Section 6.2) is likewise drawn from a three-seed median and should be labeled accordingly.","section":"Abstract and Section 6.2"},{"comment":"Figure 4's caption lists panels (c) 'Path planning - Return' and (d) 'Path planning - Constraint violation,' but the manuscript rendering shows an additional 'Task solved' axis in the path-planning row; please ensure every panel is labeled exactly as described in the caption and referenced in the text.","section":"Figure 4"},{"comment":"Appendix A states that 'Theile et al. (2024) propose to sample multiple actions per support point of the KDE, which our experiments showed to be unnecessary,' but no experiment supporting this claim is reported; either add a brief ablation or remove the assertion.","section":"Appendix A, Algorithm 2"},{"comment":"No code is released, and the projection baseline's optimization procedure is only described by reference to an example method (Appendix C.1); given that the paper's claims hinge on the exact feasibility-model and projection definitions, a reproducibility package or precise implementation details would substantially strengthen the paper.","section":"Appendix C.1 and reproducibility"},{"comment":"In Table A.2, the cost discount factor for Lagrangian PPO is set to gamma_C = 0 with the note 'Not needed,' which makes the safety critic (Eq. 17) myopic; the choice is defensible because episodes terminate upon violation, but a one-sentence justification would remove a possible fairness concern relative to Lagrangian SAC (gamma_C = 0.9).","section":"Appendix C.2, Table A.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is an empirical methods contribution that is within the journal's scope, but its flagship comparison (AM-SAC vs. SAC+Projection with an approximate feasibility model) is confounded, and the required fix (a same-model rerun of the projection baseline) is well within the authors' reach. The novelty relative to Theile et al. (2024) is moderate: the feasibility-policy training is carried over largely unchanged, with the new contribution being the objective-policy formulation and the two case studies; that is acceptable provided the empirical claims are made solid. The three-seed evidence base and the unqualified 'significantly' and 'unconstrained MDP' phrasing should be tightened before publication. The unequal feasibility-model margins documented in Appendix C.1 must be disclosed in the main text and controlled in the experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is a solid extension of Theile et al.'s feasibility-policy idea. The new piece is the objective-policy training in latent space — AM-PPO and AM-SAC — plus a two-environment comparison. That part is worth reading. The writing is clear, the method section is concrete, and the authors are honest about the limits: no strict safety guarantee, feasibility model may be too restrictive, learned mapping is not surjective.\n\nWhat is genuinely new is showing that a single Gaussian in latent space can induce a bi-modal or multi-modal distribution over feasible actions after passing through the feasibility policy. That is a real insight and the figures make it visible.\n\nThe soft spot is the empirical claim that action mapping beats projection, especially with imperfect feasibility models. In the path-planning environment, the projection baseline was given a stricter feasibility model — 'a higher distance to obstacles and a tighter curvature bound' — than the AM-SAC agent. That is in Appendix C.1. So the comparison in Figures 4c/4d is not controlled. The better results could come from handicapping projection. This is not a tiny detail; it is the main support for the 'especially with imperfect feasibility models' claim. The robot-arm experiment uses a perfect model and does not have this confound, and there action mapping's advantage over projection is modest. But the imperfect-model case is where the paper promises the most, and that is the case with the unequal baseline.\n\nThere are also the usual thinness issues: 3 seeds, no released code. The sensitivity analysis in Appendix G is a good addition and shows the method degrades as the feasibility model gets coarser, which is useful, but it also means the exact operating point matters a lot.\n\nOn the 'unconstrained MDP' framing: the paper itself concedes it is only approximate. That is fine as a motivating viewpoint, but the load-bearing assumption is that the feasibility policy covers enough of the feasible set. The KDE bandwidth is a free parameter and there is no direct measure of coverage. The multi-modal visualization helps, but it is not a formal argument.\n\nWho is this for? People working on action masking, action representations, or safety layers for continuous-action RL. They will get a clear method and a set of experiments that need a careful read. The paper deserves a serious referee, but the authors should be asked to re-run projection with the same feasibility model, or at least re-frame the claim. I would not desk-reject it.","headline":"Clever extension with a clear write-up, but the headline advantage over projection is not established because the projection baseline was given a stricter feasibility model.","tokens_in":17022,"tokens_out":2812,"would_cite":true,"duration_ms":26351,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Action mapping transforms a state-wise constrained MDP into an unconstrained MDP by pretraining a feasible-action generator, and the reported experiments show it improves both learning speed and constraint satisfaction.","keywords":["action mapping","constrained reinforcement learning","feasibility model","continuous action spaces","safe reinforcement learning","latent action space","soft actor-critic","proximal policy optimization"],"falsifier":"Train the feasibility policy on a toy feasible set made of two disconnected components with very different volumes, sample many latent codes uniformly, and count how often each component is generated. If the smaller component is generated far less often than its volume share, or not at all, the claimed surjectivity fails and the transformed MDP is not truly unconstrained.","tokens_in":16016,"feed_emoji":"🎯","tokens_out":8854,"duration_ms":77665,"temperature":0.7,"pith_summary":"This paper argues that a pretrained feasibility policy can act as a state-dependent generator of all feasible actions, so the reinforcement learning agent only has to choose a latent code instead of a raw action. If that generator covers the feasible set well enough, the state-wise constrained MDP becomes an unconstrained MDP over the latent space, and standard PPO or SAC can solve it. The experiments compare this action-mapping strategy with action projection, resampling, replacement, and Lagrangian methods in a robotic-arm pose task and a spline-based path-planning task. The reported result is that action mapping trains faster, reaches higher reward, and violates constraints less often, with the largest gap appearing when the feasibility model is only approximate.","feed_headline":"Pretrained action maps speed constrained RL and cut violations","feed_subtitle":"Pretrained feasibility generator narrows RL actions to feasible ones, beating projection and Lagrangian baselines.","key_machinery":"The load-bearing mechanism is the action map: a pretrained feasibility policy $\\pi_f$ that turns latent samples into feasible actions, trained by minimizing the Jensen-Shannon divergence between the generated density and the uniform target over feasible actions, with both densities estimated by kernel density estimates and Monte-Carlo importance sampling. The second piece is training the objective policy on the latent $z$ rather than on actions, so that disconnected components of the feasible set sit close together in latent space and a unimodal Gaussian can induce a multi-modal action distribution. Standard PPO or SAC then operates on the induced unconstrained transition function $P_f(s'|s,z)=P(s'|s,\\pi_f(s,z))$.","core_discovery":"The central claim is that allowing the objective policy to choose only among feasible actions transforms an SCMDP into an unconstrained MDP. The transformation is explicit: a feasibility policy $\\pi_f: S \\times Z \\to A_s^+$ is pretrained to map latent space uniformly onto the state-dependent feasible set, using a kernel density estimate of the Jensen-Shannon divergence gradient, and then an objective policy $\\pi_o: S \\to P(Z)$ is trained with PPO or SAC on the latent $z$. The environment the objective policy sees is $P_f(s'|s,z)=P(s'|s,\\pi_f(s,z))$, which has no constraints by construction. The reported experiments show AM-PPO and AM-SAC beating projection, resampling, replacement, and Lagrangian baselines on return and constraint violation, particularly in the path-planning environment with an approximate feasibility model.","pith_inferences":["Beyond the paper's hand-built models, the same pipeline should transfer to learned feasibility models: any classifier or safety critic that can be queried as $g(s,a)$ can replace the analytic models here, making action mapping a general wrapper around existing safe-RL approaches.","The mechanism should be most fragile exactly where the KDE approximation is stressed, namely highly fragmented or very narrow feasible sets with fixed bandwidth; a toy benchmark with two feasible components of very unequal volume would quantify how quickly surjectivity degrades.","Because the objective policy acts on a learned action representation, the same decoupling could be applied to hierarchical reinforcement learning or to any setting where a generative model can serve as a bottleneck between the policy and a constrained action space."],"forward_implications":["With a feasibility model available, action mapping removes per-step optimization: instead of projecting each proposed action, the agent runs one extra neural-network inference, which the paper reports makes training and inference substantially cheaper than projection-based methods.","Pretraining the feasibility policy decouples feasibility from reward, so the objective policy explores only feasible actions and wastes fewer samples on constraint violations; in the path-planning environment this produced an early performance jump and a later second improvement once the agent learned obstacle geometry.","Because disconnected feasible sets are close in latent space, a single Gaussian objective policy can express multi-modal action distributions, improving exploration and making it easier to jump between separated feasible regions.","When a safe replacement action exists, composing action mapping with replacement gives the best return with no constraint violations in the robotic-arm task.","The benefit is robust to a moderately imperfect feasibility model: reducing the number of spline evaluation points from 64 to 32 preserves most of the advantage, while much coarser approximation degrades toward plain SAC."],"supporting_citations":[{"why":"Supplies the pretraining method for the feasibility policy, including the KDE-based Jensen-Shannon divergence objective that this paper refines.","marker":"(Theile et al., 2024)"},{"why":"Provides the state-wise constrained Markov decision process formulation that the action-mapping transformation targets.","marker":"(Zhao et al., 2023)"},{"why":"Provides the Soft Actor-Critic algorithm on which AM-SAC is built.","marker":"(Haarnoja et al., 2018)"},{"why":"Provides the Proximal Policy Optimization algorithm on which AM-PPO is built.","marker":"(Schulman et al., 2017)"},{"why":"Provides the Lagrangian PPO baseline and safety-critic training procedure that action mapping is compared against.","marker":"(Ray et al., 2019)"},{"why":"Provides the Lagrangian SAC baseline used in the path-planning comparison.","marker":"(Ha et al., 2020)"},{"why":"Provides the DC3 gradient-based action-projection approach that is the main feasibility-model baseline and the source of the computational overhead action mapping avoids.","marker":"(Donti et al., 2021)"},{"why":"Provides the action-resampling baseline that rejects infeasible actions and resamples from the actor.","marker":"(Bharadhwaj et al., 2020)"},{"why":"Provides the action-replacement baseline that substitutes a predefined feasible action for an infeasible one.","marker":"(Srinivasan et al., 2020)"}],"fun_headline_variants":["Action mapping turns constrained RL into an unconstrained MDP","Pretrained feasibility models boost continuous constrained RL","Action mapping speeds constrained RL by narrowing to feasible actions","Decoupling feasibility from policy improves constrained RL training","Imperfect feasibility models still work with action mapping"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole benefit depends on pretraining producing a near-surjective map from the latent space to the set of feasible actions; if parts of the feasible set are never generated, the objective policy cannot see them and the constrained problem is not actually lifted.","fun_headline_variants_meta":{"raw":{"variants":["Action mapping turns constrained RL into an unconstrained MDP","Pretrained feasibility models boost continuous constrained RL","Action mapping speeds constrained RL by narrowing to feasible actions","Decoupling feasibility from policy improves constrained RL training","Imperfect feasibility models still work with action mapping"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000489,"raw_usage":{"total_tokens":2355,"prompt_tokens":842,"completion_tokens":1513,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":458,"completion_tokens_details":{"reasoning_tokens":1440}},"tokens_in":458,"tokens_out":1513,"duration_ms":12129,"temperature":1.0,"reasoning_tokens":1440,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:32:15.888674+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the feasibility policy on a toy feasible set made of two disconnected components with very different volumes, sample many latent codes uniformly, and count how often each component is generated. If the smaller component is generated far less often than its volume share, or not at all, the claimed surjectivity fails and the transformed MDP is not truly unconstrained.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the pretraining method for the feasibility policy, including the KDE-based Jensen-Shannon divergence objective that this paper refines."}],"review_version":1}