{"id":"a9af5f42-f9bc-42cc-8c48-0cc53b56670f","arxiv_id":"2506.17297","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SafeRL-Lite provides modular wrappers for safety-constrained and explainable DQN agents, with a CartPole demonstration showing decreasing violations and pole-angle-dominant SHAP attributions.","lead":"SafeRL-Lite is an open-source Python library that wraps standard Gym environments and DQN agents with runtime safety constraints and SHAP/saliency explanations. It demonstrates both features on a constrained CartPole task, where the wrapper reduces constraint violations while SHAP highlights pole angle as the main decision driver.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported zero-violation result conflates the wrapper's hard action replacement with genuine safe learning; the paper's own constraint definition makes violations impossible by construction once a safe action exists.","rationale":"The reader's weakest_assumption correctly identifies the off-policy distribution mismatch caused by replacing the DQN's selected action in the replay buffer. My stress-test focuses on a sharper version of the same problem: because the paper's wrapper replaces unsafe actions at every step (Section 3.1) and the constraint depends only on the current observation in a finite action space, the 'zero violations' result follows almost by construction once any safe action exists. The paper does not state whether the violation count is computed before or after the replacement, so the reported CVC trends cannot distinguish 'the DQN learned to propose safe actions' from 'the wrapper masked unsafe actions'. This is load-bearing because the central claim (Section 7) is that agents learn safe policies without modification to the core learning algorithm. A test that logs raw versus executed actions, and ablates the hard override versus penalty-only reward shaping, would settle whether the claimed result is a genuine property of the learned policy or an artifact of the safety filter. I agree with the reader's partial assessment on other issues (lack of baselines, error bars, quantitative convergence), but the override-distribution mismatch is the single most decisive concern. A CONDITIONAL verdict is appropriate: the paper's contribution is a plausible software wrapper, but the empirical evidence for the zero-violation learning claim is incomplete and potentially misleading.","tokens_in":4922,"tokens_out":2396,"duration_ms":20785,"concrete_test":"Run the SafeEnvWrapper with logging of both the raw DQN action and the executed post-wrapper action for 200 episodes in the constrained CartPole-v1 setup. Measure three quantities: (1) the fraction of steps where the raw action violated |v_t| <= 0.5; (2) the fraction of executed steps that resulted in a violation count; (3) the same training pipeline with the hard replacement disabled and only the penalty r'_t = r_t - lambda * I[C_i>0] enabled, under identical hyperparameters and seeds. If the raw-action violation fraction is high but executed violations are near zero, the zero-violation claim is an artifact of the override. If the penalty-only agent also reaches zero violations, the claim that the wrapper is needed is weakened; if it does not, the paper must still report corrected Q-learning or demonstrate that the override-induced bias is negligible.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (Section 7: agents learn safe policies with zero violations without modifying the core learning algorithm) is undercut by the paper's own safety mechanism. In Section 3.1, the wrapper replaces any unsafe action with a_safe = argmin_{a'} sum_i max(0, C_i(s,a')). For the CartPole experiments, the constraint |v_t| <= 0.5 is a function of the current observation only, and the action space is discrete (left/right). Because the wrapper is invoked at every step, a violation can be logged only if (i) no action in {left, right} satisfies the constraint, or (ii) a violation is recorded before the replacement is applied. The paper does not define how violations are counted relative to the override, and it never reports the fraction of steps in which the DQN proposed an unsafe action that the wrapper had to correct. If that fraction is high, the 'zero violations by episode 150' result is an artifact of the hard block: the deployed policy is not the DQN policy but the wrapper policy, and the training buffer is filled with transitions generated under a different action-selection distribution than the one being learned. The off-policy mismatch is never corrected (no importance sampling, no clipping of replaced actions in the target), so the Q-updates are chasing a mixture of the raw DQN policy and the safety-filter policy. The paper also provides no per-seed statistics, no baseline from an unconstrained DQN, and no comparison with an agent that only uses the penalty r'_t = r_t - lambda * I[C_i>0] without the hard override. Therefore the key empirical claim, that the library lets an unmodified DQN learn a safe policy, is not established; the experiments can be explained entirely by the wrapper's forced action substitution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SafeRL-Lite, an open-source Python library that wraps standard Gym environments and deep Q-network (DQN) agents to add safety constraints and post-hoc explainability. Safety is enforced through a SafeEnvWrapper that, at every step, checks user-defined constraints C_i(s,a) <= 0 and replaces unsafe actions with a_safe = argmin_{a'} sum_i max(0, C_i(s,a')). Explainability is provided through SHAP attributions and gradient-based saliency maps of the trained Q-network. The empirical section reports experiments on a constrained CartPole-v1 variant with a velocity limit |v_t| <= 0.5, claims that constraint violations decrease over 200 episodes and reach zero by episode 150, and provides qualitative SHAP and saliency visualizations. The central claim is that unconstrained off-the-shelf DQN agents can learn safe policies with zero violations without modifying the core learning algorithm.","tokens_in":5209,"tokens_out":4482,"duration_ms":47654,"significance":"If the central claim were established, SafeRL-Lite would be a convenient educational and prototyping tool for safe, interpretable RL. The work has some genuine strengths: it is released as an installable pip package, the wrapper architecture is modular and easy to extend, violation logging is built in, and the visual explanation pipeline is straightforward. However, the conceptual contribution is modest: action replacement or projection onto a safe set is a standard shielding technique in safe RL, and the empirical demonstration is qualitative and lacks the controls needed to distinguish the agent's learned behavior from the wrapper's hard enforcement. Given the explicit claims in the abstract and conclusion, the significance depends on whether the empirical claims survive a rigorous re-evaluation.","major_comments":[{"comment":"The zero-violation conclusion is not supported as a statement about the learned policy because the SafeEnvWrapper replaces every unsafe action with a_safe = argmin_{a'} sum_i max(0, C_i(s,a')) before execution. If the reported Constraint Violation Count measures violations after this replacement, then near-zero violations are expected by construction once at least one admissible action exists, and Figure 4 cannot distinguish learned safety from wrapper enforcement. If it measures violations before replacement, the manuscript never states this and never reports the fraction of steps in which the wrapper had to override the DQN's action. Please define precisely when a violation is counted, report the override rate over training, and provide per-seed statistics (mean, standard deviation, number of seeds).","section":"Section 3.1 / Section 5.4 / Section 6.1"},{"comment":"The training procedure suffers from an untreated off-policy mismatch. When the wrapper replaces the DQN's selected action, the transition stored in the replay buffer contains the safe action, not the action proposed by the Q-network. The Q-update therefore estimates values for a mixture of the raw policy and the safety-filter policy. The penalty term r'_t = r_t - lambda sum_i I[C_i(s_t,a_t)>0] is only meaningful if a_t refers to the pre-filter action, but then it is inconsistent with the stored transition. No importance sampling, no clipping of replaced actions in the Q-target, and no other correction is described. The claim that the agent learns a safe policy 'without modifying the core learning algorithm' is therefore not established as stated.","section":"Section 4.3 / Section 3.1"},{"comment":"The safety guarantee relies on the assumption that a safe action exists at every state, which is not guaranteed. In CartPole with |v_t| <= 0.5 and a binary action space, if the current velocity violates the constraint, both possible control choices may lead to states that still violate the constraint, depending on the environment dynamics and the discretization used in the constraint check. The argmin formula may then return a violating action and the wrapper cannot ensure safety. The manuscript does not define a fallback for this case, nor does it report how often no safe action exists. The statement in Section 3.3 that the framework provides 'runtime safety guarantees' is therefore too strong.","section":"Section 3.1 / Section 5.1"},{"comment":"The main quantitative claim is reported only qualitatively. No table or text gives violation counts at any specific episode, no convergence statistics (e.g., mean and standard deviation over multiple seeds) are reported, no baseline without the wrapper or an agent trained only with the penalty term is shown, and the DQN hyperparameters are essentially unspecified beyond 'trained for 200 episodes'. The statement 'By episode 150, the agent consistently selects safe actions' has no numerical support in the manuscript. This is load-bearing because the paper's contribution is an empirical demonstration of safe learning, not merely a software release.","section":"Sections 5.3-5.4 and 6.1"}],"minor_comments":[{"comment":"The SHAP notation is confusing: the text says 'missing features in S^c marginalized out' while the formula uses f(S union {j}); please clarify the exact definition of f(S) and the approximation used for KernelSHAP.","section":"Section 3.2"},{"comment":"The SHAP and saliency explanations are computed for the raw Q-network, but the action actually executed may be the wrapper's replacement. The paper should state that explanations may not correspond to the deployed action after safety filtering.","section":"Section 4.4"},{"comment":"Sections 5 and 6 describe the same CartPole experiment and report overlapping results; they should be consolidated or cross-referenced to avoid the impression of two separate evaluations.","section":"Sections 5 and 6"},{"comment":"The figures lack axis labels, units, legends, and error bars; captions such as 'Violation decreasing' do not convey quantitative information and should be replaced with descriptive captions that cite specific numbers.","section":"Figures 3-7"},{"comment":"Table 1 reports mean SHAP values without variance or details on the number of samples and kernel settings; add standard deviations or standard errors and describe the approximation parameters.","section":"Table 1"},{"comment":"Section 4.2 lists 'blocked, masked, or replaced' as three enforcement mechanisms, but Section 3.1 defines only action replacement; align the design description with the implemented mechanism.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"This is best viewed as a tool/demo paper rather than a full research contribution. The safety and learning claims are not verifiable from the current manuscript because the wrapper's action replacement can explain the reported zero-violation trend by construction. I would ask the authors for a revised version with explicit violation-counting definitions, override-rate statistics, multi-seed quantitative results, and a clean accounting of the off-policy issue; without those, the paper's central claim remains unsubstantiated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a packaging paper, and the package probably does what it says. The useful part is the code: a lightweight wrapper around Gym/DQN that does action-level constraint filtering and SHAP/saliency explanations in a few lines, installed via pip. That genuinely lowers the barrier for people who want to try constrained RL and XRL together without reading CPO or Lyapunov papers. The authors cite CPO, Lyapunov, Atari saliency correctly. No invented results, no citation fraud. As a tool, it is fine.\n\nWhat it is not is a research paper about safe RL. The central demonstration is CartPole-v1 with |v|<=0.5, 200 episodes, no numbers except a table of SHAP means. Figure 3 is a line with no axis labels. The 'zero violations by episode 150' claim is substantially by construction: Section 3.1's wrapper replaces any unsafe action with argmin sum max(0,C_i(s,a)). With two discrete actions and a velocity threshold, once a safe action exists the wrapper can always pick it. So the logged violation count is guaranteed to fall to zero regardless of what DQN learns. The paper never reports how often the DQN proposed unsafe actions or how much of the behavior is the filter rather than the learner. The stress-test note is right: the training buffer is filled with transitions from a mixed policy (raw DQN plus safety filter) and the Q-updates are not corrected for that off-policy mismatch. If you only award 'safe learning' to an agent whose own policy is safe, this evidence doesn't show it.\n\nThe other soft spots are empirical hygiene. No seeds, no error bars, no baseline from an unconstrained DQN, no comparison with the penalty-shaping variant r' - lambda*I. Sections 5 and 6 are the same CartPole experiment described twice. The SHAP result—pole angle dominates—is the known physics of the environment and previous saliency papers, so it's not new.\n\nIf this is evaluated as a software contribution, I'd be moderately positive. It ships real code, the API is simple, and educators/tinkerers will get value. But the paper's text overclaims 'agents can learn safe policies' and 'deep synergy.' Those claims should be rewritten to describe hard filtering, and the experiments should include per-seed statistics and a baseline.\n\nRecommendation: don't desk reject; send it to peer review at a software/demo/workshop track with the expectation of heavy revision. As a regular research paper, the contribution is too thin, but the artifact is a legitimate candidate for a tool paper.","headline":"A useful pip-installable safe-RL wrapper whose core demo overstates itself: the zero-violation result is mostly a hard action-mask artifact, not evidence about learning.","tokens_in":5831,"tokens_out":3400,"would_cite":false,"duration_ms":31278,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that wrapping an ordinary DQN with safety and explanation layers can enforce hard constraints and explain every decision without changing the learning algorithm.","keywords":["safe reinforcement learning","constrained Markov decision process","action masking","SHAP","saliency maps","explainable reinforcement learning","Gym wrapper","CartPole"],"falsifier":"Train the DQN with SafeRL-Lite's override active, then freeze the network and run 100 evaluation episodes with the override disabled; if constraint violations occur in those episodes, the zero-violation result belongs to the wrapper-plus-agent system, not to the learned policy alone.","tokens_in":4722,"feed_emoji":"🛡️","tokens_out":7472,"duration_ms":73861,"temperature":0.7,"pith_summary":"The paper introduces SafeRL-Lite, a Python library that wraps standard Gym environments and Deep Q-Network agents so safety constraints are enforced at runtime and decisions are explained after the fact. The central claim is that an ordinary, unconstrained DQN can be turned into a safe and interpretable agent purely through environment-side wrappers, with no change to the Q-learning update. That matters because safety-critical deployments often cannot afford the custom solvers and modified objectives required by constrained reinforcement-learning methods. The authors demonstrate the idea on a CartPole variant with a velocity limit, reporting that constraint violations fall over 200 episodes and that SHAP attributions identify pole angle as the dominant decision feature.","feed_headline":"Safety wrapper yields zero-violation RL policies without retraining","feed_subtitle":"A lightweight library wraps any Gym environment and DQN agent to enforce constraints and explain each decision.","key_machinery":"The load-bearing object is the SafeEnvWrapper and its runtime action-projection rule: return the agent's action $a_t$ if $C_i(s_t, a_t) \\leq 0$ for every constraint $i$, otherwise return $\\arg\\min_{a'} \\sum_i \\max(0, C_i(s_t, a'))$. This rule lets the library claim hard runtime safety while leaving the learning algorithm intact. The second mechanism is the SHAP value decomposition of the Q-function, $\\phi_j = \\sum_{S \\subseteq F \\setminus \\{j\\}} \\frac{|S|!(d-|S|-1)!}{d!}[f(S\\cup\\{j\\})-f(S)]$, approximated by KernelSHAP, which assigns each input feature a marginal contribution to the action value and is what generates the interpretability output.","core_discovery":"On its own terms, the paper establishes a recipe: any Gym environment can be wrapped with a safety layer that checks user-defined constraints at each step and, when they are violated, replaces the agent's action with the feasible action minimizing total constraint violation. The same wrapper can log violations and penalize them in the reward, while the DQN's update remains unchanged. The paper argues this is enough for an unconstrained DQN to converge to a policy with decreasing and eventually zero constraint violations, and that SHAP and saliency explainers can then attribute each Q-value to state features, with pole angle emerging as the dominant feature in the CartPole experiments.","pith_inferences":["The action-replacement rule operates outside the DQN's behavior policy, so the transitions stored in the replay buffer are chosen by the wrapper, not by the Q-network; correcting for this off-policy mismatch could change both the learned Q-values and the reported violation curves.","A direct test of where safety resides is to freeze the trained Q-network, disable the override at evaluation, and count violations; if violations reappear, the zero-violation result belongs to the wrapper-plus-agent system rather than to the learned policy alone.","The same wrapper idea could be applied to policy-gradient agents or continuous action spaces through safety projections, an extension the paper lists as future work, but the off-policy issue would need re-examination there as well.","SHAP dominance of a single feature may provide a cheap deployment-time early-warning signal: shifts in attribution could flag that the policy is leaving its trained safety regime, though the paper does not test this."],"forward_implications":["Users can take an existing Gym environment and DQN training loop, attach constraint wrappers, and obtain agents whose constraint violations shrink during training without rewriting the agent's learning rule.","The built-in violation log and early-termination option provide a uniform way to measure and penalize unsafe behavior across environments.","SHAP heatmaps offer a post-hoc view of which state features drive Q-values, and the paper reports the attribution given to pole angle matches physical intuition.","The wrapper pattern can be extended to new constraints by supplying constraint functions, so the safety mechanism is not tied to CartPole.","If the central claim holds, adding safety and explainability to existing RL pipelines becomes a configuration task rather than an algorithm swap."],"supporting_citations":[{"why":"Defines the constrained MDP formulation and the CPO approach that SafeRL-Lite's wrapper design contrasts with.","marker":"[1]"},{"why":"Benchmark suite for safe exploration that motivates the constrained CartPole evaluation setup.","marker":"[3]"},{"why":"Introduced saliency visualizations for RL agents, the precedent for the gradient-based explainer module.","marker":"[4]"},{"why":"Recent explainable DQN approach using heatmaps and uncertainty, the closest prior work the library extends toward modular wrappers.","marker":"[5]"}],"fun_headline_variants":["SafeRL-Lite: zero-violation policies via action masking","Lightweight RL library enforces constraints and explains decisions","Wrapper turns any Gym env into a safe, explainable RL agent","SafeRL-Lite: constraint-safe RL with SHAP explanations","Zero-violation RL achievable with a lightweight wrapper"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"At every state a constraint-satisfying action exists, and substituting it for the DQN's intended action does not distort what the DQN learns from the replay buffer.","fun_headline_variants_meta":{"raw":{"variants":["SafeRL-Lite: zero-violation policies via action masking","Lightweight RL library enforces constraints and explains decisions","Wrapper turns any Gym env into a safe, explainable RL agent","SafeRL-Lite: constraint-safe RL with SHAP explanations","Zero-violation RL achievable with a lightweight wrapper"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000825,"raw_usage":{"total_tokens":3543,"prompt_tokens":819,"completion_tokens":2724,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":435,"completion_tokens_details":{"reasoning_tokens":2642}},"tokens_in":435,"tokens_out":2724,"duration_ms":19518,"temperature":1.0,"reasoning_tokens":2642,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:47:36.182861+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the DQN with SafeRL-Lite's override active, then freeze the network and run 100 evaluation episodes with the override disabled; if constraint violations occur in those episodes, the zero-violation result belongs to the wrapper-plus-agent system, not to the learned policy alone.","supporting_citations":[{"cited_title":"Constrained Policy Optimization","cited_arxiv_id":null,"evidence_quote":"Defines the constrained MDP formulation and the CPO approach that SafeRL-Lite's wrapper design contrasts with."},{"cited_title":"Benchmarking Safe Exploration in Deep Reinforcement Learning","cited_arxiv_id":null,"evidence_quote":"Benchmark suite for safe exploration that motivates the constrained CartPole evaluation setup."},{"cited_title":"Visualizing and Understanding Atari Agents","cited_arxiv_id":null,"evidence_quote":"Introduced saliency visualizations for RL agents, the precedent for the gradient-based explainer module."}],"review_version":2}