{"id":"3fb907ae-a647-45e1-84d4-e7c8d09965e6","arxiv_id":"2412.19311","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A framework that merges risk-aware local explanations with global policy graphs to explain, attack, and patch safety-critical RL agents.","lead":"xSRL is a framework that combines local and global explanations for reinforcement learning agents, showing both why specific actions are risky and how the agent behaves overall. It lets developers find policy weaknesses through adversarial attacks and patch them without retraining, tested on continuous control tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 2's risk-critic loss omits the immediate cost c_t and the discount factor, so Q_risk as written cannot learn expected safety costs; unless the code differs, the shield (Eq. 7) and critical-state detector (Eq. 6) inherit this flaw.","rationale":"The paper's central claim is that explainability produces safety through the Q_risk-augmented pipeline. The most direct linchpin is the risk critic itself, since it drives both the global explanation annotations and the utility mechanisms described in Eqs. 6-7. The training loss in Eq. 2, as printed, is missing the cost term c_t and the discount factor \\gamma; such a loss does not define a Q-function for safety costs and would drive the critic toward a constant. This would make the fidelity numbers and all downstream conclusions uninterpretable. I am not claiming the authors deliberately misled; this may be a typographical error, and the code is available. The concrete test resolves it. If the code matches the printed equation, the framework as described fails; if not, the paper requires only a correction plus the distribution-shift checks the reader requested. I therefore agree with the CONDITIONAL verdict: the paper should not be accepted as-is until Eq. 2 is clarified and the shield's deployment behavior is verified.","tokens_in":14603,"tokens_out":9334,"duration_ms":83403,"concrete_test":"Inspect the released code at github.com/risal-shefin/xSRL for the Q_risk training loop. Check whether the TD target is c_t + \\gamma \\hat{Q}(s_{t+1}, a_{t+1}) or the bare \\hat{Q}(s_{t+1}, a_{t+1}) shown in Eq. 2. If the code matches Eq. 2, retrain Q_risk with the corrected loss (adding c_t and \\gamma), re-run Table 1, and compare the fidelity numbers and Figure 2 attack/shield results; if the results change materially, the equation error is the load-bearing flaw. If the code already includes c_t and \\gamma, the paper needs only an erratum, and the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central mechanism is the risk critic Q_risk, used for local explanations, the safety-critical state definition (Eq. 6), and the shield (Eq. 7). The training loss for this critic, Eq. 2, is written as J_risk = 1/2 (\\hat{Q}_{\\phi,risk}(s_t,a_t) - E_{a_{t+1}~\\pi(.|s_{t+1})}[\\hat{Q}_{\\phi,risk}(s_{t+1},a_{t+1})])^2. This target omits both the immediate cost c_t and the discount \\gamma, despite the text saying the loss uses sampled transitions (s_t,a_t,s_{t+1},c_t) and is with respect to the RHS of Eq. 1. As written, the minimizer is any constant Q, so the critic would not encode expected safety costs. If this is a typographical omission, the paper should state the corrected loss; if the implementation matches the equation, then every downstream use of Q_risk, including attack targeting and shield triggering, is untrustworthy. The Table 1 NRMSE values (33.2-62.8%) are consistent with an improperly trained critic and do not by themselves establish accuracy. No reference or appendix clarifies the missing term.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes xSRL, a framework for safety-aware explainable reinforcement learning that combines local explanations (learned Q_task and Q_risk critics) with global explanations (an extension of the CAPS policy-graph summarization) for agents in constrained MDPs. xSRL additionally provides adversarial explanations by using Q_risk to identify safety-critical states, launching adversarial attacks at those states, and patching the policy with a safety shield or a separately trained safety policy. The evaluation comprises computational fidelity measures (action, policy-selection, Q_risk, Q_task) on SAC, AdvExRL, RRL-MF, and SQRL agents in two MuJoCo environments, plus three user studies with 270 participants measuring comprehension and ability to identify the safer agent. The authors claim that safety can be improved as a byproduct of explainability, and that xSRL is the first framework to combine local and global explanations for RL safety and to offer adversarial explanations for vulnerability analysis and policy patching.","tokens_in":14822,"tokens_out":3753,"duration_ms":33725,"significance":"If the results hold, xSRL would be a practically useful contribution to explainable and safe RL, integrating several desiderata (local explanations, global summaries, adversarial vulnerability analysis, and post-hoc patching without retraining) into one toolkit. The paper ships code and includes a large user study, which are strengths. However, the central claims rest on the accuracy of the learned risk critic and on fidelity numbers that are, in their reported form, partly at odds with the paper's own abstract and introduction. The contribution is novel enough for AAMAS, but the evidence needs to be tightened before the safety-as-a-product-of-explainability claim is fully supported.","major_comments":[{"comment":"The risk-critic loss J_risk is written as the MSE between the current Q_risk estimate and the expected future Q_risk estimate only, omitting the immediate cost c_t and the discount factor gamma. As written, any constant Q is a minimizer, so Q_risk would not encode expected safety costs as defined in Eq. (1). This is load-bearing because Q_risk is used to define safety-critical states in Eq. (6), to trigger the shield in Eq. (7), and to generate the local and global explanations. Please correct Eq. (2) to include c_t and gamma, or supply code or an appendix demonstrating that the implemented loss actually contains these terms. The Table 1 NRMSE values (33.2% to 62.8%) do not resolve the issue by themselves, since a constant predictor can yield comparable NRMSE depending on the scale of the costs.","section":"§5.1.1, Table 1"},{"comment":"The fidelity results for patched agents are weak relative to the claims made in the abstract and Section 1. Action fidelity for AdvExRL, RRL-MF, and SQRL is 34.25%, 31.7%, and 32.4% respectively, and Q_risk NRMSE is 33.2%, 48.4%, and 62.8%. The introduction states that xSRL generates accurate policy graphs with less than 33.5% error in risk estimation, which is contradicted by the RRL-MF and SQRL rows. These numbers do not convincingly establish that xSRL provides high-fidelity explanations for the patched agents that are central to the safety claims. Please report error bars or significance tests across the five graphs, and revise the accuracy claims to match the actual ranges.","section":"§5.1.1, Table 1"},{"comment":"The user study results for H1 only partially support the claim that xSRL is the most effective explanation method. For unsafe agents, xSRL and global explanations achieve exactly the same average accuracy (60.37%), so xSRL is not superior to CAPS in the high-risk setting that motivated the paper. For safe agents, xSRL leads by 77.4% vs 73.7%, but no statistical test is reported, so it is unclear whether this difference is meaningful. The conclusion that xSRL is most effective and the related superiority claim over global explanations need to be either supported with inferential statistics or appropriately softened.","section":"§5.1.2, Table 2"},{"comment":"The utility evaluation of patching is weakened by how the attack is constructed. The explanation-guided attack in Section 4.2 targets high-risk abstract states identified from the SAC agent's own Q_risk values, and the same attack is then applied to all patched agents. Since AdvExRL, RRL-MF, and SQRL already have their own safety mechanisms, the results in Figure 2 partly reflect the pre-existing robustness of those methods rather than the benefit of xSRL's patching. Moreover, there is no comparison against a baseline that patches without xSRL's explanation guidance, so the incremental utility of xSRL in the patching loop is not established.","section":"§5.2.1, Figure 2"}],"minor_comments":[{"comment":"The introduction states that fidelity tests show 'less than 33.5% error in risk estimation', but Table 1 reports Q_risk NRMSE of 48.4% for RRL-MF and 62.8% for SQRL. Please correct this inconsistency.","section":"§1"},{"comment":"The notation in Eqs. (1) and (3) is confusing: the expectation subscript is taken over a_t while the sum runs over future timesteps, and the inserted 'task' text in Eq. (3) is misplaced. Please reformulate these expressions cleanly.","section":"§4.1, Eq. (1), Eq. (3)"},{"comment":"No measures of variance are provided for the fidelity scores, despite the text stating that scores are averaged across five generated graphs. Reporting standard deviations or confidence intervals would substantially strengthen the empirical claims.","section":"§5.1.1, Table 1"},{"comment":"The recruitment description says '270 participants (30 per study)' with three studies, which is arithmetically inconsistent; presumably the intended meaning is 30 participants per condition across nine conditions. Please clarify.","section":"§5.1.2, User Studies"},{"comment":"There is a typo: 'trustworthines' should be 'trustworthiness'.","section":"§1"}],"recommendation":"major_revision","confidential_remarks":"The critical technical issue is Eq. (2): if the code matches the equation, the risk critic is not learning expected safety costs and the entire xSRL pipeline is untrustworthy; if it is a typo, the authors must provide the corrected formula and evidence that the reported results were produced with it. The fidelity and user-study weaknesses are substantial but fixable with revised claims and additional analysis. I recommend major revision rather than rejection because the framework is well-motivated and the released code allows verification. I would also suggest the authors consider more clearly separating the contribution over CAPS, since the user study shows no advantage for unsafe agents."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What's actually new here: xSRL combines local Q-task/Q-risk explanations with CAPS-style global policy graphs, and uses those explanations to drive adversarial attacks and policy patching without retraining. That integration is real and not present in the cited prior work. The paper also does more empirical work than most XRL papers: fidelity metrics, a 270-participant user study across nine conditions, and attack/patching experiments. The authors report low fidelity numbers honestly rather than hiding them.\n\nThe strongest part is the demonstration that the combined graph gives users at least as much comprehension as global explanations alone, and better comprehension for safe agents. The debugging workflow—attack high-risk abstract states, inspect the changed graph, then patch with a safety policy and verify—is plausible and useful for practitioners.\n\nThe soft spots are real but fixable. Most importantly, Eq. 2 as written omits both the immediate cost c_t and the discount factor. The target is just the expected next-state Q-value. Minimizing that loss collapses to any constant function, so the risk critic would not encode expected safety costs. The text says the loss is with respect to the RHS of Eq. 1, so this may be a typographical omission, but the paper must state the corrected loss explicitly, and the code must match. If the implementation matches Eq. 2 as written, every downstream use of Q_risk—critical-state detection, attack targeting, shield triggering—is untrustworthy. This is not a minor nit.\n\nBeyond that: there are no error bars or significance tests, and Table 2 shows xSRL and global explanations tie on unsafe agents (60.37% both). The 'safety as a product of explainability' slogan is broader than what the experiments show; they demonstrate a useful workflow, not a causal law. Action fidelity for patched agents is low (31.7–34.25%), which the paper discusses, but it undercuts the trust argument.\n\nThe citation pattern is fine; building on the authors' own CAPS and AdvExRL is disclosed and sensible.\n\nThis paper deserves a serious referee. The core idea is worth engaging with, and the equation issue is likely fixable, but it needs to be resolved before the central mechanism can be evaluated. I would accept it for review with a request for major revision on the formalism and statistics.","headline":"A useful integration of local and global explanations for RL safety with a serious typo-level flaw in the risk-critic loss that must be fixed before the central claims can be trusted.","tokens_in":15412,"tokens_out":1767,"would_cite":false,"duration_ms":17516,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that combining local risk critics with global policy graphs makes RL safety actionable, allowing developers to attack, debug, and patch policies without retraining.","keywords":["explainable reinforcement learning","safe reinforcement learning","local explanations","global explanations","adversarial attack","policy patching","risk critic","policy graphs"],"falsifier":"A reader could run the SAC agent in Navigation 2 with the same 50% attack, but with the $Q_{\\mathrm{risk}}$ critic deliberately corrupted by noise or by training it on trajectories from a different policy, and check whether the set of safety-critical states and the shield's firing behavior change materially; if they do, the safety explanations inherit the critic's error. A cleaner quantitative check is to compare $Q_{\\mathrm{risk}}$'s predicted cumulative cost against the ground-truth cost in held-out episodes under adversarial perturbations, and ask whether its NRMSE (reported between 33% and 63%) is concentrated precisely in the high-risk states used for attack and patching.","tokens_in":14361,"feed_emoji":"🛡️","tokens_out":10386,"duration_ms":82017,"temperature":0.7,"pith_summary":"The paper claims that safety in reinforcement learning can be improved as a product of explainability: if operators can see both the agent's overall strategy and its per-state risk estimates, they can identify, attack, and patch vulnerabilities without retraining the policy. xSRL trains two post-hoc critics, one for task reward ($Q_{\\mathrm{task}}$) and one for safety cost ($Q_{\\mathrm{risk}}$), and attaches their values to abstract-state policy graphs built by an existing global explanation method. This combination is what the authors call the first framework to join local and global explanations for RL safety and to offer adversarial explanations for vulnerability analysis and policy patching. The authors support the claim with fidelity measurements, user studies, and an attack-then-patch demonstration in two safe-RL benchmark environments.","feed_headline":"xSRL turns RL explanations into attack-and-patch tools","feed_subtitle":"Local risk values on global policy graphs let developers find vulnerabilities and fix them without retraining.","key_machinery":"The load-bearing mechanism is the post-hoc risk critic $Q_{\\mathrm{risk}}$, trained by minimizing an MSE Bellman-like loss (Eq.~2) over sampled transitions of a fixed target policy, and its task counterpart $Q_{\\mathrm{task}}$. These critics are averaged over abstract states in a CAPS-style global policy graph (Eqs.~4--5), letting each node carry expected return and expected safety cost. The mechanism also includes the threshold definitions: a state is safety-critical when $Q_{\\mathrm{risk}}(s,a) > \\epsilon_{\\mathrm{safety}}$ for some policy action (Eq.~6), and the shield triggers when $Q_{\\mathrm{risk}}(s_t,a_t) > T_{\\mathrm{safety}}$ (Eq.~7), replacing the action with one from a separately trained safety policy. This machinery turns explanation values into attack targets and patch triggers.","core_discovery":"On the paper's own terms, the central discovery is that a single explanation artifact—a directed graph of abstract states with $Q_{\\mathrm{task}}$ and $Q_{\\mathrm{risk}}$ attached, and edges labelled as safety- or task-driven—gives users both comprehension and debugging utility. With $Q_{\\mathrm{risk}}$ estimated from sampled transitions for any fixed policy, safety-critical states can be defined by thresholding $Q_{\\mathrm{risk}}$ (Eq.~6), attacked adversarially by forcing alternative actions at those states, and then patched by a shield that substitutes a safety-policy action when $Q_{\\mathrm{risk}}$ exceeds a threshold (Eq.~7). The evaluation claims that xSRL explanations achieve the best user accuracy for safe agents (77.4%), that an explanation-guided attack on a SAC agent reduces its safety by roughly 72% at a 50% attack rate, and that patching with a separate safety policy and shield restores safety without retraining. The authors conclude that separate-policy safe RL methods produce more interpretable safety behaviour than joint-optimization methods, and that safety-critical decisions become explainable exactly because the risk critic is available.","pith_inferences":["Because the two-critic recipe only assumes a discounted-sum objective, the same graph could carry critics for uncertainty, fairness, or human-specified costs, turning xSRL into a general lens for what an agent worries about.","The shield inherits the risk critic's estimation error, so tightening $Q_{\\mathrm{risk}}$ with off-policy evaluation methods should improve explanation fidelity and shield reliability together; that is a testable prediction the paper does not make.","The explain-attack-patch-re-explain loop points toward a certification protocol: a policy is plausibly 'explanation-certified safe' when no explanation-guided attack at high-$Q_{\\mathrm{risk}}$ states measurably degrades safety, which could serve as a benchmark for safe RL releases."],"forward_implications":["xSRL gives users a testable map of where an RL agent is unsafe, so safety audits reduce to reading high-$Q_{\\mathrm{risk}}$ nodes before deployment.","With the same $Q_{\\mathrm{risk}}$ critic used as both explanation and shield trigger, the cost of adding safety explanations is amortized into actual safety enforcement.","Explanation-guided attacks become a debugging workflow: contrast the graph before and after attack, identify rising $Q_{\\mathrm{risk}}$ values, patch the policy, and re-explain to verify the fix.","Separate-policy safe RL methods (such as AdvExRL) are more explainable than joint-optimization ones (such as SQRL), so interpretability can act as a selection criterion for safe RL algorithms.","Users can identify the safer of two agents under attack with high confidence using the combined graph, supporting its use in operator training and incident review."],"supporting_citations":[{"why":"It introduced the CAPS global policy-summary graph that xSRL extends with Q-task and Q-risk annotations, and it serves as the main global-explanation baseline in the user studies.","marker":"[21]"},{"why":"It supplies the separate safety policy and adversarial policy used in xSRL's attack generation, patching, and policy-selection fidelity evaluation.","marker":"[26]"},{"why":"It provides the post-posed shielding strategy that xSRL reuses, with the Q-risk threshold as the shield trigger.","marker":"[2]"},{"why":"It is one of two separate-policy safe RL methods used as a patching baseline and in user studies; its recovery-zone policy is another source of safety decisions.","marker":"[33]"},{"why":"It supplies the joint-optimization SQRL agent used as a contrasting patching baseline and in the comparison of interpretability across safe RL methods.","marker":"[31]"},{"why":"It provides the Alternative Adversarial Action (AAA) attack used to select adversarial actions at xSRL-identified critical states.","marker":"[32]"},{"why":"It trains the SAC target agent whose policy is explained, attacked, and patched throughout the experiments.","marker":"[9]"},{"why":"It is the reward-decomposition local explanation baseline that xSRL's local Q-critic method is contrasted against.","marker":"[15]"}],"fun_headline_variants":["Explainable RL that attacks its own vulnerabilities and patches them","SRL framework reveals RL risks then lets you fix them without retraining","Attack your RL agent's weak spots then shield it without retraining","Safety through explainability: attack and patch RL agents","From explanations to exploits: xSRL finds RL flaws and fixes them"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole pipeline rests on the learned risk critic $Q_{\\mathrm{risk}}$ being an accurate estimate of expected safety cost for the target policy even under adversarial perturbations and in states the critic was not trained on; if that estimate is wrong, the safety-critical states, the attack targeting, and the shield decisions are all wrong.","fun_headline_variants_meta":{"raw":{"variants":["Explainable RL that attacks its own vulnerabilities and patches them","SRL framework reveals RL risks then lets you fix them without retraining","Attack your RL agent's weak spots then shield it without retraining","Safety through explainability: attack and patch RL agents","From explanations to exploits: xSRL finds RL flaws and fixes them"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001018,"raw_usage":{"total_tokens":4372,"prompt_tokens":1098,"completion_tokens":3274,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":714,"completion_tokens_details":{"reasoning_tokens":3186}},"tokens_in":714,"tokens_out":3274,"duration_ms":20962,"temperature":1.0,"reasoning_tokens":3186,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:42:15.782279+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could run the SAC agent in Navigation 2 with the same 50% attack, but with the $Q_{\\mathrm{risk}}$ critic deliberately corrupted by noise or by training it on trajectories from a different policy, and check whether the set of safety-critical states and the shield's firing behavior change materially; if they do, the safety explanations inherit the critic's error. A cleaner quantitative check is to compare $Q_{\\mathrm{risk}}$'s predicted cumulative cost against the ground-truth cost in held-out episodes under adversarial perturbations, and ask whether its NRMSE (reported between 33% and 63%) is concentrated precisely in the high-risk states used for attack and patching.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It introduced the CAPS global policy-summary graph that xSRL extends with Q-task and Q-risk annotations, and it serves as the main global-explanation baseline in the user studies."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It provides the post-posed shielding strategy that xSRL reuses, with the Q-risk threshold as the shield trigger."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It is one of two separate-policy safe RL methods used as a patching baseline and in user studies; its recovery-zone policy is another source of safety decisions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It provides the Alternative Adversarial Action (AAA) attack used to select adversarial actions at xSRL-identified critical states."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It is the reward-decomposition local explanation baseline that xSRL's local Q-critic method is contrasted against."}],"review_version":1}