{"id":"ccd68761-00e8-4ec5-ac8b-0e96201d2d2a","arxiv_id":"2512.05291","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"RSA2C is a kernel-based actor-critic that uses RKHS-SHAP state attributions from its value critic to reweight the policy kernel and advantage targets, with a claimed global non-asymptotic convergence bound under perturbations.","lead":"This paper presents RSA2C, an actor-critic algorithm that computes Shapley-value explanations of its learned value function and uses them to reweight state features in the policy's kernel during training. The authors prove a non-asymptotic convergence bound under state perturbations and report competitive returns on two of three continuous-control benchmarks, while acknowledging a plateau far below deep RL on the high-dimensional Ant task.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 2's proof analyzes score ascent, not RSA2C's advantage-weighted update, so the central convergence bound does not apply to the proposed algorithm.","rationale":"The reader's weakest assumption is the Σ^{-1/2} vs Σ^{-1} mismatch in the compatible feature. That is a genuine textual inconsistency and should be fixed, but it is not the deepest flaw: because both features are related by the invertible, w-independent linear map Σ^{1/2}, the compatible function class and the orthogonality condition are actually preserved (E[(Q−\\hat Q)Σ^{1/2}∇logπ]=0 implies E[(Q−\\hat Q)∇logπ]=0), so the proof can likely be repaired locally. The update mismatch, by contrast, cannot be repaired by a simple substitution: the proof's h_{t+1}−h_t=α_t^h \\hat g(h_t) with \\hat g an unbiased score estimator is not the algorithm's advantage-weighted gradient update. The proof of Proposition 1 is also circular (C.3 'we assume a coercivity condition' after deriving F), but this appears unused in Theorem 2 and is a side-claim issue. Therefore REJECT remains the correct verdict, but the decisive reason is the mismatch between the analyzed update and the proposed RSA2C update, not primarily the Σ notation inconsistency.","tokens_in":54272,"tokens_out":16181,"duration_ms":152262,"concrete_test":"Independently re-derive Eqs. (54)-(56) of Appendix C.5 replacing the abstract score estimator \\hat g with RSA2C's actual advantage-weighted estimator from Eq. (18): \\hat g_t = (1/n)Σ_i \\hat A_{w_A}(s_i,a_i)K(s_i,·)Σ^{-1}(a_i−h_t(s_i)). Check whether E[\\hat g_t]=E[g_t]; it equals E[\\hat A_t g_t], which is not E[g_t] unless \\hat A_t≡1. If the equality fails, Theorem 2's telescoping bound applies to a different algorithm. A minimal numerical check on the LQR benchmark in Appendix A.5 (e.g., compare the empirical mean update direction with E[∇logπ] over a batch) would also expose the mismatch.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing problem is not the Σ^{-1/2}/Σ^{-1} notation; it is that Theorem 2's proof analyzes a different update from RSA2C. Algorithm 1 line 5 and Eq. (18) define the Actor step as h_{t+1}=h_t+α_t^h(1/n)Σ_i \\hat A_{w_A}(s_i,a_i)K(s_i,·)Σ^{-1}(a_i−h_t(s_i)): the advantage-weighted score. In Appendix C.5, g(h) is defined as the score ∇_h logπ (right after (51)) and the unprojected step is written as h_{t+1}−h_t=α_t^h \\hat g(h_t) (Eq. (54)), with \\hat g called a 'score estimate'. The identities (56)-(57) and the bias bound (73)-(74) all require \\hat g to be an unbiased estimator of g itself. But the algorithm's \\hat g from Eq. (18) estimates E[\\hat A g], not E[g]; E[\\hat A g] = E[g] only if \\hat A≡1. Thus Eq. (56) is false for the actual update. Worse, D(h)=E_{ν^π⋆}[logπ_h] is an oracle cross-entropy whose gradient is E_{a∼π⋆}[∇logπ_h], while the sampled estimator uses a∼π_h, so even a pure-score update would be biased. Consequently the telescoped bound in Step 3 bounds a score-ascent/oracle algorithm, not RSA2C. Theorem 3 inherits this, so the central non-asymptotic guarantee is unproven for the proposed method.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RSA2C, a kernelized two-timescale actor-critic algorithm in which SHAP attributions computed from the Value Critic are turned into Mahalanobis weights for the Actor kernel and Advantage Critic targets. The Actor lives in a vector-valued RKHS with an operator-valued kernel; the Value and Advantage Critics live in scalar RKHSs with sparse ALD dictionaries. The main theoretical claim is a global non-asymptotic convergence bound under state perturbations (Theorems 2 and 3), decomposed into a perturbation error and a convergence error. Experiments are reported on Pendulum-v1, BipedalWalker-v3, Ant-v5, and a linear-quadratic benchmark, with code released.","tokens_in":54737,"tokens_out":8297,"duration_ms":78720,"significance":"If established, the convergence guarantee would be a notable first for SHAP-guided, kernel-based actor-critic methods under adversarial state perturbations, and the interpretability-through-training idea is timely. The manuscript has strengths: it ships executable code, reports systematic ablations, compares with deep RL baselines, and includes a dedicated LQR experiment with a closed-form optimal value function. However, the central theoretical result is not supported as written: the proof of Theorem 2 analyzes a score-ascent update and an oracle cross-entropy objective, not the advantage-weighted on-policy update actually implemented in Algorithm 1, and the compatible-feature definition is internally inconsistent. These are load-bearing problems for the main contribution.","major_comments":[{"comment":"The proof of Theorem 2 defines the Actor update as h_{t+1}-h_t = alpha_t^h \\hat g(h_t), where \\hat g is called a score estimate of g(h)=∇_h log π_h (Eqs. (51)-(54)). But Algorithm 1 line 5, with the concrete form in Eq. (18), updates h_{t+1}=h_t+alpha_t^h (1/n) Σ_i \\hat A_{w_A}(s_i,a_i) K(s_i,·)Σ^{-1}(a_i-h_t(s_i)). This estimates E[\\hat A g], not E[g]; these coincide only if \\hat A ≡ 1. Consequently the identities in Eq. (56)-(57), the bias bound in Eq. (73)-(74), and the telescoped bound in Step 3 prove convergence of a different algorithm. Moreover, D(h)=E_{ν^{π⋆}}[log π_h] is an oracle cross-entropy whose gradient is E_{a∼π⋆}[∇ log π_h], whereas RSA2C samples actions from the on-policy π_h; even a pure score estimator would be biased for this oracle gradient. Theorem 3 inherits the gap, so the paper's central non-asymptotic guarantee is unproven for the proposed method.","section":"Appendix C.5 (Theorem 2), Eq. (54); Algorithm 1 / Eq. (18)"},{"comment":"There is an internal inconsistency in the compatible-function-approximation argument. Section 3.1 defines the Advantage Critic feature as ν(s,a)=K(s,·)Σ^{-1/2}(a-h(s)) and the associated scalar kernel uses Σ^{-1/2}. The policy score derived in Proposition 2/Eq. (24) and used in the compatibility proof is ∇_h log π_h(a|s)=K(s,·)Σ^{-1}(a-h(s)). Appendix B.3 writes ∇_{w_A} A = ν(s,a)=K(s,·)Σ^{-1}(a-h(s)). Compatible function approximation requires ν = ∇_h log π_h. If the algorithm genuinely uses Σ^{-1/2}, the gradient estimate is biased and the unbiased-policy-gradient identity used in Theorem 2 collapses; if it uses Σ^{-1}, then Eq. (5) and the kernel definition in §3.1 are inconsistent. Either way, the compatibility condition (20) is not satisfied as stated.","section":"Section 3.1, Eq. (5); Appendix B.3, Eq. (20)"},{"comment":"Proposition 1 claims that the Fisher information operator satisfies F(h) ⪰ λ_F I_m, but the proof consists of assuming Eq. (26): 'We assume a coercivity condition that there exists λ_F > 0 such that ...'. This is circular; the proposition does not derive the coercivity from the stated assumptions. Since this proposition is presented as a formal part of the theoretical guarantees and the coercivity of the Fisher operator is asserted rather than established, the statement should either be proved from the assumptions or removed/reclassified as an assumption.","section":"Appendix C.3, Proposition 1 and Eq. (26)"},{"comment":"The proof repeatedly uses that the score estimator satisfies E∥g(h_t)-\\hat g(h_t)∥²_{H_K} ≤ C_b/n_eff (Assumption 4, used in Eqs. (65), (73)-(80)). For RSA2C's actual \\hat g from Eq. (18), which is advantage-weighted, this bound has not been verified and in fact the expectation of the estimator is not the score. Thus Assumption 4 is not an innocuous sampling condition; it is effectively part of the conclusion that needs to be proved for the implemented update. The mismatch makes the sample-complexity claim in Theorem 2 unsupported.","section":"Assumption 4 and Appendix C.5, Eq. (65)"}],"minor_comments":[{"comment":"Algorithm 1 line 5 says '∇_hJ(h,Σ) defined in (4)', but Eq. (4) defines the Gaussian policy, not the gradient. The gradient appears later in Eq. (15)/(18); the cross-reference should be fixed.","section":"Algorithm 1 and Section 3.1"},{"comment":"The statement says 'Under Assumptions 2 and 2' twice. This appears to be a typo; one of the two references should point to a different assumption.","section":"Theorem 1"},{"comment":"The final sentence 'Assumption 3 holds with C_ν = O(...)' conflates two different objects: Assumption 3 concerns Gram-matrix invertibility, while C_ν is the Lipschitz constant of the visitation distribution. Please reword.","section":"Assumption 5"},{"comment":"The caption reads 'onvergence of the result gap'; missing 'C' in 'Convergence'.","section":"Figure 11 caption"},{"comment":"The text says 'Weevaluateonthreecontinuous-controlenvironments...includingPendulum-v1andBipedalWalker-v3.' Ant-v5 is described later but omitted from this list; all three environments should be named.","section":"Appendix A.1"}],"recommendation":"reject","confidential_remarks":"The core theoretical contribution is not established: Theorem 2's proof analyzes a different update than the algorithm, and the compatible-feature definition is inconsistent. These are not presentation-level issues. A viable revision would require a new convergence proof for the actual advantage-weighted update and a consistent compatible-function-approximation construction, which is beyond a local fix in the current manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper pairs RKHS-SHAP attributions with a kernelized two-timescale actor-critic, and that combination is genuinely new relative to prior work (Chau et al. is post-hoc, Lever and Stafford and Yang et al. don't use SHAP-style weights). The empirical part is careful: ten seeds, FLOPs and runtime reported, noise-robustness tables, and the Ant-v5 saturation is disclosed rather than hidden. Code is available. For the algorithmic idea, the ablation on Pendulum and BipedalWalker is plausible and the CME variant does look more stable under noise. Unfortunately, the theory doesn't hold together. The most serious problem is that Theorem 2's proof analyzes score ascent, not RSA2C. Algorithm 1 / Eq. (18) defines the actor update as h_{t+1} = h_t + alpha_t^h * (1/n) sum_i A-hat(s_i,a_i) K(s_i,.) Sigma^{-1}(a_i-h_t(s_i)) - advantage-weighted. But in Appendix C.5 the unprojected step is written as h_{t+1}-h_t = alpha_t^h g-hat(h_t), with g-hat called a score estimate. The bias bounds (56)-(57) and the telescoping in Step 3 treat g-hat as an unbiased estimator of E[g], not of E[A-hat g]. Those are different unless A-hat is identically 1. So the non-asymptotic bound applies to a different algorithm - pure score ascent on D(h) = E_{nu^{pi*}}[log pi_h] - not to RSA2C. Theorem 3 inherits this. That's a load-bearing flaw. Two smaller issues: Proposition 1's proof (C.3) simply states 'we assume a coercivity condition' that is exactly what the proposition claims; there's no derivation. And the Advantage Critic feature is defined with Sigma^{-1/2} in Section 3.1 but the compatibility proof in B.3 uses Sigma^{-1}. If the algorithm uses Sigma^{-1/2}, the unbiased-gradient step is wrong; if it uses Sigma^{-1}, the definition in Section 3.1 is wrong. Either way, the compatibility step doesn't go through as written. The kernel-drift handling of the self-referential SHAP loop is legitimate and worth keeping. The empirical method may be salvageable with a corrected proof or a changed definition, but the current preprint does not support its main theoretical claims. Who this is for: someone working on kernel RL or explainable RL who wants the algorithmic idea and the empirical comparison - not the theory as it stands. A serious referee should see it, because the flaws are subtle and the empirical direction has value, but the manuscript needs major revision before its guarantees can be taken at face value.","headline":"Good idea and honest experiments, but the flagship convergence theorem analyzes a different update than the algorithm actually runs.","tokens_in":778,"tokens_out":1672,"would_cite":false,"duration_ms":46967,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that SHAP state attributions, computed from a learned value function, can be piped directly into a kernelized actor-critic, yielding a two-timescale algorithm that provably converges under state perturbations.","keywords":["Explainable reinforcement learning","Shapley values / SHAP","Reproducing kernel Hilbert spaces","Actor-critic","Kernel methods","State perturbations","Non-asymptotic convergence","Continuous control"],"falsifier":"On the LQR benchmark with a known optimal value function, compute the empirical policy-gradient estimate used by the actual implementation and compare it to the true policy gradient; if the difference does not shrink as the dictionary grows, the compatible-function premise is violated. Alternatively, inspect the Advantage Critic feature in the released code: if it uses Σ^{-1/2} as written in Section 3.1, the convergence theorem in Appendix C.5 does not apply to the implemented algorithm.","tokens_in":54126,"feed_emoji":"🤖","tokens_out":8949,"duration_ms":82133,"temperature":0.7,"pith_summary":"RSA2C puts the Actor, Value Critic, and Advantage Critic in kernel-based function spaces (RKHS) and uses SHAP attributions—computed analytically from the Value Critic via kernel mean embeddings—to rescale state features inside an adaptive Mahalanobis kernel. The point is to make interpretability intrinsic to training: the same attribution scores that explain decisions also shape policy gradients and advantage targets. The paper's central claim is that this attribution-aware mechanism does not compromise learning theory: it derives a global non-asymptotic convergence bound under state perturbations, with the performance gap decomposing into a perturbation-error term and a convergence-error term. If the claim holds, explainable RL no longer has to be a post-hoc afterthought—explanations can be a load-bearing part of the optimization loop, and a non-neural kernel method can match deep RL baselines on low-dimensional continuous control while staying substantially cheaper.","feed_headline":"Attribution-guided actor-critic provably converges under noise","feed_subtitle":"RSA2C makes SHAP explanations a training signal, with convergence guarantees under state perturbations.","key_machinery":"The central object is the adaptive Mahalanobis-weighted operator-valued kernel K(s,s_j) = κ_φ(s,s_j) Σ_K, where κ_φ is a Gaussian kernel whose per-dimension length scales are set by SHAP attributions (floored at a small positive constant). This kernel defines the policy mean h(s) in a vector-valued RKHS and is shared by the Advantage Critic; the SHAP scores themselves come from the scalar-RKHS Value Critic through kernel mean/conditional mean embeddings. The theoretical machinery that carries the convergence proof is the compatible function approximation assumption: the Advantage Critic's feature ν(s,a) must equal the policy score ∇_h log π(a|s) = K(s,·)Σ^{-1}(a−h(s)), so that the critic can","core_discovery":"The paper claims to establish a global, non-asymptotic convergence bound for a two-timescale RKHS actor-critic with SHAP-guided feature weighting under state perturbations. The algorithm estimates state attributions from the Value Critic using two routes—kernel mean embeddings for on-manifold expectations and conditional mean embeddings for off-manifold expectations—then converts them into Mahalanobis weights that modulate both the Actor's kernel and the Advantage Critic's targets. The theory shows the expected optimality gap is bounded by an approximation error, a perturbation term that vanishes as the perturbation magnitude goes to zero, and a convergence term of order O(log²T / T^{1/4}) a","pith_inferences":["Editorial inference: the convergence proof appears to require the Advantage Critic feature to be K(s,·)Σ^{-1}(a−h(s)), but the algorithm definition in Section 3.1 uses K(s,·)Σ^{-1/2}(a−h(s)); a reader or user should verify which feature the released code implements, because the theorem's unbiased-gradient premise holds only for the first form.","Editorial inference: the same Mahalanobis weighting scheme could be lifted to off-policy or deep actor-critics by using kernel approximations such as random Fourier features; the paper's own conclusion gestures in this direction for pixel-based observations.","Editorial inference: the perturbation bound assumes a deterministic, time-invariant adversary; stochastic or adaptive adversaries—common in robust control—would be a natural stress test and may require a different proof technique.","Editorial inference: if the compatible-function gap is resolved, an immediate testable prediction is that attribution-weighted updates should outperform unweighted kernel actor-critic exactly in regimes where state dimensions are heterogeneous in reward relevance; the paper's ablation study already shows this on Pendulum and BipedalWalker."],"forward_implications":["If correct, explainability becomes intrinsic: the SHAP values shown to a user are the exact signals that reweight the Actor's kernel and Advantage Critic, so the explanation is not a post-hoc overlay.","Kernel-based (non-neural) actor-critics can be competitive with deep RL on low-dimensional continuous control, with orders-of-magnitude lower FLOPs and wall-clock time per update.","The convergence bound gives a concrete stability guarantee under bounded state perturbations: performance degrades gracefully with the perturbation magnitude, rather than abruptly.","Without perturbations, the algorithm is claimed to reach ε error with O((1−γ)^{-5} ε^{-4} log²(1/ε)) samples—a finite-time guarantee for a kernel-based actor-critic.","The CME (off-manifold) variant is claimed to be more robust to state noise than the KME variant, because it models feature correlations when imputing missing states."],"fun_headline_variants":["SHAP-guided actor-critic converges under noise","Attribution-aware RL converges with SHAP proof","SHAP-weighted critic ensures provable RL convergence","Noise-robust convergence for SHAP-driven actor-critic","Explainable RL via SHAP-weighted actor-critic"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The guarantee rests on the Advantage Critic using exactly the same feature as the policy's score gradient; the paper's Section 3.1 defines this feature with a half-inverse covariance factor while the Appendix B.3 proof uses the full inverse, so the exact matching condition—and with it the unbiased-gradient argument—is not established for the algorithm as described.","fun_headline_variants_meta":{"raw":{"variants":["SHAP-guided actor-critic converges under noise","Attribution-aware RL converges with SHAP proof","SHAP-weighted critic ensures provable RL convergence","Noise-robust convergence for SHAP-driven actor-critic","Explainable RL via SHAP-weighted actor-critic"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001426,"raw_usage":{"total_tokens":5614,"prompt_tokens":794,"completion_tokens":4820,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":4744}},"tokens_in":538,"tokens_out":4820,"duration_ms":31071,"temperature":1.0,"reasoning_tokens":4744,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T18:25:11.139423+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the LQR benchmark with a known optimal value function, compute the empirical policy-gradient estimate used by the actual implementation and compare it to the true policy gradient; if the difference does not shrink as the dictionary grows, the compatible-function premise is violated. Alternatively, inspect the Advantage Critic feature in the released code: if it uses Σ^{-1/2} as written in Section 3.1, the convergence theorem in Appendix C.5 does not apply to the implemented algorithm.","supporting_citations":[],"review_version":1}