{"id":"edaaa6c4-0f09-4ab4-85e1-4df38dc61e65","arxiv_id":"2505.12701","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A reward-penalized TD3 policy generates counterfactual action sequences for continuous-action RL, evaluated on diabetes and Lunar Lander simulators.","lead":"The authors train an RL agent to produce alternative action sequences that increase reward while staying close to the original actions, using a distance penalty inside the TD3 algorithm. It is a proposed tool for explaining continuous-action RL decisions in high-stakes domains like diabetes treatment and lunar landing.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The soft-penalty relaxation in Eqs. (2)-(5) does not enforce G(τ')>G(τ), and the paper's own ρ+ results show the learner often fails to find any improving rollout, leaving the central counterfactual claim unsupported.","rationale":"The paper's central claim requires that generated action sequences are positive counterfactuals: G(τ')>G(τ) with minimal distance. The derivation from Problem 1 to Algorithm 1 weakens this requirement at two points. First, Eq. (2) converts the hard constraint into a soft penalty, so an optimizing trajectory may violate it. Second, Eq. (4) to Eq. (5) replaces an essential supremum over random rollouts of a policy with the expectation, justified only by a generic reference to concentration inequalities; no theorem, bound, or condition is stated. These are not cosmetic: the two objectives are different, and the paper's own Table 1 quantifies the consequence. For diabetes P1 single-environment, ρ+=0.53±0.01 means that for 47% of test trajectories none of the ten generated rollouts improved on G(τ). The reported metric is the fraction of trajectories with at least one valid candidate, not the fraction of candidates that are valid; a method that occasionally samples a good trajectory but mostly outputs non-improving sequences does not 'compute counterfactual trajectories that improve outcomes.' The baseline comparison is not sufficient to rescue this because the same validity issue applies to both arms, and ρ_adv only counts cases where a positive counterfactual exists. A concrete check—computing the per-rollout success rate from the released code—would directly decide whether the soft-penalty policy is generating valid counterfactuals or merely proposing candidates that are mostly invalid. I therefore agree with the reader's assessment that the central claim is not supported, and the REJECT verdict should stand.","tokens_in":10513,"tokens_out":8526,"duration_ms":89914,"concrete_test":"Using the released code, re-run the diabetes single-environment P1 evaluation and, for each test trajectory, compute the per-rollout success rate: among the N_c=10 generated rollouts, what fraction satisfy G(τ')>G(τ). Report this alongside ρ+. If the per-rollout success rate is low (e.g., <0.5) even on trajectories where ρ+ is attained, the method is producing mostly invalid candidates and does not solve Problem 1.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Algorithm 1 'efficiently generates counterfactual trajectories that improve outcomes while minimizing deviations' (Section 6). The load-bearing step is in Section 4.1: Eq. (2) relaxes the hard constraint G(τ')>G(τ) of Problem 1 into a soft penalty, Eq. (4) replaces the trajectory optimization by an essential supremum over realizations of a policy, and Eq. (5) then swaps the essential supremum for an expectation, justified only by an appeal to concentration inequalities. No bound is derived, and the two objectives are not equivalent: a policy maximizing E[G(τ'(μ))−λD] can have low probability of exceeding the specific threshold G(τ). The algorithm then trains this policy and, at evaluation, rolls it out N_c=10 times per test trajectory, reporting ρ+ as the fraction of trajectories with at least one improving rollout. The paper's own results (Table 1) show ρ+=0.53±0.01 for diabetes P1 single-env and 0.44±0.0 multi-env, and 0.31–0.82 across all variants; for a large fraction of test trajectories, no rollout satisfies the defining constraint. A candidate generation method that frequently outputs non-improving sequences does not solve Problem 1 under the paper's own definition of a positive counterfactual trajectory. The soft penalty and expectation swap are therefore not a faithful relaxation: they change the problem from 'find a trajectory with G>G(τ) at minimal distance' to 'maximize expected reward minus λ times average distance,' and the reported metrics only count whether any candidate happens to satisfy the constraint.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a method for generating counterfactual explanations in continuous-action reinforcement learning. The authors formulate Problems 1 and 2 as constrained optimization problems seeking an alternative action sequence that increases cumulative reward while minimizing a relative action distance, possibly subject to state-dependent policy constraints. They relax the hard reward constraint into a soft penalty (Eq. 2), approximate the trajectory-level optimization by a policy optimization over an essential supremum (Eq. 4), and then replace the essential supremum with an expectation (Eq. 5). The resulting objective is optimized with a modified TD3 algorithm (Algorithm 1). The approach is evaluated on diabetes control (UVA/Padova simulator) and Lunar Lander, reporting positive counterfactual percentage (ρ+) and advantage counterfactual percentage (ρadv).","tokens_in":10911,"tokens_out":3862,"duration_ms":37741,"significance":"If the method worked as claimed, it would fill a real gap: counterfactual explanations for continuous-action RL are underexplored, and the paper provides a concrete algorithm with released code and experiments in two domains. However, the significance is undermined by a major derivation gap in the relaxation from the constrained problem to the learned objective, and by experimental results that show the method frequently fails to produce any positive counterfactual. The central claim of 'efficiently generating counterfactual trajectories that improve outcomes while minimizing deviations' is therefore not supported by the evidence presented.","major_comments":[{"comment":"The replacement of the hard constraint G(τ')>G(τ) in Problem 1 with the soft penalty in Eq. (2), and the subsequent substitution of the essential supremum in Eq. (4) by the expectation in Eq. (5), is not a valid relaxation. The authors appeal to concentration inequalities and large deviation theory but provide no bound, theorem, or argument that a policy maximizing E[G(τ'(μ))-λD] will produce trajectories satisfying G(τ')>G(τ) with any guaranteed probability. This is load-bearing because Algorithm 1 optimizes the expected objective, while the evaluation tests the hard constraint.","section":"Section 4.1, Eqs. (2)-(5)"},{"comment":"The reported ρ+ values for the diabetes domain, such as P1: 0.53±0.01 in the single-environment setting and 0.44±0.0 in the multi-environment setting, are only marginally above the baseline (0.44±0.0 and 0.39±0.0) and are far from 1.0. This means that for roughly half of the test trajectories, none of the 10 generated rollouts satisfies G(τ')>G(τ), so the method does not reliably solve Problem 1 under the paper's own definition of a positive counterfactual. The conclusion that the approach 'efficiently generates counterfactual trajectories' is not supported by these results.","section":"Section 5.1, Table 1"},{"comment":"The evaluation never directly reports the action distances D(α(τ), α(τ')) of the generated counterfactual trajectories. The ρadv metric uses a ratio of distances to define advantage, but no absolute distances are presented, so the 'minimizing deviations' component of the central claim in Sections 1 and 6 is not empirically verified. The paper provides no evidence that the generated counterfactuals are proximal to the original actions.","section":"Section 5, Metrics"}],"minor_comments":[{"comment":"Equation (2) uses λ⁻¹ as the penalty weight, while Eq. (3) and Eq. (5) use λ; the relationship between the two parameters is not clearly explained, and the text should clarify that Eq. (2) and Eq. (3) are equivalent up to scaling.","section":"Section 4.1, Eq. (2)"},{"comment":"The notation in Eq. (4) is imprecise: the essential supremum should be written with respect to the random variable τ'(μ) (e.g., ess sup over realizations of τ'(μ)), and the text in the following paragraph should make the domain of the essential supremum explicit.","section":"Section 4.1, Eq. (4)"},{"comment":"The baseline method, which rolls out the baseline policy without additional training, is a weak comparator for counterfactual generation, and the paper would benefit from a discussion of why this is an appropriate reference point given that no existing method targets continuous-action counterfactual RL.","section":"Section 5, Metrics and Baselines"}],"recommendation":"reject","confidential_remarks":"The paper addresses a relevant problem and includes reproducible code, but the central derivation gap in Section 4.1 and the weak empirical results (Table 1) are substantial. The authors would need to reformulate the problem or provide rigorous guarantees to make the central claim defensible; I do not see that as a minor revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper fills a genuine gap — counterfactual explanations for continuous-action RL are underexplored, and framing the task as trajectory optimization with a minimal-deviation objective is the right instinct. The authors are honest about the gap and the code is public, which helps. But the central derivation does not hold up on close reading, and the experimental results are weaker than the text claims.\n\nWhat's new and what works: unlike Tsirtsis et al. and Amitai et al., which are discrete-action, this targets continuous action sequences directly, with a reasonable relative-distance metric (Eq. 1) and a sensible extension to constrained states via an augmented MDP (Section 4.2). The P2 formulation, where certain states must follow a fixed or baseline policy, is a useful practical twist. The code appears to be available, which is real credit.\n\nThe soft spot is load-bearing. Problem 1 asks for a trajectory satisfying the hard constraint G(τ') > G(τ) with minimal distance. Eq. (2) replaces that with a soft penalty, Eq. (4) moves to an essential supremum over policy rollouts, and Eq. (5) swaps the essential supremum for an expectation. The justification for that swap is one sentence citing concentration inequalities, with no bound and no argument that the optimizer of the expectation will, with high probability, produce trajectories that actually beat the observed trajectory. That is not a cosmetic gap; it changes the problem. A policy maximizing E[G − λD] can easily have low probability of exceeding the specific threshold G(τ), and the paper's own ρ+ numbers confirm this: for the diabetes P1 variant, 0.53 single-env and 0.44 multi-env mean that for roughly half the test trajectories, none of the 10 rollouts satisfies the defining constraint. A method that frequently outputs non-improving sequences does not solve Problem 1 as stated.\n\nI also found the baseline too weak. Comparing against a single rollout of the PPO policy in evaluation mode is not a meaningful baseline for a method that can sample many candidates; a trivial noise-added rollout baseline would be fairer. The ρadv metric is clever but only evaluated on trajectories where both methods find a positive counterfactual, and P2-base showing ρadv = 0.0 in the single-env diabetes setting suggests the metric is not telling a clean story.\n\nOn the other hand, the paper is not circular and not incoherent. The training objective is clear, the evaluation separately checks the hard condition, and the authors do not misreport their own metrics — they just overclaim what the relaxation achieves. The derivation gap is central but fixable: post-filtering candidates on G(τ') > G(τ), using a proper constrained optimization approach, or explicitly reporting the probability of improvement would substantially strengthen the work.\n\nThis paper deserves a serious referee: the problem setting is relevant and the empirical setup, while flawed, is real. I would send it to review, with the expectation that the authors need to address the relaxation gap and improve the baseline before acceptance. For my own reading group, it is a maybe — the gap discussion is instructive even if the method is not yet convincing.","headline":"A straightforward TD3-with-a-distance-penalty method for generating counterfactual action sequences; the idea is real but the central relaxation from a constrained problem to an unconstrained expectation is asserted, not proven, and the paper's own metrics show the method often fails to find any improving trajectory.","tokens_in":11387,"tokens_out":803,"would_cite":false,"duration_ms":10711,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A TD3-based method turns counterfactual 'what-if' questions for continuous-action RL into a tractable optimization over action sequences.","keywords":["counterfactual explanations","reinforcement learning","continuous action spaces","TD3","trajectory optimization","interpretability","diabetes control","reward shaping"],"falsifier":"Take any test trajectory and run Algorithm 1 with $\\lambda$ large enough that the learned counterfactual barely changes the actions; if the resulting trajectory does not improve $G$ for a substantial fraction of trajectories while the training objective keeps increasing, the expectation step and soft relaxation do not preserve the constraint. The paper's own diabetes single-environment $\\rho^+$ of 0.53 already indicates that no positive counterfactual is found for about half of the test trajectories.","tokens_in":10274,"feed_emoji":"🤖","tokens_out":5191,"duration_ms":47671,"temperature":0.7,"pith_summary":"The paper sets out to make reinforcement learning policies in continuous action spaces explainable by computing counterfactual trajectories: alternative action sequences that start from the same state, achieve a higher cumulative reward than the observed one, and deviate as little as possible from the observed actions. If it works, a clinician could ask what insulin doses would have produced better glucose control and receive a concrete alternative treatment plan rather than a saliency map. The authors formulate this as a constrained optimization problem, relax the hard reward-improvement constraint into a soft penalty, and solve the relaxed problem by extending the TD3 actor-critic algorithm with a sparse terminal reward equal to the negative action distance. They report experiments in a diabetes simulator and in Lunar Lander in which the method generates counterfactuals for a large share of test trajectories and generalizes across environments.","feed_headline":"New method finds minimal action changes that improve RL outcomes","feed_subtitle":"A TD3-based optimizer explains a policy by generating alternative trajectories with higher reward and minimal deviation.","key_machinery":"The load-bearing object is the soft-penalized objective of Eq. (5), built from the relative distance metric $D$ in Eq. (1), which sums $|a_{t+i} - a'_{t+i}| / (|a_{t+i}| + \\delta)$ over time steps and generalizes to $\\ell^p$ norms for multi-dimensional actions. The paper replaces the essential supremum in Eq. (4) with an expectation, citing concentration inequalities as justification, and then solves the expectation via TD3, adding $-\\lambda D$ as a terminal reward so that the actor learns a deterministic counterfactual policy with minimal deviation. For Problem 2, an augmented MDP removes constrained states and applies the predefined policy $\\pi_c$ inside them, so the constrained variant reduces to Problem 1.","core_discovery":"The central claim is that counterfactual explanations for continuous-action RL can be reduced to a trajectory-level optimization: minimize a relative action-sequence distance $D(\\alpha(\\tau), \\alpha(\\tau'))$ subject to $G(\\tau') > G(\\tau)$, and that this constrained problem can be solved pragmatically by optimizing the expectation of $G(\\tau'(\\mu)) - \\lambda D(\\alpha(\\tau), \\alpha(\\tau'(\\mu)))$ over deterministic policies $\\mu$, with the distance injected as a sparse terminal reward. The paper further claims that constrained states, such as glucose levels below 100 mg/dL where a doctor's policy applies, can be folded into an augmented MDP so that the same algorithm handles the constrained variant unchanged. The evaluation claims that all three variants outperform a rollout baseline on positive counterfactual percentage in both domains, and that the unconstrained variant P1 achieves the best effectiveness and efficiency.","pith_inferences":["The paper's own $\\rho^+$ values (0.53 in single-environment diabetes) imply that for nearly half of the test trajectories no improving counterfactual was found, which suggests the soft-penalty relaxation does not guarantee the hard constraint; a natural extension is a Lagrangian or constrained-policy formulation that explicitly enforces $G(\\tau') > G(\\tau)$.","The concentration-inequality step from Eq. (4) to Eq. (5) is asserted rather than derived; in environments with heavy-tailed returns, expectation-maximizing policies may not produce trajectories with a high essential supremum, so the guarantee could fail precisely where counterfactuals are most needed.","The distance metric is scale-dependent on action magnitudes; applying the method across heterogeneous action scales, such as different insulin sensitivities, may require normalizing per-trajectory distances, which is a testable modification.","One could validate counterfactual plausibility by feeding the generated action sequences back into the environment model to check whether they are realizable and actually improve outcomes, rather than relying only on the learned reward estimate."],"forward_implications":["In diabetes control, a trained counterfactual policy can produce alternative insulin dose sequences that raise cumulative reward while staying close to the original treatment, giving clinicians concrete 'what if' explanations.","Because the output is a deterministic policy rather than single-trajectory edits, the same learned policy generates counterfactuals for many observed trajectories at once.","User-specified constraints, such as a doctor's dosing rule in certain glucose ranges, can be enforced by rebuilding the MDP, so explanations respect clinical or safety rules.","The approach is method-agnostic: trajectories from any RL algorithm can be fed in, so it can explain existing trained policies without retraining them.","Training in multiple environments improves the advantage metric, suggesting that counterfactual generators benefit from diverse training conditions."],"supporting_citations":[{"why":"Supplies the TD3 actor-critic algorithm that the proposed method extends for continuous action counterfactual generation.","marker":"[Fujimoto et al., 2018]"},{"why":"Provides the definition of counterfactual trajectories as alternative action sequences with better outcomes in sequential decision making, which the paper adapts to continuous actions.","marker":"[Tsirtsis et al., 2021]"},{"why":"The UVA/PADOVA Type 1 Diabetes Simulator used for the diabetes evaluation, including virtual patient profiles and glucose-insulin dynamics.","marker":"[Man et al., 2014]"},{"why":"The OpenAI Gym Lunar Lander environment used as the second evaluation domain.","marker":"[Brockman, 2016]"},{"why":"PPO is used to train the baseline policies whose trajectories serve as observed inputs and as the naive comparison baseline.","marker":"[Schulman et al., 2017]"},{"why":"Provides the reward function for glucose control that defines the cumulative reward used in the diabetes experiments.","marker":"[Zhu et al., 2020]"},{"why":"Frames counterfactual explanations as an optimization task over minimal changes, the conceptual basis for the paper's problem formulation.","marker":"[Wachter et al., 2018]"}],"fun_headline_variants":["Minimal-change counterfactuals for continuous RL","Explain RL via least-deviation action sequences","What-if action trajectories for interpretable RL","Counterfactual optimization for continuous action RL","Improve RL outcomes with minimal action edits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that maximizing the expected value of the soft-penalized reward-minus-distance objective will, with high probability, produce trajectories that actually satisfy the original hard constraint $G(\\tau') > G(\\tau)$ and are minimal in distance, even though the hard constraint has been replaced by a soft penalty and the per-trajectory guarantee by an average.","fun_headline_variants_meta":{"raw":{"variants":["Minimal-change counterfactuals for continuous RL","Explain RL via least-deviation action sequences","What-if action trajectories for interpretable RL","Counterfactual optimization for continuous action RL","Improve RL outcomes with minimal action edits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001131,"raw_usage":{"total_tokens":4650,"prompt_tokens":844,"completion_tokens":3806,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":460,"completion_tokens_details":{"reasoning_tokens":3739}},"tokens_in":460,"tokens_out":3806,"duration_ms":29875,"temperature":1.0,"reasoning_tokens":3739,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:28:02.266911+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take any test trajectory and run Algorithm 1 with $\\lambda$ large enough that the learned counterfactual barely changes the actions; if the resulting trajectory does not improve $G$ for a substantial fraction of trajectories while the training objective keeps increasing, the expectation step and soft relaxation do not preserve the constraint. The paper's own diabetes single-environment $\\rho^+$ of 0.53 already indicates that no positive counterfactual is found for about half of the test trajectories.","supporting_citations":[{"cited_title":"Counterfactual explanations in sequential de- cision making under uncertainty.Advances in Neural In- formation Processing Systems, 34:30127–30139,","cited_arxiv_id":null,"evidence_quote":"Provides the definition of counterfactual trajectories as alternative action sequences with better outcomes in sequential decision making, which the paper adapts to continuous actions."},{"cited_title":"The uva/padova type 1 diabetes simulator: new features.Journal of Diabetes Science and Technology, 8(1):26–34,","cited_arxiv_id":null,"evidence_quote":"The UVA/PADOVA Type 1 Diabetes Simulator used for the diabetes evaluation, including virtual patient profiles and glucose-insulin dynamics."},{"cited_title":"Basal glucose control in type 1 dia- betes using deep reinforcement learning: An in silico vali- dation.IEEE Journal of Biomedical and Health Informat- ics, 25(4):1223–1232, 2020","cited_arxiv_id":null,"evidence_quote":"Provides the reward function for glucose control that defines the cumulative reward used in the diabetes experiments."},{"cited_title":"Counterfactual explanations without opening the black box: automated decisions and the gdpr.Harvard Journal of Law and Technology, 31(2),","cited_arxiv_id":null,"evidence_quote":"Frames counterfactual explanations as an optimization task over minimal changes, the conceptual basis for the paper's problem formulation."}],"review_version":1}