{"id":"df941577-ca82-472f-8d41-1e0511166874","arxiv_id":"2505.20621","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A DP-based certified defense provides lower bounds on expected cumulative reward and per-state action stability for offline RL under transition- and trajectory-level poisoning, with larger certified radii than COPA.","lead":"Offline reinforcement learning policies can be certified against data poisoning by training them with differential privacy and then bounding how much expected reward or per-state actions can change under a limited number of poisoned transitions. The authors show this DP-based certification extends to continuous and stochastic environments, where the previous COPA approach only handled discrete deterministic ones, and they report substantially larger certified radii.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 7.17% trajectory-level certificate depends on an unreported group-privacy budget; standard RDP accounting for DP-FEDA VG with q≈0.033 and σ≈3 over thousands of training steps makes Theorem 4.2's lower bound near zero at r≈70.","rationale":"The reader's weakest assumption is precisely the load-bearing gap: the empirical certificate values in Figure 2 and the 7.17% headline require a DP training mechanism with a small group-privacy epsilon over the full training run, but no privacy budget, training iterations, clipping norm, or reward bound b are reported. My independent check of the order-of-magnitude RDP accounting indicates that the required epsilon is likely enormous for trajectory-level poisoning, which would make the Theorem 4.2 lower bound vacuous. I agree with the reader that the theory is sound but the empirical support is incomplete. The verdict should remain CONDITIONAL: the paper should be accepted or rejected based on whether the authors can supply the missing privacy accounting and a recomputation of the certificates. If the accounting confirms a large epsilon, the central empirical claim would need to be withdrawn or substantially weakened; if the accounting somehow yields small epsilon, the claim stands. The concrete test is the single check that would settle this.","tokens_in":19729,"tokens_out":7322,"duration_ms":84154,"concrete_test":"Recompute the Freeway trajectory-level certificate using the paper's stated settings: D=976 trajectories, q=32/976, sigma=3, and the actual number of training iterations T for DQN (e.g., 10,000 or 100,000). Use the RDP accountant for Algorithm 2 (available in Opacus for group privacy or via direct RDP composition) to compute epsilon(alpha) at the optimal alpha for group size r=70, then plug into Equation (6) with b set to the environment's maximum cumulative reward. If the resulting J^r at r=70 is below 50% of the clean J(pi), the headline claim fails. Repeat for sigma=1,2 and for Breakout at r=75, and also report the same computation for transition-level SGM with the corresponding q=32/2,000,000.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The theoretical framework is internally consistent: Theorem 4.2 follows from the (K,r)-outcomes guarantee via tail integration, and the DKW-based lower bound on J(pi) is standard. The load-bearing problem is the quantitative link between the DP-FEDA VG instantiation and the plotted certificates. To certify a 50% performance bound at 7.17% of Freeway trajectories (r≈70 of 976), the mechanism must satisfy the (K,r)-outcomes guarantee with a small enough group-privacy epsilon. The paper never reports epsilon, T, C, b, or the RDP accountant output for the experiments (Section 5 and Appendix A.2). With q=32/976≈0.033 and sigma=3, even an optimistic per-step RDP bound of about q^2 alpha/(2 sigma^2) = 6e-5 alpha gives roughly 0.006 per step at alpha=100; after 10,000 steps and group size 70, epsilon is on the order of 4,000. The factor e^{-epsilon} then makes the certified lower bound in Equation (6) negligible regardless of the J(pi)/b term. Without the missing privacy accounting, the headline 7.17% claim and the claimed 5x improvement over COPA are not verifiable from the manuscript.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MuCD, a certified defense for offline reinforcement learning against data poisoning. The framework trains policies with differentially private mechanisms (SGM for transition-level guarantees and DP-FEDA VG for trajectory-level guarantees) and then derives two kinds of certificates: action-level robustness, which certifies that the chosen action at a state is unchanged under up to r poisoned transitions or trajectories, and policy-level robustness, which certifies a lower bound on expected cumulative reward under such poisoning. The central theoretical device is an outcomes-guarantee argument that converts differential privacy into bounds on expected rewards (Lemma 4.1, Theorem 4.2) and on inferred action scores (Theorem 4.4). Experiments on Atari (Freeway, Breakout) and MuJoCo (Half Cheetah) claim that the approach certifies no more than a 50% performance drop with up to 7.17% of Freeway trajectories poisoned and that certified action-level radii are about 5 times larger than those of COPA.","tokens_in":19990,"tokens_out":7462,"duration_ms":79120,"significance":"If the empirical certificates are correct, the paper is a meaningful advance: it extends certified poisoning defenses beyond COPA's discrete/deterministic setting to continuous and stochastic environments, and it provides a general reduction from DP outcomes guarantees to policy-level reward certificates. The theoretical skeleton is elegant and, given the (K,r)-outcomes guarantee, the derivations in Lemma 4.1 and Theorem 4.2 are sound; the post-processing argument is standard and correctly identified. The paper also deserves credit for separating transition-level from trajectory-level threat models and for attempting both action-level and policy-level certifications within one framework. However, the quantitative claims hinge on two issues that are not resolved in the manuscript: an apparent sign error in the DKW-based lower bound on clean reward, and the complete absence of the privacy-accounting parameters needed to verify the reported radii. These issues affect the headline results directly, so the current empirical claims are not verifiable from the manuscript.","major_comments":[{"comment":"The DKW lower bound on the expected cumulative reward has a sign error. The text correctly states that the true CDF satisfies F_hat(x) - epsilon <= F_X(x) <= F_hat(x) + epsilon. To lower-bound J(pi) = integral_0^b (1 - F_X(x)) dx, one must use the upper bound on F_X, giving J(pi) >= integral_0^b (1 - F_hat(x)) dx - b*epsilon. As printed, Eq. (8) uses 1 - (F_hat - epsilon) = 1 - F_hat + epsilon, which is an upper bound on 1 - F_X, not a lower bound. Because J r in Eqs. (5) and (6) is increasing in J(pi), this error inflates all policy-level certified lower bounds reported in Figures 2 and 3. The authors need to correct this sign and recompute every policy-level certificate.","section":"Section 4.2, Eq. (8)"},{"comment":"The privacy accounting that connects the DP training mechanisms to the plotted certificates is missing. The experiments report the sampling ratio q (batch size 32 over the dataset), the noise multiplier sigma, the number of policy instances p, the number of rollouts m, and delta = 0.001, but they do not report the Rényi order alpha, the per-step RDP epsilon, the number of training iterations T, the clipping norm C, the learning rate, or the cumulative-reward bound b used in Eqs. (5) and (6). Without these, the headline claim of certifying a 50% performance bound with 7.17% of Freeway trajectories poisoned (Section 5.2) cannot be checked. In particular, for trajectory-level DP-FEDA VG with q about 0.033 and sigma = 3 over thousands of training steps, standard RDP composition would yield a group-privacy epsilon for r = 70 that is very large, making e^{-epsilon} in Eq. (6) negligible. The authors must report the full accountant output at each certified radius and either substantiate the 7.17% and 2.05% claims or revise them.","section":"Section 5 and Appendix A.2"},{"comment":"The abstract's claim that the method produces certified radii '5 times larger' than COPA is not supported by a controlled comparison. COPA certifies action stability along individual trajectories in deterministic environments with truncated horizons, while MuCD certifies expected behavior of a randomized policy. The paper acknowledges the comparison is 'often implicit,' but the abstract and Section 5.1 state the ratio as though it were a direct empirical improvement. Please specify the exact certification object, trajectory-horizon settings, and threat models used for the ratio, or qualify the claim accordingly.","section":"Section 5.1, Table 1 and Figure 1"}],"minor_comments":[{"comment":"There is a typo in the definition of trajectory-level poisoning: the symmetric difference is written as |D ⊖tra D~| but should be |D ⊖trj D~|.","section":"Section 3.1, Definition 3.1"},{"comment":"The text contains a typo: 'noisey' should be 'noisy'.","section":"Section 5.1"},{"comment":"The symbol K is used both for the family of functions and for an individual element of that family (e.g., 'for any K in K'). This makes equations such as Eq. (1) harder to parse; consider denoting an element by a lowercase kappa or another symbol.","section":"Definition 3.5 and surrounding text"},{"comment":"The policy-level certification plots do not label their axes or indicate the scale of the cumulative reward bound b. Adding axis labels and stating b would make the curves interpretable and would also clarify how Eqs. (5) and (6) were evaluated.","section":"Figures 2 and 3"},{"comment":"The abstract says 'up to 7% of the training data poisoned,' while Section 5.2 reports 7.17% for Freeway and 2.05% for Breakout. The abstract should specify that the 7% figure is for Freeway under the trajectory-level threat model, and should state which RL algorithm and noise level produce it.","section":"Abstract and Section 5.2"},{"comment":"The table heading contains a typo: 'Halfcheetha' should be 'HalfCheetah', and the main text's 'Mujoco' should be 'MuJoCo' for consistency.","section":"Table 5 and Appendix A.7"}],"recommendation":"major_revision","confidential_remarks":"The theoretical framework is defensible, but the two load-bearing issues—the Eq. (8) sign error and the missing privacy accounting—mean that the empirical claims in their current form cannot be verified. If the missing accounting, once supplied, shows that the group-privacy epsilon for r = 70 is too large for a nontrivial certificate, then the 7.17% claim and the '5 times larger' comparison should be withdrawn or substantially weakened. The authors should also be asked to recompute all policy-level curves after correcting the DKW sign."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the theoretical core of MuCD is solid and the novelty is real, but the headline empirical claims—the 7.17% trajectory radius and the 5x improvement over COPA—are not verifiable from the manuscript, and with standard RDP accounting they look too optimistic.\n\nThe genuinely new thing here is the observation that expected cumulative reward is a post-processing of a DP-trained policy, so the outcomes guarantee can be pushed from probabilities to expectations by tail integration (Lemma 4.1), giving policy-level certificates in continuous and stochastic offline RL. That is a valid extension, and it is presented carefully. The action-level majority-vote certificates are also a sensible adaptation of standard certified classification machinery. I have no quarrel with Theorem 4.2 or the DKW-based lower bound on J(pi); the citation pattern to COPA and the DP literature is fair.\n\nThe soft spot is the empirical section. The figures and Table 1 do not report the privacy budget epsilon, training iterations T, clipping norm C, or the reward bound b used in Equations (5)-(6). Those are the numbers that make the certificates go. The stress-test arithmetic lands: with q=32/976 and sigma=3, an optimistic per-step RDP bound already gives on the order of 0.006 at alpha=100; over thousands of steps, group privacy for r around 70 pushes epsilon into the thousands, making e^{-epsilon} negligible in Eq (6). If that is what the experiments used, the plotted 50%-at-7.17% certificate cannot come from Theorem 4.2 as written. The comparison with COPA is also indirect—COPA certifies individual trajectories with hard limits on trajectory length and reward scale, while MuCD certifies expected reward—so the \"5x\" claim compares different objects.\n\nThis is not a case where the central idea fails; it is a case where the evidence is incomplete. The appendix even shows empirical attacks beating the certified lower bounds, which is the right direction. I would send this to a serious referee, and I would ask for the full privacy accounting, the reward bound b, and code or data for Figures 2-3. For my own work, I would cite the theory but not the empirical claims. Worth a reading-group slot if you want to discuss how DP certificates can be checked.","headline":"A clean theoretical extension of DP outcome guarantees to offline RL certificates, but the headline empirical claims are unsupported because the privacy accounting and training hyperparameters are never reported.","tokens_in":20537,"tokens_out":3385,"would_cite":true,"duration_ms":38512,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proves that differentially private training of offline reinforcement learning policies yields certified lower bounds on expected cumulative reward that survive poisoning of up to 7.17% of trajectories — roughly 900 times more…","keywords":["offline reinforcement learning","data poisoning attacks","certified robustness","differential privacy","outcomes guarantee","Rényi differential privacy","policy-level certification","action-level certification"],"falsifier":"Recompute the certified curves in Figure 2 using a standard Rényi-DP accountant for the Sampled Gaussian Mechanism with the paper's effective sampling ratio $q \\approx 0.033$ and the reported noise multipliers $\\sigma$ over the full training-iteration count, then convert to group privacy at $r$ equal to 7.17% of Freeway trajectories (about 70 trajectories) and evaluate the Theorem 4.2 floor $e^{-\\varepsilon}(J(\\pi) - b\\delta)$ at that $r$. If the floor falls below a meaningful fraction of clean reward at the 7.17% poisoning level, the headline certified-performance claim is refuted; if it stays near 50%, the claim stands.","tokens_in":19510,"feed_emoji":"🛡️","tokens_out":14866,"duration_ms":121163,"temperature":0.7,"pith_summary":"Offline reinforcement learning trains policies on datasets gathered by someone else, so a poisoned dataset can silently degrade the learned behavior. This paper claims that if the training algorithm injects differential-privacy noise, the resulting randomized policy carries a computable guarantee: no adversary who alters up to r transitions or whole trajectories can push the expected cumulative reward below a certified floor, and no adversary can flip the action chosen at a certified state. The key move is extending differential privacy's 'outcomes guarantee' from probabilities to expected rewards, which turns a privacy property into a performance bound. Empirically the method certifies a 50% performance floor with up to 7.17% of Freeway trajectories poisoned, versus 0.008% for the earlier COPA approach, with action-level certified radii about five times larger. If the claim holds, safety-critical offline reinforcement learning can be deployed with a worst-case performance promise rather than a hope.","feed_headline":"Privacy noise certifies offline RL against poisoning at 7% of data","feed_subtitle":"Certified reward floor of 50% survives ~900× more poisoned trajectories than the prior approach.","key_machinery":"The load-bearing object is the $(K,r)$-outcomes guarantee (Definition 3.5), a group-privacy form of differential privacy stating that for datasets differing in up to $r$ elements, $\\Pr[M(D_1) \\in S] \\le K(\\Pr[M(D_2) \\in S])$, with $K(x) = e^{\\varepsilon}x + \\delta$ for approximate DP and $K(x) = (e^{\\varepsilon}x)^{(\\alpha-1)/\\alpha}$ for Rényi DP. The central move is the Expected Outcomes Guarantee (Lemma 4.1), which converts this bound on probabilities into two-sided bounds on expected values by writing $\\mathbb{E}[M(\\tilde{D})]$ as the integral of the tail probability $\\Pr[M(\\tilde{D}) > t]$ via Fubini's theorem, applying the outcomes guarantee inside the integral, and using Hölder's inequality for the Rényi case. The post-processing property of DP then lets the cumulative-reward function inherit the guarantee, producing Theorem 4.2's certified lower bound; on the training side the guarantees are realized by the Sampled Gaussian Mechanism for transition-level poisoning and the DP-FEDA virtual-gradient method for trajectory-level poisoning, while on the estimation side the clean reward $J(\\pi)$ is lower-bounded from rollouts via the Dvoretzky–Kiefer–Wolfowitz inequality and per-action scores are bounded by the SIMUEM confidence-interval method.","core_discovery":"The paper's central claim is Theorem 4.2: if the offline-RL training algorithm $M$ preserves a $(K,r)$-outcomes guarantee — a differential-privacy-style condition that for any two datasets differing in up to $r$ transitions or trajectories, the probability that $M$'s output lands in any set $S$ changes only through a function $K$ — then any policy $\\tilde{\\pi}$ trained on a dataset poisoned up to size $r$ has expected cumulative reward at least $e^{-\\varepsilon}(J(\\pi) - b\\delta)$ under approximate DP and at least $e^{-\\varepsilon}(b^{-1/\\alpha}J(\\pi))^{\\alpha/(\\alpha-1)}$ under Rényi DP, where $J(\\pi)$ is the clean policy's expected reward, bounded in $[0,b]$. The proof runs through a new lemma that converts the probability-level outcomes guarantee into two-sided bounds on expectations by integrating over tail distributions, then applies the post-processing property of differential privacy so that the reward function inherits the guarantee. Because the bound is stated directly in expected cumulative reward, the certification covers discrete and continuous action spaces and stochastic and deterministic environments alike, which the prior COPA approach (limited to discrete, deterministic settings with exhaustive trajectory search) could not. The same $(K,r)$-outcomes guarantee applied to per-state inferred action scores yields action-level certification of stable actions, with the maximum tolerable poisoning radius located by binary search over privacy parameters.","pith_inferences":["Editorial caution: the paper never reports the privacy budget $\\varepsilon$, the training-iteration count, the gradient clipping norm, or the reward bound $b$ that feed Theorem 4.2, so the headline certified percentages cannot yet be reproduced from the paper alone; a standard Rényi-DP accountant at the reported sampling ratio ($q \\approx 0.033$) and noise multipliers, applied over a full deep-RL ","Because the proof only needs the certified quantity to be a bounded post-processing of the trained policy, the same construction plausibly extends to other safety metrics — probability of entering a bad state, constraint-violation counts, or worst-case episode cost — whenever those are bounded functions of the policy; the paper itself certifies only expected cumulative reward and per-state action ","The framework is agnostic to the DP training mechanism, so tighter privacy accountants or improved noise-injection schemes should enlarge certified radii without touching the certification theorems; the conclusions gesture at adaptive noise, and the same logic applies to any future accountant.","The appendix's empirical attacks land far above the certified floor, which raises a testable question the paper does not pursue: an adversary who spends the allowed $r$ poisonings on the most reward-sensitive trajectories may approach the certified bound, and whether the bound is approached in practice would reveal how tight the worst-case guarantee really is."],"forward_implications":["Any offline RL algorithm that admits a DP training mechanism — SGM for transition-level attacks or DP-FEDA VG for trajectory-level attacks — inherits a computable certified lower bound on expected cumulative reward, with no restriction to discrete actions or deterministic environments.","The two certification layers compose: a state can be certified to keep the same action under poisoning up to $r_t$, while the policy as a whole keeps a reward floor under poisoning up to $r$, protecting both critical decisions and overall performance.","Certified strength is tunable through noise: raising $\\sigma$ enlarges the certified poisoning fraction at the cost of clean reward, and the Rényi-DP arm of the bound is uniformly tighter than the approximate-DP arm in the reported experiments.","At equal certified reward floors, the trajectory-level threat model allows roughly ten times more poisoned transitions than transition-level poisoning, because trajectory corruption is a coarser but more destructive attack.","Against concrete trajectory attacks (random and reward-flipping perturbations) the observed rewards sit well above the certified lower bound, consistent with the certification being a worst-case guarantee rather than a prediction of attack success."],"supporting_citations":[{"why":"COPA, the prior practical certified defense for offline RL; supplies the baseline percentages (0.008% Freeway, 0.0075% Breakout), the 5x-radii comparison, and the discrete/deterministic-only scope this work extends.","marker":"Wu et al., 2022"},{"why":"Defines Rényi differential privacy and its composition behavior, which the paper uses for the tighter RDP arm of the certification.","marker":"Mironov, 2017"},{"why":"The Sampled Gaussian Mechanism, the transition-level DP training algorithm whose RDP guarantee feeds the certification.","marker":"Mironov et al., 2019"},{"why":"DP-FEDA VG, the client-level (here trajectory-level) DP training algorithm adapted for the trajectory threat model.","marker":"McMahan et al., 2017"},{"why":"Foundational differential privacy, including group privacy and post-processing, on which the (K,r)-outcomes guarantee and Theorem 4.2 rest.","marker":"Dwork et al., 2006"},{"why":"Certified robustness via DP; its tail-integral technique is applied in the proof of the expected outcomes guarantee for ADP.","marker":"Lecuyer et al., 2019"},{"why":"SIMUEM, the sampling-based confidence-interval method used to estimate per-action inferred scores for action-level certification.","marker":"Jia et al., 2020"},{"why":"The Dvoretzky–Kiefer–Wolfowitz inequality used to convert finite rollout samples into a 1-delta lower bound on clean expected reward J(pi).","marker":"Dvoretzky et al., 1956"},{"why":"Source of the outcomes-guarantee formulation that the paper generalizes from classification to expected cumulative reward.","marker":"Liu et al., 2023"}],"fun_headline_variants":["Offline RL certified against poisoning: 5x larger radius, 7% tolerance","DP-based certification shields offline RL from data poisoning at scale","Multi-level defense: offline RL withstands 7% poisoned data with 50% reward floor","Certified robustness for offline RL: from discrete to continuous with DP","Offline RL poisoning defense: certifying rewards and actions via privacy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole certification rests on one premise — that the noisy training procedures deliver a genuine differential-privacy guarantee with a small enough privacy loss over the entire deep-network training run, since a large privacy loss would collapse the certified reward floor toward zero and take the headline poisoning percentages with it.","fun_headline_variants_meta":{"raw":{"variants":["Offline RL certified against poisoning: 5x larger radius, 7% tolerance","DP-based certification shields offline RL from data poisoning at scale","Multi-level defense: offline RL withstands 7% poisoned data with 50% reward floor","Certified robustness for offline RL: from discrete to continuous with DP","Offline RL poisoning defense: certifying rewards and actions via privacy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000853,"raw_usage":{"total_tokens":3747,"prompt_tokens":1022,"completion_tokens":2725,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":638,"completion_tokens_details":{"reasoning_tokens":2626}},"tokens_in":638,"tokens_out":2725,"duration_ms":17664,"temperature":1.0,"reasoning_tokens":2626,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:51:30.953992+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the certified curves in Figure 2 using a standard Rényi-DP accountant for the Sampled Gaussian Mechanism with the paper's effective sampling ratio $q \\approx 0.033$ and the reported noise multipliers $\\sigma$ over the full training-iteration count, then convert to group privacy at $r$ equal to 7.17% of Freeway trajectories (about 70 trajectories) and evaluate the Theorem 4.2 floor $e^{-\\varepsilon}(J(\\pi) - b\\delta)$ at that $r$. If the floor falls below a meaningful fraction of clean reward at the 7.17% poisoning level, the headline certified-performance claim is refuted; if it stays near 50%, the claim stands.","supporting_citations":[{"cited_title":"Calibrating Noise to Sensitivity in Private Data Analysis","cited_arxiv_id":null,"evidence_quote":"Foundational differential privacy, including group privacy and post-processing, on which the (K,r)-outcomes guarantee and Theorem 4.2 rest."},{"cited_title":"Asymptotic Minimax Character of the Sample Distribution Function and of the Classical Multinomial Estimator","cited_arxiv_id":null,"evidence_quote":"The Dvoretzky–Kiefer–Wolfowitz inequality used to convert finite rollout samples into a 1-delta lower bound on clean expected reward J(pi)."},{"cited_title":"Enhancing the Antidote: Improved Pointwise Certifications Against Poisoning Attacks","cited_arxiv_id":null,"evidence_quote":"Source of the outcomes-guarantee formulation that the paper generalizes from classification to expected cumulative reward."}],"review_version":1}