{"id":"712b8f39-e928-4281-bca7-099df5e284b4","arxiv_id":"2505.23871","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ADG uses an ambient DDPM to flag corrupted RL transitions, trains a standard DDPM only on the clean subset, then refines the flagged transitions to produce a recovered dataset that improves offline RL policies.","lead":"This paper introduces ADG, a three-stage diffusion pipeline that finds and repairs corrupted samples in offline reinforcement learning datasets before any RL algorithm trains on them. It reports consistent score improvements across MuJoCo, Kitchen, and Adroit benchmarks under random and adversarial corruption, with a theoretical guarantee that is only partially established.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proposition 4.4's SNR proof drops the forward-process noise in Eq. (30), so the detection threshold at ka is theoretically unsupported; the load-bearing guarantee of ADG fails at this point.","rationale":"The pipeline is only as strong as its detector: the denoiser is trained on the subset the detector labels clean, and the recovered dataset inherits any detector miscalibration. Proposition 4.4 is the only theoretical justification for choosing the thresholded norm at ka. The concrete error is in Eq. (30), where the noised input is replaced by \\u221a\\u00af\\u03b1k times the raw sample, discarding the forward diffusion noise \\u03be. This is not a small approximation; it changes the meaning of the model output. Under the ambient DDPM objective, the minimizer predicts the posterior mean of the diffusion noise, which is nonzero for clean data, so the proof's clean-case prediction norm of zero is inconsistent with the training loss. The resulting SNR formula (5) is therefore unsupported, and with it the claimed detection guarantee. I agree with the reader that this is the weakest load-bearing assumption. I also note the internal tension between Theorem 4.3 (requires large ka for small KL under Assumption 4.2) and Proposition 4.4 (requires small ka for high SNR), which the paper only addresses qualitatively. That said, the empirical evidence is broad: Tables 1 and 2 show consistent gains across CQL, IQL, RIQL, DT, and RDT, and Figure 7 suggests the heuristic separation does occur on the tested benchmarks. The concern attacks the proof and the strength of the stated guarantee, not the existence of the empirical effect. A corrected proof, a restriction of the theoretical claim, or an empirical calibration study would resolve it. This matches the reader's CONDITIONAL verdict, so I recommend no change to the verdict.","tokens_in":27630,"tokens_out":13205,"duration_ms":135743,"concrete_test":"Compute the exact SNR(k) in Proposition 4.4 without dropping the forward diffusion noise: for x0 \\u223c N(0,I), set xk = \\u221a\\u00af\\u03b1k (x0 + \\u03b9 Inoise \\u03b5) + \\u221a(1\\u2212\\u00af\\u03b1k)\\u03be, derive the minimizer of the ambient DDPM objective (Eq. 6) in closed form, evaluate E\\u2016\\u03b5\\u03b8(xk,k)\\u2016\\u00b2 for Inoise = 0 and 1 at every k, and compare the resulting ratio with Eq. (5). If the corrected ratio differs from \\u03b9\\u00b2\\u00af\\u03b1k / [(1\\u2212\\u00af\\u03b1k)\\u03c3k\\u00b2] or does not peak at k=ka, then the detection criterion in \\u00a74.3 lacks the theoretical support claimed in the paper.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of ADG is that thresholding the noise-prediction norm at diffusion step ka separates clean from corrupted transitions. The proof of that claim, Proposition 4.4, is not sound. In Eq. (28) the paper defines the prediction target from a clean x0, but in Eq. (30) it substitutes hk(\\u010dx)=\\u221a\\u00af\\u03b1k \\u010dx for the actual forward-process sample xk, dropping the independent forward diffusion noise \\u221a(1\\u2212\\u00af\\u03b1k)\\u03be. With that term retained, xk = \\u221a\\u00af\\u03b1k (x0 + \\u03b9 Inoise \\u03b5) + \\u221a(1\\u2212\\u00af\\u03b1k)\\u03be, and the ambient DDPM's minimizer predicts the posterior mean of the diffusion noise, not the corruption \\u03b5. Consequently the proof's 'noise-free' case, which asserts \\u03b5\\u03b8 = \\u03b4k\\u03b8 with zero norm, is inconsistent with the training objective: for clean data the model outputs a nonzero diffusion-noise prediction. The SNR formula (5) therefore has an incorrect denominator and omits the \\u03be-dependence in the numerator, so the claimed maximum at k=ka is not established. Separately, Theorem 4.3 requires ka large enough that DKL is below the assumed learnability threshold (Assumption 4.2), while Proposition 4.4 wants ka small for high SNR; the paper gives no quantitative reconciliation. Because the clean subset used to train the denoiser is produced by this threshold, the recovery stage inherits the detector's miscalibration without any stated guarantee. The empirical results may still hold, but the paper's advertised 'theoretical guarantees' are not currently supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ADG, a three-stage data-only preprocessing method for offline reinforcement learning under random and adversarial corruption. Stage 1 trains an 'Ambient DDPM' on the partially corrupted dataset; Stage 2 uses the squared Frobenius norm of the noise prediction at a chosen timestep k_a to classify transitions as clean or corrupted; Stage 3 trains a standard DDPM on the predicted clean subset and uses it to refine the predicted corrupted samples before running any offline RL algorithm. Experiments on MuJoCo, Kitchen, and Adroit benchmarks, with state-only and full-element corruption, show consistent improvements for CQL, IQL, RIQL, DT, RDT, and two recent methods, A2PR and NUNO; ablations study the choice of k_a, the threshold zeta, the slice size H, and single- versus dual-model variants.","tokens_in":28056,"tokens_out":15900,"duration_ms":164077,"significance":"If the empirical claims hold, ADG would provide a practical dataset-level layer that can be composed with essentially any offline RL algorithm and consistently outperform dedicated robust variants on corrupted datasets. The paper's strengths are its broad experimental coverage (multiple tasks, corruption types, dataset scales, and algorithms), four-seed averages, detailed ablations, reported compute overhead, and visualizations of detection and recovery. The advertised theoretical guarantees, however, are not established as stated: Proposition 4.4's SNR proof drops the forward-process diffusion noise, Theorem 4.3 rests on an assumption close to the result it claims to prove, and the training objective in Eq. (6) is not clearly the objective implemented in Algorithm 1. These issues are load-bearing because the clean/corrupted partition produced by the detector is what makes the subsequent denoiser training feasible. The empirical program is strong enough to warrant a major revision rather than rejection, but the theoretical section needs substantial rework and the algorithm specification needs to be made internally consistent.","major_comments":[{"comment":"The derivation of the detection SNR omits the forward-process diffusion noise. In DDPM, the noised sample is x_k = sqrt(alpha_bar_k) x + sqrt(1-alpha_bar_k) xi, so the prediction target in Eq. (28) equals xi + [sqrt(alpha_bar_k)/sqrt(1-alpha_bar_k)] * iota * I_noise * epsilon, not the expression in Eq. (30). Consequently, the 'noise-free' case in Eq. (31) does not have a zero prediction target; the standard DDPM objective requires the model to predict the nonzero diffusion noise xi even for clean inputs. The denominator of the SNR formula in Eq. (5) therefore misses the unit-variance xi contribution, and the proof does not establish the claimed maximum at k = k_a. A corrected calculation is needed; if the xi term is retained and the network error variance is assumed constant, the ratio becomes iota^2 * alpha_bar_k / ((1 - alpha_bar_k)(1 + sigma_k^2)), which is still monotone in k under that assumption, but this is not the formula or proof given in the paper.","section":"Section 4.3, Appendix A.3, Eq. (30)"},{"comment":"Assumption 4.2 asserts the very learnability property that Theorem 4.3 is supposed to provide. The theorem only shows that DKL[q(x_k|x_0) || q(check_x_k|check_x_0)] can be made smaller than any prescribed c for k >= k_a, and then invokes Assumption 4.2; the quantification over c in the theorem is not tied to the existential c in the assumption, so the statement is close to a restatement of the assumption. No verifiable condition on the corruption rate, noise scale, or network class is given. In addition, Theorem 4.3 requires k_a large enough for the KL divergence to be small, while Proposition 4.4 requires k_a small for maximal detection SNR; the paper acknowledges a trade-off in Section 4.2 but provides no quantitative reconciliation. Since the detector's output selects the clean subset used to train the denoiser, this gap is load-bearing for the overall method.","section":"Section 4.2, Assumption 4.2 and Theorem 4.3"},{"comment":"The training loss actually implemented is not specified consistently. Equation (6) defines check_tau^k_t as a conditional noising of the observed check_tau^{k_a}_t, namely check_tau^k_t = sqrt(alpha_bar_k/alpha_bar_{k_a}) check_tau^{k_a}_t + sqrt((alpha_bar_{k_a} - alpha_bar_k)/alpha_bar_{k_a}) epsilon, which matches Corollary 4.1. Algorithm 1, however, produces noised elements as tilde_z^k_t = sqrt(alpha_bar_k) check_z_t + sqrt(1 - alpha_bar_k) epsilon, which is the standard forward noising from x0. These are different training objectives unless check_z_t is identified with x0, but the algorithm text says check_z_t is the observed component that 'may or may not contain noise'. The paper needs to state which objective was actually used in the experiments and reconcile it with the ambient theory.","section":"Section 4.4, Eq. (6) versus Algorithm 1"},{"comment":"No theoretical guarantee is stated for the recovery step. The theoretical results concern only the ambient DDPM used for detection; there is no bound on the reconstruction error of the naive DDPM applied to the detected corrupted samples, nor any connection between the recovered dataset and downstream policy performance. Given the abstract's claim of 'theoretical guarantees,' this should be scoped explicitly, or the recovery stage needs its own analysis.","section":"Section 4.4, Stage 3"}],"minor_comments":[{"comment":"Several table cells contain rendering artifacts, e.g., '34.0±13.466.3±15.9', '26.±5.0', and '0.7±0.0 20.3±7.3'; these should be cleaned up so each entry has a single mean and standard deviation.","section":"Tables 1 and 2"},{"comment":"The aggregate improvement percentages quoted for MDP-based algorithms (69.1%) and non-MDP algorithms (17.4%) do not obviously match the per-algorithm averages in Tables 1 and 2; please state the aggregation rule or correct the numbers.","section":"Section 5.2"},{"comment":"The recovery formula uses an unspecified timestep k; the text in Section 4.4 refers to a reverse DDPM process p_phi( tau^{0:k_a}_t ), but Algorithm 1 appears to perform a single denoising step. Please specify the exact reverse procedure and the starting timestep.","section":"Algorithm 1, Step 3"},{"comment":"There are several typos and proofreading issues, including 'revese', 'Appendix a A.3', 'corrputed', 'offine', and 'Datastet'.","section":"Throughout"},{"comment":"The hyperparameter table lists 'Learning Rate (lr) 5' and then a separate 'Learning Rate 1e-4'; please clarify which learning rate applies to which network component.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The empirical contribution is substantial and likely worth publishing after revision, but the theory needs careful repair. In particular, the discrepancy between Eq. (6) and Algorithm 1 should be checked against the actual code during revision; if the implementation follows Algorithm 1, the existing ambient-loss theory does not justify the detector as written."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: the empirical claim is credible and the method is a genuine addition; the theoretical guarantee as written does not survive inspection. The three-stage pipeline—ambient DDPM training on partially corrupted data, noise-prediction-norm detection, then standard DDPM refinement—is a sensible, novel combination. Prior work either assumes clean training data (DMBP) or focuses on action/reward corruption (RDT), so extending ambient diffusion to RL datasets with mixed clean/corrupted transitions is a real step forward. The experiments are unusually broad: MuJoCo, Kitchen, Adroit; state and full-element attacks; random and adversarial corruption; downsampled datasets; ablations over ka, zeta, and H. The consistent improvement across CQL, IQL, RIQL, DT, and RDT, and the comparison against filtered datasets, make the empirical case convincing. I believe the central data-only preprocessing claim will be useful to the community.\n\nThe soft spot is the theory, and it is load-bearing. Assumption 4.2 assumes the ambient DDPM can be effectively learned from the approximated distribution; Theorem 4.3 then proves a KL bound under that assumption. That is close to circular: the theorem's conclusion is essentially the assumption. Worse, Proposition 4.4's SNR derivation in Eq. (30) substitutes h_k(x) = sqrt(alpha_k) x for the forward-process sample x_k and drops the independent forward diffusion noise. With that term kept, x_k = sqrt(alpha_k)(x0 + iota I_noise epsilon) + sqrt(1-alpha_k) xi, so the model's prediction target includes both the corruption and the diffusion noise. The claimed SNR formula is therefore not derived, and the recommendation to threshold at k = ka is theoretically unsupported. Since the denoiser's training set is selected by this detector, the recovery stage inherits any miscalibration. This is a serious flaw in the paper's advertised guarantees, not a minor typo.\n\nThat said, the experimental pipeline is not circular—the denoiser is trained on predicted-clean slices and evaluated on held-out corrupted transitions—so the empirical results may well stand. The fix is straightforward: either correct the proof, replace it with a proper argument, or explicitly present the detection step as heuristic. Missing code release and the sensitivity of the method to ka, zeta, and H are secondary but worth noting.\n\nI would send this to peer review. The empirical contribution and the method itself deserve referee time, but the theory section needs major revision or removal before acceptance.","headline":"ADG's empirical data-recovery pipeline is a useful and novel contribution, but the theoretical guarantees in Proposition 4.4 and Theorem 4.3 do not hold as stated.","tokens_in":28564,"tokens_out":3197,"would_cite":true,"duration_ms":33304,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ADG recovers corrupted offline RL datasets with a two-model diffusion pipeline, letting standard algorithms beat robust baselines.","keywords":["offline reinforcement learning","data corruption","diffusion models","ambient DDPM","dataset recovery","corruption detection","state and action corruption","sequence modeling"],"falsifier":"Take an offline dataset with known corruption labels, train the ambient DDPM at the paper's $k_a = 30$, and measure the false-positive and false-negative rates of the $e_\\theta(\\check{z}_t) > \\zeta$ rule against ground truth at corruption scales $\\alpha \\in \\{0.5, 1.0, 2.0\\}$; if the false-negative rate is not small enough that the clean subset remains effectively noise-free, or if the same $\\zeta$ fails to transfer across corruption scales, then the recovery stage's premise fails.","tokens_in":27430,"feed_emoji":"🛠️","tokens_out":4198,"duration_ms":46874,"temperature":0.7,"pith_summary":"This paper argues that corrupted offline RL datasets can be repaired rather than robustly learned from, by adding a three-stage diffusion preprocessing layer. It introduces ADG, which trains an ambient DDPM on partially corrupted trajectory slices, uses the noise-prediction norm to flag corrupted transitions, trains a standard DDPM on the flagged-clean subset, and then denoises the corrupted transitions. The central claim is that any downstream offline RL algorithm—pessimistic Q-learning, policy-constraint methods, or sequence models—improves on the recovered dataset, and often outperforms its dedicated robust variant, under both random and adversarial state, action, and reward corruption. The theoretical support is that an ambient DDPM trained on approximated distributions can learn the clean data distribution once a sufficiently large diffusion timestep is chosen, and that the detection signal is strongest at that timestep.","feed_headline":"Repaired datasets let plain offline RL beat robust variants","feed_subtitle":"A three-stage diffusion pipeline cleans corrupted trajectories, improving MuJoCo, Kitchen, and Adroit scores across five baselines.","key_machinery":"The paper's central mechanism is the ambient DDPM detector paired with a selectively trained standard DDPM denoiser. The detector uses an ambient-diffusion objective with an ambient timestep $k_a$; Corollary 4.1 shows that minimizing a reweighted prediction loss over forward-noised samples yields the conditional noise predictor $\\mathbb{E}[\\epsilon \\mid x_k]$ for all $k \\ge k_a$. Theorem 4.3 then shows that, for bounded corruption scale, the clean forward distribution and the partially corrupted forward distribution can be made arbitrarily close in KL divergence by choosing $k_a$ large enough, while Proposition 4.4 shows that the detector's signal-to-noise ratio is maximized at $k_a$. The detector flags samples by thresholding $e_\\theta(\\check{z}_t) = \\|\\epsilon_\\theta(\\check{\\tau}_t, k_a)_{H+1}\\|_F^2$ with threshold $\\zeta$; the denoiser is a standard DDPM trained only on the clean subset, using a mask to ignore corrupted positions within trajectory slices, and it refines the flagged corrupted transitions.","core_discovery":"The paper's core claim is that a data-only repair pipeline can make standard offline RL algorithms more robust than algorithms explicitly designed for corrupted data. ADG works in three stages: first, an ambient DDPM is trained on trajectory slices from the partially corrupted dataset, using a loss that provably recovers the noise predictor of the clean distribution even though some training samples are corrupted; second, the squared Frobenius norm of that predictor at a chosen ambient timestep distinguishes corrupted from clean transitions; third, a standard DDPM is trained only on the detected-clean subset and used to renoise and denoise the corrupted transitions, restoring them before RL training. The paper reports consistent improvements across CQL, IQL, RIQL, DT, and RDT on MuJoCo, Kitchen, and Adroit benchmarks, under random, adversarial, and Gaussian corruption, and for missing-data settings, with the recovered datasets often outperforming both filtered datasets and the noised baselines.","pith_inferences":["If the noise-prediction norm cleanly separates clean and corrupted transitions as reported, the same detector could serve as a general data-quality audit for offline datasets, flagging suspicious transitions before any RL training begins.","The two-model recipe may transfer to other partially corrupted time-series domains beyond RL, such as robotics demonstrations or medical sensor logs, wherever clean samples form a large uncontaminated majority and corruption is sparse and additive.","The paper's theory is stated for scaled Gaussian noise, so an immediate testable extension is whether the detection guarantee survives structured or adversarial dropouts, which the missing-data experiments probe only empirically.","Because ADG is data-only, it can be stacked with robust offline RL algorithms rather than competing with them; the paper's ablations suggest this combination should be explored at higher corruption rates than the current $\\eta = 0.3$ setting."],"forward_implications":["ADG can be wrapped around any offline RL algorithm without modifying the algorithm's loss, since it only replaces the dataset that the algorithm consumes.","Standard non-robust algorithms trained on ADG-recovered data outperform their explicitly robust variants on most tested corruption scenarios, suggesting that dataset repair can substitute for robust algorithmic design.","Recovery outperforms simple filtering: filtered datasets lose sequential information and degrade performance, while ADG's recovered datasets restore trajectory structure that MDP-based and sequence-modeling methods both use.","ADG improves performance under state-only, full-element, Gaussian, adversarial, and missing-data corruption, and remains effective across dataset sizes from 1% to 100% and corruption rates up to 0.5.","Separating the detector and denoiser into two independent diffusion models is important; sharing one model creates loss interference and reduces downstream performance."],"supporting_citations":[{"why":"Supplies the ambient-diffusion objective and theory that Corollary 4.1 extends to the discrete DDPM framework; load-bearing for training on corrupted data.","marker":"[8]"},{"why":"Establishes the trajectory-based corruption setup, random and adversarial attacks, and the sequence-modeling baseline RDT that ADG builds on and compares against.","marker":"[34]"},{"why":"Provides RIQL and the corruption settings (rate $\\eta$, scale $\\alpha$) that ADG uses in its experiments, giving the robust baseline it aims to improve upon.","marker":"[37]"},{"why":"Defines the DDPM forward and reverse processes and training loss that both the ambient detector and the naive denoiser build on.","marker":"[14]"},{"why":"Provides the projected gradient descent method used to construct adversarial state and full-element corruption in the experiments.","marker":"[25]"},{"why":"Demonstrates diffusion-based state denoising for offline RL at test time, motivating the use of a standard DDPM to refine corrupted observations during training.","marker":"[38]"},{"why":"Provides the Decision Transformer sequence-modeling baseline whose performance benefits from ADG's trajectory recovery.","marker":"[6]"},{"why":"Supplies the corruption-robust offline RL framework and uncertainty weighting that the paper situates itself against, describing the problem ADG addresses from a data-only angle.","marker":"[40]"}],"fun_headline_variants":["Diffusion pipeline scrubs corrupted offline RL data","Clean up corrupt data, then use any offline RL algorithm","Data repair beats robust RL algorithms on corrupted datasets","Ambient diffusion recovers clean trajectories for offline RL","Standard offline RL outperforms robust variants after data repair"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole pipeline depends on the detector's noise-prediction norm, computed at timestep $k_a$ and thresholded by $\\zeta$, cleanly separating corrupted from clean samples; if it does not, the supposedly clean subset used to train the denoiser is itself contaminated and the recovery stage has no stated guarantee.","fun_headline_variants_meta":{"raw":{"variants":["Diffusion pipeline scrubs corrupted offline RL data","Clean up corrupt data, then use any offline RL algorithm","Data repair beats robust RL algorithms on corrupted datasets","Ambient diffusion recovers clean trajectories for offline RL","Standard offline RL outperforms robust variants after data repair"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000167,"raw_usage":{"total_tokens":1287,"prompt_tokens":1006,"completion_tokens":281,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":622,"completion_tokens_details":{"reasoning_tokens":206}},"tokens_in":622,"tokens_out":281,"duration_ms":3178,"temperature":1.0,"reasoning_tokens":206,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:47:43.247841+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take an offline dataset with known corruption labels, train the ambient DDPM at the paper's $k_a = 30$, and measure the false-positive and false-negative rates of the $e_\\theta(\\check{z}_t) > \\zeta$ rule against ground truth at corruption scales $\\alpha \\in \\{0.5, 1.0, 2.0\\}$; if the false-negative rate is not small enough that the clean subset remains effectively noise-free, or if the same $\\zeta$ fails to transfer across corruption scales, then the recovery stage's premise fails.","supporting_citations":[{"cited_title":"Consistent diffusion meets tweedie: Training exact ambient diffusion models with noisy data","cited_arxiv_id":null,"evidence_quote":"Supplies the ambient-diffusion objective and theory that Corollary 4.1 extends to the discrete DDPM framework; load-bearing for training on corrupted data."},{"cited_title":"Tackling data corruption in offline reinforcement learning via sequence modeling","cited_arxiv_id":null,"evidence_quote":"Establishes the trajectory-based corruption setup, random and adversarial attacks, and the sequence-modeling baseline RDT that ADG builds on and compares against."},{"cited_title":"Towards robust offline reinforcement learning under diverse data corruption","cited_arxiv_id":null,"evidence_quote":"Provides RIQL and the corruption settings (rate $\\eta$, scale $\\alpha$) that ADG uses in its experiments, giving the robust baseline it aims to improve upon."},{"cited_title":"Denoising diffusion probabilistic models","cited_arxiv_id":null,"evidence_quote":"Defines the DDPM forward and reverse processes and training loss that both the ambient detector and the naive denoiser build on."},{"cited_title":"Dmbp: Diffusion model-based predictor for robust offline rein- forcement learning against state observation perturbations","cited_arxiv_id":null,"evidence_quote":"Demonstrates diffusion-based state denoising for offline RL at test time, motivating the use of a standard DDPM to refine corrupted observations during training."},{"cited_title":"Corruption-robust offline reinforcement learning with general function approximation","cited_arxiv_id":null,"evidence_quote":"Supplies the corruption-robust offline RL framework and uncertainty weighting that the paper situates itself against, describing the problem ADG addresses from a data-only angle."}],"review_version":1}