{"id":"ce8c2bf7-07ac-47f8-987e-78a79b03f2df","arxiv_id":"2607.16090","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"DADiff estimates cross-domain dynamics mismatch from diffusion-model latent-state trajectories and uses it for reward modification or data selection in policy adaptation.","lead":"An RL system called DADiff uses a diffusion model's internal denoising steps to measure how different a simulator and a real environment are, then adjusts rewards or filters training data to transfer a policy. It reports better transfer performance than several baselines across MuJoCo control tasks with shifted physics.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 10 replaces the source reverse-process noise ε_src with the forward-sample ε; the KL in Eq. 8 depends on ε_src−ε_tar, so this substitution is invalid unless dynamics are deterministic, and Table I's stochastic experiments are exactly where it breaks.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: Eq. 10 substitutes the forward-process sample ε for the source model's reverse-process noise, so the estimated d is not the KL divergence in Eq. 8 and the bound in Theorem 4.2 no longer directly justifies the algorithm. I agree with this reading and with the CONDITIONAL verdict. The concern is serious but addressable: the authors could correct the estimator, justify the substitution in a restricted deterministic setting, or soften the theoretical claim. Given the empirical results are plausible and the method is clearly described, the appropriate outcome remains CONDITIONAL acceptance with requests for the corrected derivation and supporting experiment, rather than REJECT or ACCEPT. No verdict adjustment is needed.","tokens_in":14049,"tokens_out":8675,"duration_ms":81102,"concrete_test":"Train DADiff with the target domain set identical to the source domain (M_tar = M_src), so the true dynamics mismatch is zero. Train both noise models on the same source data and compute the average d(s,a,s') from Eq. 10 on held-out source transitions. If Eq. 10 faithfully estimates Eq. 8, this self-deviation should be near zero; any substantial positive value is exactly the source irreducibility term ||ε−ε_src||². Run this in both deterministic tasks and the stochastic settings of Table I (e.g., ς=0.03). If the self-deviation is comparable to or larger than the cross-domain deviations used in Figure 2, Eq. 10 cannot be considered the theoretical trajectory deviation, and the paper must either switch to a corrected estimator using ε_src(s'_k,s,a,k) or explicitly treat DADiff as a heuristic.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is the passage in Section IV-B claiming ε_src(s'_k,s,a,k)=ε for source transitions. In DDPM, the source noise model is trained to predict the conditional expectation E[ε|s'_k,s,a], not the particular ε used to construct s'_k via the forward process. For a well-trained source model, ε_src(s'_k,s,a,k) differs from ε almost surely; the difference is the source model's irreducible denoising error. Eq. 8's KL divergence between source and target latent transitions depends on ||ε_src−ε_tar||², but Eq. 10 replaces ε_src with ε, yielding ||ε−ε_tar||². This estimated quantity includes the source model's sampling noise in addition to any target mismatch, so it is not the latent transition mismatch in Theorem 4.2's bound. In deterministic dynamics and the infinite-data/perfect-model limit, the substitution is approximately valid, but the paper explicitly claims robustness in stochastic environments (Table I, ς>0), where the source model cannot perfectly predict ε. Thus the central claim that DADiff's estimated deviation controls the target-performance gap is not established by the current derivation. The empirical results may still be reasonable as a heuristic, but the theoretical justification requires either a corrected estimator using the source model's actual prediction or a repositioned claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DADiff, a diffusion-based framework for online dynamics adaptation in RL. When a policy is trained in a source domain and only limited target-domain interactions are allowed, DADiff estimates the dynamics mismatch by comparing source and target generative trajectories of the next state in a DDPM. Two variants are introduced: DADiff-modify, which penalizes source rewards by the estimated deviation, and DADiff-select, which filters source data by low deviation. The paper presents a theoretical bound (Theorem 4.2) relating the policy performance gap between domains to a generative trajectory deviation, derives an estimator of that deviation (Eq. 10), and reports experiments on MuJoCo tasks with kinematic, morphology, friction, and gravity shifts.","tokens_in":14434,"tokens_out":8870,"duration_ms":80383,"significance":"The idea of using diffusion-model latent trajectories to measure dynamics mismatch is creative and potentially useful; the paper also provides code and a reasonably broad empirical study. If the theoretical bridge between the bound and the implemented estimator were sound, the paper would offer a new principled perspective on off-dynamics RL. However, the central estimator does not actually compute the quantity in the theorem: the substitution of the forward-process noise for the source reverse-process noise is unjustified in stochastic dynamics, and the generative-trajectory construction used in the theory is conflated with the forward-process latents used in the algorithm. The empirical results may still be valuable as a heuristic, but the claimed theoretical justification is not established.","major_comments":[{"comment":"The step in Section IV-B claiming that for source transitions ε_src(s'_k,s,a,k)=ε is not valid. In DDPM, a trained noise model predicts the conditional expectation E[ε | s'_k,s,a,k], not the particular ε sample used in the forward process. Equation (8) requires ∥ε_src(s'_k,s,a,k) − ε_tar(s'_k,s,a,k)∥², while Eq. (10) uses ∥ε − ε_θ^tar(...)∥² with ε being the forward-process sample. For a well-trained source model, ε_src(s'_k,s,a,k) differs from ε almost surely, with the difference equal to the source model's irreducible denoising error. Thus d(s,a,s') in Eq. (10) is not the KL divergence in Eq. (8) and includes source sampling noise beyond any true dynamics mismatch. In deterministic dynamics with a perfect source model the substitution is approximately valid, but the paper explicitly claims robustness in stochastic environments (Table I, ς>0), where the error is non-negligible. To keep","section":"Section IV-B, Eqs. (8) and (10)"},{"comment":"Term (a) is claimed to vanish because the generative trajectories in both domains 'start from the same initial latent state s'_K'. Under the forward-process parameterization actually used in Eq. (10), s'_K = √ᾱ_K s'_0 + √(1−ᾱ_K)ε is a deterministic function of the next state s'_0 and a fresh noise ε. Since P_src(s'_0|s,a) and P_tar(s'_0|s,a) differ, the marginal distributions of s'_K also differ (unless ᾱ_K=0), so DKL(Psrc(s'_K|s,a)∥Ptar(s'_K|s,a)) does not vanish. If instead s'_K is intended to be a common noise prior independent of (s,a), then the s'_k obtained from forward noising of a source tuple in Eq. (10) are not the reverse-process latent states appearing in the theorem. The manuscript needs to state which construction is meant and prove that term (a) is zero under that construction; currently the two constructions are conflated.","section":"Definition 4.1 and Remark after Theorem 4.2"},{"comment":"Even after correcting the ε_src substitution, Eq. (10) uses the trained target model ε_θ^tar in place of the true target transition kernel P_tar that appears in Theorem 4.2. No finite-sample or model-error bound is given, so the quantity actually computed is not formally shown to bound the performance gap. This is a standard approximation in practice, but the wording 'd(s,a,s') can be practically estimated as follows' obscures the additional approximation error. Please add a formal statement of this approximation or explicitly declare that the algorithm is heuristic without a guarantee.","section":"Eq. (10) and Theorem 4.2"}],"minor_comments":[{"comment":"The first line of the proof writes η_Msrc(π)−η_Mtar(π) = γ/(1−γ) E_ρπsrc[ E_Psrc[r(s,a)] − E_Ptar[r(s,a)] ]. This should involve V_{M_tar}^{π}(s') rather than r(s,a), following Lemma 7.1; the subsequent lines correctly use the value function, so this appears to be a typo.","section":"Appendix VII-B, proof of Theorem 7.2"},{"comment":"The method name is written inconsistently as 'DADiff' and 'DADIFF' (e.g., abstract vs. Figure 2). Please unify.","section":"Throughout"},{"comment":"The stochastic-dynamics experiment compares only DADiff-modify with PAR. Adding DARC or VGDF would make the claim of robustness under stochastic transitions more convincing.","section":"Table I"}],"recommendation":"major_revision","confidential_remarks":"The core issue is the mismatch between the theorem's true reverse-process noise and the algorithm's forward-process noise. This is a technical but potentially fixable problem: the authors could train a source noise model and use its prediction in Eq. (10), or honestly reposition the method as a heuristic with the empirical claims appropriately qualified. The paper should not be accepted in its current form because the central theoretical guarantee does not apply to the implemented estimator. The empirical evaluation is reasonable but would benefit from significance testing and additional baselines in the stochastic experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper in one line: it's a neat method with a shaky theoretical core. The idea is to train a target-domain DDPM conditioned on state-action pairs, then use the denoising error on source transitions as a measure of dynamics mismatch, for reward modification or data selection. That's new, as far as I can tell from the cited literature, and the multi-step performance bound in Theorem 4.2 is a legitimate generalization of the PAR bound (K=1). The writing is clear, the method is concrete, and they ship code. Empirically they do a lot: sixteen MuJoCo tasks spanning kinematic, morphology, friction, and gravity shifts, with DARC, VGDF, PAR, SAC-IW, and SAC fine-tuning baselines, plus a hyperparameter study and a stochastic-dynamics robustness check. That's a solid experimental package, and the results look plausible.\n\nThe soft spot is the one the stress test flagged, and it's load-bearing. In Section IV-B they claim that for a source transition, the source reverse-process noise ε_src equals the forward-sample ε. But a trained DDPM noise model outputs the conditional mean E[ε | s'_k, s, a], not the particular ε used to construct s'_k. So Eq. 10 estimates ||ε − ε_tar||², while the KL in Eq. 8 is proportional to ||ε_src − ε_tar||². The difference is the source model's irreducible denoising error. In deterministic dynamics with a perfect model, the substitution is exact; in stochastic environments it is not, and the paper explicitly claims robustness there (Table I, ς>0). So the theorem doesn't currently justify the implemented estimator. The method might still work as a heuristic, and the empirical results suggest it does, but the paper needs to either use the source model's actual output in the estimator, or reposition the theory as applying only to the deterministic case and the algorithm as a heuristic extension.\n\nMinor issues: the appendix proof of Theorem 7.2 has a typo in the first line after Lemma 7.1—it writes r(s,a) where it should write the value function (later lines use V, so it's likely a typo). Also, λ and ξ are task-tuned with no stated defaults, and the parameter study shows performance is sensitive to them, which weakens the practical claim.\n\nBottom line: this deserves a serious referee, not a desk reject, but I would not accept it as-is. The idea is worth engaging, the experiments are reasonably thorough, and the flaw is identified and fixable. I'd suggest the authors either correct the estimator derivation or soften the theoretical claim.","headline":"DADiff has a genuinely new idea—measuring source-target dynamics mismatch with diffusion latent trajectories—but the estimator in Eq. 10 is not the KL divergence from the paper's own Theorem 4.2, so the theoretical justification needs real repair.","tokens_in":14878,"tokens_out":3235,"would_cite":true,"duration_ms":31220,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DADiff claims that in reinforcement learning, the performance gap between source and target domains is controlled by how well a target-domain diffusion model denoises source transitions, and uses that residual to modify rewards or filter da","keywords":["diffusion models","reinforcement learning","dynamics mismatch","domain adaptation","reward modification","data selection","generative trajectory deviation","sim-to-real transfer"],"falsifier":"Run the estimator on two identical domains—same transition dynamics but different random seeds—and compute the deviation for many transitions; any systematically nonzero value shows the residual reflects source-model noise rather than dynamics mismatch, falsifying the claim that the formula measures true generative trajectory deviation.","tokens_in":13922,"feed_emoji":"🎯","tokens_out":3642,"duration_ms":33778,"temperature":0.7,"pith_summary":"The paper tries to establish that generative trajectory deviation—the difference between source and target diffusion trajectories for the same next-state prediction—is the right quantity for online dynamics adaptation. It proves a theorem bounding the target-vs-source performance difference of any policy by the KL divergence between source and target latent-state transitions, then estimates that divergence with a diffusion model trained on limited target data. If correct, dynamics mismatch can be measured without a domain classifier or learned representation: just ask how well the target model denoises a source transition. The method works in two modes, penalizing rewards or selecting low-deviation data, and experiments across kinematic, morphology, friction, and gravity shifts show it matching or beating prior approaches.","feed_headline":"Diffusion residual bounds the cross-domain RL performance gap","feed_subtitle":"A target-trained diffusion model scores each source transition, and the score drives reward fixes or data filtering for faster adaptation.","key_machinery":"The central object is the generative trajectory in a diffusion model: the chain of latent states going from pure noise to the next state, conditioned on the current state and action. Theorem 4.2 decomposes the policy performance gap into KL divergences along this chain. The operational machinery is the estimated deviation formula, which computes a per-transition scalar by summing weighted squared residuals between the forward-process noise and the target-domain noise model at every diffusion step. This scalar is what carries the algorithm, since it can be plugged directly into reward modification or data selection.","core_discovery":"The central discovery is Theorem 4.2: for any policy, the expected-return gap between the source and target domains is bounded by two terms—the deviation of initial latent states and the sum of KL divergences of latent-state transitions between the domains. Because both generative trajectories start from the same initial latent noise, the first term vanishes, so the entire performance gap is controlled by latent transition mismatch. DADiff instantiates this bound with a denoising diffusion probabilistic model: it trains a target-conditioned noise model on target transitions, then for each source transition measures a scalar deviation by the squared residual between the forward-process noise","pith_inferences":["I infer that the same residual estimator could be built with other conditional generative models, such as flow matching, since only the chain of latent states matters; the paper hints at this connection but does not pursue it.","The practical estimator likely mixes source-model irreducibility error into the dynamics-mismatch measure, because it replaces the source reverse-process noise with the forward-process sample; this is my inference, not a claim the paper makes.","The data-selection variant suggests a natural curriculum extension: tighten the deviation threshold as more target data accumulates, a schedule the paper does not explore.","The bound assumes the two domains share the same initial latent state; if the domains ever used different noise schedules or forward processes, term (a) would not vanish and the estimator could miss part of the gap."],"forward_implications":["If the bound is correct, policy performance in the target domain can be improved by penalizing source transitions that the target diffusion model denoises poorly.","Filtering source data to the lowest-deviation quantile is a valid alternative, and the paper finds it outperforms reward modification in certain tasks where low-reward transitions matter.","Measuring mismatch along the full latent trajectory is finer-grained than comparisons at the next-state level alone, which is why the paper argues it beats classifier-based and representation-based approaches.","The bound reduces to prior representation-mismatch bounds when the number of latent states is one, so the analysis generalizes that earlier theoretical guarantee.","The method needs only limited target-domain interactions—the target noise model is trained on the same small budget as the policy—so it fits the online adaptation setting without extra simulator access."],"fun_headline_variants":["Diffusion residual bounds cross-domain RL gap","Latent transition mismatch limits policy transfer","Target diffusion model scores source transitions","Diffusion-driven adaptation closes dynamics gap"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The proof assumes that for source transitions the reverse-process noise equals the forward-process sample, so the estimated deviation equals the theoretical trajectory deviation; if a trained source model predicts noise differently from the actual forward noise, the estimator bundles source-model error into the dynamics-mismatch measure and the bound no longer justifies the algorithm.","fun_headline_variants_meta":{"raw":{"variants":["Diffusion residual bounds cross-domain RL gap","Latent transition mismatch limits policy transfer","Target diffusion model scores source transitions","Diffusion-driven adaptation closes dynamics gap"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000124,"raw_usage":{"total_tokens":944,"prompt_tokens":751,"completion_tokens":193,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":495,"completion_tokens_details":{"reasoning_tokens":142}},"tokens_in":495,"tokens_out":193,"duration_ms":2596,"temperature":1.0,"reasoning_tokens":142,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T21:23:18.426891+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the estimator on two identical domains—same transition dynamics but different random seeds—and compute the deviation for many transitions; any systematically nonzero value shows the residual reflects source-model noise rather than dynamics mismatch, falsifying the claim that the formula measures true generative trajectory deviation.","supporting_citations":[],"review_version":1}