{"id":"bbcaefa2-3f85-4a3a-80b1-bec5424798bf","arxiv_id":"2506.03154","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Guidance-first diffusion training, which trains and freezes the value guidance before the policy, improves sample efficiency and enables cross-algorithm reuse of guidance modules in offline RL.","lead":"This paper proposes training the reward guidance network first, then freezing it to guide a separate diffusion policy in offline reinforcement learning. The authors find this modular setup speeds learning and lets guidance from one algorithm be reused with another.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central modularity claim rests on an unproved assumption that a pretrained Q-gradient is positively correlated with the target diffusion policy's improvement direction; the theory in Sec. 3.2 assumes this, and the transfer experiments cover only two hybrids on an unspecified task set.","rationale":"The reader's weakest assumption correctly identifies the core load-bearing premise: guidance is independent of the diffusion policy's behavior. My reading confirms that the paper's theoretical justification in Section 3.2 assumes exactly the positive correlation that needs to be proved, and the empirical transfer evidence is too thin to carry the weight. I do not reject the paper because the idea is plausible and there are some supporting ablations: the guidance-necessity experiments, the variance-reduction result with independently seeded guidance, and the statistical tables on several tasks are real evidence. However, the central cross-module transferability claim, which is arguably the most novel contribution, is supported by only one figure, no per-task table, and an inconsistency about whether the hybrid outperformed or merely matched the DQL baseline. The proposed concrete test would settle whether the invariance holds across tasks and whether the Q-gradient alignment condition is actually satisfied in the regimes where transfer works. Since the reader already assigned CONDITIONAL, my concern reinforces that verdict rather than changing it; the path to acceptance is to supply the missing experiments and either repair or qualify the theoretical claim.","tokens_in":10076,"tokens_out":5469,"duration_ms":60512,"concrete_test":"Re-run the plug-and-play experiment from Section 4.4 on all 8 PyBullet tasks with at least 10 seeds, and for every checkpoint compute the cosine similarity between the source Q-gradient and the target diffusion policy's own Q-gradient at actions sampled from the target diffusion policy, alongside true episodic returns of the hybrid policy. If the hybrid matches the DQL baseline even when cosine similarity is near zero or negative, the independence claim survives. If hybrid performance tracks the cosine similarity, the transfer result is explained by gradient alignment rather than by module independence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that a guidance module can be trained independently of the diffusion policy and then frozen, reused, or cross-combined, because guidance depends only on offline data. For this to be true, the Q-function's gradient must point toward genuinely better actions on the support of actions actually generated by the target diffusion policy, and the Q-values must not encode policy-dependent information. Neither condition is established. Section 3.2's 'Observation' explicitly assumes cos(theta_t) > 0 between the Q-gradient and the ideal sampling direction, then Eq. (11) only shows that Q_phi increases along the guided path, not that true expected return increases. Eq. (12), which says cumulative noise vanishes in expectation, does not imply convergence to an optimal point; with fixed step size and stochastic gradient noise, the iterate typically converges to a noise ball. Moreover, DQL's Q-network is not policy-independent: its temporal-difference targets use actions sampled from the diffusion policy (Eqs. 3-4), so a Q-function trained under one diffusion policy is not automatically a valid value estimator for another. The only direct evidence for cross-module transfer is Figure 2b, which reports two hybrid combinations without a per-task breakdown, without confidence intervals across the full benchmark, and with an internal contradiction: Section 1 says the hybrid 'outperformed both' models, while Section 4.4 says it 'matched the performance baseline of DQL'. This leaves the load-bearing invariance assumption under-supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes \"Guidance-First Diffusion Training\" (GFDT), a modular training scheme for diffusion-based offline RL in which the reward-guidance (Q-function) module is trained first as a value estimator, frozen, and then used to guide a separately trained diffusion policy via classifier-free guidance. The authors claim three findings: guidance is not equally important at all training stages; freezing a pretrained guidance module accelerates training and improves final performance; and guidance modules can be transferred across different diffusion-policy algorithms (e.g., IDQL guidance with DQL diffusion) without retraining, reducing variance and sometimes improving early learning. A short theoretical argument in Section 3.2 is offered, and experiments are run on eight PyBullet D4RL tasks with three baseline algorithms (EDP, DQL, IDQL).","tokens_in":10533,"tokens_out":4324,"duration_ms":41269,"significance":"If the central modularity claim is correct, the paper would establish a practical and conceptually useful property: diffusion policies and their reward-guidance modules can be trained independently and recombined, which could reduce peak memory, enable reuse of value estimators, and improve sample efficiency and stability. Strengths of the manuscript include its explicit focus on an underexplored architectural design space, the use of multiple seeds and checkpoint-based evaluation, and the attempt to provide a theoretical rationale for decoupled training. However, the theoretical argument in Section 3.2 does not currently justify the central claim, and the empirical reporting has important gaps (no absolute normalized scores, post-hoc exclusion of Medium-Replay, contradictory statements about the plug-and-play hybrid). These issues are load-bearing because the paper's title and abstract promise a general principle of modular diffusion policy training.","major_comments":[{"comment":"The theoretical justification for independent guidance rests entirely on the assumption, stated as an 'Observation', that the Q-gradient is positively correlated with the ideal sampling direction (cos(theta_t) > 0). Eq. (11) only shows that Q_phi increases along the guided path, not that the true expected return improves. Eq. (12) shows only that zero-mean noise accumulates to zero in expectation; with a fixed step size and persistent gradient noise, the standard conclusion is convergence to a noise ball, not to an optimum. The manuscript never verifies the cos(theta_t) > 0 condition for the trained Q-functions and target diffusion policies, and a Q-function accurate on the behavior distribution can be misleading off-distribution. This is a central load-bearing point: please either prove a sufficient condition from the offline-data support assumptions or measure the cosine similarity empirically over denoising trajectories.","section":"Section 3.2, Eqs. (10)-(12)"},{"comment":"The abstract and Section 1 claim that guidance depends only on offline data and is independent of the policy module's behavior. This is not true for DQL: the Q-network is trained with temporal-difference targets that use actions sampled from the current diffusion policy, as shown in Eq. (3) and Eq. (4). The DQL Q-function therefore encodes information about the action distribution of the policy with which it was co-trained. The plug-and-play experiments in Figure 2b use IDQL guidance with DQL diffusion and vice versa, but they never test the case where a DQL-trained guidance module is transferred to a different diffusion policy, which is exactly the case where the independence premise fails. Please clarify which guidance modules satisfy the stated policy-independence property and provide direct evidence for that case.","section":"Section 2.2, Eqs. (3)-(4), and Section 4.4"},{"comment":"Absolute normalized scores are never reported; Tables 1 and 2 report only relative gains, and the relation between the two tables is unclear. Table 2 contains a row without a dagger despite the caption saying 'All differences marked with dagger are statistically significant', which is internally inconsistent. More importantly, the Medium-Replay environment is omitted 'due to significant performance degradation', and this exclusion appears to be made after observing the results. Medium-Replay is a standard D4RL data quality, so excluding it undermines the claim of consistent improvement. The note that the problem 'can be addressed by decreasing the ratio of loss_reward/loss_behavior_clone' also suggests task-specific tuning, which conflicts with the statement in Section 4.1 that no training-related components were modified. Please report full per-task absolute scores with standard deviations, clarify the table comparisons, and either include all tasks or justify the exclusion on a priori grounds.","section":"Tables 1-2 and the Medium-Replay note"},{"comment":"The paper gives contradictory descriptions of the plug-and-play hybrid: Section 1 says the forcibly combined IDQL-guidance/DQL-diffusion model 'outperformed both the standalone IDQL and DQL models', while Section 4.4 says it 'achieved final performance comparable to the DQL baseline'. Figure 2b is described without a per-task breakdown, confidence intervals, or seed counts. In addition, Section 4.4 claims that the composition 'requires only that the guidance delivers E[r(s,a)] with epsilon < 0.1 (per Theorem 1)', but Theorem 1 (from Fujimoto et al.) is about convergence of BCQL and contains no such epsilon condition. Please reconcile the contradictory statements, give the actual measured values, and either cite a theorem that supports the epsilon claim or remove it.","section":"Section 4.4 vs. Section 1"}],"minor_comments":[{"comment":"The experiments are run on PyBullet D4RL environments, but the text says all models are trained using the 'D4RLMuJoCoTD Dataset'; please clarify the dataset/environment naming.","section":"Section 4.1"},{"comment":"The text says 'removing either guidance degrades performance' but earlier in the same section states that no-guidance diffusion improves faster in the early stage; please specify which stage and which algorithm each statement refers to.","section":"Section 4.2"},{"comment":"The metric 'Early Gain' is not defined anywhere in the text; please define it and explain how it is computed.","section":"Tables 1-2"},{"comment":"The sentence beginning 'Recent work by [Wang et al., 2024]...' is incomplete and the text ends mid-word with 'Because the lin'; please fix this truncated sentence.","section":"Section 6"},{"comment":"The captions 'DDQL Double Guidance' and 'Plug-and-Play' do not describe axes, tasks, number of seeds, or aggregation; please provide full captions.","section":"Figure 2"},{"comment":"The definitions of 'Same Model' and 'Different Guidance' should state whether the guidance is pretrained, frozen, and whether it is used during training or only inference; the row label 'policy configurations' also appears to refer to groups rather than policies.","section":"Table 3"},{"comment":"The text says Theorem 1 'motivates our design choice: pre-training a guidance policy on offline data', but BCQL is a batch-constrained Q-learning method, not a guidance policy; please rephrase to avoid conflating the two.","section":"Section 3.1"},{"comment":"The paper alternates between 'guidance module', 'classifier', and 'Q-network'; please use consistent terminology and clearly define each term at first use.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and potentially useful question, and the empirical setup (multiple seeds, checkpoint evaluation, public code base) is a reasonable foundation. However, the central modularity claim depends on an assumption about Q-gradient alignment that is not tested, and the reporting choices (relative-only numbers, post-hoc exclusion, contradictory statements about the hybrid) currently make it difficult to assess whether the claimed effect is real. I believe the paper can be revised to address these points, but it needs substantial additional experiments and a rewritten theory section before it is ready for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a plausible and genuinely novel training recipe for diffusion-based offline RL, but the paper overclaims what it proves. The core idea—train the guidance module first, freeze it, then train the diffusion model—is not in the cited literature, and the ablations showing that a pretrained frozen guidance speeds up early training are worth taking seriously. The cross-module swap (IDQL guidance with DQL diffusion) is an interesting empirical observation, though the paper reports it inconsistently: Section 1 says the hybrid outperformed both, Section 4.4 says it matched DQL. That internal contradiction needs to be resolved.\n\nThe best part of the paper is the careful ablation of guidance timing and the Double Guidance variance experiment. The IQR reduction of 86% is striking if it holds. But the reader can't easily check the numbers because absolute normalized scores are never reported, only relative gains and AUC peaks, and the Medium-Replay tasks are dropped after observed degradation. That's selective reporting and it needs fixing.\n\nThe theory in Section 3.2 doesn't bear the load. It assumes the Q-gradient is positively correlated with the ideal sampling direction (cos theta > 0) and then shows Q increases along the guided path—not that true expected return increases. Eq. (12) about noise averaging is not a convergence argument. This is more a sketch than a proof, and the paper should label it as such. Also, the modularity claim rests on the assumption that the Q-function depends only on offline data, but DQL's Q is trained with TD targets that sample actions from the diffusion policy, so it is policy-dependent. That doesn't disprove the empirical result, but it means the paper's invariance premise is unverified.\n\nThe citation pattern looks okay; they build on DQL, IDQL, EDP, and cite related modular work. No obvious missing major reference, though the related work section is brief.\n\nWho is this for? Practitioners in offline RL who want a cheap training recipe and know to treat theoretical claims skeptically. It deserves peer review because the empirical claim, if reproduced, would be useful. I'd send it out, but with a strong request for code, data, absolute scores, and a rewritten theory section that honestly states the assumptions.","headline":"A genuinely novel training order with some real ablations, but the load-bearing modularity claim is supported more by assertion than evidence; worth reviewing, needs major revision.","tokens_in":10880,"tokens_out":2561,"would_cite":false,"duration_ms":22418,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that in offline RL, the reward-guidance module and the diffusion policy can be trained separately and recombined freely, making diffusion policies modular.","keywords":["offline reinforcement learning","diffusion policies","classifier-free guidance","modular training","guidance-first training","Q-learning","cross-module transferability","D4RL benchmark"],"falsifier":"Measure the cosine similarity between ∇a Qϕ(s,a) and the true action-improvement direction on held-out states for a diffusion policy that was not used to train Qϕ; a value that is not reliably positive across the action distribution would refute the transferability claim.","tokens_in":9870,"feed_emoji":"🧩","tokens_out":8044,"duration_ms":67114,"temperature":0.7,"pith_summary":"This paper tries to establish that in offline reinforcement learning, the reward-guidance module of a diffusion policy—the Q-function that steers sampling toward high-value actions—can be trained independently of the diffusion model, frozen, and then reused to guide a different diffusion model with no joint training. The motivation is that guidance depends only on the offline dataset (state, action, reward), not on the policy's own actions, so joint training is an unnecessary source of early-stage noise. The paper proposes Guidance-First Diffusion Training (GFDT): train the guidance module first, freeze it, and use classifier-free reward guidance during diffusion training. It reports across PyBullet D4RL tasks that this accelerates convergence, improves peak and area-under-curve performance (e.g., up to 11.9% peak and 25.2% AUC gains), cuts peak memory, and—by swapping in a differently seeded guidance module at inference—reduces reward variance (86% lower IQR). The payoff if true is a modular, reusable pipeline where value estimators and diffusion policies are independent building blocks.","feed_headline":"Freeze the guide first: diffusion RL modules swap freely","feed_subtitle":"Train the value estimator on offline data, freeze it, and reuse it to guide any diffusion policy—no joint training.","key_machinery":"The central mechanism is Guidance-First Diffusion Training (GFDT), which decouples the Q-value guidance module from the diffusion policy. The guidance module is a Q-function Qϕ(s,a) trained on the offline dataset, frozen, and then injected into the denoising loop as a gradient perturbation: ϵθ(at,s,t) ← ϵθ(at,s,t) + α∇a_t Q(s,a_t). A second component, Double Guidance, uses a separately seeded copy of the same architecture at inference time to break the self-reinforcing bias loop between a policy and its own value estimates. The theoretical anchor is a positive-correlation argument: if ∇aQϕ is positively correlated with the ideal sampling direction, then small-step guided updates increase expected Q, and injected Gaussian noise averages to zero over many steps, so pretrained guidance can steer an untrained or differently trained diffusion model.","core_discovery":"On its own terms, the central discovery is that a Q-value guidance module in diffusion-based offline RL is policy-independent: because both the guidance and the diffusion policy are learned from the same fixed dataset, the guidance signal can be computed from (s, a, r) data alone and does not need to be co-trained with the denoiser. The paper demonstrates this through GFDT, where the Q-network is trained first, frozen, and then used to perturb the denoising prediction as ϵθ(at,s,t) ← ϵθ(at,s,t) + α∇a_t Q(s,a_t). It further shows that the decoupling can be pushed to inference: plugging a guidance module trained for IDQL into a DQL diffusion model—two components that never trained together—yields performance at least matching the better baseline, while swapping in a separately seeded but architecturally identical guidance module at inference reduces variance (IQR down 86%).","pith_inferences":["If guidance is truly policy-independent, one could maintain a library of pretrained Q-modules per task and swap them freely as improved value estimators are developed; the paper gestures at this but does not build it.","The positive-correlation argument implies a measurable boundary condition: modular transfer should fail when the guided diffusion model samples actions where the Q-gradient is inaccurate, so testing on high-entropy or out-of-distribution policies would delimit the method's validity.","The two-seed variance reduction suggests a natural extension to ensembles: averaging gradients from several independently seeded guidance modules may reduce variance further than a single swap, a variant the paper does not test."],"forward_implications":["Value estimators can be trained once on an offline dataset and reused across multiple diffusion policies, turning guidance into a shared, frozen asset.","Training memory drops because the guidance module is not updated during diffusion training, and convergence is faster because early training uses an already-accurate reward signal.","Diffusion policies and guidance modules become interoperable across algorithms: an IDQL-trained guidance can steer a DQL-trained diffuser, so new policy architectures can inherit existing value estimators.","Seed-to-seed stability improves by decoupling the guidance used in training from the guidance used at inference, suggesting that feedback coupling between value estimates and policy is a source of variance."],"supporting_citations":[{"why":"Supplies the classifier-free guidance mechanism that the paper decouples.","marker":"Ho and Salimans [2022]"},{"why":"Defines DQL, the Q-guided diffusion baseline that GFDT modifies.","marker":"Janner et al. [2022b]"},{"why":"Provides IDQL, the inference-only Q-guidance precedent and the source of the transferable guidance module.","marker":"Hansen-Estruch et al. [2023a]"},{"why":"Introduces EDP, the one-step denoising method whose stronger dependence on guidance is used as evidence for the guidance role.","marker":"Kang et al. [2024]"},{"why":"Motivates the Double Guidance variance-reduction design through double Q-learning's bias-correction idea.","marker":"van Hasselt [2010]"},{"why":"Gives the batch-constrained convergence theorem used to justify pretraining guidance within the offline data support.","marker":"Fujimoto et al., 2019"},{"why":"Supplies the open-source implementation and untouched hyperparameters that make the plug-and-play comparisons possible.","marker":"Dong et al. [2024]"},{"why":"Defines the D4RL benchmark and datasets on which the empirical claims are evaluated.","marker":"Fu et al., 2020"}],"fun_headline_variants":["Modular diffusion RL: swap guidance modules across algorithms","Train guidance first, freeze, then guide any diffusion policy","No joint training needed: modular diffusion for offline RL","Guidance-first diffusion training cuts variance by 86%","Decouple then recombine: modular diffusion RL training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a Q-function trained on offline data gives gradients that remain accurate and positively correlated with reward improvement for whatever diffusion policy it is later attached to, even though it never saw that policy during training.","fun_headline_variants_meta":{"raw":{"variants":["Modular diffusion RL: swap guidance modules across algorithms","Train guidance first, freeze, then guide any diffusion policy","No joint training needed: modular diffusion for offline RL","Guidance-first diffusion training cuts variance by 86%","Decouple then recombine: modular diffusion RL training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000207,"raw_usage":{"total_tokens":1451,"prompt_tokens":1047,"completion_tokens":404,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":663,"completion_tokens_details":{"reasoning_tokens":325}},"tokens_in":663,"tokens_out":404,"duration_ms":4010,"temperature":1.0,"reasoning_tokens":325,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:11:24.473252+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the cosine similarity between ∇a Qϕ(s,a) and the true action-improvement direction on held-out states for a diffusion policy that was not used to train Qϕ; a value that is not reliably positive across the action distribution would refute the transferability claim.","supporting_citations":[{"cited_title":"Coupling OpenFOAM(R) with BoSSS, a discontinuous Galerkin solver written in C#","cited_arxiv_id":"2310.03573","evidence_quote":"Introduces EDP, the one-step denoising method whose stronger dependence on guidance is used as evidence for the guidance role."},{"cited_title":"Double q-learning","cited_arxiv_id":null,"evidence_quote":"Motivates the Double Guidance variance-reduction design through double Q-learning's bias-correction idea."}],"review_version":1}