{"id":"91b8f662-314e-44af-971a-673f2521b3b9","arxiv_id":"2510.08233","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"DMPO improves diffusion LLM reasoning by matching the model's distribution to the reward-tilted target via importance-weighted denoising cross-entropy plus weight-baseline subtraction, beating GRPO-style baselines on planning benchmarks without task SFT.","lead":"Diffusion large language models write text all at once instead of one word at a time, and this paper gives them a new reinforcement-learning method that trains the model to reproduce the whole distribution of good answers rather than chasing a single best answer. On planning puzzles such as Countdown and Sudoku, the method (DMPO) beats the previous best diffusion-LLM reinforcement-learning baseline by large margins without an extra supervised fine-tuning stage.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The distribution-matching guarantee in Eqs. (9)–(11) is not delivered by the implementation: negative-ELBO surrogates replace exact order-specific sequence probabilities in both the importance weights and the loss, with no bias bound.","rationale":"The reader's weakest_assumption identifies precisely the load-bearing gap: the paper's formal derivation assumes exact order-specific sequence probabilities and exact -log pθ, while the implementation substitutes negative-ELBO surrogates in the weights and the loss. This matters because the paper's headline contribution is a “principled and theoretically grounded” distribution-matching objective; if the surrogate bias is uncontrolled, Eq. (9) no longer holds and DMPO reduces to a heuristic reward-weighted denoising cross-entropy loss. The concern is not a disagreement with a widely held consensus; it is an internal inconsistency between the derivation and the described implementation, with no error bound in the manuscript. The empirical results on Countdown and Sudoku are substantial and the code is promised, so rejection would be too strong; a conditional verdict with a concrete validity check is appropriate. I agree with the reader's choice of weakest assumption and would not change the verdict.","tokens_in":24136,"tokens_out":9243,"duration_ms":81897,"concrete_test":"Use a small synthetic masked diffusion model (e.g., vocabulary size 16, sequence length 8) with a known pref and reward so that p*(o|q) can be enumerated exactly. Run DMPO as described in Secs. 3.2–3.3 (negative-ELBO surrogates in both the weights and the loss) and compare against an exact variant that computes log pref(o|q;σ), log pv(o|q;σ), and -log pθ(o|q) by enumerating all orders. Measure the final KL(p*∥pθ) and the empirical distribution of importance weights. If the ELBO variant converges to a materially higher KL than the exact variant, the implementation is not optimizing the claimed distribution-matching objective and the Table 1 results should be interpreted as heuristic reward-weighted CE. If the two variants coincide, the surrogate is benign and the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is the chain from Eq. (9) to Eq. (11). Eq. (9) is an exact identity only if (i) the importance weight w(o|q;σ) is proportional to p*(o|q;σ)/pv(o|q;σ) using exact order-specific probabilities, and (ii) the inner cost is -log pθ(o|q). The implementation replaces both with negative-ELBO surrogates. Section 3.2, under “Sequence log probability log pv(o|q)”, says the exact sequence log probability is intractable “due to the presence of expectation over the random order σ” and proposes Eq. (2) as a surrogate. But Eq. (10) needs log pref(o|q;σ)/pv(o|q;σ) for the sampled σ, not the order-averaged log probability; the order-specific quantity is well-defined and is in principle computable as a product of D conditional forward passes. Using Lθ(o|q) in place of -log pθ(o|q) in Eq. (11) makes the objective an upper bound whose gradient is not the gradient of KL(p*∥pθ). Moreover, if the log weights ℓ(o|q;σ) in Eq. (10) are themselves computed with ELBO surrogates, the softmax normalization estimates the wrong partition function and the IS correction is biased. Section B.2 acknowledges only that variance reduction “holds only approximately” because of the ELBO substitution; no error bound or sufficient condition is given. If the ELBO gaps differ between pref and pv, or across training, DMPO is no longer optimizing KL(p*∥pθ) and the central theoretical claim is unsupported. The empirical gains in Table 1 may still be real, but they would not validate the distribution-matching framework as stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Distribution Matching Policy Optimization (DMPO), an off-policy RL fine-tuning method for masked diffusion LLMs (dLLMs). The method targets the reward-tilted distribution p*(o|q) ∝ pref(o|q)e^{r(q,o)/α} and proposes to match a dLLM policy pθ to p* by minimizing an importance-weighted denoising cross-entropy (WDCE) loss, together with a weighted direct discriminative optimization (WDDO) variant. To handle small training batches, the paper introduces weight-baseline subtraction (group, individual, and model variants). Experiments on LLaDA-8B-Instruct across GSM8K, MATH500, Countdown, and Sudoku report gains over the d1 baseline, especially on Countdown and Sudoku, without task-specific SFT. The central theoretical claim is that the WDCE loss implements an unbiased importance-sampled estimator of the forward KL KL(p*∥pθ).","tokens_in":24473,"tokens_out":10131,"duration_ms":90255,"significance":"If the distribution-matching guarantee were delivered by the implementation, DMPO would be a meaningful advance: it is forward-only, off-policy, and enables replay-buffer reuse for dLLMs, while avoiding the backward-trajectory dependence of GRPO-style methods. The paper ships code, reproduces the d1 baseline, evaluates on external benchmarks, and reports large gains on Countdown and Sudoku without SFT. These are real strengths. However, the exactness of the derivation in Eqs. (9)–(11) is not realized by the implemented loss because exact order-specific sequence probabilities are replaced by negative-ELBO surrogates. This is a load-bearing gap: the empirical results may still be valid, but they do not validate the claim that DMPO is minimizing KL(p*∥pθ), and the theoretical grounding collapses to a heuristic reward-weighted loss unless the bias is quantified or removed.","major_comments":[{"comment":"The exact identity in Eqs. (9)–(11) requires the inner loss to be -log pθ(o|q) and the importance weight to use exact order-specific probabilities pref(o|q;σ)/pv(o|q;σ). The implementation substitutes negative-ELBO surrogates in both places. The stated justification—intractability due to the expectation over σ—concerns the order-averaged pv(o|q), not the order-specific pv(o|q;σ) actually needed in Eq. (10); the latter is well-defined and in principle computable as a product of conditional forward passes. With ELBO surrogates, the softmax normalization estimates the wrong partition function and the minimized loss is no longer KL(p*∥pθ). No bias bound or sufficient condition is given; Appendix B.2 only notes that variance reduction holds approximately. This directly undermines the paper's central theoretical claim.","section":"§3.2, Eqs. (9)–(11)"},{"comment":"The WDDO objective inherits the same problem: it requires exact sequence log probabilities log pθ(o|q)/pv(o|q), but the paper does not state how these are computed in the implementation. If negative-ELBO surrogates are used here as well, the proof in Appendix B.3 that the functional is minimized at pθ = p* does not transfer to the implemented loss. Please state explicitly whether exact or surrogate probabilities are used, and if surrogates are used, provide an analysis of the resulting bias or restrict the claim to the idealized objective.","section":"§3.4, WDDO"}],"minor_comments":[{"comment":"The header abstract reports different improvement numbers (39.63 and 67.97 percentage points) than the abstract in the body and Section 4 (42.9% and 55.8%). Please reconcile these numbers.","section":"Abstract / §4"},{"comment":"The reward-tilt temperature α and the buffer refresh frequency F are parameters required by Eqs. (5), (10) and Algorithm 1, but they are not reported in the hyperparameter list. Please list their values; this is needed for reproducibility.","section":"§C.2 / Algorithm 1"},{"comment":"Table 1 reports single-run accuracies with no standard deviations or number of seeds. The gains on GSM8K and MATH500 are small (+1.0–1.6 points); please report variance or rerun with multiple seeds.","section":"Table 1"},{"comment":"Legend typo: 'Invidiual' should be 'Individual'.","section":"Fig. 5"},{"comment":"The paper emphasizes speed advantages of forward-only training, but no wall-clock or throughput comparison with d1 is reported. A runtime comparison would substantiate the efficiency claim.","section":"§1 / §4"}],"recommendation":"major_revision","confidential_remarks":"The ELBO-surrogate issue is the gating point. If the authors can compute exact order-specific log probabilities from the rollouts, or provide a quantitative bias bound, the central claim can be repaired. Otherwise, the theoretical claims should be substantially scaled back and the method presented as a heuristic reward-weighted denoising objective. The empirical results on Countdown/Sudoku are interesting enough to warrant a revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should read this if you work on RL for diffusion LLMs. The core idea is clean: instead of GRPO-style reward maximization, minimize forward KL to the reward-tilted target p* using importance-weighted denoising cross-entropy. That framing, plus the off-policy replay-buffer and forward-only training, is a genuinely useful direction, and the gains on Countdown (+42.9 pp over d1) and Sudoku (+11.8 pp) are large enough to take seriously even with the caveats below.\n\nWhat's new: applying the WDCE/SOC machinery from the authors' MDNS preprint to dLLM reasoning RL, with weight-baseline subtraction to handle small batches. The derivation in (9)–(11) is internally consistent, and the code is public. That's real.\n\nThe soft spot is load-bearing. Eq (10) needs order-specific log probabilities log pref(o|q;σ)/pv(o|q;σ), which are expensive but well-defined for a sampled σ. The implementation replaces them with negative-ELBO surrogates, both in the weights and in the loss (Sec. 3.2). The order-specific quantity is not intractable; it just costs D forward passes. Using ELBO breaks the importance-sampling identity, so the objective is no longer exactly KL(p*∥pθ), and no error bound or sufficient condition is given. Section B.2 acknowledges the variance reduction is only approximate, but that's not the same as a bias bound. So the 'theoretically grounded' claim is overstated relative to what is actually optimized. This is the referee's main job.\n\nOther issues are smaller. All reported results are single-run with no error bars; the math gains (1–4 pp) are within plausible noise. The caption claim of consistent improvement is contradicted at GSM8K/128 and tied at MATH500/128. The per-task choice of group vs individual baseline is not justified by a hold-out procedure. Figure 1 swaps the LLaDA-Instruct and LLaDA-1.5 bars in all four panels, and the metadata abstract's improvement numbers (39.63/67.97) don't match the full-text abstract (42.9%/55.8%) — the table backs the full-text numbers, so it's sloppiness, not fraud.\n\nBottom line: the empirical story could survive the theory gap — DMPO may well be a good reward-weighted denoising objective — but the paper has not shown that it matches p*. It deserves a serious referee, but the authors should be pushed to quantify the surrogate bias (or use exact order-specific probabilities for a subset) and add seeds.","headline":"DMPO is a plausible forward-KL RL method for diffusion LLMs with striking planning gains, but the paper's headline guarantee of matching p* is not delivered by the implementation, which swaps exact order-specific probabilities for ELBO surrogates.","tokens_in":25145,"tokens_out":4017,"would_cite":true,"duration_ms":34144,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DMPO matches a diffusion LLM's distribution to a reward-tilted target, reporting up to 42.9 percentage points of accuracy gain over the strongest prior RL baseline, without supervised fine-tuning.","keywords":["diffusion language models","reinforcement learning","distribution matching","cross-entropy","importance sampling","masked diffusion","reasoning benchmarks","off-policy RL"],"falsifier":"Train a small masked diffusion model on a finite alphabet small enough to enumerate all sequences, compute DMPO's gradient using the ELBO surrogates and the true forward-KL gradient using exact probabilities, and measure their cosine similarity on a typical training batch; if the angle is not small (or the surrogate gradient does not descent toward p*), the distribution-matching claim is falsified. Alternatively, on such a toy model, check by exact enumeration whether the trained distribution actually equals the reward-tilted p*.","tokens_in":23835,"feed_emoji":"🧩","tokens_out":7637,"duration_ms":58386,"temperature":0.7,"pith_summary":"Diffusion language models generate text by unmasking tokens in parallel rather than left to right, which promises faster inference but has lacked a reinforcement learning recipe. This paper proposes DMPO, which trains a diffusion LLM to match a reward-tilted distribution — the original model distribution reweighted by how good each response's reward is — using an importance-weighted denoising cross-entropy loss. The method is off-policy (rollouts can be reused) and forward-only (no need to score full trajectories), and it requires no supervised fine-tuning before RL. The authors report accuracy gains of up to 42.9 percentage points over the previous best RL baseline on Countdown and 11.8 on Sudoku, with smaller gains on GSM8K and MATH500. The broader claim is that distribution matching, not reward maximization, is the right objective for fine-tuning diffusion language models.","feed_headline":"Matching, not maximizing: RL for diffusion LLMs jumps 43 points","feed_subtitle":"DMPO replaces reward maximization with distribution matching, lifting diffusion LLM reasoning up to 43 points.","key_machinery":"The reward-tilted distribution p*(o|q) ∝ pref(o|q) exp(r(q,o)/α) is the target; the weighted denoising cross-entropy (WDCE) loss carries the argument. It converts the intractable forward-KL objective into a tractable one: sample responses from a reference policy pv, weight them by exp(r/α + log pref(o|q;σ)/pv(o|q;σ)), and fine-tune with the masked-diffusion denoising cross-entropy loss. Weight baseline subtraction and weighted direct discriminative optimization are auxiliary mechanisms that fix small-batch behavior.","core_discovery":"Central claim: the right objective for RL fine-tuning a masked diffusion language model is to match p*(o|q) ∝ pref(o|q) exp(r(q,o)/α), the entropy-regularized reward-tilted distribution, rather than to maximize reward directly. Since p* cannot be sampled, DMPO estimates it via importance weighting from a reference policy: weight w ∝ exp(r/α + log pref / pv), then trains with a weighted denoising cross-entropy loss that reuses the negative-evidence-lower-bound surrogate standard in masked diffusion training. The authors argue this is mass-covering, off-policy, and forward-only. They also identify a small-batch failure mode (all positive weights promote bad responses) and fix it with weight ba","pith_inferences":["A test the paper does not run: compare DMPO's surrogate gradient to the exact forward-KL gradient on a toy model with enumerable sequence probabilities; if the angle is large, the distribution-matching guarantee fails.","The weight-baseline requirement under small batches suggests an adaptive baseline schedule — add the baseline only when sampled responses poorly cover the space — could further stabilize WDDO, which the authors observe collapsing after about 1,200 steps.","DMPO's target is the same reward-tilted distribution targeted by GFlowNet-style objectives, but via forward KL; comparing the two on a common reasoning benchmark could reveal when mass-covering beats mode-seeking in practice.","The large gain on Countdown and Sudoku versus small gains on math word problems suggests DMPO's advantage is largest where rewards are dense, verifiable, and diverse solution paths exist; testing on other combinatorial planning tasks would show whether that pattern generalizes."],"forward_implications":["RL for diffusion LLMs can run off-policy: generated responses can be stored in a replay buffer and reused for multiple gradient updates without recomputing sequence probabilities.","Training is forward-only: it relies on the cheap forward noising process and clean samples, discarding rollout trajectories, so it can inherit speedups from fast inference and KV-cache techniques.","Matching the full tilted distribution, instead of chasing the reward mode, should preserve response diversity and reduce mode collapse and reward hacking.","The method requires no supervised fine-tuning before RL, simplifying the pipeline for reasoning tasks.","The same recipe applies to any masked diffusion sequence model with a reward, opening the door to RL fine-tuning beyond the tested model and benchmarks."],"fun_headline_variants":["DMPO: distribution matching RL lifts diffusion LLM reasoning","Matching, not maximizing: RL for diffusion LLMs up 39.63 pts","DMPO matches reward-tilted distribution to boost diffusion LLM reasoning","For diffusion LLMs, RL by distribution matching beats reward max","Diffusion LLM reasoning boosted via DMPO distribution matching RL"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The theory assumes exact order-specific sequence probabilities are available for computing importance weights and the loss, but the implementation uses the negative evidence lower bound as a surrogate, and the paper provides no error bound showing the substitution preserves the distribution-matching gradient; if those ELBO gaps differ sharply between the reference and current policies, the guarantee collapses.","fun_headline_variants_meta":{"raw":{"variants":["DMPO: distribution matching RL lifts diffusion LLM reasoning","Matching, not maximizing: RL for diffusion LLMs up 39.63 pts","DMPO matches reward-tilted distribution to boost diffusion LLM reasoning","For diffusion LLMs, RL by distribution matching beats reward max","Diffusion LLM reasoning boosted via DMPO distribution matching RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001138,"raw_usage":{"total_tokens":4574,"prompt_tokens":767,"completion_tokens":3807,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":511,"completion_tokens_details":{"reasoning_tokens":3712}},"tokens_in":511,"tokens_out":3807,"duration_ms":24307,"temperature":1.0,"reasoning_tokens":3712,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T10:50:44.135876+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a small masked diffusion model on a finite alphabet small enough to enumerate all sequences, compute DMPO's gradient using the ELBO surrogates and the true forward-KL gradient using exact probabilities, and measure their cosine similarity on a typical training batch; if the angle is not small (or the surrogate gradient does not descent toward p*), the distribution-matching claim is falsified. Alternatively, on such a toy model, check by exact enumeration whether the trained distribution actually equals the reward-tilted p*.","supporting_citations":[],"review_version":1}