{"id":"cf889586-8a77-425c-b7fd-9f7bed4b2d48","arxiv_id":"2608.09424","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A training objective that keeps prompts clean and applies diffusion only to the continuation improves diffusion language models on continuation benchmarks without changing inference.","lead":"The paper proposes a new pretraining objective for diffusion language models that keeps the prompt prefix clean and autoregressive while denoising only the continuation. Early results on LLaDA2-Mini and Qwen-1.7B show consistent gains over native diffusion training, suggesting a simple, infrastructure-level fix.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported gains are attributed to clean-prefix conditioning, but every PCD variant simultaneously adds an AR next-token loss on the prefix; no control isolates the clean-prefix interface from this extra AR supervision.","rationale":"The reader's weakest_assumption identified exactly this confound: all PCD variants add an AR next-token loss on the prefix, and no clean-prefix-without-AR control is reported. Our stress-test agrees that this is the most load-bearing issue for the central claim. The empirical finding that PCD improves over the matched native baseline is credible, but the paper's causal interpretation ('aligning the pretraining context distribution with prompt-conditioned generation can recover a measurable part of the dLLM continuation gap') is not established until the AR-supervision confound is removed. Because the reader already made this the basis of a CONDITIONAL verdict, the appropriate output is UNCHANGED rather than a more severe adjustment. A single targeted control run would settle whether the mechanism claim survives.","tokens_in":12577,"tokens_out":5443,"duration_ms":55809,"concrete_test":"Run the Qwen-1.7B 50B ablation with a clean-prefix/no-AR control: same intra-sample mask and normalization as Eq. (8), but set λ_ar=0 so prefix tokens are visible and uncorrupted while only the suffix MDM loss is trained. If this control's Avg.6 gain over the native baseline is close to the reported +4.86, the clean-prefix interface is the active ingredient; if it collapses toward 0, the gain is attributable to AR prefix supervision rather than context-distribution alignment. For completeness, also run a second control that adds the AR prefix loss to native full-sequence MDM; if this matches PCD, the suffix-denoising context (clean vs. corrupted prefix) is irrelevant.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central causal claim is that matching the pretraining context distribution to prompt-conditioned generation (clean prefix, denoised suffix) drives the +2.56 Avg.6 LLaDA2-Mini and +4.86 Qwen gains. The experiments do not isolate this mechanism. In Eq. (3) and Eq. (8), PCD always combines an AR next-token loss on the clean prefix with the no-shift suffix MDM loss; the native baseline has neither a clean prefix nor an AR loss. Figure 6's 'intra-sample-only,' 'inter-only,' and 'mixed' rows all retain the AR prefix loss, so they cannot distinguish clean-prefix conditioning from the extra AR supervision. The inter-sample-only row (ppcd=0.50) still contains the AR term in its PCD examples. A possible alternative explanation is that adding an AR objective to continued pretraining strengthens left-to-right modeling enough to improve these benchmark scores on its own, independent of whether the suffix-denoising context is clean. The variational argument in Eq. (4) compares only the suffix-MDM terms under clean vs. corrupted prefixes and omits the AR term, so it provides no evidence about the confound. The Limitations section acknowledges single runs but does not mention this missing control.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Prefix-Conditioned Diffusion (PCD), a continued-pretraining objective for diffusion language models that keeps prompt prefixes clean and autoregressively supervised while training the unknown suffix with a no-shift masked-diffusion (MDM) objective. The authors argue that native diffusion pretraining corrupts prompt and continuation tokens together, creating a mismatch with prompt-conditioned evaluation, and that PCD reduces this mismatch. They report matched-budget comparisons against same-family native dLLM stable baselines on LLaDA2-Mini and Qwen-1.7B, with a +2.56 Avg.6 gain at the 400B stable budget and a +4.86 gain in the Qwen mechanism comparison, plus additional results on decay continuation, chat-SFT transfer, prompt-prefix statistics, and intra-sample versus inter-sample objective mixing.","tokens_in":12857,"tokens_out":7016,"duration_ms":73412,"significance":"The matched-budget experimental design and consistent gains across budgets and backbones are strengths: the paper ships a clear controlled comparison on the same infrastructure, data, and evaluation harness, and it reports a falsifiable prediction. If the effect is genuinely due to clean-prefix conditioning, PCD would be a practical training-interface fix compatible with WSD-style conversion. The paper also provides a clean variational identity and useful prompt-statistics diagnostics. However, the central attribution is not yet established, because every PCD variant adds an autoregressive next-token loss on the prefix, and no control isolates the clean-prefix interface from this additional AR supervision. The variational argument in Eq. (4) addresses only the suffix-MDM term and omits the AR term, so it cannot rule out the alternative explanation that the gains come from adding AR supervision to continued pretraining. The phenomena are plausible and worth publishing, but the mechanism claim needs additional controls.","major_comments":[{"comment":"The paper attributes the reported gains to clean-prefix conditioning, but every PCD variant combines an AR prefix loss with no-shift suffix MDM, while the native baseline has neither. In Eq. (8), L_intra always contains the λar term; the intra-sample-only, inter-only, and mixed rows in Figure 6 all use this objective, and the inter-only row at ppcd=0.50 still uses PCD examples with their AR prefix loss. No row sets λar=0 while keeping clean-prefix suffix MDM, and no row adds an AR loss to the native corruption baseline. The variational identity in Eq. (4) compares only the suffix-MDM terms under clean versus corrupted prefixes and omits the AR term entirely, so it does not address this confound. Please add a clean-prefix-without-AR control (e.g., λar=0) and, ideally, a native-corruption-plus-AR control, and revisit the mechanism claims in light of those results.","section":"§4.4, Figure 6, Eq. (8)"},{"comment":"The headline numbers come from single training runs, and the paper provides no variance estimates. The matched-budget gains start at +0.99 Avg.6 at 100B and reach +2.56 at 400B; without seed variance or repeated evaluations, it is difficult to assess whether the consistent ordering is statistically meaningful or within run-to-run noise. The Limitations section acknowledges that most large-scale rows are single runs, but this issue is load-bearing for the claim that PCD consistently improves over the native baseline. Please provide seed variance at smaller scale (e.g., repeated Qwen-1.7B runs), and where full repeats are infeasible, report at least bootstrap confidence intervals over benchmark examples or multiple evaluation seeds.","section":"§4.1, §4.3, Limitations"},{"comment":"The text states that PCD improves the native baseline on all 12 benchmarks, but Table 1 reports only the six Avg.6 benchmarks, and Figure 8 reports domain aggregates rather than per-benchmark scores. The breadth claim is not verifiable from the displayed data. Please include the full 12-benchmark table in the main text or appendix, with matched native-baseline scores for each benchmark, so that the 'all 12' claim can be checked.","section":"§4.3, §5"}],"minor_comments":[{"comment":"There is a missing space in 'interfacePrefix-Conditioned Diffusion'; it should read 'interface Prefix-Conditioned Diffusion'.","section":"§1"},{"comment":"The quantities V*_nat and V*_pcd are introduced without a formal definition; please define them explicitly (e.g., as conditional entropies or as optimal suffix-denoising losses over a model family) and state the independence assumptions needed for the Markov condition Y ⊥ C~ | (C, Y~, M).","section":"§3.3, Eq. (4)"},{"comment":"The text says 'We use λar = λmdm = 1 unless otherwise specified,' but no experiment varies these weights. Either add a λ sweep or remove the phrase, since the current wording implies an ablation that is not reported.","section":"§3.2 and §3.4"},{"comment":"The row label 'LLaDA2-MiniPCD stable' is missing a space; it should be 'LLaDA2-Mini PCD stable'.","section":"Table 1"},{"comment":"The AR-decoding diagnostic is not described: please specify how PCD and native checkpoints are decoded autoregressively, since the models are block-diffusion checkpoints and the AR-decoding procedure is not obvious from the main text.","section":"Figure 3"},{"comment":"The Qwen mechanism suite is said to use six benchmarks, but the specific composition is not listed; please provide the benchmark list in the caption or table so readers can interpret the Qwen Avg.6 numbers.","section":"§4.2 and Figure 6"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of a CS/CL venue and the matched-budget setup is commendable. The main risk is the missing control separating clean-prefix conditioning from the added AR loss; this is fixable with a targeted ablation rather than a fundamental flaw. I would lean toward major revision rather than rejection, provided the authors can provide the proposed controls or substantially temper the mechanism claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth your time: it makes a clean objective-level modification to continued pretraining for diffusion LMs, and reports consistent matched-budget gains across two backbones. The headline numbers are plausible: +2.56 Avg.6 over the native stable baseline at 400B, with gains at every intermediate budget, and +4.86 on Qwen-1.7B in the intra-sample ablation. The benchmark-prefix analysis in Figure 7 is a nice piece of evidence that native random corruption would indeed hit almost every evaluation prompt. That premise is solid.\n\nWhat is genuinely new: the specific combination—AR supervision on a clean prefix, no-shift MDM on the suffix, inside WSD continued pretraining—and the framing of intra-sample vs inter-sample mixing as separate degrees of freedom. The components are all off-the-shelf, but the recipe is new and cheap to adopt.\n\nThe soft spot is the one the stress-test flags: the central mechanism is not isolated. Every PCD variant pairs clean-prefix conditioning with an AR next-token loss on that prefix. No control trains the clean-prefix suffix-MDM without the AR term. So the +2.56 and +4.86 gains could in principle come from the extra AR supervision alone, not from matching the evaluation context distribution. The intra/inter-sample separation does not resolve this, because all rows retain the AR prefix term. Equation 4 only accounts for the suffix-MDM terms and omits the AR term, so it cannot support the mechanism claim. The Limitations section honestly notes the single-run issue but does not mention this missing control.\n\nThat said, the empirical result itself is not invalidated. A package that adds an AR prefix loss and clean-prefix suffix denoising does improve over native stable training at matched budget, and that is useful even if we cannot yet unpack which ingredient matters. The paper overclaims in attributing the gain to clean-prefix conditioning specifically.\n\nThis is a solid conditional paper. It deserves a serious referee, with the request to add exactly one control: clean-prefix suffix MDM without any AR prefix loss, and ideally a few seeds at small scale. The citation pattern and related-work coverage look fair. For a reader working on diffusion LMs or WSD conversion, this is a useful reference even in its current form.\n\nRecommendation: send to peer review, but the authors should be told the mechanism claim needs support before acceptance.","headline":"A useful objective-level fix for dLLM prompt continuation, but the attribution to clean-prefix conditioning is confounded by the simultaneous addition of an AR prefix loss.","tokens_in":13359,"tokens_out":4054,"would_cite":true,"duration_ms":34962,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Diffusion language models can recover a measurable part of their prompt-continuation gap by training with a clean prefix and a denoised suffix, without changing inference.","keywords":["diffusion language models","prompt continuation","pretraining-generation mismatch","prefix-conditioned diffusion","block diffusion","masked diffusion modeling","warmup-stable-decay"],"falsifier":"Train a variant that keeps the prefix clean and denoises the suffix with the same no-shift MDM objective but removes the AR prefix loss (prefix tokens observed clean but not supervised), and compare Avg.6 against both native stable and PCD at the same budget. If the clean-prefix-only variant retains the gain, clean-prefix conditioning carries the effect; if it falls back to native level, the added AR prefix supervision is the operative ingredient.","tokens_in":12405,"feed_emoji":"🎯","tokens_out":6488,"duration_ms":57214,"temperature":0.7,"pith_summary":"Diffusion language models trail autoregressive models on prompt-continuation tasks partly, the paper argues, because pretraining corrupts prompt and continuation tokens together while evaluation hands the model a clean prompt. The paper proposes Prefix-Conditioned Diffusion (PCD), a continued-pretraining objective that keeps the left prefix clean and trains it with an autoregressive next-token loss while applying masked-diffusion denoising only to the unknown suffix. On matched continued-pretraining runs, PCD improves the native diffusion stable baseline by +2.56 points on the LLaDA2-Mini six-benchmark average and +4.86 points on the Qwen-1.7B mechanism comparison, with gains at every completed budget from 100B to 400B. These results support the claim that aligning the pretraining context distribution with prompt-conditioned generation can recover a measurable part of the diffusion language model's continuation gap without changing inference. The paper's variational view makes this precise: replacing corrupted-prefix context with the clean prefix removes a non-negative conditional-mutual-information term from the suffix-denoising objective.","feed_headline":"Clean-prefix training narrows the diffusion language model gap","feed_subtitle":"A new objective beats matched native baselines on prompt continuation at every budget, up to +4.86 points.","key_machinery":"The load-bearing object is the hybrid attention/corruption mask of PCD: for a sampled prefix boundary $k$, prefix positions stay clean, use causal attention, and are trained with an AR next-token loss, while suffix positions are corrupted by the native diffusion mask and reconstructed by same-position MDM under the clean-prefix context. The effective mask $m^{\\mathrm{pcd}}_i = s_i m_i$ and the corrupted input $\\tilde{x}^{\\mathrm{pcd}}_i = x_i$ for $a_i=1$ enforce that only suffix tokens can become denoising targets. Two orthogonal knobs control the recipe: intra-sample mixing (AR prefix plus MDM suffix inside one sequence) and inter-sample mixing (mixing PCD and native diffusion examples in a minibatch via $z \\sim \\mathrm{Bernoulli}(p_{\\mathrm{pcd}})$). The variational identity $V^*_{\\mathrm{nat}} - V^*_{\\mathrm{pcd}} = I(Y; C \\mid \\tilde{C}, \\tilde{Y}, M)$ formalizes the removed corrupted-prefix context shift.","core_discovery":"The central claim is that prompt-continuation performance of diffusion language models is limited by a training-interface mismatch, not by the absence of an autoregressive decoder. Native stable-phase denoising can corrupt the very prefix that prompt-continuation evaluation keeps clean; PCD changes the attention mask, corruption mask, and label construction so that each training sequence contains a clean, autoregressively supervised prefix and a no-shift masked-diffusion suffix. In controlled same-family comparisons, PCD stays above the native stable baseline at every completed budget, reaches +2.56 Avg.6 at the matched 400B LLaDA2-Mini stable stage, and remains +1.67 after matched decay continuation; Qwen-1.7B ablations attribute the strongest signal to intra-sample clean-prefix suffix denoising rather than to batch-level objective mixing.","pith_inferences":["Because every PCD variant adds an AR next-token loss on the prefix, the paper's design does not isolate clean-prefix conditioning from the extra supervision; a control with a clean prefix but no AR prefix loss would determine whether the interface or the added loss drives the gain.","The 96.8% median prefix-ratio diagnostic suggests benchmark difficulty is dominated by long clean prompts, so gains may concentrate on tasks with longer prompts; a prompt-length-stratified evaluation would test this directly.","PCD's principle extends beyond block diffusion: leaving any prefix uncorrupted and denoising the rest with full-sequence diffusion is the same interface fix, and the same reasoning could be applied at SFT time rather than only during continued pretraining.","The optimal prefix-ratio range depends on backbone and budget in the Qwen sweep, so a validation-driven prefix schedule may beat any fixed range; the paper does not claim a universal optimum."],"forward_implications":["Every completed continued-pretraining budget from 100B to 400B shows PCD above the matched native stable baseline, with the gap growing to +2.56 Avg.6 at 400B on LLaDA2-Mini.","The benefit survives WSD decay continuation (+1.67 over the matched native decay row) and transfers to chat-SFT, where the PCD-derived checkpoint reaches 67.39 versus 66.53 for the best native-SFT run.","Intra-sample clean-prefix suffix denoising, not inter-sample mixing, is the primary alignment signal: the intra-only Qwen row gains +4.86 versus +4.70 for inter-only and +4.66 for mixed.","PCD is a training-objective change only; it requires no autoregressive decoder, verifier, or new inference mode, so it slots into existing WSD-style conversion pipelines.","The 12-benchmark stable-stage average improves on all 12 benchmarks by +2.48 points, indicating the effect is broad across QA, math, code, and reasoning."],"supporting_citations":[{"why":"Defines block diffusion, the generation interface with a clean prefix and denoised blocks that PCD matches during training.","marker":"Arriola et al., 2025"},{"why":"Provides the LLaDA2.0 WSD conversion recipe whose native stable objective is the controlled baseline in the main comparison.","marker":"Bie et al., 2025"},{"why":"Supplies the Warmup-Stable-Decay framework and the rationale for inserting PCD in the stable phase.","marker":"Wen et al., 2025"},{"why":"Establishes LLaDA-style masked-diffusion pretraining and SFT that PCD builds on as external reference.","marker":"Nie et al., 2025"},{"why":"Supplies the discrete diffusion ratio-estimation objective underlying the MDM suffix loss.","marker":"Lou et al., 2024"},{"why":"Supplies the simple masked diffusion language modeling objective used for suffix denoising.","marker":"Sahoo et al., 2024"}],"fun_headline_variants":["Diffusion LMs gain from clean-prefix training objective","PCD recovers diffusion LM gap with clean-prefix diffusion","Better diffusion LMs by preserving clean prefix in training","Clean-prefix diffusion pretraining boosts LM continuation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Every PCD variant trains the clean prefix with an added autoregressive next-token loss, and no experiment isolates a clean prefix without that AR loss, so the reported gains could in principle come from the extra AR supervision rather than from matching the evaluation context distribution.","fun_headline_variants_meta":{"raw":{"variants":["Diffusion LMs gain from clean-prefix training objective","PCD recovers diffusion LM gap with clean-prefix diffusion","Better diffusion LMs by preserving clean prefix in training","Clean-prefix diffusion pretraining boosts LM continuation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000367,"raw_usage":{"total_tokens":2008,"prompt_tokens":1019,"completion_tokens":989,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":635,"completion_tokens_details":{"reasoning_tokens":921}},"tokens_in":635,"tokens_out":989,"duration_ms":10541,"temperature":1.0,"reasoning_tokens":921,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:36:35.298815+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a variant that keeps the prefix clean and denoises the suffix with the same no-shift MDM objective but removes the AR prefix loss (prefix tokens observed clean but not supervised), and compare Avg.6 against both native stable and PCD at the same budget. If the clean-prefix-only variant retains the gain, clean-prefix conditioning carries the effect; if it falls back to native level, the added AR prefix supervision is the operative ingredient.","supporting_citations":[],"review_version":1}