{"id":"ee18f1ac-e28f-4ad0-af82-1c3c8ad57de5","arxiv_id":"2608.03769","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"MDLMPE encodes the evolving revealed/masked pattern of a masked diffusion language model as an explicit positional signal, improving performance over standard RoPE in most tested settings.","lead":"Researchers propose a new positional encoding for masked diffusion language models that tracks which tokens are revealed or masked during generation, and adds that information to the model's position signals. The method improves accuracy and perplexity on several benchmarks across fine-tuning and pretraining, suggesting this availability signal is a useful addition to these models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SFT gains may stem from MDLMPE's progressive head warmup and added parameters, not from the availability signal: no parameter- and schedule-matched RoPE baseline is provided.","rationale":"The paper is careful in many ways: the pretraining protocol fixes data order, token budget, and corruption sampler; the phase MLP is zero-initialized so the rotary branch begins at RoPE; Appendix F gives deterministic analyses of the representation's mathematical properties. These are real strengths. However, the central claim—that MDLMPE's availability signal, not auxiliary machinery, drives the reported gains—requires that the SFT comparisons isolate that signal. Eq. 27 explicitly introduces a method-specific curriculum, and the added parameters are not matched in a RoPE baseline. A curriculum or extra capacity can improve fine-tuning outcomes for reasons unrelated to the mask pattern. The proposed dummy-branch control removes only the availability content while keeping the schedule and parameter count, so it directly settles whether the confound explains the results. The 100M pretraining evidence is partly independent of the warmup confound, but it lacks variance and has regressions, so it is suggestive rather than decisive. This matches the reader's conditional verdict: the claim is plausible but currently unproven, and the missing control is the load-bearing gap. I therefore keep the verdict at CONDITIONAL (UNCHANGED).","tokens_in":22034,"tokens_out":9516,"duration_ms":125638,"concrete_test":"Run the LLaDA-8B Tulu SFT with three seeds and four arms: (A) RoPE baseline; (B) MDLMPE with the Eq. 27 schedule; (C) RoPE plus an equal-size zero-initialized phase-MLP branch fed a constant dummy input and activated with the same Eq. 27 schedule (same parameter count, no availability input); (D) MDLMPE with heads active from step 0 (schedule disabled). Compare the Table 1 metrics. If B ≈ C, the warmup/parameter confound explains the gains; if B beats C by the original margins and D ≈ B, the availability signal is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that the availability signal itself causes the reported gains. The weakest point is the SFT protocol: Eq. 27 linearly activates MDLMPE heads from 0 at 10% of updates to 100% at 90%, while the RoPE baseline gets no analogous curriculum. MDLMPE also adds an embedding projection, a phase MLP, and gates. At initialization the phase MLP is zero and the embedding gate is 0.01, so MDLMPE begins near RoPE and gradually departs during SFT. A progressive activation schedule of this kind can improve checkpoint adaptation independently of the mask pattern (e.g., by reducing forgetting), and extra parameters alone can absorb variance. The paper counts the added parameters but never trains a RoPE baseline with the same warmup schedule and parameter budget, nor an MDLMPE variant with the schedule disabled. Appendix C's ablations compare MDLMPE components, not a schedule-matched RoPE. Because all Table 1 comparisons share this confound, the SFT evidence does not yet isolate the positional signal. The 100M pretraining results are less affected by the warmup (it is disabled there), but they are single-run and include regressions (e.g., LLaDA-100M PTB 170.18 vs. RoPE 144.55/ALiBi 135.60), so they do not fully rescue the claim. This is an untested confound, not a demonstrated falsehood.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes MDLMPE, a positional encoding for masked diffusion language models. It represents the current revealed/masked configuration as a binary availability sequence, applies a normalized truncated Gaussian weighting, projects the result through a shifted-sine basis on the RoPE frequency grid, and injects the resulting features through two paths: a gated additive term on token embeddings and an MLP-produced bounded phase residual applied to roughly half the rotary pairs. The authors evaluate MDLMPE on instruction post-training of LLaDA-8B and DREAM-7B (Tulu-3 and SmolTalk), controlled 100M-parameter pretraining with zero-shot perplexity, blockwise post-training, and component ablations on LLaDA-7B. They report broad gains over RoPE and ALiBi and conclude that the availability pattern is a useful positional signal for masked diffusion models.","tokens_in":22419,"tokens_out":5987,"duration_ms":68794,"significance":"The proposal is clearly specified and addresses a genuine gap: existing positional encodings ignore the evolving revealed/masked structure of MDLM decoding. The deterministic analyses in Appendix F are a clear strength, establishing exact availability-ratio envelopes, locality bounds, endpoint invariants, and complexity accounting without relying on empirical outcomes. No circular dependence on clean target tokens is apparent; the availability signal is computed from the actual masked input. If the empirical comparisons were clean, the contribution would be solid and likely useful to the MDLM community. However, the current evidence does not yet isolate the mechanism, and a few reported numbers contradict the 'generally outperforms' summary. The paper is therefore publishable in principle, but the empirical support needs strengthening.","major_comments":[{"comment":"The headline SFT evidence is confounded. All MDLMPE runs use the progressive head-activation schedule of Eq. (27), while the RoPE baselines do not. At initialization the phase MLP is zero-initialized and the embedding gate is 0.01, so MDLMPE begins near RoPE and gradually departs during post-training. The schedule alone could improve checkpoint adaptation independently of the mask signal, and MDLMPE adds trainable parameters beyond RoPE. No schedule-matched RoPE baseline, no parameter-matched RoPE baseline, and no MDLMPE-without-schedule control are reported. Because Table 1 is the primary evidence for checkpoint compatibility, the gains cannot yet be attributed to the availability signal itself.","section":"§4.2, §B.4, Eq. (27)"},{"comment":"The pretraining results contain regressions that contradict the abstract's 'generally outperforms' claim. On LLaDA-100M, PTB perplexity is 170.18 for MDLMPE vs 144.55 for RoPE and 135.60 for ALiBi, and WT103 is 104.84 vs 103.12; DREAM-100M Text8 is 107.10 vs ALiBi's 103.46. All pretraining results are single-run with no variance. Multi-seed training, or at least a paired bootstrap over test tokens, and a clear statement of which differences are statistically distinguishable are needed before claiming general superiority.","section":"§4.3, Table 2"},{"comment":"The blockwise results are mixed and generally small. For example, LLaDA-8B block-16 ARC-Challenge is 60.1 for MDLMPE vs 60.7 for RoPE, and DREAM-7B block-64 HellaSwag is 65.9 vs 66.6; the largest positive differences are 1–2 percentage points. No error bars or significance tests are provided. This section can support robustness only if the positive direction is consistent and distinguishable from noise, which is not demonstrated.","section":"§4.4, Table 3"},{"comment":"The default geometry W/L=3/16 and rho=0.25 is selected on the 50K/2K Tulu SFT split (Tables 5–6) and then applied to all other settings. There is no held-out validation split and no sensitivity analysis across corpora. The reported gains could therefore be partly a product of tuning on one evaluation set. The paper should report the hyperparameters' sensitivity on at least one other dataset or on a separate validation split.","section":"§4.1, §C.1–C.2"}],"minor_comments":[{"comment":"The table's floating-point entries are concatenated without visible separators (e.g., '60.360.5' appears to be '60.3, 60.5'). This makes the table nearly unreadable; add explicit delimiters or columns.","section":"Table 1"},{"comment":"The first affiliation is printed as 'University Chinese Academic of Science'; this appears to be 'University of Chinese Academy of Sciences'. Please correct.","section":"Affiliation"},{"comment":"The component analysis reports evaluation perplexity on only 2,000 examples with no confidence interval or multi-seed variance. The phrase 'the strongest result' is too strong; qualify it as 'best among the single runs reported'.","section":"§4.5, Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the core idea is worth pursuing. The main weakness is experimental controllability: the SFT comparison is confounded by the progressive schedule and added parameters, and the pretraining tables show regressions. I am not recommending rejection because the central derivation is sound and the missing controls are obtainable. I would ask the authors to add a schedule-matched/parameter-matched RoPE baseline, an MDLMPE-without-warmup control, multi-seed pretraining results, and a more measured wording of the claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: MDLMPE is a real idea—conditioning positional encodings on the revealed/masked configuration in masked diffusion LMs—and the paper gives it a careful, unusually explicit treatment. The math is laid out in full, the complexity bounds are honest, and the deterministic analyses in Appendix F actually characterize the representation rather than hand-wave. That is more than most positional-encoding papers do. The novelty is real: making the availability pattern itself a positional signal, via Gaussian-weighted cosine-basis features and bounded RoPE phase residuals, is not a routine RoPE tweak.\n\nWhat I trust: the method is fully specified, the design choices are plausible, the ablations (window ratio, bandwidth, embedding projection vs. basis) are properly controlled within MDLMPE, and the pretraining results at 100M are suggestive. The paper also does the right thing by recomputing availability from the actual input state to avoid leaking clean responses.\n\nWhere it gets soft: the headline claim—'generally outperforms'—rests on Table 1, and every SFT comparison shares the same confound. MDLMPE gets a progressive head-activation schedule (Eq. 27) that RoPE does not get, plus an embedding projection, a phase MLP, and gates. The paper counts the parameters but never trains a RoPE baseline with the same warmup and parameter budget. A progressive schedule alone can improve SFT by reducing forgetting, independent of the mask pattern. So the 7B/8B gains do not isolate the availability signal. The 100M pretraining results avoid the warmup but are single-run and include regressions—LLaDA-100M on PTB is substantially worse than RoPE and ALiBi. That is not fatal, but it means 'generally outperforms' is too strong for the current evidence. The paper needs multi-seed runs with error bars, a schedule-matched and parameter-matched RoPE baseline, and a variant with the warmup disabled.\n\nAlso worth noting: the qualitative examples in Appendix E are cherry-picked and should be labeled as anecdotes, which they mostly are. The authors explicitly say they are not aggregate evidence, so no issue there.\n\nBottom line: this is a solid contribution that deserves a serious referee. The method is novel and clearly specified; the empirical support needs to be strengthened before the claim is accepted. If I were editing, I would send it to review, with a request for the matched baseline and variance estimates. I would not myself cite it as evidence of improvement yet, but I would cite it as the first explicit availability-aware positional encoding.","headline":"A genuinely novel positional encoding for masked diffusion LMs with a solid formal apparatus, but the empirical claim is undercut by a confounded SFT protocol and missing uncertainty estimates.","tokens_in":22869,"tokens_out":2059,"would_cite":true,"duration_ms":21734,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For masked diffusion language models, the changing revealed/masked pattern is itself a positional signal, and encoding it explicitly — as MDLMPE does — generally beats RoPE across fine-tuning, pretraining, zero-shot evaluation, and block di","keywords":["masked diffusion language models","positional encoding","RoPE","token availability","mask pattern","rotary phase residual","LLaDA","DREAM"],"falsifier":"Train the RoPE baseline with the same added parameters — the embedding projection and phase MLP — fed a constant input, and apply the identical progressive head-activation schedule of Eq. 27. If that matched-capacity baseline reproduces MDLMPE's numbers, the availability encoding is not the cause. A cheaper check: rerun the 100M pretraining comparisons over several seeds; overlapping variance bands would undercut the improvement claim. The paper's own deterministic analysis says the availability ratio's range collapses at revealed fractions 0 and 1, so testing MDLMPE at very high or very low m","tokens_in":21936,"feed_emoji":"🎭","tokens_out":9703,"duration_ms":90240,"temperature":0.7,"pith_summary":"Masked diffusion language models do not decode left-to-right: they reveal tokens in a changing, non-contiguous order, so the same absolute position can have very different usable context at different denoising steps. This paper argues that standard positional encodings miss that structure, and that the evolving revealed/masked configuration is itself a useful positional signal. To capture it, the paper introduces MDLMPE, which encodes which positions are revealed as a binary pattern, weighs it by distance with a Gaussian, projects it through a cosine basis aligned with RoPE's frequencies, and feeds the result into both the token embeddings and small phase adjustments on selected RoPE pairs. Across supervised fine-tuning of 7B/8B LLaDA and DREAM models, 100M-parameter pretraining, zero-shot perplexity, and block-diffusion post-training, MDLMPE generally improves over RoPE with the same checkpoints, data, and decoding budgets. If the claim holds, position encoding for diffusion language models should be state-dependent, not index-only.","feed_headline":"Mask-aware positions beat RoPE on diffusion language models","feed_subtitle":"Encoding which tokens are revealed during denoising lifts LLaDA and DREAM scores across fine-tuning and pretraining.","key_machinery":"The load-bearing object is the availability encoder: a binary vector marking revealed versus masked positions; a row-normalized truncated Gaussian kernel making each position's view local and target-relative; and shifted-sine coordinates on RoPE's own frequency grid. The ratio r_{i,f} = (weighted revealed mass)/(all-valid reference) lies in [0,1] and distinguishes mask layouts with identical revealed counts by spatial arrangement. The encoder feeds two paths: a gated embedding residual e_i + γW_p u_i, and a pair-conditioned phase residual δ_{ij,f} = τ_max tanh(MLP(arccos(2r_ij−1) − π/2)) applied to roughly half of the interleaved RoPE pairs, so attention angles become (j−i)ω_f + δ_{ij,f}. Ro","core_discovery":"The paper claims that in masked diffusion language models the evolving revealed/masked configuration is a positional signal standard encodings miss, and that encoding it explicitly improves the model. MDLMPE builds a content-free binary availability signal per position, applies target-centered Gaussian weighting, and projects the pattern through a cosine basis on RoPE's frequency grid, yielding an availability ratio that distinguishes equal-density layouts. The signal enters twice: as a gated addition to token embeddings, and, for about half the RoPE pairs, as a bounded angular residual from an MLP over the pair-averaged ratio, preserving RoPE's displacement phase. On LLaDA and DREAM, MDLMPE","pith_inferences":["A natural extension the paper does not test: the advantage of availability-aware positioning should grow as the reveal order becomes more non-contiguous — e.g., fully parallel or random-order decoding versus near-prefix schedules — because those regimes create more information-rich mask patterns.","The fixed Gaussian window ratio and bandwidth are tuned on one setting; making the window or the frequency allocation learnable per layer, or conditioning the phase residual on longer contexts, could extend the method to sequences where the mask topology changes at different scales.","The matched comparisons add parameters and a warmup schedule to MDLMPE but not to the RoPE baseline; if a capacity-matched and schedule-matched baseline erases the gap, the correct conclusion would be that the availability representation is a convenient way to spend extra capacity, not that the signal itself matters.","The deterministic analysis implies the signal saturates at fully masked and fully revealed extremes, so the method's practical value should concentrate in the intermediate denoising steps — a testable prediction for scheduling denoising so that intermediate revealed fractions get the most benefit."],"forward_implications":["MDLMPE can be added to existing RoPE-pretrained MDLM checkpoints through instruction post-training, improving knowledge, reasoning, and code tasks on both LLaDA-8B and DREAM-7B without changing the backbone, tokenizer, or decoding budget.","From-scratch 100M pretraining with MDLMPE lowers validation and zero-shot perplexity on external corpora relative to RoPE and ALiBi, suggesting the availability signal is a useful inductive bias, not only a post-training patch.","The same encoder transfers to blockwise denoising: matched block-size experiments show MDLMPE improvements over RoPE without a block identifier, because the signal is recomputed from whatever visibility pattern attention actually sees.","The component ablation indicates that binary state, Gaussian locality, spectral basis, and embedding injection are complementary; the strongest results require the complete combination, and intermediate configurations can underperform even the RoPE baseline.","Because the representation provably distinguishes equal-density mask layouts, MDLMPE gives the model a positional coordinate that can tell apart different arrangements of the same number of revealed tokens — something index-based and displacement-based encodings cannot do."],"supporting_citations":[{"why":"Defines RoPE, the positional-encoding baseline that MDLMPE builds on and must beat.","marker":"(Su et al. 2024)"},{"why":"Supplies LLaDA, one of the two MDLM backbones used for 7B/8B post-training and 100M pretraining.","marker":"(Nie et al. 2025)"},{"why":"Supplies DREAM, the second MDLM backbone used for matched post-training and pretraining experiments.","marker":"(Ye et al. 2025)"},{"why":"Provides OpenWebText, the corpus used for the controlled 100M pretraining runs.","marker":"(Gokaslan et al. 2019)"},{"why":"Provides Tulu-3, one of the two instruction-following datasets for 7B/8B supervised fine-tuning.","marker":"(Lambert et al. 2024)"},{"why":"Provides SmolTalk, the second SFT dataset used in matched post-training comparisons.","marker":"(Allal et al. 2025)"},{"why":"Provides ALiBi, the additional positional-encoding baseline in the controlled pretraining diagnostics.","marker":"(Press, Smith, and Lewis 2022)"},{"why":"Defines the blockwise denoising factorization whose visibility patterns MDLMPE is tested against.","marker":"(Arriola et al. 2025)"}],"fun_headline_variants":["Mask-aware positions beat RoPE on diffusion LMs","Why diffusion models should encode which tokens are revealed","New position encoding for masked diffusion outperforms RoPE","Encoding the revealed/masked state improves diffusion LMs","Diffusion LMs get a boost from mask-sensing positions"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The matched comparisons assume the gains come from the availability signal itself: no RoPE baseline was run with the same added parameters (embedding projection and phase MLP) and the same progressive head-activation schedule, and all results are single runs with no variance, so the reported improvements cannot be cleanly attributed to the positional signal rather than to extra capacity or the warmup.","fun_headline_variants_meta":{"raw":{"variants":["Mask-aware positions beat RoPE on diffusion LMs","Why diffusion models should encode which tokens are revealed","New position encoding for masked diffusion outperforms RoPE","Encoding the revealed/masked state improves diffusion LMs","Diffusion LMs get a boost from mask-sensing positions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000843,"raw_usage":{"total_tokens":3527,"prompt_tokens":779,"completion_tokens":2748,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":2671}},"tokens_in":523,"tokens_out":2748,"duration_ms":24715,"temperature":1.0,"reasoning_tokens":2671,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T12:52:53.820653+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the RoPE baseline with the same added parameters — the embedding projection and phase MLP — fed a constant input, and apply the identical progressive head-activation schedule of Eq. 27. If that matched-capacity baseline reproduces MDLMPE's numbers, the availability encoding is not the cause. A cheaper check: rerun the 100M pretraining comparisons over several seeds; overlapping variance bands would undercut the improvement claim. The paper's own deterministic analysis says the availability ratio's range collapses at revealed fractions 0 and 1, so testing MDLMPE at very high or very low m","supporting_citations":[],"review_version":1}