Pith. sign in

REVIEW 4 major objections 6 minor 52 references

This paper argues that strict one-to-one positional supervision misaligns masked diffusion language model training, and that a CTC-based slack token that absorbs small shifts improves open-ended generation.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Adding a CTC-style <SLACK> token during MDLM supervised fine-tuning relaxes strict positional supervision and improves open-ended generation benchmarks over a matched cross-entropy baseline.

T0 review reviewed 2026-08-03 challenge →

load-bearing objection A clean diagnostic and a plausible training intervention, but the central claim that CTC (rather than slack-augmented CE) drives the gains is not yet isolated by the experiments. the 4 major comments →

arxiv 2601.22947 v2 pith:UGO5PWNU submitted 2026-01-30 cs.CL cs.LG

Reconsidering Positional Supervision in Masked Diffusion Language Model Training

Mengyu Ye , Keito Kudo , Ryosuke Takahashi , Jun Suzuki This is my paper
classification cs.CL cs.LG
keywords masked diffusion language modelspositional alignmentconnectionist temporal classification<SLACK> tokensupervised fine-tuningopen-ended text generationLLaDApositional robustness
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Masked diffusion language models generate text by unmasking tokens in parallel, and once a token is revealed it cannot be revised. The paper claims that the standard training objective, which supervises each position independently, is mismatched with this irreversible decoding: a single token shifted by one position can cascade into major semantic degradation. It demonstrates the sensitivity with a controlled intervention that rotates already-unmasked blocks by one index, causing win rates to fall as the number of shifts grows. To fix the mismatch, the authors add a connectionist temporal classification (CTC) term with a special token that absorbs positional uncertainty, and combine it with cross-entropy during supervised fine-tuning. On LLaDA-8B-Instruct, the resulting model outperforms both the original and a matched cross-entropy-only baseline across five open-ended generation benchmarks, and degrades far more slowly under the same positional-shift intervention.

Core claim

The central claim is that strict one-to-one positional supervision is misaligned with the irreversible denoising dynamics of MDLMs. The authors show that a controlled one-position rotation of unmasked token blocks during decoding reduces generation quality, and that the degradation grows with the number of shift boundaries. They then introduce an alignment-flexible objective: a CTC loss with a <SLACK> token, added to the masked cross-entropy loss as L_SFT = L_CE + λ L_CTC. The <SLACK> token absorbs local positional mismatch and is removed by a collapse operation that also merges adjacent duplicates, so the final output surface form is preserved after collapse. Training with this objective im

What carries the argument

The central mechanism is the CTC (connectionist temporal classification) objective, adapted to MDLM fine-tuning, with a special <SLACK> token inserted into the target response at a random per-example slack ratio (max 0.5) and a collapse map that removes <SLACK> tokens and merges adjacent duplicates. The combined training loss is L_SFT = L_CE + λ L_CTC (λ=0.1), where the CE term supervises the slack-augmented target and the CTC term marginalizes over all alignments that collapse to the original clean response. The <SLACK> token acts as an alignment buffer at subword and word boundaries, and the authors add a decoding-time merge step that converts adjacent duplicate decoded tokens into <SLACK>

Load-bearing premise

The paper's diagnosis rests on treating artificially rotated unmasked blocks as a faithful proxy for real one-position misalignments in unperturbed decoding; the paper does not measure how often such shifts actually occur naturally.

What would settle it

Measure the frequency and structure of one-position misalignments in unperturbed LLaDA-8B-Instruct generations. If natural misalignment is vanishingly rare or differs from the intervention (e.g., duplication rather than shifts), then the CTC remedy may be solving a perturbation the model almost never encounters, and the benchmark gains would need another explanation.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If strict positional supervision is the bottleneck, then MDLM training should move away from per-position token prediction toward alignment-flexible objectives; CTC is one instance, and more sophisticated alignment losses could improve further.
  • Because the CTC term is combined with CE during SFT only, the gains are available without changing the model architecture or the decoding interface; any MDLM can adopt the same recipe.
  • The robustness result means models trained this way tolerate small positional perturbations better, which matters for long generations where early local errors otherwise cascade.
  • The method complements inference-time decoding changes like block diffusion, so training-side and inference-side alignment flexibility can be stacked.
  • The paper's limitation shows the approach can hurt format-sensitive tasks such as numeric strings (GSM8K), suggesting the objective needs task-aware variants before broad deployment.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: The paper never measures how often natural one-position misalignment occurs in unperturbed LLaDA decoding; the Appendix artifacts are only qualitative. A direct measurement of shift frequency in unperturbed outputs would tell whether the intervention targets the dominant error source or just an artificial one.
  • Editorial extension: The CTC-trained model's gains might partly come from the <SLACK>-augmented targets changing the effective length distribution and giving the model more positions to spread content, rather than from absorbing genuine misalignment. A control that inserts random padding tokens without the CTC collapse would separate these effects.
  • Editorial extension: The same slack-token idea could be applied during pre-training rather than only SFT, and the paper explicitly leaves this open; if it works, alignment flexibility could be native to MDLMs.
  • Editorial extension: The trailing-zero failures on GSM8K suggest a testable boundary: a CTC variant that merges only <SLACK> tokens, not adjacent duplicates, would preserve digit strings and could tell whether duplicate-merging is the culprit.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper argues that masked diffusion language models (MDLMs) suffer in open-ended generation because their training objective enforces a strict one-to-one positional alignment, while their irreversible iterative decoding is vulnerable to small positional misalignments. It first demonstrates, via a controlled shift intervention on LLaDA-8B-Instruct and LLaDA-1.5, that displacing a small fraction of tokens by one position substantially degrades generation win rates. It then adapts connectionist temporal classification (CTC) to MDLM supervised fine-tuning, introducing a special <SLACK> token that the model can emit to absorb positional uncertainty, and combines the CTC loss with a cross-entropy loss computed over a slack-augmented target. On five open-ended generation benchmarks, the CTC-trained model reportedly outperforms both the original LLaDA-8B-Instruct and a CE-only baseline, and it shows reduced sensitivity to the same shift intervention. The paper concludes that relaxing strict positional supervision is an effective design dimension for MDLM training.

Significance. If the central claim is established, the paper identifies a training-side alignment-flexibility mechanism as a useful complement to inference-time decoding changes for MDLMs, an area where the field is actively seeking improvements. The controlled intervention experiment is a clear and useful diagnostic, and the adaptation of CTC to masked diffusion is technically sensible. The authors provide a matched CE-only baseline, report confidence intervals for the main Arena-Hard rows, and include a careful analysis of a known failure mode (GSM8K trailing-zero degradation) in Appendix C. However, the comparison is confounded by several simultaneous changes to the training objective and inference procedure, and the reported statistical significance claims are not fully substantiated. These issues are load-bearing for the paper's main conclusion.

major comments (4)
  1. [§4.2, Eq. (7); §5.2; Table 1] The CE-only baseline is not matched on the CE target. Eq. (7) defines L_SFT = L_CE(\tilde r0) + λ L_CTC(r0), where \tilde r0 contains inserted <SLACK> tokens, while §5.2 states the CE-only baseline is trained with the standard CE objective on the original target r0. Thus the CTC-trained model differs from the baseline in at least five ways: slack-augmented CE target (§5.1), padding-to-slack conversion (§5.1), downweighting of <SLACK> in CE (B.1), the added CTC loss, and inference-time EOS fixation and duplicate merging (§4.3, B.3). No ablation isolates the CTC term from slack-augmented CE. The benchmark gains could therefore be due to the altered CE supervision rather than to the alignment-flexible CTC objective, undermining the paper's central claim.
  2. [Table 1; §5.3] The CE-only baseline is dramatically weaker than the original model (27.2 vs 50.0 on Arena-Hard hard-prompt; 31.4 vs 50.0 on creative-writing), which the paper attributes to differences in instruction-tuning data and pipelines. The CTC+CE model's gain over the original is small (51.4 vs 50.0) and its confidence interval includes 50. The main evidence is the large improvement over a degraded baseline. Since the baseline is not even competitive with the original, the comparison does not convincingly show that the CTC objective is responsible; the gap could reflect a recovery of quality from slack-augmented targets. A stronger baseline—ideally one whose CE-only performance matches the original—is needed to support the claim of consistent improvement.
  3. [§5.4; §3.1] The robustness result is partly circular. The CTC-trained model was explicitly trained to emit <SLACK> at random positions and to convert adjacent duplicate tokens to <SLACK> during inference (§4.3). Demonstrating that it absorbs a forced one-position shift is therefore expected and does not separate the CTC loss from the slack-augmented CE target. Running the same intervention on a model trained with slack-augmented CE and λ=0 would deconfound this. Additionally, the diagnosis in §3 rests on the assumption that controlled rotations of unmasked blocks are a faithful proxy for naturally occurring misalignment; the paper does not measure the natural rate of such shifts (Appendix A provides only qualitative, selected examples). A quantitative estimate of natural shift frequency would strengthen the causal narrative.
  4. [Abstract; Table 1] The abstract claims 'statistically significant gains on all four' benchmarks, but Table 1 reports 95% confidence intervals only for the two Arena-Hard rows. No significance tests are shown for Creative-Writing-Bench v3, MTBench, or WildBench. Moreover, the reported gain over the original model on Arena-Hard hard-prompt (51.4 vs 50.0) is not significant given the confidence interval. The authors should report bootstrap confidence intervals or pairwise significance tests for all benchmarks, and clearly state which comparisons (vs original, vs CE-only) are claimed to be significant. As written, the statistical claim in the abstract is not supported by the evidence in Table 1.
minor comments (6)
  1. [Abstract/§5.2/Conclusion] The number of benchmarks is inconsistent: the abstract says 'four', while §5.2 and the Conclusion say 'five'. Table 1 contains five rows (Arena-Hard has two subsets). Please clarify.
  2. [§5.1] The padding-to-slack conversion ratio is described as 'controlled by a separate ratio and capped at a fixed fraction', but no values are given. Please report the actual ratio and cap used in experiments.
  3. [§4.3; §B.3] The inference-time modifications—fixing the final position to <EOS> and the greedy duplicate-to-<SLACK> merging—are additional differences from the CE-only baseline. They should be stated in the main text and, ideally, ablated, since they could affect generation quality independently of the training objective.
  4. [Figures 2 and 4] Pearson r is computed from a small number of K values; report the number of points and p-values, or use a rank-based measure. The text also does not give the exact r values in the caption for Figure 2.
  5. [Eq. (5)] The CTC collapse operation is described as merging adjacent duplicates first and then removing <SLACK>, whereas standard CTC typically removes blanks first. The order matters for sequences with repeated symbols adjacent to <SLACK>; a brief clarification of the intended semantics would avoid ambiguity.
  6. [Various] Minor typos and formatting issues: 'slack' vs '<SLACK>' are used inconsistently; 'Arena-Hard' hyphenation varies; a few reference entries have missing spaces before URLs (e.g., 'arXiv preprint arXiv:2502.09992. URLhttps'). Also, the full-text title 'Relaxing Positional Alignment in Masked Diffusion Language Models' differs from the arXiv title; please make them consistent.

Circularity Check

0 steps flagged

No significant circularity: benchmark gains are externally evaluated and the CTC objective is imported from external prior work.

full rationale

The paper's central empirical claim is that fine-tuning LLaDA-8B-Instruct with L_SFT = L_CE + lambda L_CTC (Eq. 7) improves open-ended generation quality. This claim is supported by results on external benchmarks (Arena-Hard, MTBench, WildBench, etc.) that are not derived from, or fitted to, the diagnostic shift intervention; they are observed outcomes under a fixed evaluation protocol. CTC itself is an established external objective (Graves et al., 2006), so adopting it is not circular, and no argument in the paper depends on a load-bearing self-citation or on a uniqueness theorem from the authors' prior work. The controlled shift intervention in Section 3 is an empirical probe used for diagnosis, not a definitional restatement of the conclusion. The Section 5.4 robustness evaluation tests a behavior the model was explicitly trained to exhibit via random <SLACK> insertion and CTC collapse; this is a consistency check of the intended mechanism, but it is not the basis for the main benchmark conclusion, and it does not make the reported gains equal to the training objective by construction. The absence of an ablation separating the CTC term from the slack-augmented CE target is a real experimental confound and a correctness risk, but a missing control is not itself circular. No equation is shown to be definitionally identical to another, and no fitted parameter is relabeled as a prediction. Accordingly, no specific circular step can be exhibited.

Axiom & Free-Parameter Ledger

6 free parameters · 6 axioms · 1 invented entities

The central claim rests primarily on two hand-chosen slack-related hyperparameters (smax, λ, down-weight) and on the assumption that the artificial shift intervention faithfully represents naturally occurring misalignment. The CTC objective itself is a standard prior method, so the main free parameters are the experimental and training choices around <SLACK>, plus the under-reported padding-conversion ratio.

free parameters (6)
  • slack ratio max smax = 0.5
    Controls the number of <SLACK> tokens inserted into each response (⌊sL⌋, s~U[0,smax]); hand-set with no sensitivity analysis.
  • CTC loss weight lambda = 0.1
    Weight of L_CTC in Eq. (7); chosen by hand; authors state no hyperparameter search was performed.
  • slack token CE downweight = 0.01
    Down-weights <SLACK> token contribution in the cross-entropy term; hand-set, no ablation reported.
  • padding-to-slack conversion ratio = not reported
    Controls replacement of padding <EOS> tokens with <SLACK>; value is not given in the paper, undermining reproducibility.
  • shift intervention K and schedule = K varied; 10 steps, t>=0.5 every 0.05
    Diagnostic parameters for the shift intervention; relationship to naturally occurring misalignment is not established, and the abstract's '1% of tokens' is not mapped to K.
  • response-length filter = 512-1024 tokens
    Training data filtered to this band; results may not generalize to shorter or longer responses.
axioms (6)
  • domain assumption CTC conditional independence: P(a|q,rt) factorizes as a product over positions (Eq. 4).
    Standard CTC assumption; may be violated for text where adjacent tokens are strongly conditionally dependent.
  • ad hoc to paper The controlled shift intervention is a faithful proxy for natural positional misalignment in MDLM decoding.
    Section 3 uses artificial K-boundary shifts to estimate sensitivity; natural misalignment rate is not measured, only qualitative artifacts in Appendix A.
  • ad hoc to paper One-position token shifts are a dominant error mode in MDLM open-ended generation.
    Motivates the whole approach; not directly measured in unperturbed decoding.
  • domain assumption Merging adjacent duplicate tokens during decoding is acceptable for open-ended generation.
    CTC collapse removes repeated adjacent tokens, which can corrupt legitimate repetitions and numeric strings; acknowledged in Limitations and Appendix C.
  • domain assumption LLM-as-a-judge (GPT-4.1, deterministic) provides valid quality estimates.
    Standard benchmark practice, but judge bias and variance are not independently controlled.
  • domain assumption MDLM forward/reverse noising and low-confidence remasking follow prior LLaDA/block-diffusion formulations.
    The paper relies on pre-existing MDLM training and decoding assumptions (Eqs. 1-3) without re-deriving them.
invented entities (1)
  • <SLACK> special token no independent evidence
    purpose: Added to the vocabulary as an alignment buffer that absorbs positional uncertainty in MDLM decoding; removed after decoding with adjacent duplicates merged.
    The token's utility is demonstrated only within this paper's trained model and outputs; there is no falsifiable prediction outside the method's own construction, and no external checkpoint or data release to verify independently.

reviewed 2026-08-03 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Reconsidering Positional Supervision in Masked Diffusion Language Model Training." pith.science (2026). https://pith.science/paper/UGO5PWNU

@misc{pith2026260122947,
  author       = {Pith},
  title        = {Pith review of: Reconsidering Positional Supervision in Masked Diffusion Language Model Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UGO5PWNU}},
  note         = {Machine review of arXiv:2601.22947}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Masked diffusion language models (MDLMs) generate text by unmasking tokens in parallel and have recently emerged as alternatives to autoregressive language models. They can be viewed as parallel decoders trained with a position-wise cross-entropy (CE) loss, the same setup as non-autoregressive translation (NAT). In NAT, CE-trained parallel decoders have been argued to be sensitive to small positional shifts, since CE penalizes them harshly. We ask whether CE-trained MDLMs are similarly sensitive to such shifts under iterative decoding. To probe this, we apply a controlled intervention that introduces them during decoding. On LLaDA-8B-Instruct with Arena-Hard, displacing as little as 1% of generated tokens by one position substantially reduces win rates against the unintervened model, showing that MDLMs are sensitive to such small shifts under iterative parallel decoding. Motivated by this, we adapt connectionist temporal classification (CTC), an alignment-flexible objective known to mitigate it there, to MDLM supervised fine-tuning. By relaxing the strict position-wise match that CE imposes, CTC gives the loss room to absorb small positional shifts; concretely, we modified CTC objective to use a special <slack> token that absorbs positional uncertainty between target tokens and output positions, and a updated collapse map that preserves target surface forms. Across four open-ended generation benchmarks, the resulting model consistently improves over both the original model and a matched cross-entropy-trained baseline, with statistically significant gains on all four. These results identify training-side alignment flexibility as a useful design dimension for MDLM SFT, complementary to the inference-time approaches explored in prior work.

Figures

Figures reproduced from arXiv: 2601.22947 by Jun Suzuki, Keito Kudo, Mengyu Ye, Ryosuke Takahashi.

Figure 1
Figure 1. Figure 1: Under standard MDLM decoding, tokens are generated at fixed positions and become immutable once unmasked. A small position error early in decoding can therefore propagate through subsequent steps, leading to cascading semantic degradation (left). Allowing limited alignment flexibility introduces non-semantic buffer positions that absorb local misalignment, preventing error propagation while preserving the … view at source ↗
Figure 2
Figure 2. Figure 2: Win rate (%) on Arena-hard as a function of the number of shift boundaries K for LLaDA-8B-Instruct and LLaDA-1.5. Error bands indicate 95% confidence intervals. Win rates are computed against the K = 0 (no-intervention) outputs of the same model as the reference, yielding a 50% win-rate baseline (dashed line). The Pearson correlation coefficient r is between K and the win rate; larger r indicates greater s… view at source ↗
Figure 3
Figure 3. Figure 3: Illustration of <SLACK> token placement in the CTC￾trained model. <SLACK> is observed both within tokenized words ((e.g., D<SLACK>osa, mas<SLACK>ala)) and between words, where it functions as mid-text padding. This suggests <SLACK> can act as a flexible buffer at different locations in the sequence. share the same architecture, initialization, data, and training schedule, this comparison isolates the effec… view at source ↗
Figure 5
Figure 5. Figure 5: Tokens per second for the CTC-trained model with and without the merge operation during decoding. subword-level spans to word-boundary positions. Generation overhead of CTC decoding. To test whether the merge operation used in CTC decoding adds compu￾tational overhead, we measure generation throughput with and without merge across a range of generation lengths L. For each L, we set the number of diffusion … view at source ↗
Figure 6
Figure 6. Figure 6: Examples of shift-like local inconsistency artifacts in unperturbed LLaDA-8B-Instruct generations. Across diverse prompts, we observe occasional short-range token inconsistency (highlighted), such as “From From”, “the the”, and “Victoria Victoria”. These artifacts are qualitatively consistent with a small local misalignment (e.g., shift) during generation, motivating our controlled “sequence shift” interve… view at source ↗
Figure 7
Figure 7. Figure 7: Examples of shift-like local inconsistency artifacts in unperturbed LLaDA-8B-Instruct generations. In targeted test cases, we present illustrative evidence of short-range token inconsistencies (highlighted), such as cases where the character “m” should appear at the third position after “ZIJ” but is generated at the second position instead, leading to the generation of a spurious “.” token. These artifacts… view at source ↗
Figure 8
Figure 8. Figure 8: Examples of shift-like local inconsistency artifacts in unperturbed MDLM generations. In targeted test cases, we present illustrative evidence of short-range token inconsistencies (highlighted), such as the misplaced string “YNC”, where the character “Y” should appear at the second position after “Ff” but is generated at the first position, resulting in the incorrect output “YNnC” instead of “YNC”. These a… view at source ↗
Figure 9
Figure 9. Figure 9: Representative GSM8K failure case involving trailing zeros. The model drops a trailing zero from a prompt number at the start of the reasoning (e.g., 500 → 50, 800 → 80) and then proceeds with internally consistent calculations based on the altered value, leading to an incorrect final answer. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Representative GSM8K failure case involving trailing zeros. The model drops a trailing zero from a prompt number at the start of the reasoning (e.g., 600 → 60) and then proceeds with internally consistent calculations based on the altered value, leading to an incorrect final answer. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Representative GSM8K failure case involving trailing zeros. The model drops a trailing zero from a prompt number at the start of the reasoning (e.g., 200 → 20) and then proceeds with internally consistent calculations based on the altered value, leading to an incorrect final answer. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Representative GSM8K case where trailing-zero numbers are handled correctly. The model produces and uses numbers ending in zeros (e.g., 300) consistently throughout the solution and arrives at the correct final answer. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Representative GSM8K case where trailing-zero numbers are handled correctly. The model produces and uses numbers ending in zeros (e.g., 4000, 100) consistently throughout the solution and arrives at the correct final answer. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Representative GSM8K case where trailing-zero numbers are handled correctly. The model produces and uses numbers ending in zeros (e.g., 1000) consistently throughout the solution and arrives at the correct final answer. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_14.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

52 extracted references · 4 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    S., Gokaslan, A., Yang, Z., Qi, Z., Han, J., Chiu, J

    Arriola, M., Sahoo, S. S., Gokaslan, A., Yang, Z., Qi, Z., Han, J., Chiu, J. T., and Kuleshov, V. Block diffusion: Interpolating between autoregressive and diffusion language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=tyEyYT267x

    2025

  3. [3]

    Program synthesis with large language models, 2021

    Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., and Sutton, C. Program synthesis with large language models, 2021. URL https://arxiv.org/abs/2108.07732

    Pith/arXiv arXiv 2021

  4. [4]

    Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., et al. Language models are few-shot learners. In The Thirty-fourth Annual Conference on Neural Information Process...

    2020

  5. [5]

    Chang, H., Zhang, H., Jiang, L., Liu, C., and Freeman, W. T. Maskgit: Masked generative image transformer. arXiv preprint arXiv:2202.04200, 2022. URL https://arxiv.org/abs/2202.04200

    Pith/arXiv arXiv 2022

  6. [6]

    Training verifiers to solve math word problems

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. URL https://arxiv.org/abs/2110.14168

    Pith/arXiv arXiv 2021

  7. [7]

    and Blondel, M

    Cuturi, M. and Blondel, M. Soft-dtw: a differentiable loss function for time-series. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, pp.\ 894–903. JMLR.org, 2017

    2017

  8. [8]

    Lv-ctc: Non-autoregressive asr with ctc and latent variable models

    Fujita, Y., Watanabe, S., Chang, X., and Maekaku, T. Lv-ctc: Non-autoregressive asr with ctc and latent variable models. arXiv preprint arXiv:2403.19207, 2024. URL https://arxiv.org/abs/2403.19207

    Pith/arXiv arXiv 2024

  9. [9]

    The llama 3 herd of models, 2024

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., et al. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783

    Pith/arXiv arXiv 2024

  10. [10]

    Sequence transduction with recurrent neural networks

    Graves, A. Sequence transduction with recurrent neural networks. arXiv preprint arXiv:1211.3711, 2012. URL https://arxiv.org/abs/1211.3711

    Pith/arXiv arXiv 2012

  11. [11]

    Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks

    Graves, A., Fern\' a ndez, S., Gomez, F., and Schmidhuber, J. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd International Conference on Machine Learning, ICML '06, pp.\ 369–376, New York, NY, USA, 2006. Association for Computing Machinery. ISBN 1595933832. doi:10.1145/...

    arXiv 2006

  12. [12]

    Speech recognition with deep recurrent neural networks

    Graves, A., rahman Mohamed, A., and Hinton, G. Speech recognition with deep recurrent neural networks. arXiv preprint arXiv:1303.5778, 2013. URL https://arxiv.org/abs/1303.5778

    Pith/arXiv arXiv 2013

  13. [13]

    Havasi, M., Karrer, B., Gat, I., and Chen, R. T. Q. Edit flows: Flow matching with edit operations. arXiv preprint arXiv:2506.09018, 2025. URL https://arxiv.org/abs/2506.09018

    arXiv 2025

  14. [14]

    Aligning \ ai \ with shared human values

    Hendrycks, D., Burns, C., Basart, S., Critch, A., Li, J., Song, D., and Steinhardt, J. Aligning \ ai \ with shared human values. In International Conference on Learning Representations, 2021 a . URL https://openreview.net/forum?id=dNy_RKzJacY

    2021

  15. [15]

    Measuring massive multitask language understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In The Ninth International Conference on Learning Representations, 2021 b . URL https://openreview.net/forum?id=d7KBjmI3GmQ

    2021

  16. [16]

    Mask ctc: Non-autoregressive end-to-end asr with ctc and mask predict

    Higuchi, Y., Watanabe, S., Chen, N., Ogawa, T., and Kobayashi, T. Mask ctc: Non-autoregressive end-to-end asr with ctc and mask predict. arXiv preprint arXiv:2005.08700, 2020. URL https://arxiv.org/abs/2005.08700

    Pith/arXiv arXiv 2005

  17. [17]

    No compute left behind: Rethinking reasoning and sampling with masked diffusion models

    Horvitz, Z., Singhal, R., Zou, H., Domingo-Enrich, C., Yu, Z., Ranganath, R., and McKeown, K. No compute left behind: Rethinking reasoning and sampling with masked diffusion models. arXiv preprint arXiv:2510.19990, 2025. URL https://arxiv.org/abs/2510.19990

    arXiv 2025

  18. [18]

    J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W

    Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lo RA : Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9

    2022

  19. [19]

    Don't settle too early: Self-reflective remasking for diffusion language models

    Huang, Z., Wang, Y., Chen, Z., and Qi, G.-J. Don't settle too early: Self-reflective remasking for diffusion language models. arXiv preprint arXiv:2509.23653, 2025. URL https://arxiv.org/abs/2509.23653

    arXiv 2025

  20. [20]

    X., Béthune, L., Ablin, P., Kirchhof, M., Monteiro, J., Turrisi, V., Ramapuram, J., and Cuturi, M

    Jazbec, M., Olausson, T. X., Béthune, L., Ablin, P., Kirchhof, M., Monteiro, J., Turrisi, V., Ramapuram, J., and Cuturi, M. Learning unmasking policies for diffusion language models. arXiv preprint arXiv:2512.09106, 2025. URL https://arxiv.org/abs/2512.09106

    Pith/arXiv arXiv 2025

  21. [21]

    Any-order flexible length masked diffusion

    Kim, J., Cheuk-Kit, L., Domingo-Enrich, C., Du, Y., Kakade, S., Ngotiaoco, T., Chen, S., and Albergo, M. Any-order flexible length masked diffusion. arXiv preprint arXiv:2509.01025, 2025 a . URL https://arxiv.org/abs/2509.01025

    Pith/arXiv arXiv 2025

  22. [22]

    Z., Kim, H., Kakade, S., and Chen, S

    Kim, J., Kim, S., Lee, T., Pan, D. Z., Kim, H., Kakade, S., and Chen, S. Fine-tuning masked diffusion for provable self-correction. arXiv preprint arXiv:2510.01384, 2025 b . URL https://arxiv.org/abs/2510.01384

    Pith/arXiv arXiv 2025

  23. [23]

    Lookahead unmasking elicits accurate decoding in diffusion language models

    Lee, S., Kim, S., Park, J., and Park, D. Lookahead unmasking elicits accurate decoding in diffusion language models. arXiv preprint arXiv:2511.05563, 2025. URL https://arxiv.org/abs/2511.05563

    arXiv 2025

  24. [24]

    Beyond fixed: Variable-length denoising for diffusion large language models

    Li, J., Dong, X., Zang, Y., Cao, Y., Wang, J., and Lin, D. Beyond fixed: Variable-length denoising for diffusion large language models. arXiv preprint arXiv:2508.00819, 2025 a . URL https://arxiv.org/abs/2508.00819

    Pith/arXiv arXiv 2025

  25. [25]

    E., and Stoica, I

    Li, T., Chiang, W.-L., Frick, E., Dunlap, L., Zhu, B., Gonzalez, J. E., and Stoica, I. From live data to high-quality benchmarks: The arena-hard pipeline, April 2024. URL https://lmsys.org/blog/2024-04-19-arena-hard/

    2024

  26. [26]

    E., and Stoica, I

    Li, T., Chiang, W.-L., Frick, E., Dunlap, L., Wu, T., Zhu, B., Gonzalez, J. E., and Stoica, I. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. In Forty-second International Conference on Machine Learning, 2025 b . URL https://openreview.net/forum?id=KfTf9vFvSn

    2025

  27. [27]

    and Helcl, J

    Libovick \'y , J. and Helcl, J. End-to-end non-autoregressive neural machine translation with connectionist temporal classification. In Riloff, E., Chiang, D., Hockenmaier, J., and Tsujii, J. (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp.\ 3016--3021, Brussels, Belgium, October-November 2018. Associatio...

    doi:10.18653/v1/d18-1336 2018

  28. [28]

    Y., Deng, Y., Chandu, K., Ravichander, A., Pyatkin, V., Dziri, N., Bras, R

    Lin, B. Y., Deng, Y., Chandu, K., Ravichander, A., Pyatkin, V., Dziri, N., Bras, R. L., and Choi, Y. Wildbench: Benchmarking LLM s with challenging tasks from real users in the wild. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=MKEHCx25xp

    2025

  29. [29]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7

    2019

  30. [30]

    Scaling up masked diffusion models on text

    Nie, S., Zhu, F., Du, C., Pang, T., Liu, Q., Zeng, G., Lin, M., and Li, C. Scaling up masked diffusion models on text. In The Thirteenth International Conference on Learning Representations, 2025 a . URL https://openreview.net/forum?id=WNvvwK0tut

    2025

  31. [31]

    Large language diffusion models

    Nie, S., Zhu, F., You, Z., Zhang, X., Ou, J., Hu, J., Zhou, J., Lin, Y., Wen, J.-R., and Li, C. Large language diffusion models. arXiv preprint arXiv:2502.09992, 2025 b . URL https://arxiv.org/abs/2502.09992

    Pith/arXiv arXiv 2025

  32. [32]

    and Sebban, M

    Oncina, J. and Sebban, M. Learning stochastic edit distance: Application in handwritten character recognition. Pattern Recognition, 39 0 (9): 0 1575--1587, 2006. ISSN 0031-3203. doi:https://doi.org/10.1016/j.patcog.2006.03.011. URL https://www.sciencedirect.com/science/article/pii/S0031320306001245

    work page doi:10.1016/j.patcog.2006.03.011 2006

  33. [33]

    Paech, S. J. Eq-bench creative writing benchmark v3. https://github.com/EQ-bench/creative-writing-bench, 2025

    2025

  34. [34]

    Z., Bezemek, Z., Patel, S., Rector-Brooks, J., Yao, S., Bose, A

    Peng, F. Z., Bezemek, Z., Patel, S., Rector-Brooks, J., Yao, S., Bose, A. J., Tong, A., and Chatterjee, P. Path planning for masked diffusion model sampling. arXiv preprint arXiv:2502.03540, 2025. URL https://arxiv.org/abs/2502.03540

    arXiv 2025

  35. [35]

    L., Stickland, A

    Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. GPQA : A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=Ti67584b98

    2024

  36. [36]

    and Yianilos, P

    Ristad, E. and Yianilos, P. Learning string-edit distance. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20 0 (5): 0 522--532, 1998. doi:10.1109/34.682181

    work page doi:10.1109/34.682181 1998

  37. [37]

    Non-autoregressive machine translation with latent alignments

    Saharia, C., Chan, W., Saxena, S., and Norouzi, M. Non-autoregressive machine translation with latent alignments. In Webber, B., Cohn, T., He, Y., and Liu, Y. (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.\ 1098--1108, Online, November 2020. Association for Computational Linguistics. doi:10.1865...

    doi:10.18653/v1/2020.emnlp-main.83 2020

  38. [38]

    S., Arriola, M., Gokaslan, A., Marroquin, E

    Sahoo, S. S., Arriola, M., Gokaslan, A., Marroquin, E. M., Rush, A. M., Schiff, Y., Chiu, J. T., and Kuleshov, V. Simple and effective masked diffusion language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=L4uaAR4ArM

    2024

  39. [39]

    and Cohen, W

    Sarawagi, S. and Cohen, W. W. Semi-markov conditional random fields for information extraction. In Saul, L., Weiss, Y., and Bottou, L. (eds.), Advances in Neural Information Processing Systems, volume 17. MIT Press, 2004. URL https://proceedings.neurips.cc/paper_files/paper/2004/file/eb06b9db06012a7a4179b8f3cb5384d3-Paper.pdf

    2004

  40. [40]

    N., Kaiser, L

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention is all you need. In Advances in Neural Information Processing Systems, 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf

    2017

  41. [41]

    Remasking discrete diffusion models with inference-time scaling

    Wang, G., Schiff, Y., Sahoo, S., and Kuleshov, V. Remasking discrete diffusion models with inference-time scaling. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=IJryQAOy0p

    2025

  42. [42]

    Writingbench: A comprehensive benchmark for generative writing

    Wu, Y., Mei, J., Yan, M., Li, C., Lai, S., Ren, Y., Zijia, W., Zhang, J., Wu, M., Jin, Q., and Huang, F. Writingbench: A comprehensive benchmark for generative writing. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025 a . URL https://openreview.net/forum?id=Pkskg9drDQ

    2025

  43. [43]

    Dreamon: Diffusion language models for code infilling beyond fixed-size canvas, 2025 b

    Wu, Z., Zheng, L., Xie, Z., Ye, J., Gao, J., Feng, Y., Li, Z., W., V., Zhou, G., and Kong, L. Dreamon: Diffusion language models for code infilling beyond fixed-size canvas, 2025 b . URL https://hkunlp.github.io/blog/2025/dreamon

    2025

  44. [44]

    CTC -based non-autoregressive speech translation

    Xu, C., Liu, X., Liu, X., Sun, Q., Zhang, Y., Yang, M., Dong, Q., Ko, T., Wang, M., Xiao, T., Ma, A., and Zhu, J. CTC -based non-autoregressive speech translation. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 13321--13339, Toron...

    work page doi:10.18653/v1/2023.acl-long.744 2023

  45. [45]

    Xu, Z., Jiang, F., Niu, L., Deng, Y., Poovendran, R., Choi, Y., and Lin, B. Y. Magpie: Alignment data synthesis from scratch by prompting aligned LLM s with nothing. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=Pnk7vMbznK

    2025

  46. [46]

    Diffusion llm with native variable generation lengths: Let [eos] lead the way

    Yang, Y., Wang, C., Wang, S., Wen, Z., Qi, B., Xu, H., and Zhang, L. Diffusion llm with native variable generation lengths: Let [eos] lead the way. arXiv preprint arXiv:2510.24605, 2025. URL https://arxiv.org/abs/2510.24605

    arXiv 2025

  47. [47]

    Dream 7b: Diffusion large language models

    Ye, J., Xie, Z., Zheng, L., Gao, J., Wu, Z., Jiang, X., Li, Z., and Kong, L. Dream 7b: Diffusion large language models. arXiv preprint arXiv:2508.15487, 2025. URL https://arxiv.org/abs/2508.15487

    Pith/arXiv arXiv 2025

  48. [48]

    Flexible-length text infilling for discrete diffusion models

    Zhang, A., Sivakumar, A., Tang, C.-W., and Thomas, C. Flexible-length text infilling for discrete diffusion models. In Christodoulopoulos, C., Chakraborty, T., Rose, C., and Peng, V. (eds.), Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp.\ 31344--31359, Suzhou, China, November 2025 a . Association for Computatio...

    work page doi:10.18653/v1/2025.emnlp-main.1597 2025

  49. [49]

    Z., Zhang, Y., Pan, J., and Chrysos, G

    Zhang, S., Peng, F. Z., Zhang, Y., Pan, J., and Chrysos, G. G. Corrective diffusion language models. arXiv preprint arXiv:2512.15596, 2025 b . URL https://arxiv.org/abs/2512.15596

    arXiv 2025

  50. [50]

    d1: Scaling reasoning in diffusion large language models via reinforcement learning

    Zhao, S., Gupta, D., Zheng, Q., and Grover, A. d1: Scaling reasoning in diffusion large language models via reinforcement learning. In ES-FoMo III: 3rd Workshop on Efficient Systems for Foundation Models, 2025. URL https://openreview.net/forum?id=t8oYNHAvM9

    2025

  51. [51]

    E., and Stoica, I

    Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging LLM -as-a-judge with MT -bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https://openreview.net/forum?id=uccHPGDlao

    2023

  52. [52]

    LL a DA 1.5: Variance-reduced preference optimization for large language diffusion models

    Zhu, F., Wang, R., Nie, S., Zhang, X., Wu, C., Hu, J., ZHOU, J., Chen, J., Lin, Y., Wen, J.-R., and Li, C. LL a DA 1.5: Variance-reduced preference optimization for large language diffusion models. arXiv preprint arXiv:2505.19223, 2025. URL https://arxiv.org/abs/2505.19223

    Pith/arXiv arXiv 2025

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.