Pith. sign in

REVIEW 3 major objections 4 minor 3 cited by

Discrete diffusion language models lose continuous reasoning state between denoising steps; a small persistent memory layer restores it and consistently improves math and code reasoning.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 19:39 UTC pith:4CGFVNQM

load-bearing objection A novel persistent-memory add-on for discrete diffusion LLMs with clean ablations, but the headline causal claim is not fully secured by the Dream w/o-recurrence ablation and the teacher-forced training vs. self-generated inference mismatch. the 3 major comments →

arxiv 2603.01331 v3 pith:4CGFVNQM submitted 2026-03-02 cs.CL cs.AIcs.LG

MetaState: Persistent Working Memory Enhances Reasoning in Discrete Diffusion Language Models

Kejing Xia , Mingzhe Li , Lixuan Wei , Zhenbang Du , Xiangchi Yuan , Dachuan Shi , Qirui Jin , Wenke Lee This is my paper
classification cs.CL cs.AIcs.LG
keywords discrete diffusion language modelsworking memoryrecurrent augmentationInformation Islandmasked diffusionreasoningK-step unrollingcode generation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Discrete diffusion language models generate text by iteratively unmasking tokens, but each denoising step conditions only on the previous hard-masked tokens; the continuous hidden activations that encode intermediate reasoning are discarded at the sampling-and-remasking boundary. The paper names this the 'Information Island' problem and argues it is especially damaging for multi-step math and code, where the model must carry partial results across many steps. To fix it, the authors add MetaState, a set of fixed-size memory slots that are read from the backbone's activations, updated with a recurrent gate, and written back into the next step's input embeddings, while the backbone stays frozen. Trained with K-step unrolling, this adds about 0.6% parameters and improves accuracy by an average of 4.5 percentage points across four reasoning benchmarks and two discrete diffusion model families, in both base and instruction-tuned versions. The paper's central claim is that the bottleneck, not the model's single-step capability, is what limits reasoning in current discrete diffusion LMs.

Core claim

The paper identifies the Information Island issue: in standard masked diffusion decoding, each step's transition is Markovian over the discrete token sequence, so the rich continuous hidden state computed at step t is compressed into sparse discrete tokens and remasking indicators before step t+1. This repeated lossy compression forces later steps to re-derive intermediate reasoning content from scratch, causing cross-step drift that is especially harmful for mathematical reasoning and code generation. To address this, the paper introduces MetaState, a recurrent augmentation that maintains a constant number of memory slots across the denoising trajectory. A Mixer uses cross-attention to read

What carries the argument

The central object is the augmented transition pθ(xt-1 | xt, st) with state update st-1 = gθ(st, ht, t), where st is a fixed-size set of M continuous memory slots. The mechanism is the recurrent loop formed by three modules: the Mixer (cross-attention reading backbone activations into slots), the Updater (gated recurrent unit that retains and overwrites slot content), and the Injector (cross-attention adding a zero-initialized positional modulation to the backbone's input embeddings). The paper also introduces K-step iterative unrolling with backpropagation through time, which trains the modules to learn multi-step dynamics rather than single-step denoising.

Load-bearing premise

The entire argument rests on the claim that continuous hidden activations from one denoising step contain task-relevant information that the next step cannot reconstruct from the discrete masked sequence; if that premise fails, the reported gains could be due to the added trainable parameters or to exposure to the training data rather than to persistent memory.

What would settle it

Run the same MetaState training but zero out the memory state at every step while keeping all modules (no recurrence). If accuracy over the frozen baseline stays comparable, the Information Island premise is not needed to explain the gains. Alternatively, measure EVERPASS@1 minus PASS@1 on multiple models and benchmarks with repeated runs: if the gap is consistently small or absent, the claim that correct intermediate outputs are routinely lost to remasking lacks support.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the Information Island bottleneck is real, then any discrete diffusion LM that adds a cross-step continuous memory should see similar reasoning gains, regardless of architecture details.
  • Because the backbone is frozen and only external modules are trained, the method can be layered onto an already-deployed diffusion model without retraining its weights.
  • MetaState is compatible with soft-diffusion decoding; combining them gives the best results, indicating the memory path and the token-refinement path improve different parts of the pipeline.
  • The ablation shows that backpropagation through time is necessary: detaching the state across steps consistently hurts, so the memory must learn to update, not just accumulate.
  • The fixed-size memory means the extra storage and compute do not grow with sequence length, making the approach applicable to long generations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • We read the paper as predicting that the same memory mechanism would improve any generation process with a discrete interface between computation steps, such as block-wise autoregressive or semi-autoregressive models, since those also discard continuous state.
  • One could directly probe the memory slots during a math problem: if the slots do not encode intermediate quantities like partial sums or remainders shortly after they are computed, the memory is not carrying the reasoning state the paper claims.
  • The EVERPASS@1 vs PASS@1 gap suggests a cheaper baseline: simply retaining the best full prediction seen so far along the trajectory might capture part of the gain, which would test whether the memory's advantage is about preservation rather than refinement.
  • Training the memory on an unrelated corpus instead of an instruction-tuning mixture would isolate whether the reported gains come from the recurrent mechanism or from exposure to the training distribution; the paper's LoRA control argues against the latter, but that control trains the backbone while MetaState does not.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes MetaState, a recurrent augmentation for frozen discrete diffusion LLMs. It maintains M fixed memory slots across denoising steps via a Mixer, a GRU-style Updater, and an Injector with a shared time conditioner. Training uses K-step unrolling with teacher forcing and BPTT. Experiments on Dream-7B and LLaDA-8B, in Base and Instruct variants, over GSM8K, MATH-500, HumanEval, and MBPP, show an average 4.5-point gain, supported by ablations, a LoRA control, and hyperparameter sensitivity runs. The central causal claim is that persistent working memory mitigates the 'Information Island' bottleneck.

Significance. If validated, the proposal is practically attractive: it adds only ~0.6% trainable parameters, keeps the backbone frozen, and shows consistent gains across two architecturally different dLLM families. The K-step BPTT training recipe and the LoRA control are thoughtful and go beyond a simple adapter comparison. However, the current evidence does not yet isolate the cross-step memory channel from per-step trainable capacity or from training-data exposure, and the absence of uncertainty quantification makes several of the smaller reported deltas unreliable. The core idea is plausible and worth pursuing after targeted revisions.

major comments (3)
  1. [§5.1–5.2, Table 1] The central empirical claim rests on single-run accuracy values with no error bars, confidence intervals, or significance tests. HumanEval contains only 164 problems, so deltas such as Dream-Base MBPP (+1.0) or LLaDA-Instruct GSM8K/MATH (+1.0) are within binomial sampling noise; even larger deltas on MATH-500 (500 examples) need quantification. I request bootstrap confidence intervals or repeated-seed runs for every benchmark-backbone pair. Without this, 'consistently improves' is not statistically established at the reported granularity.
  2. [§4.2, Algorithm 2 vs. §5.1 inference] Training uses teacher-forced K-step unrolling: ground-truth tokens are revealed in random order, so the recurrent state is trained on contexts that are always consistent with the clean sequence. Evaluation uses the model's own sampled tokens and confidence-based remasking (block size 32, threshold 0.9), a systematically different context distribution. This exposure-bias gap is acknowledged only as a future direction in §A.10. Because the paper's mechanism claim is that the state carries information across self-generated denoising trajectories, the authors should either train with scheduled sampling or self-generated unrolling, or provide an analysis of state behavior under self-generated contexts. As it stands, the evaluation may show test-time behavior of a module optimized for cleaner contexts.
  3. [§5.4, Table 3] The w/o recurrence ablation is the most direct test of the memory mechanism, but its results weaken the causal claim. On Dream-Instruct, removing recurrence retains almost all of MetaState's average improvement (59.25 vs. 59.88 for full MetaState; backbone-only is 56.73). Thus on one of the two backbone families the cross-step state contributes only ~0.6 points on average, and the gains are mostly attributable to other components or to training. The paper's explanation that 'Dream retains stronger per-step coherence' is not tested. I ask for a clearer implementation description of the w/o recurrence variant and for per-benchmark decompositions showing what the memory channel itself buys; a parameter-matched non-recurrent control is needed to support the persistent-memory interpretation.
minor comments (4)
  1. [§5.3, Table 2, and §A.5] Soft Diffusion hyperparameters (p, r_f) are selected per benchmark by a grid search over six configurations, while MetaState uses one fixed setting. This gives Soft Diffusion a selection advantage and makes the comparison 'MetaState outperforms Soft Diffusion' difficult to interpret. Report results with fixed hyperparameters or with averaged/uncertainty-aware selection.
  2. [§A.8, Figures 7–10] The hyperparameter sensitivity figures appear garbled with unicode escape sequences in the text; the axis labels and legends are not readable. Please replace with clean figures.
  3. [§A.3.1] Random seeds and the number of independent evaluation runs are not specified. This is essential for assessing variance, especially given the small HumanEval set.
  4. [Abstract and §1] The abstract says '4.5 percentage points' while the introduction says '4.5% average improvement'; please use consistent terminology.

Circularity Check

0 steps flagged

No circular derivation: the reported gains are measured benchmark comparisons, not fitted-input predictions; only a non-load-bearing self-citation appears in related work.

full rationale

The paper's central claim is empirical rather than derivational: MetaState's 4.5% average gain is the sum of Table 1 deltas against frozen backbones, and no equation maps a fitted constant to those deltas. The Information Island premise (Eq. 1 and Appendix A.1) is a definitional observation about the standard dLLM transition, not a conclusion derived from MetaState's success. The paper includes an explicit control for the main non-circular confound: Appendix A.6/Table 8 compares against LoRA trained on the same 50k Tulu-3 sequences, which speaks to data-exposure. The A.10 limitation paragraph ('may also help close the training-to-inference extrapolation gap') and the Table 3 Dream w/o-recurrence row (59.25 vs. 59.88 full) are real threats to the causal attribution of the gains to persistent memory, but they are validity concerns, not circularity. The only self-citation is Shi et al. 2025 (SwiReasoning, with overlapping authors) in §2.2, listed as one of several latent-reasoning methods; it is contextual and load-bearing nowhere. Thus no circular step is established.

Axiom & Free-Parameter Ledger

7 free parameters · 5 axioms · 1 invented entities

The central claim depends on several architectural and data assumptions: hidden states are information-rich, additive injection into a frozen backbone is safe, K-step teacher-forced unrolling transfers to inference, and training data does not leak into eval benchmarks. The paper provides empirical support but no formal guarantee; the Soft Diffusion comparison additionally introduces per-benchmark selected hyperparameters.

free parameters (7)
  • M (number of memory slots) = 64
    Hand-chosen; sensitivity sweep shows M=48 sometimes outperforms M=64, so the default is not a sharply tuned optimum.
  • Ds, dm, db (memory and bottleneck dimensions) = 1024 / 768 / 768
    Architectural capacity choices; not swept, though they directly determine parameter overhead and state expressiveness.
  • K (unroll depth) = 4
    Training trajectory length for BPTT; sensitivity is stable over 3-5, but transfer from K=4 teacher-forced steps to the full self-sampled inference trajectory is load-bearing.
  • lambda_d (dense/reveal loss mixing ratio) = 0.75
    Hand-selected and swept over {0.60, 0.75, 0.90}; default is a moderate choice, not theoretically derived.
  • lambda_s, tau (state-norm regularizer weight and threshold) = 1e-4 / 1.0
    Auxiliary hinge regularizer to control per-slot norms; values are chosen without sensitivity analysis in the main text.
  • Soft Diffusion (p, r_f) per benchmark = Best of 6 grid pairs per benchmark (Table 7)
    The compatibility claim in Table 2 uses test-set-selected hyperparameters for each benchmark, which can inflate the reported combination gains.
  • Inference confidence threshold and block size = 0.9 / 32
    Standard dLLM decoding settings inherited from LLaDA/Dream practice; not varied for MetaState in the main results.
axioms (5)
  • standard math The Markovian discrete-state factorization of dLLM decoding is a correct description of the base system.
    Equation (1) writes p_theta(x_0:T) = ... over discrete states; MetaState augments this with a state st but relies on the original factorization being accurate for the frozen backbone.
  • domain assumption Continuous hidden activations contain information not captured by the remasked discrete tokens.
    This is the Information Island hypothesis; Appendix A.1 supports it with an EVERPASS/PASS plot for one model and benchmark, not with a proof or controlled measurement.
  • domain assumption The zero-initialized Injector bridge preserves the frozen backbone's per-step behavior at initialization and remains stable during training.
    Sec. 4.1.4 introduces N0 zero-bridge to make the augmented model begin as the unmodified backbone; stability during training is empirically assumed rather than proven.
  • domain assumption K=4 teacher-forced unrolling transfers to the longer self-sampled inference trajectory.
    Algorithm 2 trains on pre-sampled reveal trajectories with K=4; inference runs the full autoregressive remasking loop. Appendix A.10 explicitly acknowledges a 'training-to-inference extrapolation gap'.
  • domain assumption The 50k Tulu-3 SFT training sequences do not meaningfully overlap the evaluation benchmarks.
    No contamination audit is reported. The LoRA control in A.6 partially addresses data-exposure effects but does not prove overlap-free training.
invented entities (1)
  • Persistent memory state s_t (M slots of dimension D_s) no independent evidence
    purpose: Carry continuous information across denoising steps in a frozen dLLM backbone.
    This is an internal latent introduced by the paper. It has no falsifiable handle outside the paper's own benchmark results; its usefulness is measured only through the reported accuracy changes.

pith-pipeline@v1.3.0-alltime-deepseek · 27649 in / 16762 out tokens · 167187 ms · 2026-08-02T19:39:36.224977+00:00 · methodology

0 comments
read the original abstract

Discrete diffusion language models (dLLMs) generate text by iteratively denoising a masked sequence. However, standard dLLMs condition each denoising step solely on the current hard-masked sequence, while intermediate continuous representations are discarded after sampling and remasking. We term this bottleneck the \textbf{Information Island} issue: continuous information remains isolated within individual denoising steps and fails to propagate across the trajectory. This bottleneck is especially harmful for reasoning, which requires intermediate reasoning state to be preserved and updated across many denoising steps. To address this limitation, we introduce \textbf{MetaState}, a lightweight recurrent augmentation that equips a frozen dLLM backbone with persistent, fixed-size working memory. MetaState comprises three modules with a shared time conditioner: a cross-attention \textbf{Mixer} that reads backbone activations into memory slots, a GRU-style \textbf{Updater} that integrates information across steps, and a cross-attention \textbf{Injector} that writes the updated memory back into the backbone. We train these modules with a dedicated $K$-step unrolling pipeline to learn multi-step dynamics. MetaState adds only ${\sim}0.6\%$ trainable parameters while keeping the backbone frozen, and consistently improves reasoning performance over frozen baselines on mathematical reasoning and code generation benchmarks, with an average gain of 4.5 percentage points across all evaluations.

Figures

Figures reproduced from arXiv: 2603.01331 by Dachuan Shi, Kejing Xia, Lixuan Wei, Mingzhe Li, Qirui Jin, Wenke Lee, Xiangchi Yuan, Zhenbang Du.

Figure 1
Figure 1. Figure 1: The Information Island issue in discrete diffusion: sampling and remasking compress continuous hidden activations into discrete tokens, imposing a lossy bot￾tleneck between denoising steps. MetaState addresses this issue by maintaining a per￾sistent state across steps. Autoregressive (AR) language models factor￾ize the joint distribution over sequences into a product of conditional probabilities, pro￾ducin… view at source ↗
Figure 2
Figure 2. Figure 2: Performance comparison between MetaState and frozen baselines on reasoning [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview of the MetaState architecture. The three modules (Injector, Mixer, [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: A step-by-step denoising trajectory. Each denoising step shows both the full [PITH_FULL_IMAGE:figures/full_fig_p014_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Comparison of PASS@1(t) and EVER￾PASS@1(t) across denoising steps on GSM8K with LLaDA-Instruct-8B. On our test, EVERPASS@1 re￾mains significantly higher than the PASS@1, indicating that many cor￾rect results are already present at intermediate denoising steps but are lost through the subse￾quent discrete remasking opera￾tion. In other words, the model often “knows” the right answer somewhere along the traj… view at source ↗
Figure 6
Figure 6. Figure 6: Accuracy comparison between the original backbone and MetaState at generation [PITH_FULL_IMAGE:figures/full_fig_p020_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Sensitivity to the dense-reveal loss mixing ratio [PITH_FULL_IMAGE:figures/full_fig_p020_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Sensitivity to the number of memory slots [PITH_FULL_IMAGE:figures/full_fig_p020_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Sensitivity to training data size. MetaState achieves competitive performance even [PITH_FULL_IMAGE:figures/full_fig_p021_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Sensitivity to the unroll depth K. Performance is broadly stable across K ∈ {3, 4, 5}, with average accuracy varying by less than one point. Across all four sweeps, performance varies only within a relatively narrow range on both backbones and across all benchmarks. Overall, these results suggest that MetaState is not overly sensitive to a single hyperparameter, and that the default setting represents a r… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Learned Relay Representations for Forward-Thinking Discrete Diffusion Models

    cs.LG 2026-05 unverdicted novelty 7.0

    Learned Relay Representations enable masked diffusion models to propagate useful latent information across denoising steps, scaling to Fast-dLLM v2 to outperform supervised finetuning on coding tasks while cutting inf...

  2. CopT: Contrastive On-Policy Thinking with Continuous Spaces for General and Agentic Reasoning

    cs.CL 2026-05 unverdicted novelty 7.0

    CopT reverses CoT by eliciting a draft answer first then using continuous-embedding contrastive verification and on-policy thinking to reflect and correct, yielding up to 23% higher accuracy and 57% fewer tokens witho...

  3. Learned Relay Representations for Forward-Thinking Discrete Diffusion Models

    cs.LG 2026-05 unverdicted novelty 6.0

    Learned Relay Representations add a differentiable per-token channel to masked diffusion models so they can propagate latent information across iterative denoising steps, yielding better coding performance and up to 3...

Reference graph

Works this paper leans on

28 extracted references · 17 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Block diffusion: Interpolating between autoregressive and diffusion language models.arXiv preprint arXiv:2503.09573,

    Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autoregressive and diffusion language models.arXiv preprint arXiv:2503.09573,

    Pith/arXiv arXiv

  2. [3]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

    1901

  3. [5]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    Pith/arXiv arXiv

  4. [10]

    Scalable adaptive computation for iterative generation.arXiv preprint arXiv:2212.11972,

    Allan Jabri, David Fleet, and Ting Chen. Scalable adaptive computation for iterative generation.arXiv preprint arXiv:2212.11972,

    Pith/arXiv arXiv

  5. [11]

    Ladir: Latent diffusion enhances llms for text reasoning.arXiv preprint arXiv:2510.04573,

    Haoqiang Kang, Yizhe Zhang, Nikki Lijing Kuang, Nicklas Majamaki, Navdeep Jaitly, Yi-An Ma, and Lianhui Qin. Ladir: Latent diffusion enhances llms for text reasoning.arXiv preprint arXiv:2510.04573,

    Pith/arXiv arXiv

  6. [12]

    A survey on diffusion language models.arXiv preprint arXiv:2508.10875,

    Tianyi Li, Mingda Chen, Bowei Guo, and Zhiqiang Shen. A survey on diffusion language models.arXiv preprint arXiv:2508.10875,

    Pith/arXiv arXiv

  7. [13]

    dllm-cache: Accelerating diffusion large language models with adaptive caching.arXiv preprint arXiv:2506.06295,

    Zhiyuan Liu, Yicun Yang, Yaojie Zhang, Junjie Chen, Chang Zou, Qingyuan Wei, Shaobo Wang, and Linfeng Zhang. dllm-cache: Accelerating diffusion large language models with adaptive caching.arXiv preprint arXiv:2506.06295,

    Pith/arXiv arXiv

  8. [14]

    Discrete diffusion modeling by estimating the ratios of the data distribution.arXiv preprint arXiv:2310.16834,

    Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution.arXiv preprint arXiv:2310.16834,

    Pith/arXiv arXiv

  9. [16]

    Large language diffusion models.arXiv preprint arXiv:2502.09992,

    Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models.arXiv preprint arXiv:2502.09992,

    Pith/arXiv arXiv

  10. [17]

    Under review

    11 Preprint. Under review. Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data.arXiv preprint arXiv:2406.03736,

    Pith/arXiv arXiv

  11. [18]

    Candi: Hybrid discrete-continuous diffusion models.arXiv preprint arXiv:2510.22510,

    Patrick Pynadath, Jiaxin Shi, and Ruqi Zhang. Candi: Hybrid discrete-continuous diffusion models.arXiv preprint arXiv:2510.22510,

    Pith/arXiv arXiv

  12. [19]

    Codi: Compressing chain-of-thought into continuous space via self-distillation

    Zhenyi Shen, Hanqi Yan, Linhai Zhang, Zhanghao Hu, Yali Du, and Yulan He. Codi: Compressing chain-of-thought into continuous space via self-distillation. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 677–693,

    2025

  13. [20]

    Swireasoning: Switch-thinking in latent and explicit for pareto-superior reasoning llms.arXiv preprint arXiv:2510.05069,

    Dachuan Shi, Abedelkadir Asi, Keying Li, Xiangchi Yuan, Leyan Pan, Wenke Lee, and Wen Xiao. Swireasoning: Switch-thinking in latent and explicit for pareto-superior reasoning llms.arXiv preprint arXiv:2510.05069,

    arXiv

  14. [22]

    Fast-dllm: Training-free acceleration of diffusion llm by enabling kv cache and parallel decoding.arXiv preprint arXiv:2505.22618,

    Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. Fast-dllm: Training-free acceleration of diffusion llm by enabling kv cache and parallel decoding.arXiv preprint arXiv:2505.22618,

    Pith/arXiv arXiv

  15. [23]

    Dream 7b: Diffusion large language models.arXiv preprint arXiv:2508.15487,

    Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Ling- peng Kong. Dream 7b: Diffusion large language models.arXiv preprint arXiv:2508.15487,

    Pith/arXiv arXiv

  16. [25]

    Soft thinking: Unlocking the reasoning potential of llms in continuous concept space.arXiv preprint arXiv:2505.15778,

    Zhen Zhang, Xuehai He, Weixiang Yan, Ao Shen, Chenyang Zhao, Shuohang Wang, Yelong Shen, and Xin Eric Wang. Soft thinking: Unlocking the reasoning potential of llms in continuous concept space.arXiv preprint arXiv:2505.15778,

    Pith/arXiv arXiv

  17. [26]

    Under review

    12 Preprint. Under review. Huangjie Zheng, Shansan Gong, Ruixiang Zhang, Tianrong Chen, Jiatao Gu, Mingyuan Zhou, Navdeep Jaitly, and Yizhe Zhang. Continuously augmented discrete diffusion model for categorical generative modeling.arXiv preprint arXiv:2510.01329,

    arXiv

  18. [27]

    Llada 1.5: Variance-reduced preference optimization for large language diffusion models.arXiv preprint arXiv:2505.19223, 2025a

    Fengqi Zhu, Rongzhen Wang, Shen Nie, Xiaolu Zhang, Chunwei Wu, Jun Hu, Jun Zhou, Jianfei Chen, Yankai Lin, Ji-Rong Wen, et al. Llada 1.5: Variance-reduced preference optimization for large language diffusion models.arXiv preprint arXiv:2505.19223, 2025a. Fengqi Zhu, Zebin You, Yipeng Xing, Zenan Huang, Lin Liu, Yihong Zhuang, Guoshan Lu, Kangyu Wang, Xudo...

    Pith/arXiv arXiv 2025

  19. [28]

    ,nK ∼Dir-Multi 4:Sample random reveal ranks forM 5:L ←0 6:fork=1 toKdo 7:t← |still masked|/N m ▷Continuous timestep 8:(logits,s,t cond)←Forward(x,s,t cond,t)▷Alg

    3:Sample reveal countsn 1, . . . ,nK ∼Dir-Multi 4:Sample random reveal ranks forM 5:L ←0 6:fork=1 toKdo 7:t← |still masked|/N m ▷Continuous timestep 8:(logits,s,t cond)←Forward(x,s,t cond,t)▷Alg. 1 9:L ← L+L k(logits,x 0,M k,R k) 10:Revealn k positions (teacher forcing), updatex 11:end for 12:returnL+λ sReg s A.3 Experimental Details and Hyperparameters A...

    2024

  20. [2002]

    Scaling diffusion language models via adapta- tion from autoregressive models.arXiv preprint arXiv:2410.17891,

    Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, et al. Scaling diffusion language models via adapta- tion from autoregressive models.arXiv preprint arXiv:2410.17891,

    Pith/arXiv arXiv

  21. [2017]

    Time is a feature: Exploiting temporal dynamics in diffusion language models.arXiv preprint arXiv:2508.09138,

    Wen Wang, Bozhen Fang, Chenchen Jing, Yongliang Shen, Yangyi Shen, Qiuyu Wang, Hao Ouyang, Hao Chen, and Chunhua Shen. Time is a feature: Exploiting temporal dynamics in diffusion language models.arXiv preprint arXiv:2508.09138,

    arXiv

  22. [2019]

    Working memory identifies reasoning limits in language models

    Chunhui Zhang, Yiren Jian, Zhongyu Ouyang, and Soroush Vosoughi. Working memory identifies reasoning limits in language models. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 16896–16922,

    2024

  23. [2021]

    Sdar: A synergistic diffusion- autoregression paradigm for scalable sequence generation.arXiv preprint arXiv:2510.06303,

    Shuang Cheng, Yihan Bian, Dawei Liu, Linfeng Zhang, Qian Yao, Zhongbo Tian, Wen- hai Wang, Qipeng Guo, Kai Chen, Biqing Qi, et al. Sdar: A synergistic diffusion- autoregression paradigm for scalable sequence generation.arXiv preprint arXiv:2510.06303,

    arXiv

  24. [2022]

    Residual context diffusion language models.arXiv preprint arXiv:2601.22954,

    Yuezhou Hu, Harman Singh, Monishwaran Maheswaran, Haocheng Xi, Coleman Hooper, Jintao Zhang, Aditya Tomar, Michael W Mahoney, Sewon Min, Mehrdad Farajtabar, et al. Residual context diffusion language models.arXiv preprint arXiv:2601.22954,

    arXiv

  25. [2023]

    Stop-think-autoregress: Language modeling with latent diffu- sion planning.arXiv preprint arXiv:2602.20528,

    Justin Lovelace, Christian Belardi, Sofian Zalouk, Adhitya Polavaram, Srivatsa Kundurthy, and Kilian Q Weinberger. Stop-think-autoregress: Language modeling with latent diffu- sion planning.arXiv preprint arXiv:2602.20528,

    arXiv

  26. [2024]

    Training large language models to reason in a continuous latent space.arXiv preprint arXiv:2412.06769,

    Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuan- dong Tian. Training large language models to reason in a continuous latent space.arXiv preprint arXiv:2412.06769,

    Pith/arXiv arXiv

  27. [2025]

    Structured denoising diffusion models in discrete state-spaces.Advances in neural informa- tion processing systems, 34:17981–17993, 2021a

    Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces.Advances in neural informa- tion processing systems, 34:17981–17993, 2021a. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Qu...

    Pith/arXiv arXiv

  28. [2026]

    Reinforcing the diffusion chain of lateral thought with diffusion language models.arXiv preprint arXiv:2505.10446,

    Zemin Huang, Zhiyang Chen, Zijun Wang, Tiancheng Li, and Guo-Jun Qi. Reinforcing the diffusion chain of lateral thought with diffusion language models.arXiv preprint arXiv:2505.10446,

    arXiv