Pith. sign in

REVIEW 4 major objections 5 minor 38 references

Diffusion language models do not need attention: a bidirectional Mamba-2 denoiser preserves masked-diffusion quality while making decoding cost linear in sequence length.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 21:14 UTC pith:WYUW3LRA

load-bearing objection First bidirectional Mamba-2 denoiser for masked diffusion LMs in text, with a real throughput win — but the abstract oversells a different paper and the quality comparison is confounded by parameter mismatch. the 4 major comments →

arxiv 2511.15927 v4 pith:WYUW3LRA submitted 2025-11-19 cs.LG cs.AI

DiffuMamba: High-Throughput Diffusion LMs with Mamba Backbone

Vaibhav Singh , Oleksiy Ostapenko , Pierre-Andr\'e No\"el , Eugene Belilovsky , Torsten Scholak This is my paper
classification cs.LG cs.AI
keywords diffusion language modelsmasked discrete diffusionMamba-2state-space modelsDiffuAprielDiffuMambalinear-time inferencehybrid architecture
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper sets out to show that the Transformer backbone inside a masked diffusion language model can be swapped for a bidirectional state-space (Mamba) backbone without losing modeling quality, while changing per-step decoding cost from quadratic to linear in sequence length. It trains pure-Mamba (DiffuApriel) and hybrid Mamba-attention (DiffuApriel-H) denoisers at 240M, 0.5B, and 1.3B scales under the same corpus, tokenizer, noise schedule, and 128 denoising steps used for the attention baseline. The reported results are that the hybrid model consistently matches or beats the Transformer on validation and zero-shot perplexity, and at 1.3B it improves average downstream reasoning accuracy by roughly 3-4 points while delivering 2.6x measured throughput; the pure Mamba variant reaches 4.4x throughput at the same scale. If the claim holds, attention is not strictly required for iterative text denoising, and diffusion language models can adopt the linear-time scaling that state-space autoregressive models enjoy, with the caveat that short-context gains are smaller.

Core claim

At its core, the paper claims that absorbing-state masked discrete diffusion is agnostic to the sequence mixer: the denoiser's attention layers can be replaced by bidirectional Mamba-2 layers and the probabilistic structure of the diffusion process, the masked corruption, the reweighted masked cross-entropy objective, and the factorized unmasking sampler, remains unchanged. Empirically, at 1.3B parameters, the hybrid denoiser DiffuApriel-H+MLP reduces validation perplexity from 25.01 to 22.89 under one compute budget and from 22.72 to 20.17 under another, and it beats the attention baseline on all seven zero-shot perplexity benchmarks and on average downstream reasoning accuracy by roughly 3

What carries the argument

The central object is the bidirectional Mamba-2 mixer: each diffusion block runs two input-dependent selective state-space recurrences, one forward and one backward along the sequence, adds their outputs, and applies timestep-conditioned adaptive layer normalization. This entirely replaces quadratic self-attention in the denoiser, giving O(B L d) per-block cost. In the hybrid variant, one attention layer is inserted every K=5 Mamba blocks, yielding effective complexity O(B L d + B L^2 d / K), so the linear term dominates for moderate lengths. The same masked-diffusion training loss and the same factorized unmasking sampler are retained; the architectural change is confined to the denoiser's

Load-bearing premise

The paper's quality conclusion assumes its controlled comparison isolates the architecture choice: the Mamba-based models are compared to a Transformer baseline with fewer parameters at every scale (about 15-30% fewer), so if extra capacity, not the state-space design, explains the better perplexity and accuracy, the central claim that state-space denoisers are stronger would not be established.

What would settle it

Run the paper's exact 1.3B training setup with a Transformer denoiser matched to DiffuApriel-H's parameter count or FLOPs, keeping data, tokenizer, noise schedule, and 128 decoding steps identical; if the Transformer then matches or beats the hybrid on validation perplexity and downstream average, the claim that state-space denoisers are the better backbone is refuted. Separately, throughput can be settled directly: at 128K context on one H100, DiffuApriel tokens/sec should stay approximately flat while DiffuTran decays roughly as 1/L.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • At 1.3B scale, DiffuApriel+MLP measures 4.4x end-to-end tokens/sec over the attention baseline and DiffuApriel-H+MLP 2.6x, with the gap widening as sequence length grows.
  • While Transformer throughput decays roughly as 1/L beyond a critical length, the Mamba model's throughput saturates at a memory-bandwidth floor, avoiding the long-context degradation seen with KV-cache block diffusion.
  • The hybrid Mamba-attention denoiser beats the attention-only baseline on validation perplexity and all seven zero-shot benchmarks at 1.3B, and the pure Mamba variant ranks second on most.
  • Interleaving infrequent attention (every five blocks) with state-space layers gives the best quality-efficiency tradeoff in the paper, extending the autoregressive hybrid finding to diffusion.
  • The mixer swap does not disturb the absorbing-state diffusion framework, so existing DLM acceleration techniques can compose with the new backbone.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper fixes attention frequency at every 5 blocks and never varies it; if the hybrid margin over pure Mamba grows with scale, the optimal attention density may depend on model size, which is untested.
  • An equal-parameter or equal-FLOP head-to-head run would separate architecture from capacity; the paper's own configurations leave this undetermined.
  • The throughput plateau below peak memory bandwidth suggests a direct serving measurement beyond 64K tokens would test whether the near-linear scaling holds in practice, especially for the hybrid.
  • Because block masking is described as orthogonal and the hybrid already pairs local and global mixers, combining block diffusion with Mamba denoisers seems a natural next experiment that the paper does not run.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper (body title 'DiffuApriel', arXiv metadata title 'DiffuMamba') proposes replacing the Transformer denoiser of masked diffusion language models with a bidirectional Mamba-2 backbone, plus a hybrid variant with interleaved attention. It claims that these models preserve the probabilistic structure of masked discrete diffusion while enabling linear-time inference, and reports quality and throughput comparisons against a Transformer-based DiffuTran baseline at 240M, 0.5B, and 1.3B scales, with up to 4.4×/2.6× throughput gains (the metadata abstract claims 8.2×/4.3×). The paper also contains complexity analysis and wall-clock throughput measurements on an H100.

Significance. If the results hold, this would be the first demonstration that SSM denoisers are viable for discrete diffusion language modeling, a non-obvious and useful result: it decouples diffusion LM quality from quadratic attention and points to a concrete path for long-context diffusion inference. The hybrid exploration and the attempt at controlled scaling comparisons are also valuable. The throughput mechanism (linear vs. quadratic per-step cost) is sound in principle and does not depend on the confounded quality comparison. However, the central architectural-quality claim — that replacing attention with Mamba 'does not compromise quality' — is currently not established because the compared models differ substantially in parameter count.

major comments (4)
  1. [§4, Table 1] The controlled-comparison premise is violated. Table 1 lists DiffuTran-1.3B at 1.3B params but DiffuApriel-1.6B at 1.6B (+23%) and DiffuApriel-H-1.5B at 1.5B (+15%); at 0.5B the gaps are +29% (0.67B) and +17% (0.61B) versus 0.52B; at 240M the gaps are +21% and +8%. Yet §4 states 'the only difference lies in the internal mixer architecture' and claims an 'architecture-isolated evaluation.' Halving the MLP expansion ratio does not equalize total parameters. Consequently Tables 2–4 and the statement 'DiffuApriel-H+MLP surpasses DiffuTran across all datasets' may express a scale advantage rather than an SSM advantage. The throughput claims are unaffected, but the quality claim needs either matched-parameter models, same-compute retraining at equal parameter count, or a clear re-framing that the comparison is at budgeted-compute, not architecture-isolated, settings.
  2. [§4.2, Figure 2] The decoding protocol for the KV-cache baseline is inconsistent with the rest of the comparison. The text states that DiffuTran+KV caching uses 'a block size of 32 with 2 denoising steps per block,' while the caption says 'constant 128 decoding steps' for all methods. If the KV baseline uses a different total number of denoising steps or a different step schedule, the throughput comparison is not controlled. Please report the exact decoding step count, schedule, and block size for every curve, or remove the KV-cache comparison from the headline throughput claims.
  3. [Abstract (arXiv metadata) vs. body] The arXiv abstract claims 'up to 8.2x and 4.3x higher inference throughput' and states that 'cache-efficient block diffusion with Mamba mixers emerges as the only strategy that scales linearly...' — neither appears in the body. The body reports 4.4× and 2.6× throughput and explicitly defers block diffusion to future work (§4.2, §5). The version deposited under arXiv:2511.15927 therefore misrepresents the paper's actual results. The abstract and body must be aligned, and the unsupported block-diffusion claim must be removed or supported with experiments.
  4. [§3.2, Eqs. (4)–(6)] The formal definition of the bidirectional Mamba block is not faithful to Mamba-2 and is not reproducible as written. Equations (4)–(5) describe fixed learnable kernels A_f, B_f applied as 1D convolutions/scan recurrences, while Mamba-2's selective SSM is input-dependent (B, C are projections of the input) and computed via structured state-space duality, not a simple additive fusion of forward/backward scans as in Eq. (6). If the implementation actually uses the Mamba-2 kernel, the equations should be replaced with the exact block specification or a pseudocode reference; otherwise readers cannot tell whether the experiments use standard Mamba-2 or a simplified recurrent layer.
minor comments (5)
  1. [Title/header] The body title is 'DiffuApriel' while the arXiv metadata title is 'DiffuMamba.' This inconsistency is confusing and must be resolved in the revised version.
  2. [§4.3] The text says DiffuApriel-H+MLP gives 'roughly a 2% perplexity reduction' at 1.3B. From Table 2, 25.01→22.89 is a ~8.5% relative reduction and 22.72→20.17 is ~11.2% relative. The reported number is arithmetically wrong.
  3. [§5] The concluding limitation — 'performance advantages diminish on short contexts' — is not supported by any presented data. No short-context quality or throughput results are shown. Either add supporting analysis or remove the claim.
  4. [Tables 3–5] No error bars, seeds, or repeated-run statistics are reported for any perplexity or accuracy table. For a paper whose central claim is 'consistently outperform,' at least one repeated-seed evaluation or a statement about training variance is needed. This is particularly relevant in Table 4, where several differences are only 2–4 points.
  5. [Eq. (1)] The notation δ_{x_i^0} and δ_{MASK} is undefined; these are point masses on the token and [MASK], respectively. Please define them explicitly.

Circularity Check

0 steps flagged

No significant circularity: theoretical derivation is self-contained; empirical claims are measured against an external baseline, with minor self-citation burden and abstract overclaims.

full rationale

I walked the derivation chain: the masked diffusion objective (Eq. 2) is standard; the BiMamba denoiser (Eqs. 4-6) is a new architecture; the output distribution (Eq. 11) and loss are the same as MDM, so 'preserving the probabilistic structure' is a design property, not a circular derivation. The linear-time claim follows from standard O(B L d) vs O(B L^2 d) complexity and is supported by wall-clock measurements (Figure 2), not by fitted constants. The quality tables are empirical comparisons against an independently trained DiffuTran baseline; no 'prediction' is algebraically forced by a fitted parameter. Self-citations (Radhakrishna et al. 2025a,b; Ostapenko et al. 2025; Fathi et al. 2025) are motivational or taxonomic and are not load-bearing: no uniqueness theorem or ansatz is imported from them. I flag two non-circular problems: (1) the abstract's 'up to 8.2x and 4.3x' and 'cache-efficient block diffusion ... emerges as the only strategy that scales linearly' are unsupported by the body, which reports 4.4x/2.6x and says 'We leave this direction for future work'; (2) Table 1's parameter mismatch (1.6B/1.5B vs 1.3B) confounds the architecture-quality claim, since equalizing parameter counts is claimed but not achieved ('only difference lies in the internal mixer architecture'). These are correctness/validity concerns, not circularity. Score 2 reflects the minor self-citation burden only.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

No fitted scientific constants or invented entities. The load-bearing choices are all protocol/architecture hyperparameters (K=5, S=128, d_state=128, KV block size 32), plus the validity of the standard masked-diffusion objective and the LLaDA-style sampling schedule. The main hidden assumption is that the comparison is architecture-isolated despite different parameter counts (Table 1).

free parameters (4)
  • K (attention insertion period in hybrid) = 5
    Hand-chosen and fixed across all experiments; controls the hybrid throughput/quality trade-off; no ablation is provided (§3.2, Table 1).
  • Number of decoding steps S = 128
    Fixed for all models following the LLaDA protocol; every throughput and latency measurement scales with S (§4.1).
  • Mamba state dimension d_state = 128
    Architectural hyperparameter fixed for all Mamba models; not ablated (Table 1).
  • KV-cache block size for DiffuTran+KV baseline = 32 (2 denoising steps per block)
    Taken from Wu et al. 2025; determines where the baseline turns over (~10–16K tokens) and hence the shape of the throughput comparison (Figure 2 caption).
axioms (5)
  • standard math The masked-diffusion training objective (Eq. 2) is a valid reweighted variational bound for absorbing-state discrete diffusion.
    Invoked in §3.1 and cited to MDLM/RADD; not re-derived in the paper.
  • domain assumption The MCMC approximate reverse process (Eq. 3) yields valid samples with 128 decoding steps.
    Sampling protocol inherited from LLaDA (Nie et al. 2025); all quality and throughput measurements depend on it.
  • ad hoc to paper Additive fusion of forward and backward scans (Eq. 6) provides a symmetric bidirectional context for denoising.
    The paper's own BiMamba definition; it is a simplification relative to Mamba-2 selective scans, and no analysis compares additive vs concatenated/inner-outer fusion.
  • domain assumption Wall-clock throughput at batch size 1, bf16, H100, with CUDA graphs is the right efficiency measure.
    §4.1; ignores training cost, multi-batch serving throughput, and kernel-level variance.
  • domain assumption Chinchilla and Quokka compute budgets are adequately matched across the compared models.
    Table 2 reports PPL under both budgets, but Table 1 lists a single token count per scale (25/50/120B); it is unclear which budget each run followed and how the differing parameter counts were squared with each budget.

pith-pipeline@v1.3.0-alltime-deepseek · 15626 in / 18427 out tokens · 156108 ms · 2026-08-03T21:14:13.287592+00:00 · methodology

0 comments
read the original abstract

Diffusion language models (DLMs) have emerged as a promising alternative to autoregressive (AR) generation, yet their reliance on Transformer backbones limits inference efficiency due to quadratic attention or KV-cache overhead. We introduce DiffuMamba, a masked diffusion language model built on a bidirectional Mamba backbone that combines the diffusion objective with linear-time sequence modeling, and DiffuMamba-H, a hybrid variant with interleaved attention. Across scales up to 1.3B parameters, our models match Transformer-based diffusion in downstream performance while achieving up to 8.2x and 4.3x higher inference throughput, respectively, on long sequences. We further present a systematic analysis of inference efficiency across modern DLM variants combining asymptotic complexity with empirical measurements. Notably, cache-efficient block diffusion with Mamba mixers emerges as the only strategy that scales linearly with sequence length and achieves the strongest performance across all baselines, suggesting a promising direction for future diffusion-based generation systems.

Figures

Figures reproduced from arXiv: 2511.15927 by Eugene Belilovsky, Oleksiy Ostapenko, Pierre-Andr\'e No\"el, Torsten Scholak, Vaibhav Singh.

Figure 1
Figure 1. Figure 1: Schematic diagram of our proposed DiffuApriel architecture where mixer blocks replaces attention layers with bidirectional Mamba layers. In our experiments, to maintain comparability with DiffuTran, we treat the MLP layer as optional and refer to this variant as DiffuApriel+MLP. For DiffuApriel-H we have interleaved attention layers after every K mamba layers. Attention provides global token interactions w… view at source ↗
Figure 2
Figure 2. Figure 2: Inference throughput and Model’s Latency per forward pass vs. sequence length with a batch size of 1 and constant 128 decoding steps. At 1.3B scale, DiffuApriel+MLP and DiffuApriel-H+MLP yields 4.4× and 2.6× throughput improvement over DiffuTran respectively. Further DiffuTran+KV caching (block size = 32) (Wu et al., 2025) boosts throughput up to 2048 tokens but degrades beyond 16K, eventually underperform… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

38 extracted references · 29 linked inside Pith

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

    Pith/arXiv arXiv

  2. [5]

    As expected, the absolute scores remain modest at this scale, reflecting the difficulty of these reasoning and knowledge-intensive benchmarks at 1.3B scale. Nonetheless, a consistent trend emerges across all tasks:DiffuAprieland DiffuApriel-Hclearly outperformDiffuTranby 3.3 points on average, indicating that linear-time state space modeling provides a st...

    2018

  3. [7]

    A cheaper and better diffusion language model with soft-masked noise

    Jiaao Chen, Aston Zhang, Mu Li, Alex Smola, and Diyi Yang. A cheaper and better diffusion language model with soft-masked noise. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,

    2023

  4. [8]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457,

    Pith/arXiv arXiv

  5. [10]

    Griffin: Mix- ing gated linear recurrences with local attention for efficient language models.arXiv preprint arXiv:2402.19427,

    10 Soham De, Samuel L Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, et al. Griffin: Mix- ing gated linear recurrences with local attention for efficient language models.arXiv preprint arXiv:2402.19427,

    Pith/arXiv arXiv

  6. [11]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao Goel. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752,

    Pith/arXiv arXiv

  7. [12]

    Training compute-optimal large language models.arXiv preprint arXiv:2203.15556,

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models.arXiv preprint arXiv:2203.15556,

    Pith/arXiv arXiv

  8. [13]

    Vm-ddpm: Vision mamba diffusion for medical image synthesis.arXiv preprint arXiv:2405.05667,

    Zhihan Ju and Wanting Zhou. Vm-ddpm: Vision mamba diffusion for medical image synthesis.arXiv preprint arXiv:2405.05667,

    Pith/arXiv arXiv

  9. [14]

    Mercury: Ultra-fast language models based on diffusion.arXiv preprint arXiv:2506.17298,

    Inception Labs, Samar Khanna, Siddhant Kharbanda, Shufan Li, Harshit Varma, Eric Wang, Sawyer Birnbaum, Ziyang Luo, Yanis Miraoui, Akash Palrecha, et al. Mercury: Ultra-fast language models based on diffusion.arXiv preprint arXiv:2506.17298,

    Pith/arXiv arXiv

  10. [15]

    Diffuspec: Unlocking diffusion language models for speculative decoding.arXiv preprint arXiv:2510.02358, 2025a

    Guanghao Li, Zhihui Fu, Min Fang, Qibin Zhao, Ming Tang, Chun Yuan, and Jun Wang. Diffuspec: Unlocking diffusion language models for speculative decoding.arXiv preprint arXiv:2510.02358, 2025a. Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Guha, Sedrick Scott Keh, Kushal Arora, et al. Datacomp-l...

    arXiv

  11. [16]

    A survey on diffusion language models

    Tianyi Li, Mingda Chen, Bowei Guo, and Zhiqiang Shen. A survey on diffusion language models. arXiv preprint arXiv:2508.10875, 2025b. 11 Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al. Jamba: A hybrid transformer- mamba language model.arXiv preprint ...

    Pith/arXiv arXiv

  12. [17]

    Logiqa: A challenge dataset for machine reading comprehension with logical reasoning.arXiv preprint arXiv:2007.08124,

    Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning.arXiv preprint arXiv:2007.08124,

    Pith/arXiv arXiv 2007

  13. [18]

    Vision mamba: A comprehensive survey and taxonomy.IEEE Transactions on Neural Networks and Learning Systems, 2025a

    Xiao Liu, Chenxu Zhang, Fuxiang Huang, Shuyin Xia, Guoyin Wang, and Lei Zhang. Vision mamba: A comprehensive survey and taxonomy.IEEE Transactions on Neural Networks and Learning Systems, 2025a. Yangzhou Liu, Yue Cao, Hao Li, Gen Luo, Zhe Chen, Weiyun Wang, Xiaobo Liang, Biqing Qi, Lijun Wu, Changyao Tian, et al. Sequential diffusion language models.arXiv...

    arXiv

  14. [19]

    dkv-cache: The cache for diffusion language models.arXiv preprint arXiv:2505.15781,

    Xinyin Ma, Runpeng Yu, Gongfan Fang, and Xinchao Wang. dkv-cache: The cache for diffusion language models.arXiv preprint arXiv:2505.15781,

    Pith/arXiv arXiv

  15. [21]

    Training optimal large diffusion language models.arXiv preprint arXiv:2510.03280,

    Jinjie Ni, Qian Liu, Chao Du, Longxu Dou, Hang Yan, Zili Wang, Tianyu Pang, and Michael Qizhe Shieh. Training optimal large diffusion language models.arXiv preprint arXiv:2510.03280,

    arXiv

  16. [22]

    Large language diffusion models.arXiv preprint arXiv:2502.09992,

    Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji- Rong Wen, and Chongxuan Li. Large language diffusion models.arXiv preprint arXiv:2502.09992,

    Pith/arXiv arXiv

  17. [23]

    Apriel-h1: Towards efficient enterprise reasoning models.arXiv preprint arXiv:2511.02651,

    Oleksiy Ostapenko, Luke Kumar, Raymond Li, Denis Kocetkov, Joel Lamy-Poirier, Shruthan Rad- hakrishna, Soham Parikh, Shambhavi Mishra, Sebastien Paquet, Srinivas Sunkara, et al. Apriel-h1: Towards efficient enterprise reasoning models.arXiv preprint arXiv:2511.02651,

    arXiv

  18. [24]

    Your absorbing discrete diffusion secretly models the conditional distributions of clean data.arXiv preprint arXiv:2406.03736,

    Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data.arXiv preprint arXiv:2406.03736,

    Pith/arXiv arXiv

  19. [25]

    How efficient are diffusion language models? a critical examination of efficiency evaluation practices.arXiv preprint arXiv:2510.18480,

    Han Peng, Peiyu Liu, Zican Dong, Daixuan Cheng, Junyi Li, Yiru Tang, Shuo Wang, and Wayne Xin Zhao. How efficient are diffusion language models? a critical examination of efficiency evaluation practices.arXiv preprint arXiv:2510.18480,

    arXiv

  20. [27]

    From s4 to mamba: A comprehensive survey on structured state space models

    Shriyank Somvanshi, Md Monzurul Islam, Mahmuda Sultana Mimi, Sazzad Bin Bashar Polock, Gaurab Chhetri, and Subasish Das. From s4 to mamba: A comprehensive survey on structured state space models. InarXiv:2503.18970,

    Pith/arXiv arXiv

  21. [28]

    Dim: Diffusion mamba for efficient high-resolution image synthesis.arXiv preprint arXiv:2405.14224,

    Yao Teng, Yue Wu, Han Shi, Xuefei Ning, Guohao Dai, Yu Wang, Zhenguo Li, and Xihui Liu. Dim: Diffusion mamba for efficient high-resolution image synthesis.arXiv preprint arXiv:2405.14224,

    Pith/arXiv arXiv

  22. [29]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023a

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023a. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay B...

    Pith/arXiv arXiv

  23. [30]

    A systematic analysis of hybrid linear attention.arXiv preprint arXiv:2507.06457, 2025a

    Dustin Wang, Rui-Jie Zhu, Steven Abreu, Yong Shan, Taylor Kergan, Yuqi Pan, Yuhong Chou, Zheng Li, Ge Zhang, Wenhao Huang, et al. A systematic analysis of hybrid linear attention.arXiv preprint arXiv:2507.06457, 2025a. Jinhong Wang, Jintai Chen, Danny Chen, and Jian Wu. Lkm-unet: Large kernel vision mamba unet for medical image segmentation. In Marius Geo...

    Pith/arXiv arXiv 2024

  24. [31]

    Diffusion llms can do faster-than-ar inference via discrete diffusion forcing.arXiv preprint arXiv:2508.09192, 2025b

    Xu Wang, Chenkai Xu, Yijie Jin, Jiachun Jin, Hao Zhang, and Zhijie Deng. Diffusion llms can do faster-than-ar inference via discrete diffusion forcing.arXiv preprint arXiv:2508.09192, 2025b. Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. Fast-dllm: Training-free acceleration of diffusion llm by...

    Pith/arXiv arXiv

  25. [32]

    Visual mamba: A survey and new outlooks.arXiv preprint arXiv:2404.18861,

    Rui Xu, Shu Yang, Yihui Wang, Yu Cai, Bo Du, and Hao Chen. Visual mamba: A survey and new outlooks.arXiv preprint arXiv:2404.18861,

    Pith/arXiv arXiv

  26. [33]

    Gated delta networks: Improving mamba2 with delta rule.arXiv preprint arXiv:2412.06464,

    Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule.arXiv preprint arXiv:2412.06464,

    Pith/arXiv arXiv

  27. [34]

    Dream 7b: Diffusion large language models.arXiv preprint arXiv:2508.15487,

    Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b: Diffusion large language models.arXiv preprint arXiv:2508.15487,

    Pith/arXiv arXiv

  28. [36]

    Llada 1.5: Variance-reduced preference optimization for large language diffusion models.arXiv preprint arXiv:2505.19223,

    Fengqi Zhu, Rongzhen Wang, Shen Nie, Xiaolu Zhang, Chunwei Wu, Jun Hu, Jun Zhou, Jianfei Chen, Yankai Lin, Ji-Rong Wen, et al. Llada 1.5: Variance-reduced preference optimization for large language diffusion models.arXiv preprint arXiv:2505.19223,

    Pith/arXiv arXiv

  29. [37]

    Falcon-h1: A family of hybrid-head language models redefining efficiency and performance.arXiv preprint arXiv:2507.22448,

    Jingwei Zuo, Maksim Velikanov, Ilyas Chahed, Younes Belkada, Dhia Eddine Rhayem, Guillaume Kunsch, Hakim Hacid, Hamza Yous, Brahim Farhat, Ibrahim Khadraoui, et al. Falcon-h1: A family of hybrid-head language models redefining efficiency and performance.arXiv preprint arXiv:2507.22448,

    Pith/arXiv arXiv

  30. [2013]

    Minimax-m1: Scaling test-time compute efficiently with lightning attention.arXiv preprint arXiv:2506.13585,

    Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, et al. Minimax-m1: Scaling test-time compute efficiently with lightning attention.arXiv preprint arXiv:2506.13585,

    Pith/arXiv arXiv

  31. [2015]

    A survey on efficient inference for large language models.arXiv preprint arXiv:2404.14294,

    Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, et al. A survey on efficient inference for large language models.arXiv preprint arXiv:2404.14294,

    Pith/arXiv arXiv

  32. [2017]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,

    2018

  33. [2019]

    Apriel-nemotron- 15b-thinker.arXiv preprint arXiv:2508.10948, 2025a

    Shruthan Radhakrishna, Soham Parikh, Gopal Sarda, Anil Turkkan, Quaizar V ohra, Raymond Li, Dhruv Jhamb, Kelechi Ogueji, Aanjaneya Shukla, Oluwanifemi Bamgbose, et al. Apriel-nemotron- 15b-thinker.arXiv preprint arXiv:2508.10948, 2025a. Shruthan Radhakrishna, Aman Tiwari, Aanjaneya Shukla, Masoud Hashemi, Rishabh Maheshwary, Shiva Krishna Reddy Malay, Jas...

    Pith/arXiv arXiv

  34. [2020]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

    1901

  35. [2021]

    Hybrid architectures for language models: Systematic analysis and design insights.arXiv preprint arXiv:2510.04800,

    Sangmin Bae, Bilge Acun, Haroun Habeeb, Seungyeon Kim, Chien-Yu Lin, Liang Luo, Junjie Wang, and Carole-Jean Wu. Hybrid architectures for language models: Systematic analysis and design insights.arXiv preprint arXiv:2510.04800,

    Pith/arXiv arXiv

  36. [2023]

    Block diffusion: Interpolating between autoregres- sive and diffusion language models.arXiv preprint arXiv:2503.09573,

    Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Sub- ham Sekhar Sahoo, and V olodymyr Kuleshov. Block diffusion: Interpolating between autoregres- sive and diffusion language models.arXiv preprint arXiv:2503.09573,

    Pith/arXiv arXiv

  37. [2024]

    Transformers are ssms: Generalized models and efficient algorithms through structured state space duality.arXiv preprint arXiv:2405.21060,

    Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality.arXiv preprint arXiv:2405.21060,

    Pith/arXiv arXiv

  38. [2025]

    Characterizing the behavior of training mamba-based state space models on gpus.arXiv preprint arXiv:2508.17679,

    Trinayan Baruah, Kaustubh Shivdikar, Sara Prescott, and David Kaeli. Characterizing the behavior of training mamba-based state space models on gpus.arXiv preprint arXiv:2508.17679,

    Pith/arXiv arXiv