Pith. sign in

REVIEW 4 major objections 3 minor 62 references

MDLMPE: Distribution Aware Positional Encoding for Masked Diffusion Language Models

T0 review · 4 major / 3 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read For masked diffusion language models, the changing revealed/masked pattern is itself a positional signal, and encoding it explicitly — as MDLMPE does — generally beats RoPE across fine-tuning, pretraining, zero-shot evaluation, and block di

desk verdict A genuinely novel positional encoding for masked diffusion LMs with a solid formal apparatus, but the empirical claim is undercut by a confounded SFT protocol and missing uncertainty estimates. read the letter →

arxiv 2608.03769 v1 pith:EF6CTVU7 submitted 2026-08-04 cs.CL cs.AI

Tong Ling , Hang Lei , Feng Xiao , Changhui Sun , Jiahang Xie , Hao Liu , Lu Liu , Yanlong Du This is my paper · ORCID
classification cs.CLcs.AI
keywords maskeddiffusionlanguagemodelspositionalencodingRoPEtokenavailabilitymaskpatternrotaryphaseresidualLLaDADREAM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Masked diffusion language models do not decode left-to-right: they reveal tokens in a changing, non-contiguous order, so the same absolute position can have very different usable context at different denoising steps. This paper argues that standard positional encodings miss that structure, and that the evolving revealed/masked configuration is itself a useful positional signal. To capture it, the paper introduces MDLMPE, which encodes which positions are revealed as a binary pattern, weighs it by distance with a Gaussian, projects it through a cosine basis aligned with RoPE's frequencies, and feeds the result into both the token embeddings and small phase adjustments on selected RoPE pairs. Across supervised fine-tuning of 7B/8B LLaDA and DREAM models, 100M-parameter pretraining, zero-shot perplexity, and block-diffusion post-training, MDLMPE generally improves over RoPE with the same checkpoints, data, and decoding budgets. If the claim holds, position encoding for diffusion language models should be state-dependent, not index-only.

What carries the argument

The load-bearing object is the availability encoder: a binary vector marking revealed versus masked positions; a row-normalized truncated Gaussian kernel making each position's view local and target-relative; and shifted-sine coordinates on RoPE's own frequency grid. The ratio r_{i,f} = (weighted revealed mass)/(all-valid reference) lies in [0,1] and distinguishes mask layouts with identical revealed counts by spatial arrangement. The encoder feeds two paths: a gated embedding residual e_i + γW_p u_i, and a pair-conditioned phase residual δ_{ij,f} = τ_max tanh(MLP(arccos(2r_ij−1) − π/2)) applied to roughly half of the interleaved RoPE pairs, so attention angles become (j−i)ω_f + δ_{ij,f}. Ro

What would settle it

Train the RoPE baseline with the same added parameters — the embedding projection and phase MLP — fed a constant input, and apply the identical progressive head-activation schedule of Eq. 27. If that matched-capacity baseline reproduces MDLMPE's numbers, the availability encoding is not the cause. A cheaper check: rerun the 100M pretraining comparisons over several seeds; overlapping variance bands would undercut the improvement claim. The paper's own deterministic analysis says the availability ratio's range collapses at revealed fractions 0 and 1, so testing MDLMPE at very high or very low m

Watch

Extended reading notes

Core claim

The paper claims that in masked diffusion language models the evolving revealed/masked configuration is a positional signal standard encodings miss, and that encoding it explicitly improves the model. MDLMPE builds a content-free binary availability signal per position, applies target-centered Gaussian weighting, and projects the pattern through a cosine basis on RoPE's frequency grid, yielding an availability ratio that distinguishes equal-density layouts. The signal enters twice: as a gated addition to token embeddings, and, for about half the RoPE pairs, as a bounded angular residual from an MLP over the pair-averaged ratio, preserving RoPE's displacement phase. On LLaDA and DREAM, MDLMPE

Load-bearing premise

The matched comparisons assume the gains come from the availability signal itself: no RoPE baseline was run with the same added parameters (embedding projection and phase MLP) and the same progressive head-activation schedule, and all results are single runs with no variance, so the reported improvements cannot be cleanly attributed to the positional signal rather than to extra capacity or the warmup.

Editorial extensions

If this is right

  • MDLMPE can be added to existing RoPE-pretrained MDLM checkpoints through instruction post-training, improving knowledge, reasoning, and code tasks on both LLaDA-8B and DREAM-7B without changing the backbone, tokenizer, or decoding budget.
  • From-scratch 100M pretraining with MDLMPE lowers validation and zero-shot perplexity on external corpora relative to RoPE and ALiBi, suggesting the availability signal is a useful inductive bias, not only a post-training patch.
  • The same encoder transfers to blockwise denoising: matched block-size experiments show MDLMPE improvements over RoPE without a block identifier, because the signal is recomputed from whatever visibility pattern attention actually sees.
  • The component ablation indicates that binary state, Gaussian locality, spectral basis, and embedding injection are complementary; the strongest results require the complete combination, and intermediate configurations can underperform even the RoPE baseline.
  • Because the representation provably distinguishes equal-density mask layouts, MDLMPE gives the model a positional coordinate that can tell apart different arrangements of the same number of revealed tokens — something index-based and displacement-based encodings cannot do.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not test: the advantage of availability-aware positioning should grow as the reveal order becomes more non-contiguous — e.g., fully parallel or random-order decoding versus near-prefix schedules — because those regimes create more information-rich mask patterns.
  • The fixed Gaussian window ratio and bandwidth are tuned on one setting; making the window or the frequency allocation learnable per layer, or conditioning the phase residual on longer contexts, could extend the method to sequences where the mask topology changes at different scales.
  • The matched comparisons add parameters and a warmup schedule to MDLMPE but not to the RoPE baseline; if a capacity-matched and schedule-matched baseline erases the gap, the correct conclusion would be that the availability representation is a convenient way to spend extra capacity, not that the signal itself matters.
  • The deterministic analysis implies the signal saturates at fully masked and fully revealed extremes, so the method's practical value should concentrate in the intermediate denoising steps — a testable prediction for scheduling denoising so that intermediate revealed fractions get the most benefit.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. This paper proposes MDLMPE, a positional encoding for masked diffusion language models. It represents the current revealed/masked configuration as a binary availability sequence, applies a normalized truncated Gaussian weighting, projects the result through a shifted-sine basis on the RoPE frequency grid, and injects the resulting features through two paths: a gated additive term on token embeddings and an MLP-produced bounded phase residual applied to roughly half the rotary pairs. The authors evaluate MDLMPE on instruction post-training of LLaDA-8B and DREAM-7B (Tulu-3 and SmolTalk), controlled 100M-parameter pretraining with zero-shot perplexity, blockwise post-training, and component ablations on LLaDA-7B. They report broad gains over RoPE and ALiBi and conclude that the availability pattern is a useful positional signal for masked diffusion models.

Significance. The proposal is clearly specified and addresses a genuine gap: existing positional encodings ignore the evolving revealed/masked structure of MDLM decoding. The deterministic analyses in Appendix F are a clear strength, establishing exact availability-ratio envelopes, locality bounds, endpoint invariants, and complexity accounting without relying on empirical outcomes. No circular dependence on clean target tokens is apparent; the availability signal is computed from the actual masked input. If the empirical comparisons were clean, the contribution would be solid and likely useful to the MDLM community. However, the current evidence does not yet isolate the mechanism, and a few reported numbers contradict the 'generally outperforms' summary. The paper is therefore publishable in principle, but the empirical support needs strengthening.

major comments (4)
  1. [§4.2, §B.4, Eq. (27)] The headline SFT evidence is confounded. All MDLMPE runs use the progressive head-activation schedule of Eq. (27), while the RoPE baselines do not. At initialization the phase MLP is zero-initialized and the embedding gate is 0.01, so MDLMPE begins near RoPE and gradually departs during post-training. The schedule alone could improve checkpoint adaptation independently of the mask signal, and MDLMPE adds trainable parameters beyond RoPE. No schedule-matched RoPE baseline, no parameter-matched RoPE baseline, and no MDLMPE-without-schedule control are reported. Because Table 1 is the primary evidence for checkpoint compatibility, the gains cannot yet be attributed to the availability signal itself.
  2. [§4.3, Table 2] The pretraining results contain regressions that contradict the abstract's 'generally outperforms' claim. On LLaDA-100M, PTB perplexity is 170.18 for MDLMPE vs 144.55 for RoPE and 135.60 for ALiBi, and WT103 is 104.84 vs 103.12; DREAM-100M Text8 is 107.10 vs ALiBi's 103.46. All pretraining results are single-run with no variance. Multi-seed training, or at least a paired bootstrap over test tokens, and a clear statement of which differences are statistically distinguishable are needed before claiming general superiority.
  3. [§4.4, Table 3] The blockwise results are mixed and generally small. For example, LLaDA-8B block-16 ARC-Challenge is 60.1 for MDLMPE vs 60.7 for RoPE, and DREAM-7B block-64 HellaSwag is 65.9 vs 66.6; the largest positive differences are 1–2 percentage points. No error bars or significance tests are provided. This section can support robustness only if the positive direction is consistent and distinguishable from noise, which is not demonstrated.
  4. [§4.1, §C.1–C.2] The default geometry W/L=3/16 and rho=0.25 is selected on the 50K/2K Tulu SFT split (Tables 5–6) and then applied to all other settings. There is no held-out validation split and no sensitivity analysis across corpora. The reported gains could therefore be partly a product of tuning on one evaluation set. The paper should report the hyperparameters' sensitivity on at least one other dataset or on a separate validation split.
minor comments (3)
  1. [Table 1] The table's floating-point entries are concatenated without visible separators (e.g., '60.360.5' appears to be '60.3, 60.5'). This makes the table nearly unreadable; add explicit delimiters or columns.
  2. [Affiliation] The first affiliation is printed as 'University Chinese Academic of Science'; this appears to be 'University of Chinese Academy of Sciences'. Please correct.
  3. [§4.5, Table 4] The component analysis reports evaluation perplexity on only 2,000 examples with no confidence interval or multi-seed variance. The phrase 'the strongest result' is too strong; qualify it as 'best among the single runs reported'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: MDLMPE's availability feature is an input-derived signal, not a fitted prediction or self-citation-dependent claim.

full rationale

The paper's central novelty is a positional feature computed from the current revealed/masked configuration (Eq. 1: b_t = I[x_t != x_mask] a_t). This is an input feature, not a target quantity: Appendix A.2 (Eq. 14) explicitly proves invariance of the aggregated feature to the clean token y_t, ruling out target leakage. The empirical claim that MDLMPE outperforms RoPE is supported by matched experiments (Tables 1-3) and component ablations (Table 4). Hyperparameters are tuned on the evaluation set (Appendix C), which is standard model selection rather than a circular derivation, and no fitted parameter is renamed as a prediction. No load-bearing self-citations or imported uniqueness theorems appear; citations to LLaDA, DREAM, and RoPE are external foundations. The progressive head-activation schedule (Eq. 27) and added parameters are potential experimental confounds for isolating the availability signal, but confounds are control issues, not circularity: the MDLMPE output is not equal to its inputs by construction, and no derived quantity is assumed from the result it is supposed to explain. The deterministic mechanism analysis (Appendix F) is explicitly separated from task-quality claims. Therefore no circular step is present.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The method relies on a domain premise (availability is a useful positional signal), several hand-chosen geometric hyperparameters, and an untested matching assumption in the SFT protocol. No new physical or conceptual entities are introduced.

free parameters (6)
  • Window ratio W/L = 3/16 (W=192 at L=1024)
    Selected by ablation in Appendix C.1 as the best among six settings on the same 2K-example Tulu evaluation set.
  • Bandwidth ratio rho (sigma/W) = 0.25
    Selected by ablation in Appendix C.2 on the same evaluation set.
  • Max phase residual tau_max = 0.25
    Hand-set cap on the phase residual; no sensitivity analysis is reported.
  • Embedding gate gamma max and init = 0.10 / 0.01
    Hand-set gate bound and initialization; ensures a small perturbation at the start of fine-tuning.
  • MLP hidden size H = max(16, F/2)
    Default hidden size for the phase MLP, not ablated.
  • Progressive head activation schedule = linearly from 0 at 10% of updates to 1 at 90%
    MDLMPE-specific warmup applied only to MDLMPE, confounding the post-training comparison with the RoPE baseline.
assumptions (5)
  • domain assumption The revealed/masked binary pattern is a useful positional signal for MDLM denoising.
    Central premise of the paper; the entire method is built on this assumption, stated in Section 1 and restated in the conclusion.
  • domain assumption Adding a bounded phase residual to approximately half of the RoPE pairs preserves the pretrained relative-position geometry.
    Invoked in Section 3.2 and Appendix A.3 (Eq. 20); the bound is shown, but it is assumed that the perturbation does not degrade the backbone.
  • domain assumption The Gaussian-weighted, cosine-basis aggregate of the mask pattern is a sufficient representation of the availability distribution.
    The representation is designed in Section 3.1; its sufficiency is argued via deterministic analyses in Appendix F but not proven to be optimal.
  • ad hoc to paper The hand-chosen geometric hyperparameters (W/L=3/16, rho=0.25) generalize beyond the Tulu 50K validation set.
    These values are selected by ablations in Appendix C on the same evaluation set used for the main component analysis, with no held-out validation for the selection.
  • domain assumption The evaluation protocol's matched controls make RoPE and MDLMPE comparable despite the MDLMPE-specific head warmup and additional parameters.
    Section B.4 describes the warmup but provides no matching warmup for the baseline, so this assumption is untested and load-bearing for the post-training results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MDLMPE: Distribution Aware Positional Encoding for Masked Diffusion Language Models." pith.science (2026). https://pith.science/paper/EF6CTVU7

@misc{pith2026260803769,
  author       = {Pith},
  title        = {Pith review of: MDLMPE: Distribution Aware Positional Encoding for Masked Diffusion Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EF6CTVU7}},
  note         = {Machine review of arXiv:2608.03769}
}
read the original abstract

Masked diffusion language models (MDLMs) enable parallel generation and bidirectional context modeling, but their positional context differs fundamentally from that of autoregressive (AR) models. Whereas AR decoding exposes a contiguous prefix, MDLM denoising produces dynamic, non-contiguous configurations of revealed and masked tokens. Conventional positional encodings such as RoPE capture sequence order and pairwise displacement but remain insensitive to this evolving token-availability structure. To address this limitation, we propose MDLMPE, a positional encoding designed specifically for masked diffusion. To the best of our knowledge, MDLMPE is the first method to make positional representations explicitly aware of the changing revealed/masked configuration. It represents token availability as a binary sequence, applies distance-aware Gaussian weighting, and projects the resulting pattern through a cosine basis to obtain distribution-aware positional features. These features are added to token embeddings and mapped by a lightweight MLP to angular offsets that modulate the standard RoPE phases. Extensive experiments on LLaDA and DREAM demonstrate that MDLMPE generally outperforms conventional positional encoding methods across supervised fine-tuning, pretraining, zero-shot evaluation, and block-diffusion settings. Further ablations show that the complete combination of availability state, Gaussian locality, spectral basis, and embedding injection yields the strongest result. These results establish the evolving token-availability distribution as a useful positional signal for masked diffusion language models.

Figures

Figures reproduced from arXiv: 2608.03769 by the authors.

Figure 1
Figure 1. Position availability in AR and masked diffusion generation. AR exposes a contiguous prefix, whereas MDLM [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Instruction post-training accuracies for the four table settings, ordered left to right as LLaDA-8B Tulu, DREAM-7B [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Gaussian local encoding for one position (top) and [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Equal-density availability patterns produce dif [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: System-level view of the high-level two-branch architecture. The token feature [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Blocked-context and full-sequence block diffusion. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 8
Figure 8. Figure 8: Frequency–distance transfer for one revealed source [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 10
Figure 10. Figure 10: Deterministic ratio-to-intermediate-coordinate [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 9
Figure 9. Figure 9: Exact single-state influence and analytical local [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 11
Figure 11. Figure 11: Boundary normalization for length 256, W = 64, and σ = 16. (a) Truncation reduces unnormalized Gaussian row mass near sequence ends. (b) Renormalized kernels at the left boundary, center, and right boundary each have unit mass despite asymmetric support. (c) Consequen…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 27 canonical work pages

  1. [1]

    Advances in Neural Information Processing Systems , volume=

    Attention is all you need , author=. Advances in Neural Information Processing Systems , volume=

  2. [2]

    Neurocomputing , volume=

    RoFormer: Enhanced transformer with rotary position embedding , author=. Neurocomputing , volume=. 2024 , publisher=

    work page 2024

  3. [3]

    International Conference on Learning Representations , year=

    Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation , author=. International Conference on Learning Representations , year=

  4. [4]

    arXiv preprint arXiv:2306.15595 , year=

    Extending context window of large language models via positional interpolation , author=. arXiv preprint arXiv:2306.15595 , year=

    arXiv

  5. [5]

    International Conference on Learning Representations , year=

    YaRN: Efficient Context Window Extension of Large Language Models , author=. International Conference on Learning Representations , year=

    work page

  6. [6]

    Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=

    Self-Attention with Relative Position Representations , author=. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=

    work page 2018

  7. [7]

    Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina , booktitle=

  8. [8]

    Advances in Neural Information Processing Systems , volume=

    Denoising Diffusion Probabilistic Models , author=. Advances in Neural Information Processing Systems , volume=

Show all 62 references
  1. [9]

    Advances in Neural Information Processing Systems , volume=

    Structured Denoising Diffusion Models in Discrete State-Spaces , author=. Advances in Neural Information Processing Systems , volume=

  2. [10]

    Diffusion-

    Li, Xiang Lisa and Thickstun, John and Gulrajani, Ishaan and Liang, Percy and Hashimoto, Tatsunori B , booktitle=. Diffusion-

  3. [11]

    International Conference on Machine Learning , year=

    Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution , author=. International Conference on Machine Learning , year=

  4. [12]

    Advances in Neural Information Processing Systems , volume=

    Simplified and Generalized Masked Diffusion for Discrete Data , author=. Advances in Neural Information Processing Systems , volume=

  5. [13]

    International Conference on Learning Representations , year=

    Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data , author=. International Conference on Learning Representations , year=

  6. [14]

    Advances in Neural Information Processing Systems , year=

    Large Language Diffusion Models , author=. Advances in Neural Information Processing Systems , year=

  7. [15]

    arXiv preprint arXiv:2508.15487 , year=

    Dream 7B: Diffusion Large Language Models , author=. arXiv preprint arXiv:2508.15487 , year=

    arXiv

  8. [16]

    arXiv preprint arXiv:2512.15745 , year=

    LLaDA2.0: Scaling Up Diffusion Language Models to 100B , author=. arXiv preprint arXiv:2512.15745 , year=

    arXiv

  9. [17]

    Mahabadi, Rabeeh Karimi and Ivison, Hamish and Tae, Jaesung and Henderson, James and Beltagy, Iz and Peters, Matthew E and Cohan, Arman , booktitle=

  10. [18]

    arXiv preprint arXiv:2508.02193 , year=

    Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference , author=. arXiv preprint arXiv:2508.02193 , year=

    arXiv

  11. [19]

    OpenAI blog , volume=

    Language models are unsupervised multitask learners , author=. OpenAI blog , volume=

  12. [20]

    arXiv preprint arXiv:2302.13971 , year=

    LLaMA: Open and Efficient Foundation Language Models , author=. arXiv preprint arXiv:2302.13971 , year=

    arXiv

  13. [21]

    OpenWebText Corpus , author=

  14. [22]

    Lambert, Nathan and Morrison, Jacob and Pyatkin, Valentina and Huang, Shengyi and Ivison, Hamish and Brahman, Faeze and Miranda, Lester James V and Liu, Alisa and Dziri, Nouha and Lyu, Shane and others , journal=. T

  15. [23]

    2025 , eprint=

    SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model , author=. 2025 , eprint=

    2025

  16. [24]

    Paperno, Denis and Kruszewski, Germ. The. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics , pages=

  17. [25]

    arXiv preprint arXiv:1609.07843 , year=

    Pointer Sentinel Mixture Models , author=. arXiv preprint arXiv:1609.07843 , year=

    arXiv

  18. [26]

    Building a Large Annotated Corpus of

    Marcus, Mitchell P and Santorini, Beatrice and Marcinkiewicz, Mary Ann , journal=. Building a Large Annotated Corpus of

  19. [27]

    Text8 Dataset , author=

  20. [28]

    arXiv preprint arXiv:1312.3005 , year=

    One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling , author=. arXiv preprint arXiv:1312.3005 , year=

    arXiv

  21. [29]

    International Conference on Learning Representations , year=

    Measuring Massive Multitask Language Understanding , author=. International Conference on Learning Representations , year=

  22. [30]

    Wang, Yubo and Ma, Xueguang and Zhang, Ge and Ni, Yuansheng and Chandra, Abhranil and Guo, Shiguang and Ren, Weiming and Arulraj, Aaran and He, Xuan and Jiang, Ziyan and others , journal=

  23. [31]

    Measuring Mathematical Problem Solving With the

    Hendrycks, Dan and Burns, Collin and Kadavath, Saurav and Arora, Akul and Basart, Steven and Tang, Eric and Song, Dawn and Steinhardt, Jacob , booktitle=. Measuring Mathematical Problem Solving With the

  24. [32]

    Transactions on Machine Learning Research , year=

    Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models , author=. Transactions on Machine Learning Research , year=

  25. [33]

    arXiv preprint arXiv:2107.03374 , year=

    Evaluating Large Language Models Trained on Code , author=. arXiv preprint arXiv:2107.03374 , year=

    arXiv

  26. [34]

    arXiv preprint arXiv:2108.07732 , year=

    Program Synthesis with Large Language Models , author=. arXiv preprint arXiv:2108.07732 , year=

    arXiv

  27. [35]

    C-Eval: A Multi-Level Multi-Discipline

    Huang, Yuzhen and Bai, Yuzhuo and Zhu, Zhihao and Zhang, Junlei and Zhang, Jinghan and Su, Tangjun and Liu, Junteng and Lv, Chuancheng and Zhang, Yikai and Lei, Jiayi and others , journal=. C-Eval: A Multi-Level Multi-Discipline

  28. [36]

    Think you have Solved Question Answering? Try

    Clark, Peter and Cowhey, Isaac and Etzioni, Oren and Khot, Tushar and Sabharwal, Ashish and Schoenick, Carissa and Tafjord, Oyvind , journal=. Think you have Solved Question Answering? Try

  29. [37]

    Zellers, Rowan and Holtzman, Ari and Bisk, Yonatan and Farhadi, Ali and Choi, Yejin , booktitle=

  30. [38]

    Sakaguchi, Keisuke and Le Bras, Ronan and Bhagavatula, Chandra and Choi, Yejin , booktitle=

  31. [39]

    Lin, Stephanie and Hilton, Jacob and Evans, Owain , booktitle=

  32. [40]

    Bisk, Yonatan and Zellers, Rowan and Le Bras, Ronan and Gao, Jianfeng and Choi, Yejin , booktitle=

  33. [41]

    Advances in Neural Information Processing Systems , volume=

    Language Models are Few-Shot Learners , author=. Advances in Neural Information Processing Systems , volume=

  34. [42]

    Lewis, Mike and Liu, Yinhan and Goyal, Naman and Ghazvininejad, Marjan and Mohamed, Abdelrahman and Levy, Omer and Stoyanov, Veselin and Zettlemoyer, Luke , booktitle=

  35. [43]

    Journal of Machine Learning Research , volume=

    Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer , author=. Journal of Machine Learning Research , volume=

  36. [44]

    International Conference on Machine Learning , pages=

    Insertion Transformer: Flexible Sequence Generation via Insertion Operations , author=. International Conference on Machine Learning , pages=

  37. [45]

    Transformer-

    Dai, Zihang and Yang, Zhilin and Yang, Yiming and Carbonell, Jaime and Le, Quoc V and Salakhutdinov, Ruslan , booktitle=. Transformer-

  38. [46]

    He, Pengcheng and Liu, Xiaodong and Gao, Jianfeng and Chen, Weizhu , booktitle=

  39. [47]

    International Conference on Machine Learning , pages=

    Improved Denoising Diffusion Probabilistic Models , author=. International Conference on Machine Learning , pages=

  40. [48]

    International Conference on Learning Representations , year=

    Score-Based Generative Modeling through Stochastic Differential Equations , author=. International Conference on Learning Representations , year=

  41. [49]

    Advances in Neural Information Processing Systems , volume=

    Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions , author=. Advances in Neural Information Processing Systems , volume=

  42. [50]

    International Conference on Learning Representations , year=

    DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models , author=. International Conference on Learning Representations , year=

  43. [51]

    Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing , pages=

    Mask-Predict: Parallel Decoding of Conditional Masked Language Models , author=. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing , pages=

    2019

  44. [52]

    Advances in Neural Information Processing Systems , volume=

    Levenshtein Transformer , author=. Advances in Neural Information Processing Systems , volume=

  45. [53]

    Chang, Huiwen and Zhang, Han and Jiang, Lu and Yang, Ce and Liu, Yiming , booktitle=. Mask

  46. [54]

    arXiv preprint arXiv:2110.14168 , year=

    Training Verifiers to Solve Math Word Problems , author=. arXiv preprint arXiv:2110.14168 , year=

    arXiv

  47. [55]

    Ding, Yiran and Zhang, Li Lyna and Zhang, Chengruidong and Xu, Yuanyuan and Shang, Ning and Xu, Jiahang and Yang, Fan and Yang, Mao , journal=

  48. [56]

    Zhu, Dawei and Yang, Nan and Wang, Liang and Song, Yifan and Wu, Wenhao and Wei, Furu and Li, Sujian , booktitle=

  49. [57]

    Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics , year=

    A Length-Extrapolatable Transformer , author=. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics , year=

  50. [58]

    International Conference on Learning Representations , year=

    Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models , author=. International Conference on Learning Representations , year=

  51. [59]

    2602.06036 , archivePrefix=

    Chen, Jian and Liang, Yesheng and Liu, Zhijian , year=. 2602.06036 , archivePrefix=

    arXiv

  52. [60]

    2023 , doi=

    He, Zhengfu and Sun, Tianxiang and Tang, Qiong and Wang, Kuanning and Huang, Xuanjing and Qiu, Xipeng , booktitle=. 2023 , doi=

    2023

  53. [61]

    Liu, Xiaoran and Song, Yuerong and Liu, Zhigeng and Huang, Zengfeng and Guo, Qipeng and He, Ziwei and Qiu, Xipeng , journal=

  54. [62]

    He, Guangxin and Nie, Shen and Zhu, Fengqi and Zhao, Yuankang and Bai, Tianyi and Yan, Ran and Fu, Jie and Li, Chongxuan and Yuan, Binhang , journal=

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.