Pith. sign in

REVIEW 1 major objections 6 minor 46 references

Masked diffusion language models trained on only 100 million words can match hybrid autoregressive-masked baselines on BabyLM, making diffusion a viable alternative for data-restricted language learning.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-05 05:38 UTC pith:QQLHGRNB

load-bearing objection A useful BabyLM empirical result with an honest write-up, but the theoretical-validity claim for frequency-informed masking is unsupported as written and needs a derivation before it can be taken at face value. the 1 major comments →

arxiv 2509.05056 v1 pith:QQLHGRNB submitted 2025-09-05 cs.CL

Masked Diffusion Language Models with Frequency-Informed Training

Despoina Kosmopoulou , Efthymios Georgiou , Vaggelis Dorovatas , Georgios Paraskevopoulos , Alexandros Potamianos This is my paper
classification cs.CL
keywords masked diffusion language modelsBabyLMdata-efficient pretrainingfrequency-informed maskingnoise schedulesNELBOmasked language modelingdiffusion language models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Masked diffusion language models—models trained to reconstruct text from progressively masked versions through a diffusion-style objective—can be pretrained end-to-end on just 100 million words and match the performance of the hybrid GPT-BERT baseline that needs both a next-token and a masked-language objective. The authors show that a cosine noise schedule and a frequency-informed masking policy, which makes rarer tokens more likely to be masked, preserve or improve scores on BabyLM's zero-shot, finetuning, and human-likeness suites. They argue this establishes diffusion-based training as a viable alternative for data-restricted language learning, not merely a large-scale pretraining method. If true, it means bidirectional context and generative power can come from a single loss instead of a weighted combination of two.

Core claim

On the BabyLM 2025 Strict track (100M words, 10 epochs), a masked diffusion language model trained with the simplified MDLM NELBO, a cosine masking schedule, and frequency-informed per-token masking is competitive with the top hybrid GPT-BERT baseline: it trails by a few points on BLiMP, BLiMP Supplement, EWoK, and COMPS, beats it on Entity Tracking, on finetuned tasks such as MRPC, RTE, and WSC, and on human-likeness measures Reading and Adjective Nominalization. Ablations attribute part of the gain to frequency-informed masking—a consistent absolute +1 point on BLiMP Supplement and an average +7.5 points on Adjective Nominalization in paired-seed runs—and part to schedule design: cosine be

What carries the argument

The engine is the simplified continuous-time NELBO of masked diffusion (Eq. 1): at each step a masking rate 1−α_t is sampled, tokens are masked accordingly, and the model is trained to reconstruct them, with losses across rates weighted by α′_t/(1−α_t). The paper's novel addition is frequency-informed masking: tokens are ranked by global frequency (rarer = higher rank), min-max normalized, softened by a power p, and scaled so their mean equals the target masking rate 1−α_t (Eq. 2). This makes rare tokens more likely to be the ones the model must reconstruct, and increasing the softness p over training acts as a curriculum toward rarer tokens. Two further mechanisms carry much of the empirica

Load-bearing premise

The method stands on the assumption that masking tokens with different probabilities while using a single scalar noise schedule still optimizes the diffusion ELBO; if per-token masking breaks that identity, the frequency-informed objective is a heuristic reweighting rather than the principled diffusion bound.

What would settle it

Take the same corpus and model, and compare three pretraining runs: uniform token masking, the paper's frequency-informed masking, and a variant that computes the ELBO per token with its own schedule. If the frequency-informed run does not match the per-token-ELBO run on held-out pseudo-likelihood and on a synthetic distribution with known likelihoods, then Eq. (1) is not the ELBO under non-uniform masking. A cheaper check: derive the per-token importance weight and ask whether α′_t/(1−α_t) is still the correct weight when mask probabilities differ by token.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Diffusion pretraining can replace hybrid AR+MLM objectives in data-scarce settings, so competitive BabyLM training does not require a two-loss setup.
  • Any intra-token masking strategy (frequency, salience, span-based) can be plugged into the MDLM objective, so future work can tune masking policy independently of the diffusion schedule.
  • Noise schedule selection and derivative weighting are first-order controls: cosine beats linear, and bimodal Gaussian approaches top baselines only when the ELBO derivative is scaled down or dropped.
  • Because the MLM pseudo-likelihood backend is deterministic and cheap, MDLM zero-shot evaluation does not require time conditioning, simplifying the evaluation pipeline.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The conditional-scaling formula (Eq. 2) is generic: any token-level salience signal could replace frequency ranks, so the method is a template for importing salience-, span-, or PMI-style masking into discrete diffusion—the paper demonstrates only frequency.
  • The derivative-scaling trick (p in {0.1, 0.0}) is not part of the original NELBO derivation, which suggests the practical objective trades ELBO tightness for training stability; a principled reweighting that yields the same schedules would place the result on firmer ground.
  • The reported zero-shot scores use an MLM pseudo-likelihood backend that only exercises single-token denoising steps, so they likely understate the model's generative ability; Monte-Carlo integration over the full denoising trajectory could move the MDLM's ranking relative to the hybrid baseline.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 6 minor

Summary. The paper adapts masked diffusion language models (MDLMs) to the BabyLM 2025 strict track (100M words, up to 10 epochs). It proposes frequency-informed masking, which assigns token-specific mask probabilities based on normalized frequency ranks, and explores noise schedules (linear, cosine, Gaussian mixtures) and NELBO weighting variants. The authors report zero-shot, fine-tuning, and human-likeness scores for a 126.6M-parameter model and conclude that MDLMs are competitive with hybrid autoregressive-masked baselines. The main theoretical claim is that the frequency-informed masking 'preserves the theoretical validity' of the diffusion objective.

Significance. If the theoretical claim held, the paper would show a simple way to introduce curriculum-like, token-frequency-aware corruption into MDLM training while retaining the ELBO interpretation, and it would provide the first BabyLM evaluation of MDLMs. The empirical study is useful: it compares multiple schedules, runs several ablations with five seeds (Tables 1 and 3), and reports a broad set of benchmarks. However, the load-bearing theoretical assertion is not derived, and as written the training objective does not match the stated ELBO; the paper's main novelty is therefore currently unsupported. The empirical results are nonetheless interesting enough to warrant a careful revision.

major comments (1)
  1. [Section 3.1, Eq. (1)-(2)] The assertion that frequency-informed masking 'preserves the theoretical validity of the diffusion objective' (Abstract and §3.1) is not supported. Eq. (1) is the NELBO for a uniform schedule in which every token has the same unmask probability α_t. Under Eq. (2), token i has unmask probability u_i(t)=1−p_i(t). In the branch μ≤1−α_t, u_i(t)=d_i α_t with d_i=(1−w_i^p)/(1−μ), so the per-token NELBO weight is u_i'(t)/(1−u_i(t)) = d_i α'_t/(1−d_i α_t), which depends on i and reduces to the uniform α'_t/(1−α_t) only if d_i=1 for all i. The branch μ>1−α_t does give the uniform weight, but no derivation covers the whole schedule. Thus, as written, the loss in Eq. (1) is not the ELBO of the masking process defined by Eq. (2); it is an unweighted heuristic. Please provide a correct derivation or revise the theoretical claim.
minor comments (6)
  1. [Section 4.3, Tables 4-6] The footnote '†: results refer to cosine schedule' appears in each table but no † marks are present; please indicate which rows use which schedule.
  2. [Section 4.2, Table 2 and text] The symbol p is overloaded: it denotes both the frequency-softening power and the derivative-scaling power. This makes Table 2 confusing, especially because the text says 'p=1.0' and 'p=0.0'. Please rename one of the two quantities.
  3. [Section 4.2, paragraph on Adjective Nominalization] There is a typo: 'Acjective' should be 'Adjective'. Please correct it.
  4. [Section 3.1, Eq. (2)] The quantity μ is never defined. It appears to be the sequence-level mean of the softened weights w_i^p; please state this explicitly before Eq. (2).
  5. [Section 4.2 vs. Section 4.3] The bimodal Gaussian schedule is reported as the strongest in Table 2, but the submission model uses the cosine schedule, with a footnote saying stronger results will be updated later. Please clarify this choice and, if possible, report the final model with the configuration actually used for the leaderboard.
  6. [Section 4.2, Table 3] The claimed consistent +1 point improvement on BLiMP Supplement should be supported by significance testing; the reported confidence intervals overlap substantially (70.74±1.35 vs. 71.77±0.86).

Circularity Check

0 steps flagged

No significant circularity: the central empirical claims are measured on external BabyLM benchmarks and the frequency-informed masking is a training heuristic, not a fitted prediction.

full rationale

The paper's load-bearing empirical claim—that MDLM training is competitive with hybrid AR-masked baselines in a 100M-word setting—is supported by direct comparisons on the external BabyLM evaluation suite (Tables 4–6), not by any parameter fitted to those benchmarks being relabeled as a prediction. The frequency-informed masking scheme (Eq. 2) is an explicit training design; its reported effects are ablations (Table 3), and the softening power p is selected through experiments, not derived from the evaluation target. There are no load-bearing self-citations: the cited MDLM works (Sahoo et al., Shi et al.) are external and provide the base NELBO, while the present architecture and data setup are independently described. One non-circular concern is flagged: Section 3.1 asserts without derivation that per-token frequency-informed masking 'preserves the theoretical validity of the diffusion objective,' even though Eq. (1) is the uniform-masking NELBO and Eq. (2) assigns token-dependent mask probabilities; this is an omitted proof or potential correctness issue, not a circular reduction, because the paper does not define the masking distribution in terms of Eq. (1) nor derive Eq. (1) from the masking distribution. Accordingly the circularity score is 0.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central empirical claim rests on standard diffusion theory and the BabyLM evaluation framework. The main ad hoc element is the unproved assertion that frequency-informed, per-token masking maintains the validity of a uniform-masking NELBO. Free parameters are the softening powers and the noise schedule choices, which are tuned through ablations.

free parameters (4)
  • frequency softening power p = 0.02 (linearly increased from 0 across epochs)
    Chosen by hand to avoid over-emphasis on extremely rare tokens; controls the sharpness of frequency-based masking probabilities.
  • NELBO derivative scaling power p = 0.0, 0.1, 1.0 explored
    In Table 2, the derivative term alpha'_t is raised to a power p; the paper reports the objective with p=1.0, 0.1, and 0.0, and this scaling materially changes which noise schedule works best.
  • Bimodal Gaussian schedule parameters = w1=0.6, mu1=0.12, sigma1=0.02, mu2(t)=0.4+(0.85-0.4)(1-e^{-tau}), sigma2=0.08
    Hand-set for the two-peaked noise schedule ablation; results from this schedule are preliminary and run over one seed.
  • Cosine schedule = alpha_t = cos(pi/2 (1-t))
    The noise schedule is a design choice selected after comparing linear and cosine variants in Table 1; the submission uses this schedule.
axioms (4)
  • standard math The NELBO objective in Eq (1) is a valid lower bound for the forward masking process with uniform per-token masking rates.
    Foundation of Section 3.1, cited to Sahoo et al. (2024).
  • ad hoc to paper Frequency-informed masking with per-token probabilities scaled to a common mean preserves the validity of the NELBO.
    Asserted in Section 3.1 ('preserving the theoretical validity') but not derived; per-token schedules would normally change the ELBO weighting.
  • domain assumption Pseudo-likelihood computed with the MLM backend ranks MDLMs consistently with true likelihood on zero-shot tasks.
    Used throughout Section 3.2 and for all reported zero-shot scores; the authors explicitly note this may be suboptimal.
  • domain assumption The BabyLM corpus and evaluation pipeline are appropriate proxies for linguistic competence and human-likeness.
    Background framing for all experiments and conclusions.

pith-pipeline@v1.4.0-alltime-deepseek-medium · 9927 in / 13302 out tokens · 141513 ms · 2026-08-05T05:38:24.468262+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Masked Diffusion Language Models with Frequency-Informed Training." pith.science (2026). https://pith.science/paper/QQLHGRNB

@misc{pith2026250905056,
  author       = {Pith},
  title        = {Pith review of: Masked Diffusion Language Models with Frequency-Informed Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QQLHGRNB}},
  note         = {Machine review of arXiv:2509.05056}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

We present a masked diffusion language modeling framework for data-efficient training for the BabyLM 2025 Challenge. Our approach applies diffusion training objectives to language modeling under strict data constraints, incorporating frequency-informed masking that prioritizes learning from rare tokens while maintaining theoretical validity. We explore multiple noise scheduling strategies, including two-mode approaches, and investigate different noise weighting schemes within the NELBO objective. We evaluate our method on the BabyLM benchmark suite, measuring linguistic competence, world knowledge, and human-likeness. Results show performance competitive to hybrid autoregressive-masked baselines, demonstrating that diffusion-based training offers a viable alternative for data-restricted language learning.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 12 canonical work pages · 4 internal anchors

  1. [1]

    Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg

    Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. 2023. https://arxiv.org/abs/2107.03006 Structured denoising diffusion models in discrete state-spaces . Preprint, arXiv:2107.03006

    Pith/arXiv arXiv 2023

  2. [2]

    Yoshua Bengio, J\' e r\^ o me Louradour, Ronan Collobert, and Jason Weston. 2009. https://doi.org/10.1145/1553374.1553380 Curriculum learning . In Proceedings of the 26th Annual International Conference on Machine Learning, ICML '09, page 41–48, New York, NY, USA. Association for Computing Machinery

    arXiv 2009

  3. [3]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, and 12 others. 2020. https://arxiv.org/abs/2005.14165 Lan...

    Pith/arXiv arXiv 2020

  4. [4]

    Andrew Campbell, Joe Benton, Valentin De Bortoli, Tom Rainforth, George Deligiannidis, and Arnaud Doucet. 2022. https://arxiv.org/abs/2205.14987 A continuous time framework for discrete denoising models . Preprint, arXiv:2205.14987

    Pith/arXiv arXiv 2022

  5. [5]

    Chang and Benjamin K

    Tyler A. Chang and Benjamin K. Bergen. 2022. https://doi.org/10.1162/tacl_a_00444 Word acquisition in neural language models . Transactions of the Association for Computational Linguistics, 10:1--16

    doi:10.1162/tacl_a_00444 2022

  6. [6]

    Lucas Charpentier, Leshem Choshen, Ryan Cotterell, Mustafa Omer Gul, Michael Hu, Jaap Jumelet, Tal Linzen, Jing Liu, Aaron Mueller, Candace Ross, Raj Sanjay Shah, Alex Warstadt, Ethan Wilcox, and Adina Williams. 2025. https://arxiv.org/abs/2502.10645 Babylm turns 3: Call for papers for the 2025 babylm workshop . Preprint, arXiv:2502.10645

    Pith/arXiv arXiv 2025

  7. [7]

    Lucas Georges Gabriel Charpentier and David Samuel. 2024. https://aclanthology.org/2024.conll-babylm.24/ GPT or BERT : why not both? In The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning, pages 262--283, Miami, FL, USA. Association for Computational Linguistics

    work page 2024

  8. [8]

    Hyesong Choi, Hyejin Park, Kwang Moo Yi, Sungmin Cha, and Dongbo Min. 2024. https://arxiv.org/abs/2404.08327 Salience-based adaptive masking: Revisiting token dynamics for enhanced pre-training . In European Conference on Computer Vision (ECCV), pages 343--359. Springer

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  9. [9]

    Le, and Christopher D

    Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. https://arxiv.org/abs/2003.10555 Electra: Pre-training text encoders as discriminators rather than generators . Preprint, arXiv:2003.10555

    Pith/arXiv arXiv 2020

  10. [10]

    Andrea Gregor de Varda, Marco Marelli, and Simona Amenta. 2024. https://doi.org/10.3758/s13428-023-02261-8 Cloze probability, predictability ratings, and computational estimates for 205 E nglish sentences, aligned with existing EEG and reading time data . Behavior Research Methods, 56(5):5190--5213

    doi:10.3758/s13428-023-02261-8 2024

  11. [11]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://arxiv.org/abs/1810.04805 Bert: Pre-training of deep bidirectional transformers for language understanding . Preprint, arXiv:1810.04805

    Pith/arXiv arXiv 2019

  12. [12]

    Philip Gage. 1994. A new algorithm for data compression. The C Users Journal, 12(2):23--38

    1994

  13. [13]

    Lucas Georges Gabriel Charpentier and David Samuel. 2023. https://doi.org/10.18653/v1/2023.conll-babylm.20 Not all layers are equally as important: Every layer counts BERT . In Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning, pages 238--252, Singapore. Association for Computational Linguistics

    doi:10.18653/v1/2023.conll-babylm.20 2023

  14. [14]

    Richards, Steven F

    Jill Gilkerson, Jeffrey A. Richards, Steven F. Warren, and 1 others. 2017. https://doi.org/10.1044/2016_AJSLP-15-0169 Mapping the early language environment using all-day recordings and automated analysis . 26(2):248--265

    work page doi:10.1044/2016_ajslp-15-0169 2017

  15. [15]

    Valentin Hofmann, Leonie Weissweiler, David Mortensen, Hinrich Schütze, and Janet Pierrehumbert. 2024. https://arxiv.org/abs/2411.07990 Derivational morphology reveals analogical generalization in large language models . Preprint, arXiv:2411.07990

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  16. [16]

    Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. 2021. https://arxiv.org/abs/2102.05379 Argmax flows and multinomial diffusion: Learning categorical distributions . Preprint, arXiv:2102.05379

    Pith/arXiv arXiv 2021

  17. [17]

    Ivanova, Aalok Sathe, Benjamin Lipkin, Unnathi Kumar, Setayesh Radkani, Thomas H

    Anna A. Ivanova, Aalok Sathe, Benjamin Lipkin, Unnathi Kumar, Setayesh Radkani, Thomas H. Clark, Carina Kauf, Jennifer Hu, R. T. Pramod, Gabriel Grand, Vivian Paulun, Maria Ryskina, Ekin Akyürek, Ethan Wilcox, Nafisa Rashid, Leshem Choshen, Roger Levy, Evelina Fedorenko, Joshua Tenenbaum, and Jacob Andreas. 2025. https://arxiv.org/abs/2405.09605 Elements ...

    Pith/arXiv arXiv 2025

  18. [18]

    Weld, Luke Zettlemoyer, and Omer Levy

    Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. https://arxiv.org/abs/1907.10529 Spanbert: Improving pre-training by representing and predicting spans . Preprint, arXiv:1907.10529

    Pith/arXiv arXiv 2020

  19. [19]

    Najoung Kim and Sebastian Schuster. 2023. https://doi.org/10.18653/v1/2023.acl-long.213 Entity tracking in language models . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3835--3855, Toronto, Canada. Association for Computational Linguistics

    doi:10.18653/v1/2023.acl-long.213 2023

  20. [20]

    Vladimir Kulikov, Shahar Yadin, Matan Kleiner, and Tomer Michaeli. 2023. Sinddm: A single image denoising diffusion model. In Proceedings of the 40th International Conference on Machine Learning, pages 17920--17930. PMLR

    work page 2023

  21. [21]

    Yoav Levine, Barak Lenz, Opher Lieber, Omri Abend, Kevin Leyton-Brown, Moshe Tennenholtz, and Yoav Shoham. 2020. https://arxiv.org/abs/2010.01825 Pmi-masking: Principled masking of correlated spans . Preprint, arXiv:2010.01825

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  22. [22]

    Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, and Weizhu Chen. 2024. https://arxiv.org/abs/2404.07965 Rho-1: Not all tokens are what you need . arXiv preprint arXiv:2404.07965

    Pith/arXiv arXiv 2024

  23. [23]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. https://arxiv.org/abs/1907.11692 Roberta: A robustly optimized bert pretraining approach . Preprint, arXiv:1907.11692

    Pith/arXiv arXiv 2019

  24. [24]

    Haoming Lu, Hazarapet Tunanyan, Kai Wang, Shant Navasardyan, Zhangyang Wang, and Humphrey Shi. 2023. Specialist diffusion: Plug-and-play sample-efficient fine-tuning of text-to-image diffusion models to learn any unseen style. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14267--14276

    work page 2023

  25. [25]

    Kanishka Misra, Julia Rayz, and Allyson Ettinger. 2023. https://doi.org/10.18653/v1/2023.eacl-main.213 COMPS : Conceptual minimal pair sentences for testing robust property knowledge and its inheritance in pre-trained language models . In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 292...

    work page doi:10.18653/v1/2023.eacl-main.213 2023

  26. [26]

    Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. 2025. https://arxiv.org/abs/2406.03736 Your absorbing discrete diffusion secretly models the conditional distributions of clean data . Preprint, arXiv:2406.03736

    Pith/arXiv arXiv 2025

  27. [27]

    William Peebles and Saining Xie. 2023. Scalable diffusion models with transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision

    work page 2023

  28. [28]

    Emmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig, Barnabas Poczos, and Tom Mitchell. 2019. Competence-based curriculum learning for neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages...

    work page 2019

  29. [29]

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. https://arxiv.org/abs/2412.15115 Qwen2.5 technical report . Preprint, arXiv:2412.15115

    Pith/arXiv arXiv 2025

  30. [30]

    Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, and Volodymyr Kuleshov. 2024. https://arxiv.org/abs/2406.07524 Simple and effective masked diffusion language models . Preprint, arXiv:2406.07524

    Pith/arXiv arXiv 2024

  31. [31]

    Subham Sekhar Sahoo, Justin Deschenaux, Aaron Gokaslan, Guanghan Wang, Justin Chiu, and Volodymyr Kuleshov. 2025. https://arxiv.org/abs/2506.10892 The diffusion duality . Preprint, arXiv:2506.10892

    arXiv 2025

  32. [32]

    Nguyen, and Katrin Kirchhoff

    Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020. https://doi.org/10.18653/v1/2020.acl-main.240 Masked language model scoring . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics

    doi:10.18653/v1/2020.acl-main.240 2020

  33. [33]

    David Samuel, Andrey Kutuzov, Lilja Øvrelid, and Erik Velldal. 2023. https://arxiv.org/abs/2303.09859 Trained on 100 million words and still in shape: Bert meets british national corpus . Preprint, arXiv:2303.09859

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  34. [34]

    Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K. Titsias. 2025. https://arxiv.org/abs/2406.04329 Simplified and generalized masked diffusion for discrete data . Preprint, arXiv:2406.04329

    Pith/arXiv arXiv 2025

  35. [35]

    Weiss, Niru Maheswaranathan, and Surya Ganguli

    Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. 2015. https://arxiv.org/abs/1503.03585 Deep unsupervised learning using nonequilibrium thermodynamics . Preprint, arXiv:1503.03585

    Pith/arXiv arXiv 2015

  36. [36]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. https://arxiv.org/abs/2302.13971 Llama: Open and efficient foundation language models . Preprint, arXiv:2302.13971

    Pith/arXiv arXiv 2023

  37. [37]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. https://arxiv.org/abs/1706.03762 Attention is all you need . Preprint, arXiv:1706.03762

    Pith/arXiv arXiv 2023

  38. [38]

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2020. https://arxiv.org/abs/1905.00537 Superglue: A stickier benchmark for general-purpose language understanding systems . Preprint, arXiv:1905.00537

    Pith/arXiv arXiv 2020

  39. [39]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. https://arxiv.org/abs/1804.07461 Glue: A multi-task benchmark and analysis platform for natural language understanding . Preprint, arXiv:1804.07461

    Pith/arXiv arXiv 2019

  40. [40]

    Xiyu Wang, Baijiong Lin, Daochang Liu, Ying-Cong Chen, and Chang Xu. 2024. Bridging data gaps in diffusion models with adversarial noise-based transfer learning. In Proceedings of the 41st International Conference on Machine Learning, pages 1--11. PMLR

    work page 2024

  41. [41]

    Alex Warstadt, Leshem Choshen, Aaron Mueller, Adina Williams, Ethan Wilcox, and Chengxu Zhuang. 2023 a . https://arxiv.org/abs/2301.11796 Call for papers -- the babylm challenge: Sample-efficient pretraining on a developmentally plausible corpus . Preprint, arXiv:2301.11796

    Pith/arXiv arXiv 2023

  42. [42]

    Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2023 b . https://arxiv.org/abs/1912.00582 Blimp: The benchmark of linguistic minimal pairs for english . Preprint, arXiv:1912.00582

    Pith/arXiv arXiv 2023

  43. [43]

    Mortensen

    Leonie Weissweiler, Valentin Hofmann, Anjali Kantharuban, Anna Cai, Ritam Dutt, Amey Hengle, Anubha Kabra, Atharva Kulkarni, Abhishek Vijayakumar, Haofei Yu, Hinrich Schütze, Kemal Oflazer, and David R. Mortensen. 2023. https://arxiv.org/abs/2310.15113 Counting the bugs in chatgpt's wugs: A multilingual investigation into the morphological capabilities of...

    Pith/arXiv arXiv 2023

  44. [44]

    Jingyuan Zhu, Huimin Ma, Jiansheng Chen, and Jian Yuan. 2022. Few-shot image generation with diffusion models. arXiv preprint arXiv:2211.03264

    Pith/arXiv arXiv 2022

  45. [45]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  46. [46]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...