Pith. sign in

REVIEW 3 major objections 3 minor 36 references

Reweighting discrete-flow token updates by local context improves generation quality while preserving per-token marginals.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 07:27 UTC pith:FG4XNL7T

load-bearing objection Useful empirical contribution with a clean loss trick, but the marginal-preservation theorem only holds for masked sources; the uniform-source experiments rest on an unproven claim. the 3 major comments →

arxiv 2607.21427 v1 pith:FG4XNL7T submitted 2026-07-23 cs.LG

Context-weighted Discrete Flow Matching

Daniil Cherniavskii , Daniel Severo , Karen Ullrich This is my paper
classification cs.LG
keywords discrete flow matchingcontinuous-time Markov chainlocal contexttoken uncertaintyscaled cross-entropyany-order generationtext generationmolecular generation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to show that in discrete flow matching, token prediction difficulty is largely determined by local context—how many neighboring tokens are already unmasked—and that this signal can be exploited by a simple modification to the underlying continuous-time Markov chain. It proposes multiplying each per-coordinate update rate by a context weight α_i that depends only on the current state, and proves that under a normalization condition the total number of updated tokens, and with a shift-equivariance condition each token's marginal unmasking probability, are preserved. This yields two practical tools: an inference-time sampler that reweights updates toward well-contextualized positions with no extra training, and a scaled cross-entropy loss that reweights training signal. Empirically, the loss reduces generative perplexity by up to 63% on a large web-text corpus and increases valid molecule generations by up to 2.8×, while matching a strong semi-autoregressive baseline and retaining any-order generation.

Core claim

The paper's central claim is that modifying the CTMC update rates of discrete flow matching with local-context weights—either as an inference-time sampler or as a scaled cross-entropy training loss—improves generation quality while preserving the per-token marginal probability of being unmasked. Concretely, it reports up to 63% lower generative perplexity on a large web-text corpus (82.17 to 30.20), higher MAUVE, and up to 2.8× more valid molecular samples on a small SMILES dataset, matching a strong semi-autoregressive block diffusion baseline while retaining any-order generation. The key is a context weight function α_i(x_t) multiplied into the factorized velocity; a normalization conditio

What carries the argument

The central object is the context weight function α: D^N → R^N_+ that multiplies each coordinate-wise velocity in the factorized CTMC. Because α depends only on the current state, not the jump target, the modified velocity remains a valid probability velocity. Theorem 1 gives two conditions: the weights average to 1 over currently masked coordinates, preserving the distribution of the number of unmasked tokens; and, for per-coordinate marginals, α must be equivariant under circular shifts of the sequence and depend only on the reveal mask. For training, Algorithm 1 samples from the context-weighted path exactly by first sampling the number of unmasked tokens from Binomial(N, κ_t) and then se

Load-bearing premise

The proof that local-context weighting preserves per-token marginals assumes ∑_i α_i(x_t)δ(x_i^t,x_i^0)=N−m, which holds only when no source token equals its data token; for a uniform noise source that equality can fail.

What would settle it

Train a small discrete flow matching model with a uniform source, run Neighbor-weighted sampling, and empirically measure the per-coordinate marginal P(X_i^t = x_i^1) across many samples. If it deviates from κ_t, the marginal-preservation claim is false; equivalently, check whether the identity ∑_i α_i δ(x_i^t,x_i^0) = N−m holds on states where some x_i^0 = x_i^1.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Inference-time context-weighted sampling improves MAUVE, perplexity, and molecular validity with negligible compute and no fine-tuning.
  • Scaled cross-entropy closes most of the quality gap to semi-autoregressive baselines while preserving any-order generation.
  • Context-weighted paths help most in low-data regimes, nearly doubling valid molecule generations for masked noise.
  • Neighbor-weighted sampling stacks with predictor–corrector schemes, giving extra gains.
  • Simple locality statistics are effective, cheap proxies for prediction difficulty.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The marginal-preservation theorem is stated for both masked and uniform sources, but its proof uses an identity that can fail when a source token already equals its data token; this caveat is not flagged in the paper.
  • A natural extension is to replace the fixed neighbor-count statistic with a learned or adaptive context scorer, possibly improving the low-NFE regime where neighbor-weighting is currently weak.
  • The entropy-weighted solver is not covered by the per-coordinate marginal result because model predictions are not shift-equivariant; empirical checking of its marginals would quantify the gap.
  • The same local-context principle could extend to image or graph generation by defining context as unmasked neighbors in a spatial or graph neighborhood.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes context-aware modifications to Discrete Flow Matching (DFM). A context weight α_i(x) multiplies the coordinate-wise CTMC velocity, and the authors state conditions (Theorem 1) under which the resulting path preserves endpoints, the distribution of the number of unmasked tokens, and per-coordinate marginals. They derive two inference-time samplers (Neighbor-weighted and Entropy-weighted) and a training-time Scaled Cross-Entropy (SCE) loss, and report substantial gains on OpenWebText and QM9: reduced generative perplexity, improved MAUVE, and more valid/novel molecules, while matching a semi-autoregressive block-diffusion baseline. The main empirical results use the uniform source distribution; a smaller set of experiments uses the masked source.

Significance. If the theoretical and empirical claims hold, the paper makes a useful practical contribution: a simple, nearly free inference-time reweighting improves text and molecule generation, and a one-line training loss change closes much of the gap to structured-order baselines while preserving any-order generation. The empirical connection between local context and token-level prediction uncertainty is clearly demonstrated in Fig. 2 and is a valuable observation. However, the central marginal-preservation theorem is proven only for the masked-source case, whereas the headline OpenWebText results use the uniform source. This gap must be addressed before the paper can be accepted; the empirical results may still stand, but the theoretical backing for the main experimental regime is currently missing.

major comments (3)
  1. [Appendix A.2, Proposition 2 and Theorem 1] The proof of Proposition 2 uses the identity ∑_i α_i(x)δ(x_i,x_i^0) = ∑_i δ(x_i,x_i^0) = N−m, where m = ∑_i δ(x_i,x_i^1). This identity is valid only when no coordinate has x_i^0 = x_i^1. For the uniform source used in the main OpenWebText experiments, such coincidences occur with positive probability. If K is the number of initially coincident coordinates, then ∑_i δ(x_i,x_i^0) = N−m+K, so the stated normalization condition does not imply equality of stay probabilities. Consequently the count process of the context-weighted path is not Binomial(N,κ_t); the standard path itself has law K + Binomial(N−K,κ_t). Proposition 3 also assumes B0 = 0, which is false for the uniform source. Since the theorem is stated for both sources, this is a load-bearing gap: Algorithm 1's exactness and the per-token marginal preservation claim are not established for the uniform-source setting. Please restric
  2. [Section 3.2 and Appendix A.3] The paper's central claim is that the context-weighted CTMC 'preserves the per-token marginal probability of being unmasked' (Section 3.2 and the abstract). However, the last paragraph of A.3 states that Entropy-weighted sampling is not covered by the per-coordinate marginal preservation result because model predictions are not guaranteed to be circular-shift equivariant. This caveat does not appear in the main text or the contributions list. Please move this limitation forward and adjust the wording so that the theoretical claim is not stated more broadly than what is proven.
  3. [Appendix A.3, Algorithm 1] The exactness proof of Algorithm 1 relies on Proposition 2's conclusion that the number of unmasked tokens is Binomial(N,κ_t). For uniform sources, this is false, and the proof's normalization step is also inconsistent with the algorithm's selection rule: the text defines M_t = {i : x_i^t = x_i^0}, but line 4 of Algorithm 1 selects from {i : x_i^t ≠ x_i^1}. These sets coincide only when there are no initial source/data coincidences. Table 3 uses Algorithm 1 for the uniform source; the current proof does not justify this use. If the implementation normalizes over {i : x_i^t ≠ x_i^1}, please state that explicitly and adapt the theoretical conditions accordingly.
minor comments (3)
  1. [Throughout] The symbol M_t is used both for the count ∑_i 1[x_i^t = x_i^1] (Proposition 1, Appendix A.2) and for the set of masked coordinates (Appendix A.3). Please use different notation to avoid confusion.
  2. [Appendix A.4, Proposition 4] Proposition 4 proves homogeneity of the CTMC rate divergence when both target and learned velocities are multiplied by α_i(x). The SCE objective in Section 3.2, however, is defined on the cross-entropy loss, not on the Bregman divergence. The logical connection between the proposition and SCE should be clarified, or the proposition should be presented as motivation rather than as an equivalence.
  3. [Table 6 and Section B.3] SCE-trained models have higher standard test perplexity on Wikitext-103, LAMBADA, and FineWeb-Edu while improving generative perplexity. The discussion in B.3 is helpful, but the abstract's phrase 'reduces generative perplexity' could easily be misread as test perplexity. Please explicitly distinguish generative perplexity from test perplexity in the abstract and main text.

Circularity Check

0 steps flagged

No circularity: context weights are defined from local context, Theorem 1's condition is imposed by normalization, and empirical results are benchmarked externally; the uniform-source proof gap is a correctness caveat, not a circular reduction.

full rationale

The paper's central derivation is self-contained rather than circular. The context weight function α is defined from local neighbor counts or model predictive entropy—external signals—not from the evaluation metrics or from the fitted constants of the derivation. Theorem 1's sufficient condition Σ_i α_i(x_t)δ(x_i^t,x_i^0) = Σ_i δ(x_i^t,x_i^0) is deliberately enforced by normalizing the weights over currently masked coordinates (Appendix A.3), and the conclusions about the count process and per-coordinate marginals follow mathematically from that condition plus shift-equivariance. The per-coordinate marginal is not assumed by construction: it is derived from count preservation and exchangeability, and the paper explicitly notes that Entropy-weighted sampling is not covered by the marginal-preservation result. The main empirical claims (63% lower generative perplexity, MAUVE gains, QM9 validity improvements) are evaluated against external benchmarks and independent metrics, so they are not fitted inputs renamed as predictions. There are self-citations to prior work with overlapping authors ([13], [23]), but these are used as baselines or as a standard ELBO expression, not as the load-bearing justification for the paper's central claim. The most serious issue is a correctness/scope gap: Proposition 3's proof assumes B0=0, which holds for the masked source but not for the uniform source used in the main OpenWebText experiments, so the per-token marginal claim is not established in that regime. This is an unproven or false assumption, not a circular reduction of the conclusion to the premise, and therefore does not raise the circularity score.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

No new physical entities are introduced. The free parameters are the designed context-weight hyperparameters (r, s, β, corrector strength), which are tuned on validation data. The main hidden assumption is the implicit distinct-source condition in the marginal-preservation proof, which conflicts with the uniform-source experiments.

free parameters (4)
  • neighborhood radius r for Neighbor-weighted α = r=1 (inference-time sweep), r=3 (training-path ablation peak)
    Controls the size of the local context window; swept over {1,2,3} and selected per dataset/source (Table 5, Fig 9).
  • inverse temperature s for softmax normalization of α = s=4.0–5.0 (inference), s=0.5–1.0 (CW-DFM training)
    Controls how selectively the weights favor well-contextualized tokens; swept over {0.5,1,2,3,4,5} and tuned per setting.
  • entropy-weight exponent β = β=6–7
    Swept over {1,...,7} for the Entropy-weighted solver (Table 5).
  • corrector strength a=b in predictor-corrector combination = a=b swept in Fig 5 (no single best reported)
    Hyperparameter of the combined corrector; used to test complementarity with Neighbor-weighted sampling.
axioms (4)
  • standard math The CTMC/Kolmogorov-forward-equation framework for discrete flow matching, including the convex mixture path and factorized velocity.
    Adopted from Gat et al. [1]; the paper builds on this framework without re-deriving it.
  • domain assumption Local context—the number and position of unmasked neighbors—is a faithful proxy for token-level prediction uncertainty.
    Empirically motivated in §3.1/Fig 2, but used as a design principle for α; if the proxy is wrong, the weighting scheme is unjustified.
  • ad hoc to paper Source tokens are distinct from data tokens (x_i^0 ≠ x_i^1) for the marginal-preservation theorem.
    The proof of Proposition 2 uses ∑δ(x_i,x0)=N−m, which holds only in this regime; the paper states Theorem 1 for both sources without this caveat.
  • domain assumption Weights normalized over currently masked coordinates yield the same count process as the standard convex path.
    This is the normalization condition of Theorem 1; it is proven in Appendix A.2 but only under the distinct-source assumption.

pith-pipeline@v1.3.0-alltime-deepseek · 17958 in / 14835 out tokens · 144027 ms · 2026-08-01T07:27:43.971008+00:00 · methodology

0 comments
read the original abstract

Discrete flow matching provides a flexible framework for generative modeling on discrete structures. However, the standard factorized training objective exposes the model to targets of varying difficulty, mixing well-conditioned, predictable tokens with ambiguous, high-entropy ones. We empirically demonstrate that the uncertainty over the value of each token is closely related to the density of available context in its neighborhood. Motivated by this observation, we propose a simple modification to the underlying continuous-time Markov chain (CTMC) that incorporates local context information. Our context-weighted sampler improves generation quality with negligible computational overhead, while our scaled cross-entropy loss function reweights the training signal from different tokens and reduces generative perplexity by up to 63% on OpenWebText. Moreover, our approach matches a strong semi-autoregressive block diffusion baseline in quality while retaining the ability to perform generation in any order. These results highlight the role of local context as an important factor in discrete generative modeling and show that simple context-aware modifications can significantly improve both sampling and training efficiency.

Figures

Figures reproduced from arXiv: 2607.21427 by Daniel Severo, Daniil Cherniavskii, Karen Ullrich.

Figure 1
Figure 1. Figure 1: A motivational example. The target word “Red” is much more predictable in the second [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The average entropy (left) and negative log-likelihood (right) of the masked token vs. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Neighbor-weighted sampling improves generation quality without fine-tuning or computational overhead. At larger NFE, it outperforms both the Euler baseline and the model￾uncertainty-based Entropy-weighted solver in MAUVE and generative perplexity, while maintaining comparable token-level entropy. 8 16 32 64 128 256 100 200 300 400 500 Valid Count 8 16 32 64 128 256 50 100 150 200 250 Novel Count Euler Entr… view at source ↗
Figure 4
Figure 4. Figure 4: Local context can be more reliable than model uncertainty in low-data regimes. On the small QM9 dataset, Neighbor-weighted sampling outperforms both Euler and Entropy-weighted baselines across all NFE levels, nearly tripling the number of valid molecules and doubling the number of novel molecules. approximately 92M parameters. For text, we report generative perplexity (Gen. PPL), computed with GPT-2[24], e… view at source ↗
Figure 5
Figure 5. Figure 5: Context-weighted sampling is complementary to predictor–corrector sampling. Adding Neighbor-weighted updates to a predictor–corrector sampler further lowers generative perplexity and slightly improves MAUVE, showing that local-context weighting provides an additional quality￾improving mechanism. Context-weighted sampling is complementary to predictor–corrector schemes. To test whether local-context weighti… view at source ↗
Figure 6
Figure 6. Figure 6: The distance to the nearest unmasked token provides another proxy for the token-level uncertainty. Both the entropy and negative negative log likelihood increase with the distance, but saturate quite early. Vicinity radius r = 16, κ(t) = t 2 . 19 [PITH_FULL_IMAGE:figures/full_fig_p019_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Context-weighted sampling remains effective with masked source noise. With a masked source distribution, context-weighted solvers improve generative perplexity over Euler sampling while maintaining comparable token-level entropy. 8 16 32 64 128 256 200 300 400 500 600 700 Valid Count 8 16 32 64 128 256 100 150 200 250 300 350 Novel Count Euler Entropy-weighted (ours) Neighbor-weighted (ours) [PITH_FULL_IM… view at source ↗
Figure 8
Figure 8. Figure 8: Neighbor-weighted sampling improves molecular validity with uniform source noise. On QM9, Neighbor-weighted sampling generates more valid molecules than the baselines across NFE, while maintaining a comparable number of novel molecules out of 1024 samples. Training dynamics. fig. 10 shows that SCE follows a smooth training trajectory and leads to substantially lower generative perplexity. The validation ob… view at source ↗
Figure 9
Figure 9. Figure 9: Moderate local neighborhoods give the best quality–diversity trade-off. For DFM trained with SCE on OpenWebText, increasing the neighborhood radius reduces the perplexity but also the entropy; MAUVE peaks at an intermediate radius, suggesting that too little or too much context aggregation is suboptimal. 0 200k 400k 600k 800k 1000k 3.8 4.0 4.2 4.4 4.6 4.8 Validation Loss DFM CW-DFM 0 200k 400k 600k 800k 10… view at source ↗
Figure 10
Figure 10. Figure 10: SCE changes the optimization trajectory toward lower generative perplexity. Al￾though the weighted SCE validation loss is not directly comparable to standard CE, SCE decreases smoothly during training and yields generations with lower perplexity and a moderate reduction in entropy. result, SCE may be less favorable under the standard perplexity objective, but better aligned with the iterative sampling pro… view at source ↗
Figure 11
Figure 11. Figure 11: Distribution of the number of per-token updates for Euler and Neighbor-weighted solver, [PITH_FULL_IMAGE:figures/full_fig_p022_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

36 extracted references · 8 linked inside Pith

  1. [1]

    Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky T. Q. Chen, Gabriel Synnaeve, Yossi Adi, and Yaron Lipman. Discrete flow matching. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id= GTDKo3Sv9p

    2024

  2. [2]

    Openwebtext corpus

    Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http://Skylion007.github.io/ OpenWebTextCorpus, 2019

    2019

  3. [3]

    Block diffusion: Interpolating between autoregressive and diffusion language models.arXiv preprint arXiv:2503.09573, 2025

    Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and V olodymyr Kuleshov. Block diffusion: Interpolating between autoregressive and diffusion language models.arXiv preprint arXiv:2503.09573, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    A continuous time framework for discrete denoising models.Advances in Neural Information Processing Systems, 35:28266–28279, 2022

    Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models.Advances in Neural Information Processing Systems, 35:28266–28279, 2022

    2022

  5. [5]

    Vector quantized diffusion model for text-to-image synthesis

    Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10696–10706, 2022

    2022

  6. [6]

    Digress: Discrete denoising diffusion for graph generation

    Clement Vignac, Igor Krawczuk, Antoine Siraudin, Bohan Wang, V olkan Cevher, and Pas- cal Frossard. Digress: Discrete denoising diffusion for graph generation. InThe Eleventh International Conference on Learning Representations, 2023

    2023

  7. [7]

    Wyckoffdiff–a generative diffusion model for crystal symmetry

    Filip Ekström Kelvinius, Oskar B Andersson, Abhijith S Parackal, Dong Qian, Rickard Armiento, and Fredrik Lindsten. Wyckoffdiff–a generative diffusion model for crystal symmetry. InForty-second International Conference on Machine Learning, 2025

    2025

  8. [8]

    Simple and effective masked diffusion language models

    Subham Sekhar Sahoo, Marianne Arriola, Aaron Gokaslan, Edgar Mariano Marroquin, Alexan- der M Rush, Yair Schiff, Justin T Chiu, and V olodymyr Kuleshov. Simple and effective masked diffusion language models. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URLhttps://openreview.net/forum?id=L4uaAR4ArM

    2024

  9. [9]

    Yaron Lipman, Marton Havasi, Peter Holderrieth, Neta Shaul, Matt Le, Brian Karrer, Ricky T. Q. Chen, David Lopez-Paz, Heli Ben-Hamu, and Itai Gat. Flow matching guide and code,

  10. [10]

    Kakade, and Sitan Chen

    Jaeyeon Kim, Kulin Shah, Vasilis Kontonis, Sham M. Kakade, and Sitan Chen. Train for the worst, plan for the best: Understanding token ordering in masked diffusions. InForty-second International Conference on Machine Learning, 2025. URL https://openreview.net/ forum?id=DjJmre5IkP

    2025

  11. [11]

    Halton scheduler for masked generative image transformer

    Victor Besnier, Mickael Chen, David Hurych, Eduardo Valle, and Matthieu Cord. Halton scheduler for masked generative image transformer. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  12. [12]

    Set block decoding is a language model inference accelerator.arXiv preprint arXiv:2509.04185, 2025

    Itai Gat, Heli Ben-Hamu, Marton Havasi, Daniel Haziza, Jeremy Reizenstein, Gabriel Synnaeve, David Lopez-Paz, Brian Karrer, and Yaron Lipman. Set block decoding is a language model inference accelerator.arXiv preprint arXiv:2509.04185, 2025

    Pith/arXiv arXiv 2025

  13. [13]

    Accelerated sampling from masked diffusion models via entropy bounded unmasking.arXiv preprint arXiv:2505.24857, 2025

    Heli Ben-Hamu, Itai Gat, Daniel Severo, Niklas Nolte, and Brian Karrer. Accelerated sampling from masked diffusion models via entropy bounded unmasking.arXiv preprint arXiv:2505.24857, 2025. 10

    Pith/arXiv arXiv 2025

  14. [14]

    Quantum chemistry structures and properties of 134 kilo molecules.Scientific data, 1(1):1–7, 2014

    Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and O Anatole V on Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules.Scientific data, 1(1):1–7, 2014

    2014

  15. [15]

    Neighboring autoregressive modeling for efficient visual generation.arXiv preprint arXiv:2503.10696, 2025

    Yefei He, Yuanyu He, Shaoxuan He, Feng Chen, Hong Zhou, Kaipeng Zhang, and Bohan Zhuang. Neighboring autoregressive modeling for efficient visual generation.arXiv preprint arXiv:2503.10696, 2025

    Pith/arXiv arXiv 2025

  16. [16]

    Seed diffusion: A large-scale diffusion language model with high-speed inference.arXiv preprint arXiv:2508.02193, 2025

    Yuxuan Song, Zheng Zhang, Cheng Luo, Pengyang Gao, Fan Xia, Hao Luo, Zheng Li, Yuehang Yang, Hongli Yu, Xingwei Qu, et al. Seed diffusion: A large-scale diffusion language model with high-speed inference.arXiv preprint arXiv:2508.02193, 2025

    Pith/arXiv arXiv 2025

  17. [17]

    Large language diffusion models

    Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, JUN ZHOU, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. InICLR 2025 Workshop on Deep Generative Model in Machine Learning: Theory, Principle and Efficacy, 2025

    2025

  18. [18]

    Llada 1.5: Variance-reduced preference optimization for large language diffusion models.arXiv preprint arXiv:2505.19223, 2025

    Fengqi Zhu, Rongzhen Wang, Shen Nie, Xiaolu Zhang, Chunwei Wu, Jun Hu, Jun Zhou, Jianfei Chen, Yankai Lin, Ji-Rong Wen, et al. Llada 1.5: Variance-reduced preference optimization for large language diffusion models.arXiv preprint arXiv:2505.19223, 2025

    Pith/arXiv arXiv 2025

  19. [19]

    Llada2.0: Scaling up diffusion language models to 100b, 2025

    Tiwei Bie, Maosong Cao, Kun Chen, Lun Du, Mingliang Gong, Zhuochen Gong, Yanmei Gu, Jiaqi Hu, Zenan Huang, Zhenzhong Lan, Chengxi Li, Chongxuan Li, Jianguo Li, Zehuan Li, Huabin Liu, Ling Liu, Guoshan Lu, Xiaocheng Lu, Yuxin Ma, Jianfeng Tan, Lanning Wei, Ji-Rong Wen, Yipeng Xing, Xiaolu Zhang, Junbo Zhao, Da Zheng, Jun Zhou, Junlin Zhou, Zhanchao Zhou, L...

    Pith/arXiv arXiv 2025

  20. [20]

    Llada2.1: Speeding up text diffusion via token editing, 2026

    Tiwei Bie, Maosong Cao, Xiang Cao, Bingsen Chen, Fuyuan Chen, Kun Chen, Lun Du, Daozhuo Feng, Haibo Feng, Mingliang Gong, Zhuocheng Gong, Yanmei Gu, Jian Guan, Kaiyuan Guan, Hongliang He, Zenan Huang, Juyong Jiang, Zhonghui Jiang, Zhenzhong Lan, Chengxi Li, Jianguo Li, Zehuan Li, Huabin Liu, Lin Liu, Guoshan Lu, Yuan Lu, Yuxin Ma, Xingyu Mou, Zhenxuan Pan...

    arXiv 2026

  21. [21]

    Autoregressive image generation with randomized parallel decoding.arXiv preprint arXiv:2503.10568, 2025

    Haopeng Li, Jinyue Yang, Guoqi Li, and Huan Wang. Autoregressive image generation with randomized parallel decoding.arXiv preprint arXiv:2503.10568, 2025

    arXiv 2025

  22. [22]

    A reparameterized discrete diffusion model for text generation

    Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. A reparameterized discrete diffusion model for text generation. InFirst Conference on Language Modeling, 2024. URL https: //openreview.net/forum?id=PEQFHRUFca

    2024

  23. [23]

    Flow matching with general discrete paths: A kinetic-optimal perspective

    Neta Shaul, Itai Gat, Marton Havasi, Daniel Severo, Anuroop Sriram, Peter Holderrieth, Brian Karrer, Yaron Lipman, and Ricky TQ Chen. Flow matching with general discrete paths: A kinetic-optimal perspective. InThe Thirteenth International Conference on Learning Represen- tations, 2025

    2025

  24. [24]

    Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    2019

  25. [25]

    Mauve: Measuring the gap between neural text and human text using divergence frontiers.Advances in Neural Information Processing Systems, 34:4816–4828, 2021

    Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. Mauve: Measuring the gap between neural text and human text using divergence frontiers.Advances in Neural Information Processing Systems, 34:4816–4828, 2021

    2021

  26. [26]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InInternational Conference on Learning Representations, 2019. URL https://openreview.net/forum? id=Bkg6RiCqY7

    2019

  27. [27]

    Discrete diffusion modeling by estimating the ratios of the data distribution

    Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. InForty-first International Conference on Machine Learning,

  28. [28]

    Pointer sentinel mixture models, 2016

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016

    2016

  29. [29]

    URLhttps://openreview.net/forum?id=CNicRIVIPA. 11

  30. [30]

    Fineweb-edu: the finest collection of educational content, 2024

    Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Fineweb-edu: the finest collection of educational content, 2024. URL https://huggingface.co/datasets/ HuggingFaceFW/fineweb-edu. A Theoretical results A.1 Discrete Flow Matching background We consider discrete sequences x∈ DN of length N, where D is a finite vocabulary. We denote the i-t...

    2024

  31. [31]

    The lambada dataset: Word prediction requiring a broad discourse context

    Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc-Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. InProceedings of the 54th annual meeting of the association for computational linguistics (volume 1: Long papers), pages 1525–...

    2016

  32. [33]

    The corresponding coordinate-wise velocity is then given by ui t(xi, z|x 0, x1) = ˙κt 1−κ t δ(xi, xi 1)−δ(x i, z)

    +κ tδ(xi, xi 1), where κt is a monotonically increasing scheduler. The corresponding coordinate-wise velocity is then given by ui t(xi, z|x 0, x1) = ˙κt 1−κ t δ(xi, xi 1)−δ(x i, z) . This construction leads to a simple and efficient training and sampling procedure, but treats all coordinates uniformly, regardless of their conditioning context. At inferenc...

  33. [34]

    = NX i=1 δ(xi t, xi 0) Proof. First, note that probability of Mt not changing in [t, t+h) is the same as probability of Xt not changing: P(Xt+h =x|X t =x, X0 =x 0, X1 =x 1) = 1 +h NX i=1 ui t(xi, x|x 0, x1) +o(h) = 1 +h NX i=1 ˙κt 1−κ t δ(xi, xi 1)−1 +o(h) = 1−h ˙κt 1−κ t (N− NX i=1 δ(xi, xi 1)) +o(h) = 1−h ˙κt 1−κ t (N−m) +o(h) =P(M t+h =m|M t =m) which ...

  34. [35]

    +o(h) so if PN i=1 αi(x)δ(xi, xi

  35. [36]

    Thus the normalization condition preserves the distribution of the number of unmasked tokens

    =N−m, we have P( ˜Xt+h =x| ˜Xt =x, ˜X0 =x 0, ˜X1 =x 1) =P(X t+h =x|X t =x, X0 =x 0, X1 =x 1) and so ˜Mt is following the same CTMC asM t. Thus the normalization condition preserves the distribution of the number of unmasked tokens. Since κ0 = 0andκ 1 = 1, it follows that ˜M0 = 0and ˜M1 =Nalmost surely, which implies ˜p0(x|x 0, x1) =δ(x, x0),˜p 1(x|x 0, x1...

    2048

  36. [2024]

    URLhttps://arxiv.org/abs/2412.06264

    Pith/arXiv arXiv