REVIEW 5 major objections 6 minor 1 cited by
Unifying Continuous and Discrete Text Diffusion with Non-simultaneous Diffusion Processes
T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Giving every token its own noise schedule unifies discrete and continuous text diffusion.
desk verdict New per-token noise-control scheme with real but modest gains; the training objective contradicts the derived ELBO, and the code is missing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the two-time formulation: extrinsic time $t$ for the sentence and intrinsic time $\tau_t \in [0,1]$ for each token, connected to the noisy latent $z_t$ through a Gaussian with noise schedule $\bar{\alpha}(\tau_t), \bar{\beta}(\tau_t)$. The per-token noise level $\tau_t$ is generated by a Poisson diffusion process, a random jump process in which the token's noise state increments over time, normalized and variance-rescaled so that fine-grained, still-discrete differences between tokens survive as the state count grows. The reverse direction is carried by a time predictor $\tau_\theta(z_\theta(z_t,\tau_t,t), t', x)$ that estimates the target noise level for each token from the denoised sample, not from the noisy latent, plus a Bayesian-optimized schedule of extrinsic times $\{t_1,\ldots,t_K\}$ chosen on the validation set. Together these components convert uniform sentence-level noise into token-specific, semantically guided denoising, which is the mechanism the paper credits for its results.
What would settle it
Train NeoDiff on one translation task and replace the forward process with a true Markov chain that has the same per-step noise marginals; if the translation quality is unchanged, the independent-product forward process is not what carries the result. As a second check, record the time predictor's outputs $p_\theta(\tau_{t'} | z_t, \tau_t)$ after training and compare them with the forward marginal $q(\tau_{t'})$; if they coincide at every step, the predictor is only re-learning the schedule.
Extended reading notes
Core claim
The central claim is that discrete and continuous text diffusion are two ends of a single spectrum, and that a model can be built at the general point on that spectrum where each token has its own continuous, fine-grained noise level. NeoDiff formalizes this with a per-token intrinsic noise variable $\tau_t \in [0,1]$ driven by a Poisson process, while extrinsic time $t$ schedules the whole sentence; the forward distribution is written as $q(z_t,\tau_t|z_0) = \mathcal{N}(z_t; \sqrt{\bar{\alpha}(\tau_t)}\,z_0, \bar{\beta}(\tau_t)I)\,q(\tau_t)$, with a variance-controlled rescaling that keeps token states distinct even when the number of discrete states is large. The reverse process is a chain that includes a context-aware time predictor $p_\theta(\tau_{t'} | z_t, \tau_t)$, trained with pseudo labels derived from the model's own denoising confidence, so that less-noised tokens help guide the recovery of more-noised ones. The authors report consistent BLEU improvements over non-autoregressive discrete and continuous diffusion models, iterative refinement methods, and autoregressive diffusion baselines on six datasets, and they show that setting $\tau=1$ at a single position enables localized, token-level controlled edits to the output.
Load-bearing premise
The method rests on the assumption that a step-by-step denoising chain can be trained to undo a forward process defined as a set of independent per-time noise distributions, so the usual diffusion training objective is assumed to match the generative model.
Editorial extensions
If this is right
- Text diffusion can assign each token its own continuous noise level within one sentence, which the paper presents as a strict generalization of discrete diffusion ($\tau_t\in\{0,1\}$) and continuous diffusion ($\tau_t=t$).
- A context-aware time predictor lets low-noise tokens guide the recovery of high-noise tokens, which the paper reports as faster convergence and better word-level choices in translation.
- Post-training Bayesian optimization of the extrinsic time schedule yields task-specific denoising steps that improve BLEU with a one-time tuning cost of roughly 6% of training time on WMT14.
- The same per-token noise control enables targeted generation: corrupting only one token's intrinsic time to 1 changes a single concept in the output while preserving the rest of the sentence.
- On IWSLT14, WMT14, WMT16, QQP, Wiki-Auto, and Quasar-T, NeoDiff reports BLEU scores above both non-autoregressive and autoregressive diffusion baselines, including single-sample decoding at $b=1$.
Reading between the lines
- If the two-time framework is sound, the same variance-rescaled Poisson construction could serve as a general notation for per-token masking or noise curves, letting future work compare discrete, continuous, and hybrid text diffusion in one language; the paper itself only instantiates a few special cases.
- The time predictor's pseudo labels come from the network's own denoising confidence, so the cleanest test of its contribution would be to train it against labels from a separate, fixed confidence model and see whether the BLEU gains persist.
- Since the forward distribution factorizes over time steps while the reverse is a chain, checking whether learned reverse transitions reproduce the forward marginals after training would separate the Poisson design's effect from the network simply memorizing the schedule.
- The framework's claim to unify the two diffusion families suggests a direct experiment: taking an existing continuous diffusion model and an existing discrete model, rewriting each in the two-time notation, and verifying that the abstraction adds no extra hyperparameters beyond those each model already has.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes NeoDiff, a bi-temporal text diffusion framework in which extrinsic time t indexes the global diffusion step and intrinsic time τ controls a per-token noise level. The forward process combines a Poisson model for τ with a Gaussian conditional for the token embedding z_t; the reverse process is written as a Markov chain with a learned time predictor that estimates τ_{t'} from the current denoised prediction, and the inference schedule for t is optimized on validation BLEU via Bayesian optimization. Experiments on machine translation, paraphrasing, text simplification, and question generation report BLEU gains over discrete, continuous, and hybrid diffusion baselines, with ablations attributing gains to the Poisson process, the time predictor, and the optimized schedule.
Significance. The token-level noise control idea is well motivated, and the empirical study is broad: multiple tasks, many baselines, LLM-based evaluation, diversity metrics, step-wise generation traces, and token-manipulation examples. The ablation in Table 5 is a useful component analysis. If the theoretical construction and the actually optimized loss were made coherent, this could be a meaningful contribution to non-uniform text diffusion. As written, however, the forward process in Eq. (8), the reverse Markov chain in Eq. (10), and the training procedure in Section 3.3 are three mutually inconsistent objects; the variational derivation in Appendix A does not describe the implemented loss. The contribution is therefore conditional on substantial revision.
major comments (5)
- [3.1 / Appendix A, Eqs. (8)-(20)] The forward process in Eq. (8) is a product measure q(z_{>0}, τ_{>0}|z0) = ∏_{t>0} q(z_t, τ_t|z0), so under q, z_t and z_{t'} are independent for t ≠ t'. The reverse process in Eq. (10) is a Markov chain pθ(z_{t'}, τ_{t'}|z_t, τ_t), and the ELBO in Eq. (12) is an expectation over the product measure. Consequently, each reverse-step term in Eqs. (14)-(15) is paired with an independent draw of the future latent z_t, and no conditional q(z_{t'}, τ_{t'}|z_t, τ_t, z0) exists that supports the usual diffusion derivation. The Poisson process in Section 3.2 is a Markov chain in s_t, which is inconsistent with the product-form marginals in Eq. (9). Please either define a joint Markov forward process whose time-reversal is Eq. (10), or present the objective as a mean-field variational bound and justify the independence assumption; as written, the paper does not establish that NeoDiff reverses a Poisson forward diffusion process.
- [3.3 / Eqs. (5), (19)] The Lτ term in Eq. (5) and Eq. (19) is a KL divergence between q(τ_{t'}), the forward Poisson marginal, and the time predictor pθ(τ_{t'}|z_t, τ_t). The training procedure described in the 'Pseudo Label for Training the Time Predictor' subsection instead constructs labels by ranking the denoiser's own loss Lz + Lanchor within a sample and mapping the ranks through the inverse Poisson CDF via Eq. (7). These labels are functions of the current model output, not of the forward corruption process, and the paper does not specify the exact loss (e.g., cross-entropy against pseudo-labels) or the gradient path for τθ. The objective actually optimized is therefore not the ELBO in Appendix A, and the +0.22 BLEU gain from '+PT' in Table 5 cannot be attributed to the described adaptive-denoising mechanism. Please give the exact training loss and run an ablation using forward-sampled τ_{t'} as the target.
- [3.4 / Algorithm 1, Tables 1-3] The extrinsic time schedule is selected by Bayesian optimization against validation BLEU (Algorithm 1), and NeoDiff is reported with this optimized schedule in Tables 1-3 and Table 5, while most baselines use fixed schedules. This is a legitimate hyperparameter search, but it is not part of the generative model; comparing a validation-tuned schedule to fixed baselines conflates method with tuning. Please report NeoDiff with a uniform schedule as well, and ideally give selected baselines the same validation-based schedule optimization, or quantify how much of the reported gain depends on the 100-round optimization budget.
- [3.2 / Eq. (7), Eq. (19)] After the rescaling τ_t = Clip(Round((s_t - λ(t))/√λ(t) σ(t) + λ(t)), smax)/smax with σ(t) = λ(t), the distribution of τ_t is no longer Poisson(λ(t)); it is the distribution of a rounded, rescaled, and clipped Poisson variable. Eq. (19) nevertheless treats q(τ_{t'}) as the Poisson pmf used in the forward process. The correct pmf of the transformed variable is not derived, so the KL term is not computable as stated. Please derive the distribution of τ_t (or use untransformed s_t in the loss) and check whether the claim that the process remains Poisson is needed.
- [3.1 / paragraph after Eq. (11)] The paragraph claims that discrete diffusion models, continuous diffusion models, and DiffuSeq-V2 can be 'easily derived' from the bi-temporal framework by choosing τ_t appropriately. No derivation is supplied, and given that the framework's forward process is a product measure while these baselines are Markov chains, the reductions are not immediate. Please provide explicit constructions or temper the unification claim.
minor comments (6)
- [Section 3 heading] The heading 'Non-simultaeous Continuous Diffusion Models' misspells 'Non-simultaneous'; please correct it throughout the paper.
- [Table 3] In the last two rows, the cell '41.57H' and the placement of the 'H' marker for the NeoDiff 10 row are garbled; please fix the row labels and superscripts.
- [Eq. (7)] The operations Round(·) and Clip(·) are not defined precisely. As written, Clip(Round(...), smax) appears to enforce only an upper bound, while the earlier definition of τ_t used Clip(..., 1) to enforce the range [0, 1]. Please define the order of operations and the intended output range.
- [3.3] The sentence 'we treat both pθ(τ_{t'}|z_t, τ_t) and q(τ_{t'}) as standard discrete distributions and learn them using cross-entropy loss' is imprecise: q(τ_{t'}) is fixed and not learned, and minimizing cross-entropy with a fixed q is equivalent to minimizing KL(q∥pθ) only up to the entropy of q.
- [References] The references list two separate entries for DiNoiSer (Ye et al., 2023 and Ye et al., 2024) that appear to be the same preprint; please merge them.
- [3.4 / Appendix B.4] The optimized extrinsic time schedules obtained for each dataset are not reported; please include the final schedules in the appendix for reproducibility.
Circularity Check
Time-predictor training in §3.3 is self-referential: the pseudo-labels are the model's own loss ranks, so the claimed prediction of intrinsic time reduces to a reparameterization of the model's confidence.
-
fitted input called prediction
[Section 3.3, 'Pseudo Label for Training the Time Predictor'.]
"we first compute a confidence score for each generated output using the combined loss Lz + Lanchor from the denoised prediction zθ. ... we compute the normalized rank r for each token's loss within the single sample and map these ranks through the inverse cumulative distribution function (ICDF) of the Poisson distribution ... The resulting ˜s (t) values are then transformed via Eq. (7) to obtain the final pseudo labels."
The derived objective (Eq. 5, Appendix Eq. 19) defines Lτ = KL(q(τt′)∥pθ(τt′|zt, τt)), so the time predictor should match the forward-process distribution q(τt′). The implemented training target, however, is an inverse-CDF/rank transform of the model's own Lz+Lanchor losses. Because these losses come from the very network being trained, the 'predicted' intrinsic time is by construction a monotone reparameterization of the model's current confidence, not an estimate of forward corruption. The paper says τθ is meant to serve 'dual purposes: noise level prediction and semantic quality assessment of the generated output,' and trains it with cross-entropy on these pseudo-labels rather than on the Lτ term of Eq. (5).
full rationale
The clearest circular step is in §3.3: the time predictor is trained on pseudo-labels obtained by rank/inverse-CDF transforming the model's own Lz+Lanchor losses, while the derived objective (Eq. 5, Appendix Eq. 19) requires matching the forward-process q(τt′). The implemented target is a self-referential reparameterization of the denoiser's current confidence, so the claimed prediction of intrinsic time reduces by construction to the model's own error ranking; the +0.22 BLEU gain from +PT in Table 5 cannot be straightforwardly attributed to the stated forward-process mechanism. I do not count the non-Markov forward process (Eq. 8 is a product of independent factors with no consistent reverse conditional q(zt′|zt, τt, τt′)) as circularity, because that is a mathematical correctness concern rather than an equivalence-by-construction. Likewise, the validation-BLEU optimization of the extrinsic schedule (§3.4) is a fitted inference schedule, but it is disclosed and is a standard model-selection caveat, not a circular reduction of the derivation. The self-citations to Gao et al. (2024) and Li et al. (2024) supply auxiliary components (anchor loss, DGSMAX, Bayesian-optimization machinery) and are not load-bearing for the Poisson/time-predictor claim. The unification claim in §3.1 is a formal parameterization rather than a derivation that presupposes its conclusion. Overall, the denoiser's Lz objective is still trained against real z0, so the paper has independent content; the circularity is concentrated in the time-predictor component, making the finding partial rather than total.
Assumptions & free parameters
free parameters (6)
- smax =
100
- k =
2
- tmax =
0.99
- σ(t) scaling =
σ(t)=λ(t)
- Optimized extrinsic time schedule =
task-specific, not reported
- DGSMAX =
0.2
assumptions (5)
- ad hoc to paper The forward process as a product of independent q(zt,τt|z0) factors supports the reverse Markov chain in Eq. (10).
- ad hoc to paper The simplified MSE loss Lz is a valid bound for the bi-temporal framework.
- ad hoc to paper The rescaled, clipped τt can still be treated as a standard discrete random variable with q(τt') matching the Poisson pmf.
- ad hoc to paper Pseudo-labels from inverse transform sampling of the model's own loss ranks produce unbiased targets for the time predictor.
- domain assumption Validation-BLEU optimization of the schedule generalizes to the test set.
invented entities (1)
-
Intrinsic time τ (per-token diffusion progress)
Cite this review
Pith. "Pith review of Unifying Continuous and Discrete Text Diffusion with Non-simultaneous Diffusion Processes." pith.science (2026). https://pith.science/paper/L4MYNWUB
@misc{pith2026250522165,
author = {Pith},
title = {Pith review of: Unifying Continuous and Discrete Text Diffusion with Non-simultaneous Diffusion Processes},
year = {2026},
howpublished = {\url{https://pith.science/paper/L4MYNWUB}},
note = {Machine review of arXiv:2505.22165}
}
read the original abstract
Diffusion models have emerged as a promising approach for text generation, with recent works falling into two main categories: discrete and continuous diffusion models. Discrete diffusion models apply token corruption independently using categorical distributions, allowing for different diffusion progress across tokens but lacking fine-grained control. Continuous diffusion models map tokens to continuous spaces and apply fine-grained noise, but the diffusion progress is uniform across tokens, limiting their ability to capture semantic nuances. To address these limitations, we propose \textbf{\underline{N}}on-simultan\textbf{\underline{e}}ous C\textbf{\underline{o}}ntinuous \textbf{\underline{Diff}}usion Models (NeoDiff), a novel diffusion model that integrates the strengths of both discrete and continuous approaches. NeoDiff introduces a Poisson diffusion process for the forward process, enabling a flexible and fine-grained noising paradigm, and employs a time predictor for the reverse process to adaptively modulate the denoising progress based on token semantics. Furthermore, NeoDiff utilizes an optimized schedule for inference to ensure more precise noise control and improved performance. Our approach unifies the theories of discrete and continuous diffusion models, offering a more principled and effective framework for text generation. Experimental results on several text generation tasks demonstrate NeoDiff's superior performance compared to baselines of non-autoregressive continuous and discrete diffusion models, iterative-based methods and autoregressive diffusion-based methods. These results highlight NeoDiff's potential as a powerful tool for generating high-quality text and advancing the field of diffusion-based text generation.
Figures
Forward citations
Cited by 1 Pith paper
-
Sentence Curve Language Models
Predicting a B-spline 'sentence curve' target instead of static word embeddings improves diffusion language models on translation benchmarks.
Reference graph
Works this paper leans on
-
[1]
**Accuracy**: Faithfulness to source meaning
-
[2]
**Fluency**: Naturalness in target language
-
[3]
**Completeness**: Information retention
-
[4]
**Creativity**: Handling of ambiguous or open-ended source content Return ONLY 4 numbers separated by commas, NO text. Evaluate this paraphrase generation (0-100 score): [Original] {source} [Reference] {reference} [Paraphrase] {paraphrase} Score these aspects STRICTLY IN THIS ORDER:
-
[5]
arXiv preprint arXiv:2302.05737
A reparameterized discrete diffusion model for text generation. arXiv preprint arXiv:2302.05737. A Detailed Derivation of the Training Objective of NeoDiff Let z represent a token embedding and zt its la- tent representation at time t, with z0 = z and z1 ∼ N(0, I). The joint distribution of the for- ward process is then given by: q(z>0, τ>0|z0) := Y t>0 q...
arXiv 2020
-
[6]
for tokenization. Unlike previous work, we did not employ knowledge distillation (Kim and Rush, 2016) for preprocessing to evaluate our model’s performance under more challenging con- ditions. B.2 Model Configuration For our experiments, we set the maximum noise state smax to 100 and used the sqrt schedule for training, optimized schedule for inference. T...
work page 2024
-
[11]
**Semantic Faithfulness**: Meaning preservation from original
-
[12]
**Fluency**: Naturalness in language
Show all 14 references
-
[13]
**Completeness**: Retention of all information
-
[14]
Figure 3: Prompt templates used for LLM-based evaluation
**Phrasing Diversity**: Variation in wording/structure while preserving meaning Return ONLY 4 numbers separated by commas, NO text. Figure 3: Prompt templates used for LLM-based evaluation. Top: Translation evaluation prompt. Bottom: Paraphrase evaluation prompt. 17 <src> und ...
-
[2016]
In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (V olume 1: Long Papers), pages 1715–1725, Berlin, Germany
Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (V olume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Lin- guistics. Alexander Shabalin...
2025 arXiv
-
[2022]
In The Eleventh International Conference on Learning Rep- resentations
Analog bits: Generating discrete data us- ing diffusion models with self-conditioning. In The Eleventh International Conference on Learning Rep- resentations. Yunyen Chuang, Hung-Min Hsu, Kevin Lin, Chen- Sheng Gu, Ling Zhen Li, Ray-I Chang, and Hung yi Lee. 2024. Meta-diffu$b...
2024 arXiv
-
[2023]
In Proceedings of the 61st An- nual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), pages 11575– 11596
Ssd-lm: Semi-autoregressive simplex-based diffusion language model for text generation and modular control. In Proceedings of the 61st An- nual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), pages 11575– 11596. Jonathan Ho, Ajay Jain, and Pi...
2020
-
[2024]
arXiv preprint arXiv:2310.16834
Discrete diffusion modeling by estimating the ratios of the data distribution. arXiv preprint arXiv:2310.16834. Alexander Quinn Nichol and Prafulla Dhariwal. 2021. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning , pages 8162–8...
2021 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.