Lost in Interpolation: Why Predictive Feedback Fails in Diffusion Language Models

Aryan Sood; Gaurav Kumar Nayak; Ishaan Bansal; Lavanya Nigam; Vidit Aggarwal

REVIEW 3 major objections 6 minor 12 references

Lost in Interpolation: Why Predictive Feedback Fails in Diffusion Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Masked diffusion language models should interpolate feedback on a sphere, not along straight lines; doing so nearly doubles MAUVE and lowers generative perplexity at every tested budget.

desk verdict A genuinely new feedback operator for soft-masking in MDLMs, with honest, well-controlled experiments; the missing renormalized-LERP control leaves the hyperspherical mechanism unproven, but the operator itself is worth reviewing. read the letter →

arxiv 2608.06529 v1 pith:X4YU5NRA submitted 2026-08-06 cs.CL

Lavanya Nigam , Ishaan Bansal , Aryan Sood , Vidit Aggarwal , Gaurav Kumar Nayak This is my paper · ORCID

classification cs.CL

keywords maskeddiffusionlanguagemodelssoft-maskingsphericallinearinterpolationFréchetmeanhypersphericalembeddinggeometrygenerativeperplexityMAUVEtextgeneration

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Masked diffusion language models generate text by repeatedly unmasking tokens, and soft-masking feeds the model's own predictions back into masked positions as a continuous blend. Existing soft-masking builds that blend with linear interpolation in raw embedding space, implicitly treating that space as Euclidean. This paper argues that the embedding geometry of these models is actually hyperspherical: the mask embedding and the mean of the model's top-k predictions sit at a near-constant angle of about 73 degrees throughout training, and embedding norms stay flat across vocabulary-frequency rank. The paper introduces Spherical Soft-Masking (S-SM), which aggregates top-k predictions with a Fréchet mean on the unit hypersphere, blends the mask direction with that mean by spherical linear interpolation (SLERP), and restores the mask token's native norm. In continued pre-training experiments, S-SM improves MAUVE by up to 2x over the no-feedback baseline and by 27.5 to 56.1 percent over TopK/LERP, while reducing generative perplexity by 12 to 20 percent, with output entropy essentially unchanged.

What carries the argument

The load-bearing object is the Spherical Soft-Masking (S-SM) feedback operator. It replaces the Euclidean weighted mean of top-k token embeddings with a Fréchet mean computed intrinsically on the unit hypersphere (approximated by Karcher flow, initialized at the top-1 direction), and replaces the linear blend $(1-\lambda)m+\lambda\mu$ with spherical linear interpolation (SLERP) along the connecting geodesic, $s = [\sin((1-\lambda)\Omega)/\sin\Omega]\,\hat{m} + [\sin(\lambda\Omega)/\sin\Omega]\,\mu^*$, where $\Omega = \arccos\langle \hat{m}, \mu^*\rangle$; the output is rescaled to the mask token's native norm $r_m$. This construction guarantees endpoint fidelity, constant norm for every mixing weight, and a path that stays on the sphere, with numerical fallbacks when the mask and target directions nearly coincide.

What would settle it

Run an ablation that replaces only the norm-restoration step of S-SM with a normalized LERP blend, keeping the same confidence schedule and training recipe: if renormalized LERP matches S-SM's MAUVE and generative perplexity, then the geodesic interpolation is not the active ingredient. Also monitor the 73 degree angle across training steps, seeds, and model sizes: if the angle drifts or the norms show a frequency trend, the hyperspherical premise weakens.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that the standard linear (LERP) soft-masking operator in masked diffusion language models is geometrically wrong, and that this is why predictive feedback underperforms. The evidence is a pair of geometric diagnostics on a 169M-parameter model: the geodesic angle between the normalized mask embedding and the normalized mean of top-k predictions stays near 73 degrees over training, and token-embedding norms are essentially flat across frequency rank. Together these indicate that embedding vectors are confined to a thin spherical shell, where a straight-line blend leaves the manifold. The paper's proposed fix, S-SM, computes a confidence-weighted Fréchet mean of the top-k candidate directions on the unit hypersphere, interpolates between that mean and the mask direction along the great-circle arc using SLERP, and rescales the result to the mask token's native norm. As a drop-in replacement that leaves the training objective, unmasking rule, and confidence schedule untouched, S-SM yields higher MAUVE and lower generative perplexity than both the vanilla MDLM and TopK/LERP at every inference step budget tested, and the learned mixing weight converges to a roughly 1.9 times larger value under SLERP, consistent with the geometric account.

Load-bearing premise

The argument rests on the premise that the flat embedding norms and the stable 73 degree angle reveal a true hyperspherical geometry, so the measured gains come from geodesic interpolation rather than from simply restoring the mask token's norm.

Editorial extensions

If this is right

S-SM is a drop-in replacement: it changes only the interior of the feedback step, so any existing soft-masked MDLM pipeline can adopt the spherical operator without retuning the objective, unmasking rule, or confidence schedule.
If the reported results hold, generation quality improves at every tested NFE budget, with MAUVE roughly doubling at larger budgets and generative perplexity dropping by 16.9 to 19.6 percent relative to the no-feedback baseline.
The learned confidence schedule converges to a larger mixing weight under SLERP (about 0.056 versus 0.030), implying the optimizer retains more predictive feedback when interpolation follows the sphere, a direct behavioral signature of the geometric mechanism.
The quality gains are not bought with diversity: output entropy stays essentially unchanged across the vanilla, LERP, and SLERP configurations at matched budgets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Not tested in the paper: a renormalized LERP baseline that restores the mask norm but keeps chord interpolation would separate the norm-restoration effect from the geodesic-path effect; the authors note this baseline but do not evaluate it.
The near-constant 73 degree angle is reported as a training summary; tracking its evolution and variance across steps, seeds, and model scales would show whether the hyperspherical premise is stable or an artifact of the single 169M checkpoint studied.
A natural extension is to apply S-SM at billion-parameter scale and with different unmasking schedules, since the paper's own limitation statement notes that the SLERP-LERP gap is untested beyond 169M parameters.
The same geometric argument suggests other straight-line operations in MDLM pipelines, such as logit or embedding mixing during inference-time remasking, may also deserve reparameterization on the sphere.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The new thing is S-SM: replacing the Euclidean top-k mean and LERP blend in soft-masked diffusion LMs with a Fréchet mean and SLERP on the hypersphere, then restoring the mask norm before feeding the backbone. That specific combination is not in the prior work; Hersche et al. use LERP, and the geometric tools have been used elsewhere but not as a soft-masking feedback. The empirical work is stronger than usual for 169M continued-pretraining papers: three seeds at T=64/128/256, per-seed numbers and CIs in the appendix, the same training recipe for all arms, and a stochastic time band applied identically to every configuration. The ranking SLERP > LERP > vanilla holds on generative perplexity and MAUVE at every budget, with non-overlapping PPL intervals, and the Karcher/k ablations are informative. The limitations section is honest: single scale, no human eval, inherited hyperparameters, single seed at T=512.

The soft spot is the causal story. S-SM changes three things at once: Fréchet mean vs LERP mean, SLERP vs LERP, and explicit norm restoration to r_m. The plain LERP output has a λ-dependent norm smaller than r_m, so LERP feedback is a distribution shift regardless of direction. The paper itself flags this in §4.1, saying renormalizing FLERP to norm r_m satisfies P2 but not P3, and then never runs that control. Without a renormalized-LERP baseline, the MAUVE gains in Table 1 could come entirely from norm restoration. The geometric diagnostics are also asserted rather than quantified: norms are "essentially flat" and the angle holds at ≈73°, but there are no numbers on the variance across tokens or steps, and no training curve for the angle. From the text alone, the 73° figure might be a single checkpoint. That matters because the same diagnostic is both motivation and design target. The learned-λ asymmetry (0.056 vs 0.030) is interesting but uninterpretable until a norm-restored LERP learns its own λ.

More minor: absolute MAUVE values are tiny (0.01–0.04), no human evaluation, and T=512 is one seed. These the authors mostly acknowledge.

Who should read this: anyone working on masked diffusion LMs or continuous feedback mechanisms. The operator is drop-in, cheap, and clearly improves quality at this scale; the empirical contribution stands even if the hyperspherical explanation does not survive the missing ablation. But the paper's central claim — that geodesic interpolation is why S-SM helps — is currently confounded. Send it to peer review. A competent referee can demand the renormalized-LERP baseline and angle statistics, and the paper is solid enough that those requests are cheap. I would engage with this work seriously.

Referee Report

3 major / 6 minor

Summary. The paper proposes Spherical Soft-Masking (S-SM) for masked diffusion language models (MDLMs), replacing the existing linear interpolation (LERP) soft-masking operator with a hyperspherical construction: a Fréchet (Karcher) mean of the top-k predicted token embeddings on the unit sphere, a spherical linear interpolation (SLERP) blend between the mask direction and that mean, and restoration of the mask token's native norm. The authors present geometric diagnostics (near-constant embedding norms across frequency rank and a stable ≈73° angle between mask and prediction directions) to motivate the hyperspherical model, and report that S-SM improves MAUVE and generative perplexity over both a vanilla MDLM baseline and the TopK/LERP baseline across inference-time step budgets, while keeping training perplexity essentially unchanged.

Significance. If the causal claim is established, S-SM is a principled, drop-in replacement for soft-masking feedback that could improve MDLM generation at negligible architectural cost. The paper has notable strengths: a controlled comparison across three seeds, per-seed results with confidence intervals (Appendix I), a detailed algorithmic specification (Algorithms 1-3), and explicit numerical safeguards. However, the central attribution of the gains to geodesic interpolation is weakened by a missing control that separates norm restoration from SLERP, and the geometric diagnostics are asserted without quantitative support. These issues are addressable within the manuscript's scope, so the work is promising but not yet conclusive.

major comments (3)

[§4.1, §5, Table 1] The comparison between S-SM and FLERP changes three factors simultaneously: the aggregation (Fréchet mean vs. Euclidean top-k mean), the interpolation (SLERP vs. LERP), and the norm of the feedback vector (restored to rm vs. a λ-dependent shrinking norm). The paper itself notes in §4.1 that 'simply renormalizing FLERP's output to norm rm satisfies (P2) but not (P3),' yet it never runs or reports that renormalized-LERP control. Without this baseline, the MAUVE and generative-perplexity gains in Table 1 cannot be uniquely attributed to geodesic interpolation; they could in principle arise entirely from restoring the training-time norm. Please add the renormalized-FLERP baseline and report its MAUVE, Gen PPL, and learned λ for the same 7,000-step continued pre-training budget.
[§4, Figure 2 and geodesic-angle diagnostic] The motivating evidence for the hyperspherical geometry is not quantified. The claim that embedding norms are 'essentially flat' across frequency rank should be supported by a numerical measure (e.g., relative standard deviation or slope versus log-frequency), and the claim that the mask-to-mean angle 'holds steady at ≈73°' should be supported by a plot of the angle over training steps, plus a comparison to a null distribution (e.g., angles between random unit vectors) to show that 73° is not an artifact of the embedding dimension. This evidence is load-bearing for the paper's central premise that the embedding space is effectively hyperspherical.
[§7.3, Table 2] The interpretation that 'the optimizer trusts geometrically well-formed feedback more' because λ_SLERP ≈ 0.056 versus λ_LERP ≈ 0.030 is confounded by the same missing control. The confidence schedule parameters are learned jointly with the backbone, and the magnitude of λ interacts with the feedback vector's norm; under a norm-restored LERP, λ may rise as well. The λ-based evidence should be re-examined after the renormalized-LERP control is run, and the conclusion adjusted accordingly.

minor comments (6)

[Abstract] The sentence 'We evaluate S-SM on continued pre-training ... step budgets, SLERP feedback avoids ...' is a run-on and should be split into two sentences for readability.
[§6] In the dataset description, 'GPT-2 BPE' is duplicated: 'GPT-2 BPE, GPT-2 BPE, vocabulary 50,257'.
[Figure 2] Please clarify which checkpoint the embeddings are taken from (the 169M checkpoint before continued pre-training, or after?) and specify the corpus used for frequency counts.
[§7.3] The phrase 'This comparison uses centre −4.0' should be 'center −4.0' and briefly explained (it refers to the initial sigmoid center ω_b^(0)).
[Appendix A and Table 5] The notation for the number of Karcher iterations is inconsistent: 'Niter' in the main text, 'N_iter' in the appendix, and 'Niter' in Table 5. Please unify.
[Appendix I, Table 7] Entropy values are reported only for T=64; reporting entropy for all three budgets would strengthen the diversity-trade-off discussion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: S-SM is constructed from standard spherical geometry and evaluated against external baselines; the untested renormalized-LERP control is a confound, not a circular step.

full rationale

The paper's derivation chain is self-contained and empirically grounded. The S-SM operator is defined by standard Riemannian formulas (Karcher flow in Eq. 6, SLERP in Eq. 7, norm restoration in Eq. 8), and none of its parameters is fitted to the MAUVE or generative-perplexity numbers that are later reported. The geometric diagnostics (near-73° angle, flat norms) are used as motivation for choosing a spherical interpolation primitive, but the claimed improvements are measured against external baselines (vanilla MDLM and TopK/LERP) under a shared training recipe, so the central result does not reduce to the paper's own definitions. There is no load-bearing self-citation: the cited soft-masking precursor is by Hersche et al., not by the present authors, and the other references serve as related work or motivation. The missing renormalized-LERP control, which the paper itself identifies as satisfying (P2) but not (P3), is a genuine experimental confound between norm restoration and geodesic interpolation, but confounding is not circularity: nothing in the paper's equations forces the observed gains. The learned-lambda comparison is an interpretive outcome, not a prediction derived from a fitted quantity. Overall, the construction, evaluation, and claims are not circular.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The central contribution is a new feedback geometry, not a new physical law. The method relies on several domain assumptions (constant-norm embeddings, geodesic metric, norm-preserving feedback) and on standard Riemannian geometry. The only parameters fitted to data are the learned confidence schedule; other hyperparameters are inherited from prior work and lightly ablated. No new entities are introduced.

free parameters (6)

Confidence schedule parameters (ω_s, ω_a, ω_b) = λ ≈ 0.056 (SLERP) and 0.030 (LERP) at step 7000
Three learnable scalars control the mixing weight λ via Eq. (1); they are trained jointly with the backbone and their fitted values differ between SLERP and LERP.
Timestep band (b_l, b_h) = 0.2, 0.8
Chosen following prior block-diffusion work; restricts training noise levels and affects both baselines equally.
Soft-masking activation probability p_sm = 0.5
Hyperparameter controlling stochastic activation of the soft-masking path; inherited from the LERP baseline.
Top-k width k = 3
Ablated (k=1,3,5); k=3 chosen as accuracy-compute tradeoff.
Karcher iterations N_iter = 3
Ablated (1,3,5); differences within noise, 3 chosen as balance.
Learning rates η_bb, η_sm = 3e-5, 1e-2
Separate learning rates for backbone and confidence parameters; standard hyperparameters.

assumptions (6)

domain assumption Embeddings lie on or near a hypersphere of constant norm.
Section 4 diagnostics (flat norms vs frequency rank, stable angle) support this, but flatness is not quantified with error bars.
domain assumption Soft-masked feedback should preserve the native mask norm (property P2).
Section 4.1 lists norm compatibility as a design target without empirical justification; the paper does not test whether varying norm is harmful.
domain assumption Geodesic interpolation on the sphere is the correct primitive for this manifold (property P3).
Section 4.1 assumes that tracing the great-circle arc is superior to a chord, but this is not isolated from norm restoration in experiments.
domain assumption Karcher flow converges in very few iterations because top-k softmax weights are peaked.
Appendix A.2 and Section 7.1 report small PPL differences across N_iter, supporting this but without formal convergence analysis.
standard math Standard Riemannian geometry formulas (log/exp maps, SLERP) are correct and applicable to the embedding table.
Appendix A uses standard identities; no novel math is introduced.
domain assumption Two-pass training with a gradient-free first pass and stochastic gating is a valid training setup.
Section 6.1 and Appendix B.2 describe this recipe; it is inherited from prior soft-masking work and shared across all conditions.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Lost in Interpolation: Why Predictive Feedback Fails in Diffusion Language Models." pith.science (2026). https://pith.science/paper/X4YU5NRA

@misc{pith2026260806529,
  author       = {Pith},
  title        = {Pith review of: Lost in Interpolation: Why Predictive Feedback Fails in Diffusion Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X4YU5NRA}},
  note         = {Machine review of arXiv:2608.06529}
}

read the original abstract

Soft-masking accelerates the convergence of Masked Diffusion Language Models (MDLMs). Existing formulations build this blend with linear interpolation (LERP) in the raw embedding space, which implicitly treats that space as Euclidean. We analyze the embedding space of MDLMs and find that the mask and predicted-token embeddings maintain a near-constant angle of (\approx 73^\circ) throughout training, while embedding norms remain essentially flat across vocabulary-frequency rank. These indicate a hyperspherical geometry, for which LERP is the wrong interpolation primitive. We introduce Spherical Soft-Masking (S-SM), a drop-in replacement that aggregates the top-(k) predictions with a Fr'echet mean on the hypersphere and blends this mean with the mask direction using spherical linear interpolation (SLERP), then restores the native mask norm. We evaluate S-SM on continued pre-training of a released 169M-parameter MDLM checkpoint across a wide range of inference-time step budgets, SLERP feedback avoids the training degradation that LERP feedback induces and delivers MAUVE gains of up to 2x over the vanilla MDLM baseline and 27.5-56.1% over TopK/LERP at various sampling budgets, alongside consistently lower generative perplexity (16.9-19.6% over the baseline), while leaving output entropy and convergence essentially unchanged.

Figures

Figures reproduced from arXiv: 2608.06529 by the authors.

**Figure 1.** LERP and SLERP paths between two embeddings projected onto the unit hypersphere. rule is simple and trains well, but it throws away information the model knew before deciding whether to reveal the token or not. The [MASK] embedding left behind is identical at every position and every step and fails to carry the information gathered for predictions on prior steps. Soft-masking (SM) was introduced to stop this infor… view at source ↗

**Figure 2.** L2 norm of every token embedding E[v] against its GPT-2 byte-pair-encoding frequency rank on OpenWebText. geodesic angle θ = arccos⟨mˆ , µˆ⟩ between the normalized mask embedding and the normalized Euclidean mean of the top-k predictions. Averaged across positions and steps, θ holds at ≈73°(≈ 1.27 rad) essentially for the entire duration of training. Both diagnostics point the same way: embeddings live on (or ver… view at source ↗

**Figure 3.** Iterative denoising in MDLMs using spherical soft-masking (S-SM). [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 5.** Figure 5: Mean learned λ vs. training step (SLERP vs. TopK/LERP), up to step 7000. that all comparisons isolate the effect of the interpolation geometry rather than total training time. We test a learned mixing weight (via the confidence schedule of Equation (1), initialized ne…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 9 canonical work pages

[1]

, V}B×L): standard lookupE[x]

Token IDs( x∈ {0, . . . , V}B×L): standard lookupE[x]

work page
[2]

Sparse top- k representation(indices and weights): weighted gather-and-sum, used by FLERP

work page
[3]

Pre-embedded tensor( X∈R B×L×D): pass- through, used by S-SM (Algorithm 1)

work page
[4]

Dense probability distribution( Q∈ RB×L×V ): matrix multiply QE, used by the original (dense, non-top-k) simplex-space 12 Algorithm 3S-SM Feedback (batched) Require: token IDs xt ∈ {0, . . . , V}B×L; pass-1 distributions P∈R B×L×V ; embedding table E∈R V×D ; mask token id; top-k; Karcher iterationsN iter; tolerancesε, δ Ensure:embedding tensorX∈R B×L×D 1:...

work page 2026
[5]

arXiv preprint arXiv:2503.04482

Generalized interpolating discrete diffusion. arXiv preprint arXiv:2503.04482. Guanghan Wang, Yair Schiff, Subham Sahoo, and V olodymyr Kuleshov. 2026. Remasking discrete diffusion models with inference-time scaling.Ad- vances in Neural Information Processing Systems, 38:147282–147339. Yunshu Wu, Jiayi Cheng, Longxuan Yu, Partha Thakuria, Rob Brekelmans, ...

arXiv 2026
[10]

At a typical mask- ing rate of 30-70%, this reduces FLOPs and intermediate memory by a factor proportional to the masking fraction

Masked-only computation.All operations in Algorithm 1 (top- k selection, embedding gather, Fréchet mean, SLERP) are applied only to the M=|M| masked positions rather than the full B×L tensor. At a typical mask- ing rate of 30-70%, this reduces FLOPs and intermediate memory by a factor proportional to the masking fraction

work page
[11]

No-gradient first pass.The feedback for- ward pass (Pass 1, Appendix B.2) is wrapped in torch.no_grad(), which avoids con- structing the backward graph and substantially reduces peak activation memory

work page
[12]

H Distributed Training Considerations The S-SM pathway is fully compatible with Py- Torch DDP (Distributed Data Parallel)

Selective precision.Only the trigonomet- ric core ( arccos, sin, normalisation) runs in float32; the embedding gathers and the fi- nal norm rescaling (Equation (14)) operate in the backbone’s nativebfloat16. H Distributed Training Considerations The S-SM pathway is fully compatible with Py- Torch DDP (Distributed Data Parallel). All op- erations in Algori...

work page 1973

Show all 12 references

[2023]

InProceedings of the 61st An- nual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), pages 11575– 11596

Ssd-lm: Semi-autoregressive simplex-based diffusion language model for text generation and modular control. InProceedings of the 61st An- nual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), pages 11575– 11596. Haoyu He, Katrin Renz, Yong Cao...
[2024]

Justin Deschenaux and Caglar Gulcehre

Fisher flow matching for generative modeling over discrete data.Advances in Neural Information Processing Systems, 37:139054–139084. Justin Deschenaux and Caglar Gulcehre. 2026. Lan- guage modeling with hyperspherical flows.arXiv preprint arXiv:2605.11125. Sander Dieleman, Lau...

2026 arXiv
[2025]

Michael Hersche, Samuel Moor-Smith, Thomas Hof- mann, and Abbas Rahimi

Mdpo: Overcoming the training-inference divide of masked diffusion language models.arXiv preprint arXiv:2508.13148. Michael Hersche, Samuel Moor-Smith, Thomas Hof- mann, and Abbas Rahimi. 2026. Soft-masked diffu- sion language models. InInternational Conference on Learning Rep...

2026
[2026]

Xiaochuang Han, Sachin Kumar, and Yulia Tsvetkov

Learning in compact spaces with approxi- mately normalized transformer.Advances in Neural Information Processing Systems, 38:75765–75803. Xiaochuang Han, Sachin Kumar, and Yulia Tsvetkov

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

, V}B×L): standard lookupE[x]

Token IDs( x∈ {0, . . . , V}B×L): standard lookupE[x]

work page

[2] [2]

Sparse top- k representation(indices and weights): weighted gather-and-sum, used by FLERP

work page

[3] [3]

Pre-embedded tensor( X∈R B×L×D): pass- through, used by S-SM (Algorithm 1)

work page

[4] [4]

Dense probability distribution( Q∈ RB×L×V ): matrix multiply QE, used by the original (dense, non-top-k) simplex-space 12 Algorithm 3S-SM Feedback (batched) Require: token IDs xt ∈ {0, . . . , V}B×L; pass-1 distributions P∈R B×L×V ; embedding table E∈R V×D ; mask token id; top-k; Karcher iterationsN iter; tolerancesε, δ Ensure:embedding tensorX∈R B×L×D 1:...

work page 2026

[5] [5]

arXiv preprint arXiv:2503.04482

Generalized interpolating discrete diffusion. arXiv preprint arXiv:2503.04482. Guanghan Wang, Yair Schiff, Subham Sahoo, and V olodymyr Kuleshov. 2026. Remasking discrete diffusion models with inference-time scaling.Ad- vances in Neural Information Processing Systems, 38:147282–147339. Yunshu Wu, Jiayi Cheng, Longxuan Yu, Partha Thakuria, Rob Brekelmans, ...

arXiv 2026

[6] [10]

At a typical mask- ing rate of 30-70%, this reduces FLOPs and intermediate memory by a factor proportional to the masking fraction

Masked-only computation.All operations in Algorithm 1 (top- k selection, embedding gather, Fréchet mean, SLERP) are applied only to the M=|M| masked positions rather than the full B×L tensor. At a typical mask- ing rate of 30-70%, this reduces FLOPs and intermediate memory by a factor proportional to the masking fraction

work page

[7] [11]

No-gradient first pass.The feedback for- ward pass (Pass 1, Appendix B.2) is wrapped in torch.no_grad(), which avoids con- structing the backward graph and substantially reduces peak activation memory

work page

[8] [12]

H Distributed Training Considerations The S-SM pathway is fully compatible with Py- Torch DDP (Distributed Data Parallel)

Selective precision.Only the trigonomet- ric core ( arccos, sin, normalisation) runs in float32; the embedding gathers and the fi- nal norm rescaling (Equation (14)) operate in the backbone’s nativebfloat16. H Distributed Training Considerations The S-SM pathway is fully compatible with Py- Torch DDP (Distributed Data Parallel). All op- erations in Algori...

work page 1973