Discrete Tilt Matching

Jaeyeon Kim; Michael S. Albergo; Peter Potaptchik; Shiyi Wang; Yuyuan Chen

arxiv: 2604.18739 · v3 · pith:AZNW4OFNnew · submitted 2026-04-20 · 💻 cs.LG · stat.ML

Discrete Tilt Matching

Yuyuan Chen , Shiyi Wang , Peter Potaptchik , Jaeyeon Kim , Michael S. Albergo This is my paper

Pith reviewed 2026-05-21 00:24 UTC · model grok-4.3

classification 💻 cs.LG stat.ML

keywords discrete tilt matchingmasked diffusion llmsreinforcement learning fine-tuningreward tiltingunmasking posteriorscontrol variatesdiffusion language models

0 comments

The pith

Discrete Tilt Matching allows likelihood-free fine-tuning of masked diffusion LLMs by matching tilted local unmasking posteriors.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Masked diffusion large language models face challenges in reinforcement learning fine-tuning because sequence-level marginal likelihoods are intractable to compute. The authors derive Discrete Tilt Matching as a method that reformulates the problem as matching state-level local unmasking posteriors under a reward-based tilt. This leads to a weighted cross-entropy objective that has an explicit minimizer and supports control variates for better stability. Tests on a synthetic maze task examine the effects of annealing schedules and control variates on preventing mode collapse. Scaling up, fine-tuning an 8-billion parameter instruct model with DTM brings notable improvements on Sudoku and Countdown tasks while matching performance on MATH500 and GSM8K.

Core claim

The central discovery is that Discrete Tilt Matching recasts dLLM fine-tuning as state-level matching of local unmasking posteriors under reward tilting. This produces a likelihood-free weighted cross-entropy loss with an explicit solution and control variates that enhance training stability. On synthetic tasks, annealing and control variates help avoid mode collapse. At scale, the approach delivers strong gains on Sudoku and Countdown after fine-tuning LLaDA-8B-Instruct, while staying competitive on MATH500 and GSM8K.

What carries the argument

Discrete Tilt Matching, which matches local unmasking posteriors under reward tilting using a weighted cross-entropy objective with control variates.

If this is right

DTM provides an explicit minimizer for the fine-tuning objective without needing sequence marginals.
Control variates can be used to improve training stability in dLLM RL.
Fine-tuning with DTM leads to performance gains on planning tasks like Sudoku and Countdown.
The method remains competitive on math reasoning benchmarks such as MATH500 and GSM8K.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

DTM might extend to other generative modeling settings where sequence likelihoods are hard to access.
The state-level focus could lead to more efficient optimization in high-dimensional reward landscapes.
Adjusting the tilting function or masking schedule may require careful validation to avoid unintended biases.

Load-bearing premise

Matching state-level local unmasking posteriors under reward tilting is sufficient to optimize the sequence-level objective without bias from the masking schedule or tilting function.

What would settle it

Demonstrating that DTM-optimized models achieve lower rewards than a feasible sequence-level RL baseline on a small-scale dLLM task, or showing performance degradation when the masking schedule changes independently of the tilt.

Figures

Figures reproduced from arXiv: 2604.18739 by Jaeyeon Kim, Michael S. Albergo, Peter Potaptchik, Shiyi Wang, Yuyuan Chen.

**Figure 1.** Figure 1: Evaluation accuracy of DTM and baseline methods on benchmarks. All methods are of length 256 in 128 denoising steps. and often outperforms, prior RL-based fine-tuning efforts (Zhao et al., 2025; Tang et al., 2025; Wang et al., 2025a; Rojas et al., 2025; Yang et al., 2025) on LLaDA8B-Instruct (Nie et al., 2025) across Sudoku, Countdown, MATH500, and GSM8K. We highlight our main contributions: • Derivation… view at source ↗

**Figure 2.** Figure 2: Comparison of performance on maze planning task for DTM with and without the control variate. 4.1. Practical Interventions DTM is adaptive to semi-autoregressive decoding. Many state-of-the-art dLLMs are deployed with semiautoregressive (SAR) decoding, generating blocks autoregressively while allowing any-order updates within each block (Han et al., 2023; Arriola et al., 2025; Nie et al., 2025; Kim et al… view at source ↗

**Figure 3.** Figure 3: Ablation on annealing step size h on Countdown. Left figure shows the correct fraction on the evaluation set of the model checkpoints. Right figure shows the training reward trajectory. A moderate step size h = 6 achieves the best result. rising from 36.0 and 81.6 at generation length 256 to 40.2 and 83.2 at length 512, which is consistent with the view that stronger local predictions can be converted into… view at source ↗

**Figure 4.** Figure 4: Wallclock comparison for DTM and SPG on Sudoku, both trained on 8 H100 GPUs. DTM attains a higher reward and is more efficient. Since DTM reward is evaluated for frozen model πa within each a 7→ a + h phase, the reward is roughly constant per phase, and jumps at the phase boundary when the model πa is updated to πθ ≈ πa+h as in Algorithm 1. The SPG reward is evaluated on the training batch, thus showing a … view at source ↗

**Figure 5.** Figure 5: Proportion of valid paths (a) mean rewards (b), and diversity of paths (c) against degrees of tilt a. Our model is trained on three sets of control variate c and annealing steps h. Small step size and control variate 1 has higher path diversity, validity and rewards. (a) Effective generation length of DTM on Countdown (b) d1 (light green), WD1 (red), UniGRPO (dark green), SPG (blue) (from [PITH_FULL_IMAG… view at source ↗

**Figure 6.** Figure 6: Effective generation length of DTM versus RL baselines. With direct training on state-level posteriors, DTM achieves stable reasoning. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_6.png] view at source ↗

**Figure 7.** Figure 7: The fixed 41-by-41 maze with door fraction 0.4. i.e. the L 1 -distance. For each completion of the form z = (s, g, SEP, z1, z2, ..., zn, PAD, ..., PAD), we say the path is valid if z1 = s, zn = g, all zi’s are non-wall cells in the maze, and all consecutive cells (zi , zi+1) are direct neighbors in the maze (i.e. with Manhattan distance 1). The base model produces valid paths with probability 0.313 under s… view at source ↗

**Figure 8.** Figure 8: Prompt used for MATH500 and GSM8K. The problem statement is appended directly after the template. Respond in the following format: <reasoning> ... </reasoning> <answer> ... </answer> Using only the numbers [38, 92, 52] create an arithmetic expression that evaluates to exactly 78. You must use all numbers from the list, and each number must be used exactly once. You may use the operations +, -, *, and / as … view at source ↗

**Figure 9.** Figure 9: Prompt used for Countdown. In each instance, the list [38, 92, 52] is replaced by the provided numbers in the actual question and 78 is replaced by the actual target value. Sudoku We experiment on the 4×4 Sudoku dataset5 . We adopted SPG’s modification on the original split to avoid train-test leakage: the dataset contains 1M puzzles spanning all 288 possible completed 4×4 solution grids. They randomly sel… view at source ↗

**Figure 10.** Figure 10: Sudoku prompt template. We use 3-shot prompting: three solved puzzle exemplars are inserted; the evaluation set uses disjoint underlying solutions from the exemplars. To avoid repetition, we refer to Appendix D.3 of Wang et al. (2025a) for the 3 exemplars. D.2. Hyperparameters and Implementation Details Following the baselines, we employ LoRA with a rank of r = 128 and scaling factor α = 64, 4-bit quantiz… view at source ↗

**Figure 11.** Figure 11: Comparison of performance on Countdown for DTM with random interpolant versus SAR-aligned interpolant. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_11.png] view at source ↗

read the original abstract

Masked diffusion large language models (dLLMs) are a promising alternative to autoregressive generation. While reinforcement learning (RL) methods have recently been adapted to dLLM fine-tuning, their objectives typically depend on sequence-level marginal likelihoods, which are intractable for masked diffusion models. To address this, we derive Discrete Tilt Matching (DTM), a likelihood-free method that recasts dLLM fine-tuning as state-level matching of local unmasking posteriors under reward tilting. DTM takes the form of a weighted cross-entropy objective with explicit minimizer, and admits control variates that improve training stability. On a synthetic maze-planning task, we analyze how DTM's annealing schedule and control variates affect training stability and prevent mode collapse. At scale, fine-tuning LLaDA-8B-Instruct with DTM yields strong gains on Sudoku and Countdown while remaining competitive on MATH500 and GSM8K.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

DTM turns reward fine-tuning for masked diffusion LLMs into weighted cross-entropy on tilted local unmasking posteriors, with stability tools that show up in the maze experiments and some task gains at scale.

read the letter

Hi, the central new piece is the derivation of Discrete Tilt Matching. They start from tilted posteriors on individual unmasking steps and arrive at a weighted cross-entropy loss that has an explicit minimizer and accepts control variates. This sidesteps the intractable sequence marginals that usual RL adaptations run into for these models. On the maze task they walk through how the annealing schedule and control variates affect stability and mode collapse, which is concrete and helpful. Scaling to LLaDA-8B-Instruct then produces clear lifts on Sudoku and Countdown while staying in line with baselines on MATH500 and GSM8K. The construction reads as first-principles rather than post-hoc fitting, and the control variates are a straightforward practical addition that should help training. The softer spot is the step from local state matching to global sequence-level reward optimization. The masking schedule and choice of tilting function could inject bias, and while the maze analysis covers annealing effects, an explicit bound or ablation that isolates schedule-induced shifts would tighten the claim. If those checks are only sketched, the reported gains might partly reflect implementation choices rather than pure reward alignment. This is aimed at people working on non-autoregressive or diffusion-based language models who need a likelihood-free fine-tuning route. Readers already following dLLM work will see the most direct value. It is worth sending for peer review because the derivation is clean, the stability experiments are useful, and the larger-scale results are concrete enough for referees to evaluate the equivalence question properly.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces Discrete Tilt Matching (DTM) for fine-tuning masked diffusion large language models (dLLMs). It derives DTM as a likelihood-free weighted cross-entropy objective that matches state-level local unmasking posteriors under reward tilting, claiming an explicit minimizer and support for control variates to improve stability. The work analyzes annealing schedules and control variates on a synthetic maze task to address mode collapse and stability, then applies DTM to fine-tune LLaDA-8B-Instruct, reporting strong gains on Sudoku and Countdown while remaining competitive on MATH500 and GSM8K.

Significance. If the derivation establishes that local posterior matching under tilting is equivalent to sequence-level reward optimization without systematic bias from the masking schedule or tilting function, DTM would offer a practical, tractable alternative to intractable marginal-likelihood RL methods for dLLMs. The explicit minimizer and control variates are strengths that support stability and reproducibility; the maze analysis and large-scale results on reasoning tasks indicate potential impact for non-autoregressive generative models.

major comments (2)

[§2 (DTM Derivation)] §2 (DTM Derivation), paragraph on equivalence: The central claim that recasting dLLM fine-tuning as state-level matching of local unmasking posteriors under reward tilting optimizes the intractable sequence-level objective requires that the masking schedule and tilting function induce no bias. The abstract notes analysis of annealing and control variates on the maze task but provides no explicit bound, exact equivalence proof, or generalization to the LLaDA experiments; this is load-bearing for interpreting the Sudoku/Countdown gains as true reward optimization.
[§3.2 (Maze Analysis)] §3.2 (Maze Analysis), control variates paragraph: The introduction of control variates and the annealing schedule lacks ablations that isolate their contribution from the core weighted cross-entropy objective or external benchmarks, which is needed to substantiate the stability and mode-collapse prevention claims before scaling to 8B models.

minor comments (2)

[Abstract] Abstract: The phrase 'strong gains' on Sudoku and Countdown is not quantified with specific metrics or baselines; adding deltas or absolute scores would improve precision.
[§4 (Large-scale Experiments)] §4 (Large-scale Experiments): The tilting function and its parameterization across tasks are not fully specified (e.g., functional form or hyperparameter ranges), which could affect reproducibility.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive feedback and detailed comments on the manuscript. We address each major comment below with clarifications and indicate where revisions will be made to strengthen the presentation of the theoretical claims and empirical analyses.

read point-by-point responses

Referee: [§2 (DTM Derivation)] §2 (DTM Derivation), paragraph on equivalence: The central claim that recasting dLLM fine-tuning as state-level matching of local unmasking posteriors under reward tilting optimizes the intractable sequence-level objective requires that the masking schedule and tilting function induce no bias. The abstract notes analysis of annealing and control variates on the maze task but provides no explicit bound, exact equivalence proof, or generalization to the LLaDA experiments; this is load-bearing for interpreting the Sudoku/Countdown gains as true reward optimization.

Authors: We appreciate the referee's emphasis on rigorously establishing the lack of bias in the equivalence. The derivation in §2 shows that the weighted cross-entropy objective matches the tilted local unmasking posteriors by direct construction, with the masking schedule entering as a fixed distribution independent of the reward tilt; this yields an explicit minimizer without requiring marginal likelihoods. We acknowledge that the current manuscript does not include an explicit bias bound or full proof of zero bias under arbitrary schedules. In the revised version, we will add a dedicated paragraph stating the assumptions (uniform masking and consistent local tilting) under which the sequence-level equivalence holds, along with a brief discussion of potential residual biases and how the maze-task analysis empirically supports generalization to the LLaDA-8B experiments, where the identical DTM formulation produces the reported gains on Sudoku and Countdown. revision: yes
Referee: [§3.2 (Maze Analysis)] §3.2 (Maze Analysis), control variates paragraph: The introduction of control variates and the annealing schedule lacks ablations that isolate their contribution from the core weighted cross-entropy objective or external benchmarks, which is needed to substantiate the stability and mode-collapse prevention claims before scaling to 8B models.

Authors: We agree that clearer isolation of components strengthens the stability claims. Section 3.2 already reports results across annealing schedules and with/without control variates on the maze task, showing reduced variance and mode collapse relative to the base objective. To address the request for dedicated ablations, the revised manuscript will include additional experiments that directly compare (i) the core weighted cross-entropy, (ii) the same objective plus annealing only, and (iii) the full DTM with both annealing and control variates. These will report quantitative stability metrics (loss variance across seeds, success rate, and mode-collapse indicators) and will be presented prior to the 8B-scale results to better substantiate the contributions. revision: yes

Circularity Check

0 steps flagged

No significant circularity; DTM is a derived objective with independent content

full rationale

The paper presents DTM as a first-principles derivation that recasts sequence-level reward optimization for dLLMs into a tractable state-level weighted cross-entropy matching of local unmasking posteriors under tilting, with an explicit minimizer. This construction does not reduce to a fitted parameter renamed as a prediction, nor does it rely on self-citation chains or imported uniqueness theorems for its load-bearing steps. The annealing schedule and control variates are introduced and analyzed on a separate synthetic maze task rather than presupposed in the core equivalence. Downstream gains on Sudoku and Countdown are reported as empirical outcomes, not inputs to the derivation. The central claim therefore remains self-contained against external benchmarks and does not exhibit the enumerated circularity patterns.

Axiom & Free-Parameter Ledger

2 free parameters · 1 axioms · 0 invented entities

Review performed on abstract only; full derivation and any hidden modeling choices are unavailable. The ledger therefore records only the elements explicitly named in the abstract.

free parameters (2)

annealing schedule
Explicitly analyzed for stability on the maze task; schedule parameters are chosen to prevent mode collapse.
control variates
Introduced to improve training stability; their exact form is not specified in the abstract.

axioms (1)

domain assumption Local unmasking posteriors exist and can be tilted by a reward function without changing the overall diffusion process.
Invoked when recasting fine-tuning as state-level matching.

pith-pipeline@v0.9.0 · 5689 in / 1368 out tokens · 38505 ms · 2026-05-21T00:24:52.523079+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

we derive Discrete Tilt Matching (DTM), a likelihood-free method that recasts dLLM fine-tuning as state-level matching of local unmasking posteriors under reward tilting... ρ_{1,A}(x)∝ρ_1(x)e^{A r(x)}

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

6 extracted references · 6 canonical work pages · 2 internal anchors

[1]

Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen

PMLR. URL https://proceedings.mlr. press/v235/campbell24a.html. Poster. Chang, H., Zhang, H., Jiang, L., Liu, C., and Freeman, W. T. Maskgit: Masked generative image transformer, 2022. URLhttps://arxiv.org/abs/2202.04200. Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Trainin...

work page doi:10.18653/v1/2023 2022
[2]

URL https://openreview.net/forum? id=KnqiC0znVF. Oral. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, 10 Discrete Tilt Matching J., and Lowe, R. Training language models to ...

work page internal anchor Pith review Pith/arXiv arXiv 2022
[3]

URL https://openreview.net/forum? id=tcvMzR2NrP. Oral. Shi, J., Han, K., Wang, Z., Doucet, A., and Titsias, M. K. Simplified and generalized masked diffusion for discrete data, 2024. URL https://arxiv.org/abs/2406. 04329. Tang, X., Dolga, R., Yoon, S., and Bogunovic, I. wd1: Weighted policy optimization for reasoning in diffusion language models, 2025. UR...

work page arXiv 2024
[4]

URL https://openreview.net/forum? id=wczmXLuLGd. Poster. Zhao, S., Gupta, D., Zheng, Q., and Grover, A. d1: Scaling reasoning in diffusion large language mod- els via reinforcement learning. InAdvances in Neu- ral Information Processing Systems (NeurIPS 2025),

work page 2025
[5]

LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models

URL https://openreview.net/forum? id=7ZVRlBFuEv. Spotlight. Zhu, F., Wang, R., Nie, S., Zhang, X., Wu, C., Hu, J., Zhou, J., Chen, J., Lin, Y ., Wen, J.-R., and Li, C. Llada 1.5: Variance-reduced preference optimiza- tion for large language diffusion models, 2025. URL https://arxiv.org/abs/2505.19223. 11 Discrete Tilt Matching Algorithm 1DTM Training Requ...

work page internal anchor Pith review Pith/arXiv arXiv 2025
[6]

still masked at local time u

+O(h 2), G 1 =µ(x t, i) +hr(x 1) µ(xt, i) +ϕ(x i 1) +O(h 2), where theO(h 2)terms are inL 2 providedrandϕhave finite second moments. Ath= 0, we haveG 0|h=0 =ϕ(x i 1)andG 1|h=0 =µ(x t, i) =E a[ϕ(xi 1)|x t, i]. By the law of total variance, Var(ϕ(xi 1)) = Var Ea[ϕ(xi 1)|x t, i] +E Var(ϕ(xi 1)|x t, i) >Var(µ(x t, i)), so Var(G0)>Var(G 1) holds at h= 0 . Sinc...

work page 2026

[1] [1]

Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen

PMLR. URL https://proceedings.mlr. press/v235/campbell24a.html. Poster. Chang, H., Zhang, H., Jiang, L., Liu, C., and Freeman, W. T. Maskgit: Masked generative image transformer, 2022. URLhttps://arxiv.org/abs/2202.04200. Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Trainin...

work page doi:10.18653/v1/2023 2022

[2] [2]

URL https://openreview.net/forum? id=KnqiC0znVF. Oral. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, 10 Discrete Tilt Matching J., and Lowe, R. Training language models to ...

work page internal anchor Pith review Pith/arXiv arXiv 2022

[3] [3]

URL https://openreview.net/forum? id=tcvMzR2NrP. Oral. Shi, J., Han, K., Wang, Z., Doucet, A., and Titsias, M. K. Simplified and generalized masked diffusion for discrete data, 2024. URL https://arxiv.org/abs/2406. 04329. Tang, X., Dolga, R., Yoon, S., and Bogunovic, I. wd1: Weighted policy optimization for reasoning in diffusion language models, 2025. UR...

work page arXiv 2024

[4] [4]

URL https://openreview.net/forum? id=wczmXLuLGd. Poster. Zhao, S., Gupta, D., Zheng, Q., and Grover, A. d1: Scaling reasoning in diffusion large language mod- els via reinforcement learning. InAdvances in Neu- ral Information Processing Systems (NeurIPS 2025),

work page 2025

[5] [5]

LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models

URL https://openreview.net/forum? id=7ZVRlBFuEv. Spotlight. Zhu, F., Wang, R., Nie, S., Zhang, X., Wu, C., Hu, J., Zhou, J., Chen, J., Lin, Y ., Wen, J.-R., and Li, C. Llada 1.5: Variance-reduced preference optimiza- tion for large language diffusion models, 2025. URL https://arxiv.org/abs/2505.19223. 11 Discrete Tilt Matching Algorithm 1DTM Training Requ...

work page internal anchor Pith review Pith/arXiv arXiv 2025

[6] [6]

still masked at local time u

+O(h 2), G 1 =µ(x t, i) +hr(x 1) µ(xt, i) +ϕ(x i 1) +O(h 2), where theO(h 2)terms are inL 2 providedrandϕhave finite second moments. Ath= 0, we haveG 0|h=0 =ϕ(x i 1)andG 1|h=0 =µ(x t, i) =E a[ϕ(xi 1)|x t, i]. By the law of total variance, Var(ϕ(xi 1)) = Var Ea[ϕ(xi 1)|x t, i] +E Var(ϕ(xi 1)|x t, i) >Var(µ(x t, i)), so Var(G0)>Var(G 1) holds at h= 0 . Sinc...

work page 2026