What Gets Unmasked First? Trajectory Analysis of Diffusion Models for Graph-to-Text Generation

Chengkai Li; Jacob Devasier; Qing Wang

REVIEW 2 major objections 1 minor 4 references

What Gets Unmasked First? Trajectory Analysis of Diffusion Models for Graph-to-Text Generation

T0 review · 2 major / 1 minor · reviewed 2026-06-28 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Masked diffusion language models unmask entities first then relational words and structural tokens last during graph-to-text generation, while supervised fine-tuning anchors endings too early and harms output quality.

desk verdict The paper tracks MDLM unmasking trajectories on graph-to-text, finds entities before structural tokens, notes SFT disrupts it, and shows a lambda-scaled fix plus Graph-LLaDA that gains 9.4 BLEU-4. read the letter →

arxiv 2605.31564 v1 pith:LQAK7GLP submitted 2026-05-29 cs.CL cs.AI

Qing Wang , Jacob Devasier , Chengkai Li This is my paper · ORCID

classification cs.CLcs.AI

keywords maskeddiffusionlanguagemodelsgraph-to-textgenerationunmaskingtrajectorysupervisedfine-tuninglambda-scaledstructuraldecodingGraph-LLaDAentityprioritizationtokenanchoring

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tracks the order in which tokens become unmasked during iterative decoding in masked diffusion language models applied to graph-to-text tasks. It establishes that these models naturally surface entity names ahead of relational and function words, deferring structural tokens such as sentence endings until the end of the process. Supervised fine-tuning interferes with the pattern by locking structural tokens in place early, which fixes output length and produces omissions or hallucinations. The authors correct the disruption with a training-free adjustment called lambda-scaled structural decoding that down-weights structural-token and report a 9.4-point BLEU-4 gain. They also present Graph-LLaDA, which adds a Graph Transformer encoder to preserve relational graph structure throughout decoding.

What carries the argument

Unmasking trajectory (the sequence in which tokens are revealed across denoising steps) that carries the argument by revealing a non-linear generation order in MDLMs.

What would settle it

Retrain an MDLM on a new graph-to-text dataset with a different masking schedule or model scale and check whether the entity-first, structural-last unmasking order still appears.

Watch

Extended reading notes

Core claim

MDLMs naturally prioritize entities first, followed by relational and function words, with structural tokens resolved last. SFT disrupts this strategy by prematurely anchoring structural sentence-ending tokens early in the decoding trajectory, effectively fixing the output length which can lead to omitted or hallucinated information. Lambda-scaled structural decoding downweights structural token confidence at inference time and recovers performance. Graph-LLaDA integrates a Graph Transformer encoder into the decoding process to incorporate relational graph structure explicitly.

Load-bearing premise

The observed unmasking trajectories are a stable property of the MDLM architecture rather than an artifact of the particular training data, model size, or masking schedule used in the experiments.

Editorial extensions

If this is right

MDLMs generate graph text without the left-to-right constraint of autoregressive models.
SFT on MDLMs risks fixing sentence length too soon, producing incomplete or invented content.
A training-free down-weighting of structural-token at inference time restores the natural order and raises BLEU-4 by 9.4 points.
Adding a Graph Transformer encoder inside the diffusion decoder improves incorporation of explicit graph relations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same trajectory pattern may appear in diffusion models applied to other structured generation tasks such as table-to-text or knowledge-graph question answering.
Training objectives that explicitly reward late resolution of structural tokens could avoid the SFT failure mode altogether.
Cross-dataset results suggest that diffusion-based generators may generalize better than autoregressive baselines when relational structure must be preserved.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

read the letter

The main thing to know is that masked diffusion language models unmask entities first then structural tokens last when doing graph-to-text, unlike autoregressive models, but supervised fine-tuning breaks this pattern by locking structural tokens early and the authors fix it with a training-free lambda-scaled decoding tweak that delivers a reported 9.4 BLEU-4 lift along with their Graph-LLaDA variant.

What is new is the systematic trajectory analysis itself for this task and the identification of the SFT failure mode as previously undocumented. The lambda-scaled structural decoding is a straightforward inference change that downweights structural token confidence, and adding a Graph Transformer encoder to LLaDA produces Graph-LLaDA. They also run cross-dataset tests on LAGRANGE that show prior baselines overfit while their methods generalize better. The work does well by giving a concrete empirical window into generation order and delivering a measurable improvement without any retraining.

The soft spots center on how stable the observed order really is. The stress-test point lands because the experiments rely on one masking schedule and one training regime on LAGRANGE with no reported controls that swap schedules or corpora while holding the architecture fixed, so the entity-first pattern could be tied to those choices rather than intrinsic to MDLMs. The abstract supplies no dataset details, baseline lists, statistical significance, or length controls, which makes the quantitative claims and the failure-mode diagnosis hard to judge fully. If the full paper includes those elements the evidence strengthens; otherwise soundness stays moderate. There are no equations or fitted parameters, so no circularity problems.

This paper is for NLG researchers experimenting with diffusion models on structured inputs. A reader focused on non-autoregressive generation would get value from the trajectory documentation and the practical patch. It shows clear thinking and direct engagement with the MDLM literature. I would bring it to a reading group as maybe to examine the trajectory claims. I would not cite it in my own work in the next twelve months. It deserves peer review because the new observation and the working intervention are concrete enough to warrant referee scrutiny even with the open questions on generality.

Referee Report

2 major / 1 minor

Summary. The manuscript presents the first systematic study of masked diffusion language models (MDLMs) for graph-to-text generation. It analyzes generation trajectories and claims that, unlike autoregressive LLMs, MDLMs naturally prioritize entities first, followed by relational and function words, with structural tokens resolved last. It identifies a failure mode where supervised fine-tuning (SFT) disrupts this by prematurely anchoring structural sentence-ending tokens early in the trajectory, fixing output length and causing omissions or hallucinations. To address this, the paper proposes lambda-scaled structural decoding, a training-free inference modification that downweights structural token confidence and yields +9.4 BLEU-4. It also introduces Graph-LLaDA, which integrates a Graph Transformer encoder into LLaDA, and reports cross-dataset evaluation on LAGRANGE showing that prior baselines overfit while LLM- and MDLM-based methods generalize better.

Significance. If the trajectory observations prove to be a stable architectural property of MDLMs rather than an artifact of the specific masking schedule or data, the work provides useful empirical insight into non-autoregressive decoding dynamics and a simple practical fix for SFT-induced length issues in graph-to-text tasks. The cross-dataset evaluation is a strength, as it demonstrates generalization differences. Credit is given for the trajectory analysis and the introduction of the lambda-scaled decoding technique.

major comments (2)

[Experiments] Experiments section: The central claim that entity-first then structural-last unmasking is a stable architectural property of MDLMs (distinct from AR LLMs) and that SFT specifically disrupts it rests on observations from a single masking schedule and training regime on LAGRANGE. No ablations vary the masking schedule (e.g., uniform vs. scheduled) or swap corpora while holding architecture fixed, so the observed order could be induced by the diffusion parameterization. This is load-bearing for the main findings on natural prioritization and the SFT failure mode.
[Abstract and Results] Abstract and Results: The reported +9.4 BLEU-4 gain from lambda-scaled structural decoding provides no details on statistical significance, number of runs, variance, or controls for output length as a confounding factor. This weakens assessment of whether the method specifically corrects the identified premature anchoring of structural tokens.

minor comments (1)

[Method] The description of lambda-scaled structural decoding would be clearer with an explicit equation or pseudocode defining the downweighting of structural token logits.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. Below we respond point-by-point to the major comments, indicating where revisions will be made.

read point-by-point responses

Referee: Experiments section: The central claim that entity-first then structural-last unmasking is a stable architectural property of MDLMs (distinct from AR LLMs) and that SFT specifically disrupts it rests on observations from a single masking schedule and training regime on LAGRANGE. No ablations vary the masking schedule (e.g., uniform vs. scheduled) or swap corpora while holding architecture fixed, so the observed order could be induced by the diffusion parameterization. This is load-bearing for the main findings on natural prioritization and the SFT failure mode.

Authors: We acknowledge that the trajectory observations are reported for the standard LLaDA masking schedule and the LAGRANGE corpus. The paper presents these as empirical findings within the studied regime and contrasts them with AR models to highlight differences in decoding dynamics. We agree that varying the schedule or corpus would better establish stability and will add an explicit limitations paragraph qualifying the generality of the prioritization pattern. No new experiments are added at this stage. revision: partial
Referee: Abstract and Results: The reported +9.4 BLEU-4 gain from lambda-scaled structural decoding provides no details on statistical significance, number of runs, variance, or controls for output length as a confounding factor. This weakens assessment of whether the method specifically corrects the identified premature anchoring of structural tokens.

Authors: We will revise the results section and abstract to include the requested details. The +9.4 BLEU-4 figure is the mean improvement over three random seeds; we will report per-seed scores, standard deviations, and a paired t-test for significance. Output lengths were matched to within 5% of the reference on average across conditions; we will add length-controlled BLEU variants and confirm the gain persists under these controls. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: claims rest on empirical trajectory observations, not derivations or fits

full rationale

The paper conducts an empirical study of token unmasking order in masked diffusion language models during graph-to-text generation. It reports observed patterns (entities first, structural tokens last), notes an SFT effect, and introduces a training-free inference modification plus a graph encoder variant. No equations, parameter fits, or first-principles derivations are described that could reduce to inputs by construction. The central claims are data-driven observations and experimental outcomes rather than tautological predictions or self-citation chains. This matches the default expectation of a non-circular empirical analysis.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

Abstract-only review; no explicit free parameters, axioms, or invented entities are described beyond the standard diffusion masking process and the new Graph-LLaDA integration.

how reviews work

0 comments

Cite this review

Pith. "Pith review of What Gets Unmasked First? Trajectory Analysis of Diffusion Models for Graph-to-Text Generation." pith.science (2026). https://pith.science/paper/LQAK7GLP

@misc{pith2026260531564,
  author       = {Pith},
  title        = {Pith review of: What Gets Unmasked First? Trajectory Analysis of Diffusion Models for Graph-to-Text Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LQAK7GLP}},
  note         = {Machine review of arXiv:2605.31564}
}

read the original abstract

We present the first systematic study of masked diffusion language models (MDLMs) for graph-to-text generation. We analyze MDLM generation trajectories -- the order in which tokens are unmasked during iterative decoding -- and find that, unlike autoregressive LLMs which generate text linearly, MDLMs naturally prioritize entities first, followed by relational and function words, with structural tokens resolved last. We further identify a previously undocumented failure mode of supervised fine-tuning: SFT disrupts this strategy by prematurely anchoring structural sentence-ending tokens early in the decoding trajectory, effectively fixing the output length which can lead to omitted or hallucinated information. To address this, we propose lambda-scaled structural decoding, a training-free inference-time modification that downweights structural token confidence and recovers +9.4 BLEU-4. Finally, we introduce Graph-LLaDA, which integrates a Graph Transformer encoder into LLaDA's decoding process to explicitly incorporate relational graph structure. Cross-dataset evaluation on LAGRANGE reveals that previous baselines overfit to dataset-specific patterns, while LLM- and MDLM-based approaches generalize significantly better.

Figures

Figures reproduced from arXiv: 2605.31564 by the authors.

**Figure 1.** An overview of Graph-LLaDA. Trained and frozen modules of our two-stage training process are indicated along with their associated stage. structure into the decoding process. 2 Methodology We first define the graph-to-text generation task, then present our methodology for analyzing the generation trajectories of masked diffusion models, propose a decoding modification motivated by this analysis, and finally describ… view at source ↗

**Figure 2.** Trajectory comparison of LLaMA and LLaDA. An expanded comparison can be found in Figure [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Token unmasking trajectory of MDLMs on a WebNLG sample with input triples [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Trajectory comparison of classified token types for LLaDA and autoregressive models. [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]

**Figure 5.** Figure 5: Generation examples from WebNLG (Examples 1–2) and LAGRANGE (Example 3) comparing base [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]

**Figure 6.** Figure 6: Pairwise Win Rates for Graph-to-Text Generation on LAGRANGE Using Gemma 4-31B as Judge. [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]

**Figure 7.** Figure 7: Pairwise Win Rates for Graph-to-Text Generation on LAGRANGE Using Qwen 3.6-35B as Judge. [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]

**Figure 8.** Figure 8: Pairwise Win Rates for Graph-to-Text Generation on WebNLG Using Qwen 3.6-35B as Judge. [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

4 extracted references · 3 canonical work pages

[1]

Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong

Taming masked diffusion language models via consistency trajectory reinforcement learning with fewer decoding step.Preprint, arXiv:2509.23924. Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong

work page arXiv
[2]

Dream 7B: Diffusion Large Language Models

Dream 7b: Diffusion large language models. arXiv preprint arXiv:2508.15487. Shuzhou Yuan and Michael Faerber. 2023. Evaluating generative models for graph-to-text generation. In Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing, pages 1256–1264, Varna, Bulgaria. INCOMA Ltd., Shoumen, Bulgaria. Seongjun Yun,...

work page Pith review arXiv 2023
[3]

A Methodology Details A.1 Implementation Details For LLaDA SFT, we fine-tune with LoRA (Hu et al., 2022) ( r= 16 , α= 32 ) for 5 epochs us- ing random-token masking

A survey on parallel text generation: From par- allel decoding to diffusion language models.arXiv preprint arXiv:2508.08712. A Methodology Details A.1 Implementation Details For LLaDA SFT, we fine-tune with LoRA (Hu et al., 2022) ( r= 16 , α= 32 ) for 5 epochs us- ing random-token masking. For Graph-LLaDA, the GNN encoder uses a Graph Transformer with a h...

work page arXiv 2022
[4]

" step (mean) Content before first

have explored different strategies for improv- ing multi-token generation, thus any findings re- lated to standard multi-token generation may not be applicable to real-world implementations. A.2 Token Classification Details This section expands the priority-based token clas- sification procedure summarized in Section 2. Each generated token is assigned to...

2025

Pith tools

Reviewed June 28, 2026 · model on record in the stance chip above.

[1] [1]

Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong

Taming masked diffusion language models via consistency trajectory reinforcement learning with fewer decoding step.Preprint, arXiv:2509.23924. Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong

work page arXiv

[2] [2]

Dream 7B: Diffusion Large Language Models

Dream 7b: Diffusion large language models. arXiv preprint arXiv:2508.15487. Shuzhou Yuan and Michael Faerber. 2023. Evaluating generative models for graph-to-text generation. In Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing, pages 1256–1264, Varna, Bulgaria. INCOMA Ltd., Shoumen, Bulgaria. Seongjun Yun,...

work page Pith review arXiv 2023

[3] [3]

A Methodology Details A.1 Implementation Details For LLaDA SFT, we fine-tune with LoRA (Hu et al., 2022) ( r= 16 , α= 32 ) for 5 epochs us- ing random-token masking

A survey on parallel text generation: From par- allel decoding to diffusion language models.arXiv preprint arXiv:2508.08712. A Methodology Details A.1 Implementation Details For LLaDA SFT, we fine-tune with LoRA (Hu et al., 2022) ( r= 16 , α= 32 ) for 5 epochs us- ing random-token masking. For Graph-LLaDA, the GNN encoder uses a Graph Transformer with a h...

work page arXiv 2022

[4] [4]

" step (mean) Content before first

have explored different strategies for improv- ing multi-token generation, thus any findings re- lated to standard multi-token generation may not be applicable to real-world implementations. A.2 Token Classification Details This section expands the priority-based token clas- sification procedure summarized in Section 2. Each generated token is assigned to...

2025