REVIEW 6 cited by
Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokens
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Recent endeavors in Multimodal Large Language Models (MLLMs) aim to unify visual comprehension and generation by combining LLM and diffusion models, the state-of-the-art in each task, respectively. Existing approaches rely on spatial visual tokens, where image patches are encoded and arranged according to a spatial order (e.g., raster scan). However, we show that spatial tokens lack the recursive structure inherent to languages, hence form an impossible language for LLM to master. In this paper, we build a proper visual language by leveraging diffusion timesteps to learn discrete, recursive visual tokens. Our proposed tokens recursively compensate for the progressive attribute loss in noisy images as timesteps increase, enabling the diffusion model to reconstruct the original image at any timestep. This approach allows us to effectively integrate the strengths of LLMs in autoregressive reasoning and diffusion models in precise image generation, achieving seamless multimodal comprehension and generation within a unified framework. Extensive experiments show that we achieve superior performance for multimodal comprehension and generation simultaneously compared with other MLLMs. Project Page: https://DDT-LLaMA.github.io/.
Forward citations
Cited by 6 Pith papers
-
FocusDiff: Advancing Fine-Grained Text-Image Alignment for Autoregressive Visual Generation through RL
FocusDiff improves autoregressive text-to-image generation by training on paired similar prompts with a modified GRPO objective, achieving state-of-the-art alignment on PairComp and gains on GenEval and T2I-CompBench.
-
D-AR: Diffusion via Autoregressive Models
D-AR recasts pixel-space diffusion as vanilla autoregressive next-token prediction using a diffusion-ordered discrete tokenizer, reaching 2.09 FID on ImageNet 256x256 with a 775M Llama backbone.
-
Towards Meta-Cognitive Knowledge Editing for Multimodal LLMs
CogEdit and MIND shift multimodal knowledge editing toward evaluating and enabling meta-cognitive skills: self-awareness, boundary monitoring, and noise robustness.
-
What Limits Virtual Agent Application? OmniBench: A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilities
A self-generating graph benchmark produces 36k GUI agent tasks with controllable complexity and ten capability scores, and fine-tuning on its trajectories gives small gains on AndroidControl and OmniAct.
-
FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocities
A 1.5B unified multimodal model trained with discrete flow matching and metric-induced probability paths matches autoregressive baselines of similar size on generation and understanding benchmarks.
-
KRIS-Bench: Benchmarking Next-Level Intelligent Image Editing Models
A new benchmark, KRIS-Bench, evaluates image editing models on knowledge-grounded reasoning across factual, conceptual, and procedural tasks, and finds large performance gaps in current models.
Discussion (0). Sign in to comment.