{"work":{"id":"c1a2d4de-4439-4005-8c56-e0124e4ed5fa","openalex_id":null,"doi":null,"arxiv_id":"2510.11690","raw_key":null,"title":"Diffusion Transformers with Representation Autoencoders","authors":null,"authors_text":"Boyang Zheng, Nanye Ma, Shengbang Tong, Saining Xie","year":2025,"venue":"cs.CV","abstract":"Latent generative modeling, where a pretrained autoencoder maps pixels into a latent space for the diffusion process, has become the standard strategy for Diffusion Transformers (DiT); however, the autoencoder component has barely evolved. Most DiTs continue to rely on the original VAE encoder, which introduces several limitations: outdated backbones that compromise architectural simplicity, low-dimensional latent spaces that restrict information capacity, and weak representations that result from purely reconstruction-based training and ultimately limit generative quality. In this work, we explore replacing the VAE with pretrained representation encoders (e.g., DINO, SigLIP, MAE) paired with trained decoders, forming what we term Representation Autoencoders (RAEs). These models provide both high-quality reconstructions and semantically rich latent spaces, while allowing for a scalable transformer-based architecture. Since these latent spaces are typically high-dimensional, a key challenge is enabling diffusion transformers to operate effectively within them. We analyze the sources of this difficulty, propose theoretically motivated solutions, and validate them empirically. Our approach achieves faster convergence without auxiliary representation alignment losses. Using a DiT variant equipped with a lightweight, wide DDT head, we achieve strong image generation results on ImageNet: 1.51 FID at 256x256 (no guidance) and 1.13 at both 256x256 and 512x512 (with guidance). RAE offers clear advantages and should be the new default for diffusion transformer training.","external_url":"https://arxiv.org/abs/2510.11690","cited_by_count":null,"metadata_source":"pith","metadata_fetched_at":"2026-05-25T07:40:28.627587+00:00","pith_arxiv_id":"2510.11690","created_at":"2026-05-09T23:09:26.695232+00:00","updated_at":"2026-05-25T07:40:28.627587+00:00","title_quality_ok":true,"display_title":"Diffusion Transformers with Representation Autoencoders","render_title":"Diffusion Transformers with Representation Autoencoders"},"hub":{"state":{"work_id":"c1a2d4de-4439-4005-8c56-e0124e4ed5fa","tier":"hub","tier_reason":"10+ Pith inbound or 1,000+ external citations","pith_inbound_count":65,"external_cited_by_count":null,"distinct_field_count":8,"first_pith_cited_at":"2025-11-17T18:59:57+00:00","last_pith_cited_at":"2026-05-22T17:59:42+00:00","author_build_status":"not_needed","summary_status":"needed","contexts_status":"needed","graph_status":"needed","ask_index_status":"not_needed","reader_status":"not_needed","recognition_status":"not_needed","updated_at":"2026-06-01T17:33:38.936863+00:00","tier_text":"hub"},"tier":"hub","role_counts":[{"context_role":"background","n":17},{"context_role":"baseline","n":3},{"context_role":"method","n":1},{"context_role":"other","n":1}],"polarity_counts":[{"context_polarity":"background","n":16},{"context_polarity":"baseline","n":3},{"context_polarity":"support","n":1},{"context_polarity":"unclear","n":1},{"context_polarity":"use_method","n":1}],"runs":{"context_extract":{"job_type":"context_extract","status":"succeeded","result":{"enqueued_papers":25},"error":null,"updated_at":"2026-05-14T18:10:19.415038+00:00"},"graph_features":{"job_type":"graph_features","status":"succeeded","result":{"co_cited":[{"title":"Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think","work_id":"1aff8ef8-079b-4afe-9e6a-148e6fd08e6a","shared_citers":12},{"title":"SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features","work_id":"50eec732-2d41-432f-9dcf-ac7fff235ea5","shared_citers":12},{"title":"arXiv preprint arXiv:2510.15301 (2025)","work_id":"ab5fd0eb-3300-410d-a690-19e04b2a7770","shared_citers":11},{"title":"DINOv2: Learning Robust Visual Features without Supervision","work_id":"26b304e5-b54a-4f26-be7e-83299eca52e4","shared_citers":11},{"title":"Emerging Properties in Unified Multimodal Pretraining","work_id":"e0cfd82c-f5d4-44fd-b531-ec73ab0a805b","shared_citers":9},{"title":"Scaling text-to-image diffusion transformers with representation autoencoders","work_id":"f6b79552-4f2d-4b7a-9874-6ef1938b8a54","shared_citers":9},{"title":"Flow Matching for Generative Modeling","work_id":"6edb71c4-5d64-40af-a394-9757ea051a36","shared_citers":8},{"title":"Score-Based Generative Modeling through Stochastic Differential Equations","work_id":"d9110e53-a5d4-4794-a4c5-a575e91c31ad","shared_citers":8},{"title":"Back to Basics: Let Denoising Generative Models Denoise","work_id":"37973de8-a5e6-4d92-897b-a98fa9f7f2f3","shared_citers":7},{"title":"Classifier-Free Diffusion Guidance","work_id":"acf2c588-c088-4a6c-938e-150ad7c666d7","shared_citers":7},{"title":"ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment","work_id":"94248955-4bc5-4517-98a0-66224a36d865","shared_citers":7},{"title":"SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis","work_id":"8034c587-fba6-4941-87ba-c98f2ac962cb","shared_citers":7},{"title":"Wan: Open and Advanced Large-Scale Video Generative Models","work_id":"ad3ebc3b-4224-46c9-b61d-bcf135da0a7c","shared_citers":7},{"title":"An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale","work_id":"e96730e3-129b-4db6-b981-15ab7932e297","shared_citers":6},{"title":"BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset","work_id":"86d896d2-592f-4d9b-938e-dfeb11f9388f","shared_citers":6},{"title":"DDT: Decoupled diffusion Transformer","work_id":"8625279b-e85e-4c87-ba22-16d4d9e1f709","shared_citers":6},{"title":"DINOv3","work_id":"c8b07deb-8fe7-4e18-9620-f3569d3529ce","shared_citers":6},{"title":"Repa-e: Unlocking vae for end-to-end tuning with latent diffusion transformers","work_id":"4ad471a0-3426-4e98-818c-cb238de280a9","shared_citers":6},{"title":"Auto-Encoding Variational Bayes","work_id":"97d95295-30e1-42b4-bbf6-85f0fa4edb44","shared_citers":5},{"title":"Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation","work_id":"41efe203-9377-4c63-b1d6-e499cd6e46f6","shared_citers":5},{"title":"Chameleon: Mixed-Modal Early-Fusion Foundation Models","work_id":"2661b9a6-25cc-41a1-8100-612d2b801289","shared_citers":5},{"title":"CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer","work_id":"f38fc088-12aa-4bf4-9ecd-08d3e797ccb7","shared_citers":5},{"title":"Decoupled Weight Decay Regularization","work_id":"07ef7360-d385-4033-83f7-8384a6325204","shared_citers":5},{"title":"Emu3: Next-Token Prediction is All You Need","work_id":"720d288e-fac0-464c-9929-19efd9a52afc","shared_citers":5}],"time_series":[{"n":1,"year":2025},{"n":32,"year":2026}],"dependency_candidates":[]},"error":null,"updated_at":"2026-05-14T18:09:19.811370+00:00"},"identity_refresh":{"job_type":"identity_refresh","status":"succeeded","result":{"items":[{"title":"Qwen3 Technical Report","outcome":"unchanged","work_id":"25a4e30c-1232-48e7-9925-02fa12ba7c9e","resolver":"local_arxiv","confidence":0.98,"old_work_id":"25a4e30c-1232-48e7-9925-02fa12ba7c9e"}],"counts":{"fixed":0,"merged":0,"unchanged":1,"quarantined":0,"needs_external_resolution":0},"errors":[],"attempted":1},"error":null,"updated_at":"2026-05-14T18:10:23.590004+00:00"},"summary_claims":{"job_type":"summary_claims","status":"succeeded","result":{"title":"Diffusion Transformers with Representation Autoencoders","claims":[{"claim_text":"Latent generative modeling, where a pretrained autoencoder maps pixels into a latent space for the diffusion process, has become the standard strategy for Diffusion Transformers (DiT); however, the autoencoder component has barely evolved. Most DiTs continue to rely on the original VAE encoder, which introduces several limitations: outdated backbones that compromise architectural simplicity, low-dimensional latent spaces that restrict information capacity, and weak representations that result from purely reconstruction-based training and ultimately limit generative quality. In this work, we ex","claim_type":"abstract","evidence_strength":"source_metadata"}],"why_cited":"Pith tracks Diffusion Transformers with Representation Autoencoders because it crossed a citation-hub threshold.","role_counts":[]},"error":null,"updated_at":"2026-05-14T18:10:07.464893+00:00"}},"summary":{"title":"Diffusion Transformers with Representation Autoencoders","claims":[{"claim_text":"Latent generative modeling, where a pretrained autoencoder maps pixels into a latent space for the diffusion process, has become the standard strategy for Diffusion Transformers (DiT); however, the autoencoder component has barely evolved. Most DiTs continue to rely on the original VAE encoder, which introduces several limitations: outdated backbones that compromise architectural simplicity, low-dimensional latent spaces that restrict information capacity, and weak representations that result from purely reconstruction-based training and ultimately limit generative quality. In this work, we ex","claim_type":"abstract","evidence_strength":"source_metadata"}],"why_cited":"Pith tracks Diffusion Transformers with Representation Autoencoders because it crossed a citation-hub threshold.","role_counts":[]},"graph":{"co_cited":[{"title":"Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think","work_id":"1aff8ef8-079b-4afe-9e6a-148e6fd08e6a","shared_citers":12},{"title":"SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features","work_id":"50eec732-2d41-432f-9dcf-ac7fff235ea5","shared_citers":12},{"title":"arXiv preprint arXiv:2510.15301 (2025)","work_id":"ab5fd0eb-3300-410d-a690-19e04b2a7770","shared_citers":11},{"title":"DINOv2: Learning Robust Visual Features without Supervision","work_id":"26b304e5-b54a-4f26-be7e-83299eca52e4","shared_citers":11},{"title":"Emerging Properties in Unified Multimodal Pretraining","work_id":"e0cfd82c-f5d4-44fd-b531-ec73ab0a805b","shared_citers":9},{"title":"Scaling text-to-image diffusion transformers with representation autoencoders","work_id":"f6b79552-4f2d-4b7a-9874-6ef1938b8a54","shared_citers":9},{"title":"Flow Matching for Generative Modeling","work_id":"6edb71c4-5d64-40af-a394-9757ea051a36","shared_citers":8},{"title":"Score-Based Generative Modeling through Stochastic Differential Equations","work_id":"d9110e53-a5d4-4794-a4c5-a575e91c31ad","shared_citers":8},{"title":"Back to Basics: Let Denoising Generative Models Denoise","work_id":"37973de8-a5e6-4d92-897b-a98fa9f7f2f3","shared_citers":7},{"title":"Classifier-Free Diffusion Guidance","work_id":"acf2c588-c088-4a6c-938e-150ad7c666d7","shared_citers":7},{"title":"ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment","work_id":"94248955-4bc5-4517-98a0-66224a36d865","shared_citers":7},{"title":"SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis","work_id":"8034c587-fba6-4941-87ba-c98f2ac962cb","shared_citers":7},{"title":"Wan: Open and Advanced Large-Scale Video Generative Models","work_id":"ad3ebc3b-4224-46c9-b61d-bcf135da0a7c","shared_citers":7},{"title":"An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale","work_id":"e96730e3-129b-4db6-b981-15ab7932e297","shared_citers":6},{"title":"BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset","work_id":"86d896d2-592f-4d9b-938e-dfeb11f9388f","shared_citers":6},{"title":"DDT: Decoupled diffusion Transformer","work_id":"8625279b-e85e-4c87-ba22-16d4d9e1f709","shared_citers":6},{"title":"DINOv3","work_id":"c8b07deb-8fe7-4e18-9620-f3569d3529ce","shared_citers":6},{"title":"Repa-e: Unlocking vae for end-to-end tuning with latent diffusion transformers","work_id":"4ad471a0-3426-4e98-818c-cb238de280a9","shared_citers":6},{"title":"Auto-Encoding Variational Bayes","work_id":"97d95295-30e1-42b4-bbf6-85f0fa4edb44","shared_citers":5},{"title":"Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation","work_id":"41efe203-9377-4c63-b1d6-e499cd6e46f6","shared_citers":5},{"title":"Chameleon: Mixed-Modal Early-Fusion Foundation Models","work_id":"2661b9a6-25cc-41a1-8100-612d2b801289","shared_citers":5},{"title":"CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer","work_id":"f38fc088-12aa-4bf4-9ecd-08d3e797ccb7","shared_citers":5},{"title":"Decoupled Weight Decay Regularization","work_id":"07ef7360-d385-4033-83f7-8384a6325204","shared_citers":5},{"title":"Emu3: Next-Token Prediction is All You Need","work_id":"720d288e-fac0-464c-9929-19efd9a52afc","shared_citers":5}],"time_series":[{"n":1,"year":2025},{"n":32,"year":2026}],"dependency_candidates":[]},"authors":[]}}