{"id":"ee565af6-a89c-4e6b-b32a-97ba5aedd23a","arxiv_id":"2605.18324","paper_version":1,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"RAE v2 reaches gFID 1.06 on ImageNet-256 in 80 epochs by combining multi-layer encoder sums, complementary REPA targets, and free guidance via output reparameterization.","lead":"The paper improves Representation Autoencoders by summing the last k layers of a pretrained vision encoder instead of using only the final layer, showing that RAE and REPA can be used together on the same representation, and reparameterizing the diffusion output to enable classifier-free guidance without a second model. These changes yield over 10x faster convergence and state-of-the-art image generation metrics on ImageNet-256.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.3","headline":"Complementarity of RAE and REPA plus reliance on specific pretrained encoders may require retuning outside tested setups","rationale":"The reader's weakest assumption directly identifies the same load-bearing empirical generalization risk. Because the paper supplies code and the improvements are purely empirical, the proposed check is a minimal, decisive replication step that would either confirm or falsify the central claim without requiring new theory.","tokens_in":1931,"tokens_out":387,"duration_ms":40103,"concrete_test":"Re-run the 80-epoch ImageNet-256 RAEv2 training (same DiT, same optimizer schedule) but replace the CLIP-style encoder with DINOv2; measure EP_FID@2 and final unguided gFID. If either metric degrades by more than 20% relative to the reported RAEv2 numbers, the complementarity claim does not generalize.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline gains (10x faster convergence to gFID 1.06 at 80 epochs, free guidance via re-parameterization) rest on two empirical observations: (1) summing the last k layers of a fixed pretrained encoder improves reconstruction without finetuning, and (2) RAE and REPA are complementary so the same representation can serve as both encoder and intermediate-layer target. These are shown via large-scale ablations on ImageNet-256 with CLIP-style encoders and DiT. If the complementarity is an interaction effect tied to the particular encoder statistics or the x-prediction re-parameterization rather than a general mechanism, the combined RAEv2 recipe will not deliver the reported speed-up or SOTA numbers when the encoder or backbone changes. The paper reports consistent gains on text-to-image and navigation, yet those domains still use similar vision encoders; no systematic sweep over encoder families or architectures is described that would bound the assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces RAEv2, an improved version of Representation Autoencoders that replaces VAEs with pretrained vision encoders. Key contributions include: (1) defining the representation as the sum of the last k encoder layers rather than only the final layer, improving reconstruction without finetuning; (2) empirical demonstration that RAE and REPA are complementary, allowing the same pretrained representation to serve as both encoder and intermediate-layer target in diffusion models; (3) re-parameterizing the DiT output to enable free classifier-free guidance without training a second model. These changes yield >10x faster convergence, with state-of-the-art gFID of 1.06 and FDr^k of 2.17 on ImageNet-256 after only 80 epochs (vs. prior best at 800 epochs), plus consistent gains on text-to-image and navigation tasks. A new efficiency metric EP_FID@k is proposed, and code is released.","tokens_in":2143,"tokens_out":701,"duration_ms":27428,"significance":"If the empirical results hold under broader conditions, this provides a strong, simplified baseline for diffusion-based generative models that leverages pretrained representations for both encoding and alignment. The reported 10x speedup in convergence to high-quality samples, free guidance mechanism, and new efficiency metric could influence training practices in image synthesis and related domains. The large-scale ablations on ImageNet-256 with DiT and cross-task validation add credibility, while the code release supports reproducibility.","major_comments":[{"comment":"The central claim of complementarity between RAE and REPA (allowing the same representation for encoder and REPA target) is load-bearing for the RAEv2 recipe and the reported speedups. However, the experiments primarily use CLIP-style encoders on ImageNet-256 with DiT; if this interaction depends on specific encoder statistics or the x-prediction re-parameterization, the gains may not generalize without retuning. Additional ablations with alternative encoder families (e.g., DINOv2 or non-CLIP variants) would directly test this assumption.","section":"Complementarity analysis and ImageNet-256 experiments"},{"comment":"The free guidance via re-parameterization of the DiT output is presented as a key simplification over AutoGuidance. The manuscript should clarify whether this re-parameterization preserves the exact equivalence to REPA's intermediate-layer distillation or introduces any approximation that could affect guidance strength at different scales.","section":"Guidance re-parameterization section"}],"minor_comments":[{"comment":"The exact values of k (number of summed layers) and the specific pretrained encoder checkpoints used in the main results should be stated explicitly in the experimental setup, as these are free parameters in the method.","section":"Method and experimental details"},{"comment":"Training schedules, learning rates, and batch sizes for the 80-epoch RAEv2 runs versus the 800-epoch baselines should be tabulated for direct comparison to ensure the efficiency claims are not confounded by optimization differences.","section":"Training details"},{"comment":"Figure captions and axis labels for the convergence plots could more clearly indicate the number of epochs at which each method reaches the reported gFID thresholds.","section":"Figures"}],"recommendation":"minor_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the positive evaluation and constructive comments. We respond to each major comment below and indicate the corresponding revisions to the manuscript.","responses":[{"response":"We thank the referee for this observation. Our primary large-scale ablations and ImageNet-256 results do center on CLIP-style encoders with DiT, as these constitute the standard experimental setting for such models. The complementarity finding is supported by extensive controlled ablations within this regime. We also report consistent gains when transferring the full RAEv2 recipe to text-to-image generation and navigation world models, which employ different data distributions and encoder families. In the revision we will expand the discussion section to explicitly link these cross-task results to the question of generalization and to note that the core mechanisms (multi-layer summation, joint encoder-target usage, and output re-parameterization) are architecture-agnostic. Full-scale DINOv2 ablations on ImageNet-256 would require substantial additional compute; we will therefore flag this as valuable future work rather than claim to have performed it.","revision_made":"partial","referee_comment":"The central claim of complementarity between RAE and REPA (allowing the same representation for encoder and REPA target) is load-bearing for the RAEv2 recipe and the reported speedups. However, the experiments primarily use CLIP-style encoders on ImageNet-256 with DiT; if this interaction depends on specific encoder statistics or the x-prediction re-parameterization, the gains may not generalize without retuning. Additional ablations with alternative encoder families (e.g., DINOv2 or non-CLIP variants) would directly test this assumption."},{"response":"We appreciate the request for clarification. The re-parameterization is an exact algebraic rewriting that treats the REPA target as an x-prediction objective inside the RAE latent space; no distributional approximation is introduced. Because the transformation is linear, the equivalence holds for any classifier-free guidance scale. In the revised manuscript we will insert a short derivation (either in the main text or as an appendix) that makes this equivalence explicit and confirms scale independence.","revision_made":"yes","referee_comment":"The free guidance via re-parameterization of the DiT output is presented as a key simplification over AutoGuidance. The manuscript should clarify whether this re-parameterization preserves the exact equivalence to REPA's intermediate-layer distillation or introduces any approximation that could affect guidance strength at different scales."}],"tokens_in":1681,"tokens_out":528,"duration_ms":47861,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"This paper's main result is that three straightforward changes to representation autoencoders—summing the last k layers instead of just the final one, running RAE and REPA together on the same pretrained representation, and reparameterizing the DiT output to get classifier-free guidance without a second model—produce more than 10x faster convergence while hitting a gFID of 1.06 at 80 epochs on ImageNet-256. They also introduce EP_FID@k as a practical way to track training efficiency and show the recipe works on text-to-image and navigation tasks too.","headline":"Summing the last k layers from a pretrained encoder, pairing RAE with REPA on the same rep, and reparameterizing for free guidance delivers over 10x faster convergence to strong FID on ImageNet-256.","tokens_in":2659,"tokens_out":208,"would_cite":true,"duration_ms":33603,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":{"model":"grok-4.3","evidence":[{"relation":"unclear","rs_module":"IndisputableMonolith/Cost/FunctionalEquation.lean","rs_theorem":null,"paper_passage":"Generalized Representation Encoder... sum of the last K layer features... RAE and REPA exhibit complementary working mechanisms... REPA is x-prediction in RAE latent space"},{"relation":"unclear","rs_module":"IndisputableMonolith/Foundation/DimensionForcing.lean","rs_theorem":null,"paper_passage":"10x faster convergence... gFID of 1.06 in just 80 epochs on ImageNet-256"}],"headline":"Empirical improvements to representation autoencoders for diffusion models lie outside RS scope","alignment":"orthogonal","rationale":"The paper's machinery (multi-layer summation of pretrained encoder features, complementarity of RAE and REPA, x-prediction reparameterization for free guidance, EP_FID@k efficiency metric) consists of standard ML engineering and large-scale ablations on ImageNet/CLIP-style encoders. It contains no J-cost functions, golden-ratio identities, ratio-symmetric forcing, 8-tick periodicity, or derivations from a single distinction. RS modules such as Cost.FunctionalEquation, Foundation.RealityFromDistinction, and AlexanderDuality are irrelevant here.","tokens_in":62380,"confidence":"high","tokens_out":308,"duration_ms":15128,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Representation autoencoders reach SOTA image generation by summing last k encoder layers, combining with REPA, and reparameterizing for free guidance.","keywords":["representation autoencoders","diffusion models","image generation","representation alignment","classifier-free guidance","ImageNet","training efficiency"],"falsifier":"Training RAEv2 on a new dataset or architecture and finding that convergence speed and final quality match the original RAE only after extensive retuning would show the claimed generality does not hold.","tokens_in":2835,"feed_emoji":"📈","tokens_out":723,"duration_ms":40860,"temperature":0.7,"pith_summary":"The paper shows that representation autoencoders, which replace VAEs with pretrained vision encoders in diffusion models, can be improved through three targeted changes. Defining the representation as the sum of the last k encoder layers rather than only the final layer raises reconstruction quality without encoder fine-tuning or specialized data. Large-scale tests reveal that RAE and representation alignment (REPA) complement each other, so the same pretrained features can serve as both the encoder and the target for intermediate diffusion layers. Reparameterizing the DiT output lets REPA supply classifier-free guidance without training a second weaker model. Together these steps produce RAEv2, which converges more than ten times faster while reaching higher final quality on ImageNet generation.","feed_headline":"RAE v2 reaches SOTA gFID 1.06 in 80 epochs on ImageNet","feed_subtitle":"Summing last k encoder layers plus REPA complementarity delivers free guidance and over 10x faster convergence.","key_machinery":"Generalized multi-layer sum representation in RAE together with its complementarity to REPA that enables free guidance via output re-parameterization.","core_discovery":"By adopting a generalized formulation where the representation sums the last k layers of a pretrained encoder, recognizing that RAE and REPA are complementary so the same representation can be used for both encoding and alignment, and re-parameterizing the diffusion model output to obtain guidance for free, RAEv2 achieves more than 10x faster convergence, a state-of-the-art gFID of 1.06 in 80 epochs on ImageNet-256, and a state-of-the-art FDr^k of 2.17 at 80 epochs without post-training.","pith_inferences":["The same layer-summation and complementarity pattern could be tested in video or 3D diffusion models to check for similar efficiency gains.","If the free-guidance trick generalizes, many existing diffusion training pipelines could drop the cost of separate guidance models.","The approach suggests that assumptions about whether alignment replaces or augments autoencoding should be re-examined in other generative settings."],"forward_implications":["RAEv2 attains EP_FID@2 of 35 epochs versus 177 epochs for the original RAE.","State-of-the-art FDr^k of 2.17 is reached at 80 epochs compared with the prior best of 3.26 at 800 epochs.","No second diffusion model is needed for AutoGuidance.","Consistent gains appear in text-to-image generation and navigation world models."],"fun_headline_variants":["RAEv2 sums last k layers to reach 1.06 gFID in 80 epochs","RAE and REPA are complementary for free guidance","RAEv2 more than 10x faster to 1.06 gFID in 80 epochs","RAEv2 reaches 2.17 FDr at 80 epochs without post-training"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Pretrained vision encoders supply representations general enough that the improvements transfer to new domains and architectures without major hyperparameter retuning.","fun_headline_variants_meta":{"raw":{"variants":["RAEv2 sums last k layers to reach 1.06 gFID in 80 epochs","RAE and REPA are complementary for free guidance","RAEv2 more than 10x faster to 1.06 gFID in 80 epochs","RAEv2 reaches 2.17 FDr at 80 epochs without post-training"]},"model":"grok-4.3","cost_usd":0.020711,"raw_usage":{"total_tokens":8822,"prompt_tokens":860,"num_sources_used":0,"completion_tokens":86,"cost_in_usd_ticks":207112000,"prompt_tokens_details":{"text_tokens":860,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":7876,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":860,"tokens_out":86,"duration_ms":67274,"temperature":1.0,"reasoning_tokens":7876,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-20T11:35:44.861778+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Training RAEv2 on a new dataset or architecture and finding that convergence speed and final quality match the original RAE only after extensive retuning would show the claimed generality does not hold.","supporting_citations":[],"review_version":1}