Pith. sign in

REVIEW 3 major objections 4 minor 106 references

A third pretraining axis: train on the best of K generated candidates, not the average

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 08:34 UTC pith:7XKD2IJV

load-bearing objection Best-of-K training as a scaling axis is empirically convincing and worth referee time, but the paper's own theory doesn't cover the objective it actually runs. the 3 major comments →

arxiv 2607.27372 v1 pith:7XKD2IJV submitted 2026-07-29 cs.LG cs.AIcs.CLcs.CV

Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation

Alexi Gladstone , Heng Ji , Yilun Du This is my paper
classification cs.LG cs.AIcs.CLcs.CV
keywords Explorative Modelingbest-of-K traininggenerative expressivitythird pretraining axisend-to-end generative modelingmode blurringmultimodal distributionstraining-loop factorization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that generative models have a neglected, scaleable capacity axis: exploration. Instead of changing how the model generates, the training loop is modified: at each step the model draws K candidate outputs for a data point and is trained only on the closest match. This best-of-K objective changes the loss minimizer from the blurred mean of many valid targets toward the targets themselves, so the number of modes a model can represent grows with K. Across images, video, and language, increasing K monotonically improves fidelity and generalization, and the gains grow with model and data scale rather than saturating. If true, this is significant because it implies current recipes can be improved by a cheap training-time loop while also enabling fully end-to-end generation with a single forward pass at inference.

Core claim

The central claim is that factorizing the training loop can substitute for factorizing the generation procedure. Plain single-step regression fails on multimodal data because the loss minimizer of many valid targets is their average, which lies off the data manifold. Explorative Modeling instead minimizes min_i J(yhat_i, x) over K generated candidates per data point (Forward XM) or over K data targets per generation (Reverse XM). The paper argues, and proves for a smooth soft-min surrogate, that this is maximum likelihood of a K-component mixture and that generative expressivity scales with K; the objective recovers the data distribution as K grows. Empirically, adding exploration to diffusi

What carries the argument

The load-bearing object is the best-of-K objective L(theta) = min_i J(yhat_i, x). Forward XM fixes a real data target and searches K model generations; Reverse XM fixes one generation and searches K data targets. The objective's role is to move the loss minimizer off the mean and onto individual modes: each explored candidate can specialize to a different mode, so K directly controls generative expressivity. The paper's formal statements (maximum-likelihood, mode-covering, expressivity at least K) are proven for a smooth soft-min surrogate built from Gaussian kernels; the implemented hard min differs by at most log K, but, as the paper concedes, its large-K limit is a coverage objective with

Load-bearing premise

The load-bearing premise is that the hard-min objective actually used in every experiment behaves like the smooth soft-min objective the theory analyzes: that at finite K, best-of-K training commits predictions to distinct modes rather than merely covering the data's support. The paper's own appendix concedes that the hard-min large-K limit is a coverage objective without distributional control.

What would settle it

Train identical models with hard-min and soft-min XM at several K on a multimodal benchmark and compare sample quality and likelihood. If hard-min stops improving, drops modes, or diverges from soft-min behavior, the mode-commitment mechanism is not what the experiments measure. A second check: measure whether an XM-K model's generation distribution matches the data distribution beyond support coverage (for example, calibrated density or mode weights), since the coverage objective alone would not guarantee this.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Compute-optimal training would allocate a budget among parameters, data, and K; models trained without exploration increasingly fall short of what the same compute could achieve.
  • Because exploration substitutes for generation factorization, the optimal number of generation steps falls as K grows, so models become more end-to-end and less exposed to train-inference mismatch.
  • End-to-end XMs make single-forward-pass policies and world models competitive with diffusion-based ones, shifting the cost of multimodality from inference to training.
  • More exploration buys better generalization on fixed data, giving a compute-for-generalization tradeoff.
  • The reported gains grow with scale, so if the trend holds, foundation-scale training stands to benefit more than the experiments show.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If hard-min and soft-min behavior coincide at moderate K, exploration is effectively implicit energy minimization over latents; gradient-based candidate search (which the paper floats) is a natural next test of the mechanism.
  • The proof's reliance on separable Gaussian kernels suggests exploration's benefits may depend on loss geometry; perceptual, bounded, or non-normalizable losses may not support the same mode-commitment story.
  • Reverse XM with a vector database over all training data could push K to dataset size, turning mode selection into approximate nearest-neighbor search; the paper reports initial language-model success, and a clean scaling curve would test it.
  • Autoregressive LLMs are the hardest case precisely because they lack a natural latent to search over; multi-token prediction or learned latents are more direct tests of whether exploration helps discrete generation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces Explorative Modeling (XM), a training-time procedure in which a generative model draws K candidate matches between its own generations and the data and trains on the closest one (Forward XM), or matches one generation to the closest of K data points (Reverse XM). The authors argue that this best-of-K loop constitutes a new 'third pretraining axis' beyond parameters and data, because it increases the generative expressivity of a model—the number of modes its training objective can retain—without factoring the generation procedure. Empirical results are reported across images (ImageNet 256x256), video (Something-Something V2), language (masked diffusion language models), and control (behavior cloning, Maze2D), showing monotonic improvements with K, large gains in FLOP/sample/parameter efficiency, and end-to-end generation that matches diffusion baselines with far fewer inference steps. A theoretical appendix (F) claims that a smooth soft-min variant of Forward XM is maximum likelihood and has expressivity at least K, and that Reverse XM targets a reverse-KL objective.

Significance. If the empirical findings hold, this is a broadly useful and simple enhancement to generative pretraining: a few lines of code, applicable to existing diffusion/flow and language models, with efficiency and quality gains that appear to grow with scale. The paper's strengths include extensive and internally consistent experiments, comparisons to strong baselines under reproduced setups, an instructive ablation against minibatch optimal transport, and unusually candid limitations. The paper also ships code (github.com/alexiglad/XM). However, the theoretical framing is not fully matched to the implemented algorithm, and the conceptual core—'predictions commit to modes rather than blurring them'—is not formally established for the hard-min objective actually used. The work is therefore significant as an empirical contribution, but the theoretical claims and the 'third axis' narrative need substantial revision before publication.

major comments (3)
  1. [Appendix F.2 / Algorithm 1 / Eq. (1)] The formal results (Propositions 1–3) are proven for the smooth soft-min surrogate −log((1/K) Σ_i k_σ(ŷ_i, x)), but Algorithm 1 and every experiment in the paper implement the hard-min objective in Eq. (1). The paper itself concedes in F.2 that the hard-min large-K limit is a coverage objective 'without distributional control', and F.1 states that the expressivity claims 'belong to the smooth objective'. The log K bound between the hard and soft min bounds the loss value, not the minimizer's mode structure. Since Proposition 3 is the stated basis for the mechanism 'predictions commit to modes rather than blurring them' (abstract and Section 3.1), the implemented method has no formal mode-commitment guarantee at finite K. Please either (a) prove a mode-commitment result for the hard-min objective, or (b) add experiments using the smooth soft-min objective that validate the theory, and adj
  2. [Section 2.1 / Section F.1 / Ref. [19]] The generative expressivity framework and the 'predictions' in Section 5 are built on Mode Forcing [19], an unpublished draft by the same authors. Section F.1 says the paper 'inherit[s] the regularity assumptions of Mode Forcing [19]' and refers to [19] for the mean-blurring lemma behind E=1. As a result, the theoretical contribution is not self-contained, and the Section 5 claims that the theory predicted the experiments cannot be independently verified from this manuscript. Please include the necessary definitions and lemmas in an appendix, or explicitly state which theoretical claims are conditional on the unpublished draft.
  3. [Section 2.1 / Section F.1] The formal definition of Generative Expressivity E in Section F.1 'registers no deficit for any consistent objective' and 'saturates' for autoregression and diffusion (E reaches V^L or unbounded). This contradicts the paper's central narrative that existing generative models are limited by generative expressivity. The intended notion is an informal 'per-prediction expressivity' with a shortfall max(M_t(c_t) − e_t, 0) that is never formally defined. This gap makes it unclear what exactly the 'third axis' is, and why exploration relieves a bottleneck that the formal E does not measure. Please replace the informal per-prediction discussion with a precise definition and show how it connects to the empirical scaling results (Figures 7–10), or revise the conceptual claim accordingly.
minor comments (4)
  1. [Section 4.1 / Figure 10] The claim that gains 'rise from 7% to 36% as data scales and from 13% to 23% as models grow' appears to be based on a small number of scale points. Additional intermediate scales and/or error bars would make the 'grows with scale' conclusion more robust.
  2. [Section 3.2 / Footnote 6] The footnote introduces the smooth surrogate but the surrounding text and Section 4 continue to describe the hard-min objective as 'the loss'. Please consistently state which objective is being analyzed when discussing maximum likelihood or mode-commitment.
  3. [Tables 2 and 3] The claim of 'matching diffusion' is well-supported by the reproduced baselines, but the tasks (Robomimic, Maze2D) are relatively low-dimensional and may not stress multimodal coverage. A sentence acknowledging the limited multimodality of these benchmarks would help calibrate the end-to-end claim.
  4. [Section F.1] The definition of M(q) as 'the number of strict local maxima' is used throughout, but Proposition 3 also requires a separation condition and near-equal masses. Please state explicitly that the expressivity lower bound E≥K is conditional on these additional assumptions, and clarify how the mode count is read at the loss scale for continuous densities.

Circularity Check

1 steps flagged

Empirical gains are independent, but the 'third axis' framing and claimed predictions rest on an unpublished same-author draft (Mode Forcing), producing partial circularity.

specific steps
  1. self citation load bearing [Section 5, 'Mode Forcing as a Predictive Theory'; see also Section 2.1 and Appendix F.1]
    "Because this work builds on Mode Forcing [19], most of its results came about in the opposite order, where the theory predicted them before the experiments were run."

    Mode Forcing [19] is an unpublished draft by the same three authors (Gladstone, Du, Ji). The paper's central interpretive claims—that generative expressivity is an overlooked third axis and that the experiments confirm Mode Forcing's predictions—are justified by this self-citation rather than by an independently checkable source. Appendix F.1 explicitly defers the E=1 mean-blurring lemma and the regularity assumptions to [19], so the theoretical foundation of the 'third axis' framing is not established within this paper. This is load-bearing for the paper's narrative, though the FID/FVD/perplexity and control-task results are measured against external baselines and would stand without the self-citation.

full rationale

The core empirical evidence is self-contained against external benchmarks: XM gains are measured with FID/FVD/perplexity and control-task success rates against DiT, SiT, RAE, Diffusion Policy, and Diffuser, with no fitted parameter renamed as a prediction. The best-of-K objective is openly acknowledged as prior work, so the paper is not renaming a known result into a new one. The only load-bearing circularity is the reliance on the authors' own unpublished Mode Forcing draft for the generative-expressivity concept, the E=1 lemma, and the claim that results were predicted before experiments; this makes the theoretical framing partly self-referential. I also note the hard-min/soft-min issue flagged in Appendix F.2: the mode-commitment guarantee (Proposition 3, E>=K) is proven only for the smooth soft-min surrogate, while Algorithm 1 and all experiments use the hard-min objective, whose large-K limit the paper itself calls a coverage objective 'without distributional control.' That is a real correctness/applicability gap, but it is not circularity, because the empirical measurements do not reduce to the theory by construction. Hence score 4 rather than higher.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 2 invented entities

The central claim rests on (i) hand-chosen exploration levels K, (ii) a theoretical bridge from hard-min to soft-min that is not proven, (iii) the authors' unpublished Mode Forcing definition of generative expressivity, and (iv) kernel/KL assumptions for the theoretical claims. The raw empirical measurements are largely independent of these, but the 'third axis' interpretation depends on them.

free parameters (3)
  • Exploration level K = Varies: K=2 for RAE, K=8 for MDLM, K=10 for control, K up to 25 for FID sweeps
    Central axis of the paper; K is chosen per experiment and task. The paper recommends sweeping K and gives no principled selection rule, so the reported gains are conditional on hand-selected exploration levels.
  • Autoguidance scale for guided RAE results = 1.35, 1.42, 1.5 depending on model and checkpoint
    Guided FDr6/gFID numbers in Table 1 depend on guidance scale and guiding checkpoint chosen by best metric; these are post-hoc selections.
  • Train coverage percent for Reverse XM language models = e.g., 50%
    A coverage heuristic used in place of an entropy term to prevent Reverse XM collapse; tuned to make the method work.
axioms (5)
  • ad hoc to paper The hard-min objective can be analyzed through the smooth soft-min surrogate; the difference is at most log K and behavior transfers.
    All proofs in Appendix F.2 are for the smooth objective, while Algorithm 1 uses the hard min. The paper concedes the hard-min large-K limit is a coverage objective without distributional control, so transfer is not established.
  • domain assumption Generative expressivity E, defined via sup over data distributions of mode count of loss minimizers, is a meaningful capacity fixed by the objective at design time.
    The 'third axis' argument requires E to be the bottleneck at scale; the definition and scope are inherited from the authors' unpublished Mode Forcing draft [19] and are not independently operationalized.
  • domain assumption Reconstruction losses are Bregman divergences and squared error maps to a Gaussian kernel with width σ; blurring model and data by kσ gives the KL identities in Appendix F.
    Propositions 1-2 rely on this kernel interpretation; Section F.2 notes it fails for non-normalizable or perceptual losses.
  • domain assumption Per-prediction residual multimodality remains a bottleneck for existing scalable generative models even after their sampling has converged.
    Used to explain why exploration helps diffusion models even when sample quality has converged (Section 5, F.1); not directly measured.
  • standard math Standard asymptotic and regularity conditions (LLN, domination, finite entropies, minimizers range over all densities) justify the K→∞ and σ→0 limits.
    Appendix F states these assumptions for the convergence claims.
invented entities (2)
  • Generative Expressivity E no independent evidence
    purpose: A training-objective capacity—the largest number of modes an objective can retain—used to argue that exploration is a third scaling axis.
    E is defined from the training objective and mode count, not directly measured; it is inferred from the paper's own FID/FVD/perplexity proxies. There is no external falsifiable handle beyond those proxies.
  • Per-prediction residue / shortfall max(M_t(c_t) - e_t, 0) no independent evidence
    purpose: Explains why factored models are still mode-limited and why exploration helps even when the modeled distribution is weakly multimodal.
    Introduced in Section F.1 as an explanatory ledger; not estimated from data.

pith-pipeline@v1.3.0-daily-deepseek · 34200 in / 19357 out tokens · 210197 ms · 2026-08-01T08:34:54.269031+00:00 · methodology

0 comments
read the original abstract

The deep learning revolution, kicked off by AlexNet, taught us that end-to-end training beats decomposing a problem into hand-designed stages. Generative modeling, however, has remained the exception-despite generative models being remarkably capable, they are still not trained end-to-end. This is because, at its core, generative modeling is about handling distributions with many modes, and existing scalable approaches handle this the same way, by factoring the generation procedure, which prevents end-to-end generation. In this work, we introduce Explorative Modeling, a new paradigm that instead factors the training loop, exploring K candidate matches between model generations and data, and training on the best, so predictions commit to modes rather than blurring them. We find Explorative Models (XMs) useful in two settings. First, increasing exploration adds a third pretraining axis beyond parameters and data for existing generative models-where scaling exploration monotonically improves performance across both continuous and discrete domains (images, video, and language). Notably, gains from exploration increase with scale, climbing from 7% to 36% as data scales and from 13% to 23% as models grow, with efficiency gains more than doubling at 3x the compute. Concretely, exploration improves FLOP efficiency by 4.1x, sample efficiency by 6.2x, parameter efficiency by 47%, lifts the strongest of image-generation recipes to a near-state-of-the-art 1.43 FID on ImageNet without guidance, enables scaling how end-to-end existing models are, and unlocks scaling generalization. Second, XMs enable end-to-end reconstructive generative modeling, matching diffusion on control tasks with 16-256x fewer inference steps. Together, these results establish XMs as both a new pretraining axis for existing generative models and a standalone end-to-end generative modeling paradigm.

Figures

Figures reproduced from arXiv: 2607.27372 by Alexi Gladstone, Heng Ji, Yilun Du.

Figure 1
Figure 1. Figure 1: Factorization Axes of Generative Modeling. A generative model can factor either generation (x-axis) or training (y-axis). Factoring generation breaks sampling into many steps during training, making a model not end-to-end (right column); factoring training involves exploration, which trains on the modes a model captures best (top row). With neither, direct regression blurs distinct modes into their average… view at source ↗
Figure 2
Figure 2. Figure 2: Increasing Exploration Scales Generative Expressivity and Reduces Blurring. Each row shows trained model generations varying only the amount of exploration (XM-K, denoting XMs with K modes explored): 2D mixture generation (top), image generation (middle), and masked diffusion language modeling (MDLM) [27] (bottom). With direct end-to-end regression (XM-1), models can only predict the mean of all samples—a … view at source ↗
Figure 3
Figure 3. Figure 3: Explorative Modeling Visualized. Explorative Modeling explores possible matches between what the model generates and the data, and trains on the best match. This increases models’ generative expressivity, capturing multiple modes as opposed to predicting their mean (as in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Exploration Improves Data and FLOP Efficiency at Scale. We add exploration to RAE [45], the state-of-the-art image generation recipe as of three months before this work (we report FDr6 , as FID at this performance level is saturated [43, 44]). Exploration reaches the baseline’s best performance with 6.2× less data (Figure 4a) and 4.1× fewer FLOPs (Figure 4b)—more than doubling the gains of similar experime… view at source ↗
Figure 5
Figure 5. Figure 5: Exploration Improves Sample and FLOP Efficiency. We add exploration to an optimally tuned SiT baseline [29], training at roughly a third of the compute of [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Exploration Improves Parameter Ef￾ficiency. A Large model with 5 modes explored scales better than an XLarge model with 47% more parameters and no exploration, demonstrating how exploration can improve parameter efficiency. We begin by adding exploration to a strong image generation recipe (RAE [45]), training models that differ only in the amount of exploration. We find that exploration significantly impr… view at source ↗
Figure 7
Figure 7. Figure 7: Increasing Exploration Monotonically Improves Performance. As the number of modes explored increases, both FID (left) and FVD (right) improve monotonically for Explorative Diffusion (XDiffusion) and Explorative Jumpy (XJumpy) models. In both cases, XJumpy benefits more from exploration than XDiffusion, a gap we examine in more detail below. (a) 8 Step Language Generation. (b) 256 Step Language Generation … view at source ↗
Figure 8
Figure 8. Figure 8: Exploration Improves Masked Diffusion Language Modeling Performance. Switching from a baseline Masked Diffusion Language Model (MDLM) [27, 48] to an Explorative MDLM (XMDLM) by exploring 8 modes significantly improves performance, achieving a better Perplexity-Entropy frontier for all points. This demonstrates exploration can improve generative models in both discrete and continuous spaces. Does Performanc… view at source ↗
Figure 9
Figure 9. Figure 9: Exploration Improves Generalization. As the number of modes explored increases, 4-step XJumpy models achieve a better absolute minimum FVD due to overfitting less. The right panel shows the same runs as the left, adding the most-explored model (XM-12) to show this trend continues to the maximum exploration we test. Models overfit in this setting due to training on the relatively small Something-Something V… view at source ↗
Figure 10
Figure 10. Figure 10: Performance Gains from Exploration Increase as Scale Increases. We measure the performance gain from exploring 5 modes (XM-5) over no exploration as we scale model size (left) and training data (right). In both cases the gains from doing exploration grow with scale, rising from 13% to 23% with model size and from 7% to 36% with data. out as performing worse than XDiffusion models with no exploration, but … view at source ↗
Figure 11
Figure 11. Figure 11: More End-to-End Models Scale Better with Ex￾ploration. We compare Explorative Jumpy (XJumpy) models with a different number of jumps across an increasing num￾ber of modes explored. If the amount of exploration is low, XJumpy models with more jumps perform best. However, as exploration increases, the optimal number of jumps decreases, demonstrating how models that are more end-to-end (fewer jumps) scale be… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

106 extracted references · 30 linked inside Pith

  1. [1]

    Imagenet classification with deep convolutional neural networks.Advances in neural information processing systems, 25, 2012

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks.Advances in neural information processing systems, 25, 2012. 2, 5

    2012

  2. [2]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 2

    2016

  3. [3]

    End-to-end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. InEuropean Conference on Computer Vision (ECCV), 2020. 2

    2020

  4. [4]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. InProceedings of the IEEE/CVF international conference on computer vision, pages 4015–4026, 2023. 2

    2023

  5. [5]

    Do imagenet classifiers generalize to imagenet? InInternational Conference on Machine Learning (ICML), 2019

    Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? InInternational Conference on Machine Learning (ICML), 2019. 2, 5

    2019

  6. [6]

    Benchmarking neural network robustness to common corruptions and perturbations

    Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. InInternational Conference on Learning Representations (ICLR), 2019

    2019

  7. [7]

    Earnshaw, Imran S

    Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS...

    2021

  8. [8]

    Sequence level training with recurrent neural networks

    Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. InInternational Conference on Learning Representations (ICLR), 2016. 2, 5, 6

    2016

  9. [9]

    Scheduled sampling for sequence prediction with recurrent neural networks

    Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. InAdvances in Neural Information Processing Systems (NeurIPS), 2015. 2

    2015

  10. [10]

    Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A. Smith. How language model hallucina- tions can snowball, 2023

    2023

  11. [11]

    Elucidating the exposure bias in diffusion models.arXiv preprint arXiv:2308.15321, 2023

    Mang Ning, Mingxiao Li, Jianlin Su, Albert Ali Salah, and Itir Onal Ertugrul. Elucidating the exposure bias in diffusion models.arXiv preprint arXiv:2308.15321, 2023

    Pith/arXiv arXiv 2023

  12. [12]

    Alleviating exposure bias in diffusion models through sampling with shifted time steps.arXiv preprint arXiv:2305.15583, 2023

    Mingxiao Li, Tingyu Qu, Ruicong Yao, Wei Sun, and Marie-Francine Moens. Alleviating exposure bias in diffusion models through sampling with shifted time steps.arXiv preprint arXiv:2305.15583, 2023. 2, 5, 6

    Pith/arXiv arXiv 2023

  13. [13]

    Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020. 2, 4, 6, 9

    2020

  14. [14]

    Consistency models, 2023

    Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models, 2023. 2, 6, 30

    2023

  15. [15]

    Mean flows for one-step generative modeling.Advances in Neural Information Processing Systems, 38:75460–75482, 2026

    Zhengyang Geng, Mingyang Deng, Xingjian Bai, Zico Kolter, and Kaiming He. Mean flows for one-step generative modeling.Advances in Neural Information Processing Systems, 38:75460–75482, 2026. 2, 6, 16, 30

    2026

  16. [16]

    Training compute-optimal large language models.arXiv preprint arXiv:2203.15556, 2022

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models.arXiv preprint arXiv:2203.15556, 2022. 2, 10, 17, 35, 36

    Pith/arXiv arXiv 2022

  17. [17]

    Diffusion policy: Visuomotor policy learning via action diffusion.The International Journal of Robotics Research, page 02783649241273668, 2023

    Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion.The International Journal of Robotics Research, page 02783649241273668, 2023. 3, 15, 29

    2023

  18. [18]

    Planning with diffusion for flexible behavior synthesis.arXiv preprint arXiv:2205.09991, 2022

    Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis.arXiv preprint arXiv:2205.09991, 2022. 3, 15, 29

    Pith/arXiv arXiv 2022

  19. [19]

    Mode forcing: A unifying and predictive theory of genera- tive modeling

    Alexi Gladstone, Yilun Du, and Heng Ji. Mode forcing: A unifying and predictive theory of genera- tive modeling. Draft manuscript, 2026. URL https://alexiglad.github.io/assets/pdf/mode_ forcing_draft.pdf. 3, 4, 15, 28, 31, 32 20

    2026

  20. [20]

    Generative adversarial nets.Advances in neural information processing systems, 27, 2014

    Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets.Advances in neural information processing systems, 27, 2014. 3, 6, 30

    2014

  21. [21]

    Training products of experts by minimizing contrastive divergence.Neural computa- tion, 14(8):1771–1800, 2002

    Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence.Neural computa- tion, 14(8):1771–1800, 2002. 3, 6, 30

    2002

  22. [22]

    Implicit generation and modeling with energy based models.Advances in neural information processing systems, 32, 2019

    Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models.Advances in neural information processing systems, 32, 2019. 3, 6, 30

    2019

  23. [23]

    Flow matching for generative modeling.arXiv preprint arXiv:2210.02747, 2022

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling.arXiv preprint arXiv:2210.02747, 2022. 4, 6, 9

    Pith/arXiv arXiv 2022

  24. [24]

    A note on the evaluation of generative models

    Lucas Theis, Aäron van den Oord, and Matthias Bethge. A note on the evaluation of generative models. InInternational Conference on Learning Representations, 2016. 4

    2016

  25. [25]

    Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598,

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598,

    Pith/arXiv arXiv

  26. [26]

    Guiding a diffusion model with a bad version of itself

    Tero Karras, Miika Aittala, Tuomas Kynkäänniemi, Jaakko Lehtinen, Timo Aila, and Samuli Laine. Guiding a diffusion model with a bad version of itself. InAdvances in Neural Information Processing Systems (NeurIPS), 2024. 5, 29

    2024

  27. [27]

    Simple and effective masked diffusion language models

    Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, and V olodymyr Kuleshov. Simple and effective masked diffusion language models. In Advances in Neural Information Processing Systems (NeurIPS), 2024. arXiv:2406.07524. 5, 11, 29

    Pith/arXiv arXiv 2024

  28. [28]

    Murphy, and Tim Salimans

    Ruiqi Gao, Emiel Hoogeboom, Jonathan Heek, Valentin De Bortoli, Kevin P. Murphy, and Tim Salimans. Diffusion meets flow matching: Two sides of the same coin. 2024. URL https://diffusionflow. github.io/. 6

    2024

  29. [29]

    Albergo, Nicholas M

    Nanye Ma, Mark Goldstein, Michael S. Albergo, Nicholas M. Boffi, Eric Vanden-Eijnden, and Saining Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers,

  30. [30]

    Jumpy generative models: Unleashing the hidden spectrum in generative modeling

    Alexi Gladstone, Yilun Du, and Heng Ji. Jumpy generative models: Unleashing the hidden spectrum in generative modeling. Manuscript in preparation, 2026. 6, 9, 10, 29

    2026

  31. [31]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023. 6

    2023

  32. [32]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 6, 12

    2022

  33. [33]

    Stochastic multiple choice learning for training diverse deep ensembles

    Stefan Lee, Senthil Purushwalkam, Michael Cogswell, Viresh Ranjan, David Crandall, and Dhruv Batra. Stochastic multiple choice learning for training diverse deep ensembles. InAdvances in Neural Information Processing Systems (NeurIPS), 2016. 6, 30, 35

    2016

  34. [34]

    Implicit maximum likelihood estimation.arXiv preprint arXiv:1809.09087,

    Ke Li and Jitendra Malik. Implicit maximum likelihood estimation.arXiv preprint arXiv:1809.09087,

    Pith/arXiv arXiv

  35. [35]

    Diverse imitation learning via self-organizing generative models.IEEE Transactions on Neural Networks and Learning Systems, 36(4):7145–7157, 2024

    Arash Vahabpour, Tianyi Wang, Qiujing Lu, Omead Pooladzandi, and Vwani Roychowdhury. Diverse imitation learning via self-organizing generative models.IEEE Transactions on Neural Networks and Learning Systems, 36(4):7145–7157, 2024. 6, 30, 31, 35

    2024

  36. [36]

    Auto-encoding variational bayes.arXiv preprint arXiv:1312.6114,

    Diederik P Kingma and Max Welling. Auto-encoding variational bayes.arXiv preprint arXiv:1312.6114,

    Pith/arXiv arXiv

  37. [37]

    NV AE: A deep hierarchical variational autoencoder

    Arash Vahdat and Jan Kautz. NV AE: A deep hierarchical variational autoencoder. InAdvances in Neural Information Processing Systems (NeurIPS), 2020. 7

    2020

  38. [38]

    Tackling the generative learning trilemma with denoising diffusion gans.arXiv preprint arXiv:2112.07804, 2021

    Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans.arXiv preprint arXiv:2112.07804, 2021. 7

    Pith/arXiv arXiv 2021

  39. [39]

    Improving and generalizing flow-based generative models with minibatch optimal transport.arXiv preprint arXiv:2302.00482, 2023

    Alexander Tong, Kilian Fatras, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector-Brooks, Guy Wolf, and Yoshua Bengio. Improving and generalizing flow-based generative models with minibatch optimal transport.arXiv preprint arXiv:2302.00482, 2023. 7, 26, 30 21

    Pith/arXiv arXiv 2023

  40. [40]

    Learning with minibatch Wasserstein: asymptotic and gradient properties

    Kilian Fatras, Younes Zine, Rémi Flamary, Rémi Gribonval, and Nicolas Courty. Learning with minibatch Wasserstein: asymptotic and gradient properties. InProceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics (AISTATS), volume 108 ofProceedings of Machine Learning Research, pages 2131–2141, 2020. 7, 26

    2020

  41. [41]

    The bitter lesson.Incomplete Ideas (blog), 13(1):38, 2019

    Richard Sutton. The bitter lesson.Incomplete Ideas (blog), 13(1):38, 2019. 7

    2019

  42. [42]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. InInternational conference on machine learning, pages 2256–2265. pmlr, 2015. 9

    2015

  43. [43]

    Representation Fréchet loss for visual generation.arXiv preprint arXiv:2604.28190, 2026

    Jiawei Yang, Zhengyang Geng, Xuan Ju, Yonglong Tian, and Yue Wang. Representation Fréchet loss for visual generation.arXiv preprint arXiv:2604.28190, 2026. 9, 13, 26, 29

    Pith/arXiv arXiv 2026

  44. [44]

    Improved baselines with representation autoencoders.arXiv preprint arXiv:2605.18324, 2026

    Jaskirat Singh, Boyang Zheng, Zongze Wu, Richard Zhang, Eli Shechtman, and Saining Xie. Improved baselines with representation autoencoders.arXiv preprint arXiv:2605.18324, 2026. 9, 13, 26, 29

    Pith/arXiv arXiv 2026

  45. [45]

    Diffusion transformers with representation autoencoders.arXiv preprint arXiv:2510.11690, 2025

    Boyang Zheng, Nanye Ma, Shengbang Tong, and Saining Xie. Diffusion transformers with representation autoencoders.arXiv preprint arXiv:2510.11690, 2025. 9, 10, 12, 13, 18, 29

    Pith/arXiv arXiv 2025

  46. [46]

    Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling

    Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming-Yu Liu, Jun Zhu, and Qinsheng Zhang. Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling. InInternational Conference on Learning Representations (ICLR), 2025. arXiv:2409.02908. 11

    Pith/arXiv arXiv 2025

  47. [47]

    Generative frontiers: Why evaluation matters for diffusion language models.arXiv preprint arXiv:2604.02718, 2026

    Patrick Pynadath, Jiaxin Shi, and Ruqi Zhang. Generative frontiers: Why evaluation matters for diffusion language models.arXiv preprint arXiv:2604.02718, 2026. 11

    Pith/arXiv arXiv 2026

  48. [48]

    Discrete diffusion modeling by estimating the ratios of the data distribution.arXiv preprint arXiv:2310.16834, 2023

    Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution.arXiv preprint arXiv:2310.16834, 2023. 11

    Pith/arXiv arXiv 2023

  49. [49]

    something something

    Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. InProceedings of the IEEE international conference on computer vision, pages 5842–585...

    2017

  50. [50]

    Diffusion beats autoregressive in data-constrained settings.Advances in Neural Information Processing Systems, 38:10581–10606, 2026

    Mihir Prabhudesai, Mengning Wu, Amir Zadeh, Katerina Fragkiadaki, and Deepak Pathak. Diffusion beats autoregressive in data-constrained settings.Advances in Neural Information Processing Systems, 38:10581–10606, 2026. 11, 12

    2026

  51. [51]

    Deep double descent: Where bigger models and more data hurt.Journal of Statistical Mechanics: Theory and Experiment, 2021(12):124003, 2021

    Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt.Journal of Statistical Mechanics: Theory and Experiment, 2021(12):124003, 2021. 12

    2021

  52. [52]

    Deep learning is not so mysterious or different.arXiv preprint arXiv:2503.02113,

    Andrew Gordon Wilson. Deep learning is not so mysterious or different.arXiv preprint arXiv:2503.02113,

    Pith/arXiv arXiv

  53. [53]

    Slowrun: Language modeling with infinite compute, fixed data, 2026

    Akshay Vegesna, Samip Dahal, Chinmay Karkar, Bishwas Mandal, Shmuel Berman, and Zhiwei Xu. Slowrun: Language modeling with infinite compute, fixed data, 2026. URL https://github.com/ qlabs-eng/slowrun. 12, 16

    2026

  54. [54]

    Pre-training under infinite compute

    Konwoo Kim, Suhas Kotha, Percy Liang, and Tatsunori Hashimoto. Pre-training under infinite compute. arXiv preprint arXiv:2509.14786, 2025. 12, 16

    arXiv 2025

  55. [55]

    sd-vae-ft-mse, 2023

    Stability AI. sd-vae-ft-mse, 2023. URL https://huggingface.co/stabilityai/sd-vae-ft-mse . Accessed: 2024-05-21. 12, 29

    2023

  56. [56]

    Scalable diffusion models with transformers, 2023

    William Peebles and Saining Xie. Scalable diffusion models with transformers, 2023. 13, 28

    2023

  57. [57]

    Fast training of diffusion models with masked transformers, 2023

    Hongkai Zheng, Weili Nie, Arash Vahdat, and Anima Anandkumar. Fast training of diffusion models with masked transformers, 2023. 13

    2023

  58. [58]

    Mdtv2: Masked diffusion transformer is a strong image synthesizer, 2023

    Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Mdtv2: Masked diffusion transformer is a strong image synthesizer, 2023. 13

    2023

  59. [59]

    Reconstruction vs

    Jingfeng Yao, Bin Yang, and Xinggang Wang. Reconstruction vs. generation: Taming optimization dilemma in latent diffusion models, 2025. 13

    2025

  60. [60]

    Representation alignment for generation: Training diffusion transformers is easier than you think,

    Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think,

  61. [61]

    Ddt: Decoupled diffusion transformer, 2025

    Shuai Wang, Zhi Tian, Weilin Huang, and Limin Wang. Ddt: Decoupled diffusion transformer, 2025. 13

    2025

  62. [62]

    Repa-e: Unlocking vae for end-to-end tuning with latent diffusion transformers, 2025

    Xingjian Leng, Jaskirat Singh, Yunzhong Hou, Zhenchang Xing, Saining Xie, and Liang Zheng. Repa-e: Unlocking vae for end-to-end tuning with latent diffusion transformers, 2025. 13

    2025

  63. [63]

    Dinov2: Learning robust visual features without supervision, 2023

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick La...

    2023

  64. [64]

    D4rl: Datasets for deep data-driven reinforcement learning.arXiv preprint arXiv:2004.07219, 2020

    Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning.arXiv preprint arXiv:2004.07219, 2020. 15

    Pith/arXiv arXiv 2004

  65. [65]

    Scaling up test-time compute with latent reasoning: A recurrent depth approach.Advances in Neural Information Processing Systems, 38:41340–41391, 2026

    Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach.Advances in Neural Information Processing Systems, 38:41340–41391, 2026. 15, 29

    2026

  66. [66]

    Visualizing the loss landscape of neural nets.Advances in neural information processing systems, 31, 2018

    Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets.Advances in neural information processing systems, 31, 2018. 16

    2018

  67. [67]

    Better & faster large language models via multi-token prediction, 2024

    Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve. Better & faster large language models via multi-token prediction, 2024. URL https://arxiv.org/abs/2404. 19737. 16

    2024

  68. [68]

    The free transformer, 2025

    François Fleuret. The free transformer, 2025. URLhttps://arxiv.org/abs/2510.17558. 16

    arXiv 2025

  69. [69]

    Energy-based transformers are scalable learners and thinkers.arXiv preprint arXiv:2507.02092, 2025

    Alexi Gladstone, Ganesh Nanduru, Md Mofijul Islam, Peixuan Han, Hyeonjeong Ha, Aman Chadha, Yilun Du, Heng Ji, Jundong Li, and Tariq Iqbal. Energy-based transformers are scalable learners and thinkers.arXiv preprint arXiv:2507.02092, 2025. 16, 30

    Pith/arXiv arXiv 2025

  70. [70]

    Self-supervised learning from images with a joint-embedding predictive architecture, 2023

    Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture, 2023. 16

    2023

  71. [71]

    A path towards autonomous machine intelligence version 0.9

    Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27.Open Review, 62, 2022. 17

    2022

  72. [72]

    Yujia Li, Kevin Swersky, and Richard S. Zemel. Generative moment matching networks. In Francis R. Bach and David M. Blei, editors,Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, JMLR Workshop and Conference Proceedings, pages 1718–

    2015

  73. [73]

    Learning iterative reasoning through energy minimization

    Yilun Du, Shuang Li, Joshua Tenenbaum, and Igor Mordatch. Learning iterative reasoning through energy minimization. InInternational Conference on Machine Learning, pages 5570–5582. PMLR, 2022. 17

    2022

  74. [74]

    Distribution matching in variational inference.arXiv preprint arXiv:1802.06847, 2018

    Mihaela Rosca, Balaji Lakshminarayanan, and Shakir Mohamed. Distribution matching in variational inference.arXiv preprint arXiv:1802.06847, 2018. 17

    Pith/arXiv arXiv 2018

  75. [75]

    Unified latents (ul): How to train your latents.arXiv preprint arXiv:2602.17270, 2026

    Jonathan Heek, Emiel Hoogeboom, Thomas Mensink, and Tim Salimans. Unified latents (ul): How to train your latents.arXiv preprint arXiv:2602.17270, 2026. 17

    arXiv 2026

  76. [76]

    KL-regularized reinforcement learning is designed to mode collapse.arXiv preprint arXiv:2510.20817, 2025

    Anthony GX-Chen, Jatin Prakash, Jeff Guo, Rob Fergus, and Rajesh Ranganath. KL-regularized reinforcement learning is designed to mode collapse.arXiv preprint arXiv:2510.20817, 2025. 17

    arXiv 2025

  77. [77]

    Pass@k training for adaptively balancing exploration and exploitation of large reasoning models.arXiv preprint arXiv:2508.10751, 2025

    Zhipeng Chen, Xiaobo Qin, Youbin Wu, Yue Ling, Qinghao Ye, Wayne Xin Zhao, and Guang Shi. Pass@k training for adaptively balancing exploration and exploitation of large reasoning models.arXiv preprint arXiv:2508.10751, 2025. 17

    Pith/arXiv arXiv 2025

  78. [78]

    Inference-aware fine-tuning for best-of-n sampling in large language models.arXiv preprint arXiv:2412.15287, 2024

    Yinlam Chow, Guy Tennenholtz, Izzeddin Gur, Vincent Zhuang, Bo Dai, Sridhar Thiagarajan, Craig Boutilier, Rishabh Agarwal, Aviral Kumar, and Aleksandra Faust. Inference-aware fine-tuning for best-of-n sampling in large language models.arXiv preprint arXiv:2412.15287, 2024. 17

    arXiv 2024

  79. [79]

    Diffusion art or digital forgery? investigating data replication in diffusion models.arXiv preprint arXiv:2212.03860,

    Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion art or digital forgery? investigating data replication in diffusion models.arXiv preprint arXiv:2212.03860,

    Pith/arXiv arXiv

  80. [80]

    Generalization in diffusion models arises from geometry-adaptive harmonic representations.arXiv preprint arXiv:2310.02557, 2024

    Zahra Kadkhodaie, Florentin Guth, Eero P Simoncelli, and Stéphane Mallat. Generalization in diffusion models arises from geometry-adaptive harmonic representations.arXiv preprint arXiv:2310.02557, 2024

    Pith/arXiv arXiv 2024

Showing first 80 references.