Pith. sign in

REVIEW 3 major objections 7 minor 2 cited by

Decentralized Diffusion Models

T0 review · 3 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Decentralized Diffusion Models partitions a dataset among isolated expert diffusion models and shows their router-weighted ensemble optimizes the same flow-matching objective as a single monolithic model, beating it FLOP-for-FLOP.

desk verdict Solid empirical paper with a clean math core, but the headline gains come from top-1 selection, not the ensemble the theory justifies. read the letter →

arxiv 2501.05450 v2 pith:SC6VBVKS submitted 2025-01-09 cs.CV cs.DCcs.LG

David McAllister , Matthew Tancik , Jiaming Song , Angjoo Kanazawa This is my paper · ORCID
classification cs.CVcs.DCcs.LG
keywords decentralizedtrainingdiffusionmodelsflowmatchingmixtureofexpertsdataclusteringrouternetworkknowledgedistillationscalableimagegeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes training a diffusion model not as a single monolithic run but as K expert diffusion models, each trained in full isolation on its own partition of the dataset, and shows that at inference the experts can be recombined by a lightweight router into an ensemble that optimizes the same global flow-matching objective as one model trained on all the data. The central identity is Eq. (4): the marginal flow of the whole data distribution equals a sum over clusters of the cluster-conditional flow weighted by the posterior probability that a noisy sample came from that cluster. If this holds, large-scale training no longer requires a centralized high-bandwidth cluster; compute can be spread across independent clusters, 'compute islands', or on-demand GPU nodes with no gradient synchronization. Empirically the paper reports that eight-expert decentralized diffusion models beat standard monoliths FLOP-for-FLOP (FID 6.081 vs 8.494 on ImageNet at 800k steps), that top-1 expert selection works best at inference, and that a 24-billion-parameter ensemble can be trained on eight GPU nodes in under a week. A dense student distilled from the ensemble matches the monolith at one-third the training FLOPs.

What carries the argument

The load-bearing object is the router-weighted decomposition of the marginal flow. For clusters $S_1,\dots,S_K$, Eq. (4) writes $$u_t(x_t)=\sum_{k=1}^K \frac{p_{t,S_k}(x_t)}{p_t(x_t)} \frac{\sum_{x_0\in S_k} u_t(x_t|x_0)p_t(x_t|x_0)q(x_0)}{p_{t,S_k}(x_t)},$$ where the first factor is exactly the posterior $p(k|x_t,t)$ learned by a small DiT classifier and the second factor is the expert flow for that cluster. This identity turns one global regression problem into K independent regression problems plus one classification problem, and the appendix re-derives the same decomposition through score matching. Everything else in the paper — the router's cross-entropy training, top-1 sparse inference, and distillation into a dense student — is machinery for making that decomposition practical.

What would settle it

Train the same DDM and a monolith with equal total parameters and equal training FLOPs, and also record the router's agreement with the true cluster posterior at each timestep; if the equal-parameter monolith matches the DDM's FID, or if router errors at high noise remove the benefit, the claimed objective equivalence is not the source of the gains.

Watch

Extended reading notes

Core claim

The discovery is that the flow-matching objective partitions cleanly across data clusters. Writing $u_t(x_t)$ for the marginal flow at noise level $t$, the paper derives $$u_t(x_t) = \sum_{k=1}^K \frac{p_{t,S_k}(x_t)}{p_t(x_t)} \, \frac{\sum_{x_0\in S_k} u_t(x_t|x_0)p_t(x_t|x_0)q(x_0)}{p_{t,S_k}(x_t)},$$ so the whole-dataset flow is exactly a weighted average of per-cluster flows, with weights given by the cluster posterior $p(k|x_t,t)$. Consequently a collection of experts, each trained with ordinary flow matching on its own cluster, collectively optimizes the same objective as a monolith; a separately trained classifier router supplies the weights. The authors then show that with eight experts and top-1 routing, the decentralized model reaches FID 6.081 on ImageNet at 800k steps versus 8.494 for a monolith matched in total training FLOPs, that the same ranking holds on LAION Aesthetics, and that distillation into one dense model recovers monolith quality at a fraction of the compute.

Load-bearing premise

The equivalence between the ensemble and a monolithic model holds only if the learned router predicts the true cluster posterior at every timestep and each expert fits its own cluster's conditional flow, yet the headline results use top-1 expert selection rather than the full weighted sum that the derivation requires.

Editorial extensions

If this is right

  • Diffusion training can be sharded across independent compute islands with no gradient synchronization, because each expert optimizes the same objective it would optimize as part of the monolithic run.
  • Under equal total training FLOPs, an 8-expert DDM reaches FID 6.081 on ImageNet versus 8.494 for a monolith at 800k steps; on LAION Aesthetics it reaches FID 6.48 at 200k steps versus 6.52 for the monolith at 800k steps.
  • At inference, activating only the top expert (334 GFLOPs) beats both the full ensemble and the monolith on FID, so sparse expert selection is both cheaper and better.
  • The ensemble can be distilled into a single dense model that matches monolith FID (7.76 vs 7.82) at roughly one-third the training FLOPs.
  • The recipe scales: eight 3B-parameter MMDiT experts were trained with 16 GPUs per expert, equivalent to one on-demand GPU node per expert for about 6.5 days, and fine-tuned into a 24B-parameter system.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: because the theoretical equivalence covers the full ensemble, the reported top-1 gains are not directly explained by Eq. (4); they likely also reflect added capacity and specialization. A monolith matched in total parameters under equal FLOPs would test whether the advantage comes from the decomposition or from parameter count.
  • Inference: measuring the router's top-1 agreement with the true posterior $p(k|x_t,t)$ across timesteps would show whether the objective-equivalence story holds at high noise, where cluster posteriors are nearly uniform.
  • Inference: the method's privacy framing is implicit: experts can train locally on raw data, so only cluster-level information needs to be shared; pairing DDM with federated averaging could push decentralization further without sending raw data.
  • Inference: because flow matching covers rectified flows and score-based diffusion as special cases, the same decomposition should transfer to video, robotics policies, and audio generation whenever the data admits meaningful clusters; the paper only demonstrates images.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes Decentralized Diffusion Models (DDM), a framework that partitions the training data into K clusters, trains an independent flow-matching expert on each cluster, and trains a small router to combine expert predictions at inference. The central theoretical claim is that the global marginal flow decomposes exactly into a router-weighted sum of per-cluster expert flows (Eq. 4), so that the ensemble 'collectively optimizes the same objective' as a monolithic model. Experiments on ImageNet and LAION Aesthetics report FLOP-for-FLOP FID improvements over monolithic training, a distillation procedure that compresses the ensemble into a dense model, and a large-scale demonstration with eight 3B-parameter experts trained on commodity hardware.

Significance. If the central equivalence held for the actual trained components, the paper would offer a principled way to decentralize diffusion-model training across isolated compute islands, with genuine systems value. The derivation in Eq. (4) is exact and clean, and the training-FLOP accounting is careful: the monolith batch is divided equally among experts and the 4% router overhead is included. The scaling experiment to 24B parameters on eight compute nodes is a concrete practical strength. However, the main experiments use top-1 expert selection rather than the full router-weighted ensemble of Eq. (7), and Table 1 shows the full ensemble is worse than top-1. The paper therefore currently demonstrates an MoE-style sparse-specialization benefit, while the specific theoretical contribution—the DFM objective equivalence—is not directly validated by the headline results.

major comments (3)
  1. [§4.2, Table 1, Figs. 4 and 6] The main FLOP-for-FLOP comparisons and the scaling study use top-1 expert selection, not the full router-weighted ensemble of Eq. (7). The paper states that 'selecting only the top expert is the most efficient approach and does not sacrifice quality,' and Table 1 shows that for 8 experts the full ensemble achieves FID 10.52 while top-1 achieves FID 9.84. Consequently, the reported gains are not explained by the exact decomposition in Eq. (4), which applies to the full ensemble with an ideal router and ideal experts. To support the paper's central claim, please report full-ensemble FID as a function of training FLOPs for the same setups as Figures 4d and 4e, and either show that the full ensemble matches or beats the monolith at matched training FLOPs or explicitly separate the contribution of the exact ensemble from the contribution of top-1 sparsity. Without this, the conceptual contribution is not validated by the main experiments.
  2. [§3.3, Algorithm 1, Eq. (7)] The equivalence in Eq. (7) requires rθ(xt,t) = p(k|xt,t) and each vθ,k(xt,t) to equal the true cluster-conditional marginal flow. The router is trained with a simple cross-entropy loss over cluster labels, but the paper reports no empirical measure of router accuracy, nor any per-expert flow-matching error. The posterior p(k|xt,t) is a nontrivial function of the noise level, and it is not obvious that a cross-entropy-trained DiT approximates it well for all t, especially at high noise. Please add quantitative router diagnostics (for example, top-1 accuracy or calibration versus timestep, and a comparison with the Oracle row of Table 1 across noise levels) and state explicitly how the learned router and finite-capacity experts affect the equality in Eq. (4). This is needed to substantiate the phrase 'collectively optimize the same objective.'
  3. [§4.1, Appendix B] All reported FID numbers use classifier-free guidance (CFG scale 3 for ImageNet and 7.5 for LAION). The identity in Eq. (4) concerns the raw marginal flow; it does not automatically hold for CFG-adjusted expert predictions. The paper does not specify whether CFG is applied to each expert independently before routing, to the combined flow, or to the router inputs, nor whether the router sees conditional or unconditional embeddings. Since every headline comparison uses CFG, the link between the theoretical decomposition and the measured FID is incomplete. Please clarify the CFG procedure for DDMs and, if possible, include a no-guidance comparison to isolate the effect of the DFM objective.
minor comments (7)
  1. [§3.2] The phrase 'the ensemble collectively optimizes the same objective' should be qualified: each expert optimizes its own cluster-conditional flow-matching objective, and the global objective is recovered only at the exact optimum of all experts and the router. This is a fixed-point property, not an equivalence of training dynamics.
  2. [§4.1] The evaluation section states that a fixed set of 50,000 samples is drawn 'from each training dataset' for FID computation. Please clarify whether these are held-out samples or part of the training set, since FID reference samples that overlap with training data can bias the metric.
  3. [§4.4, Figure 5] The visual correlation test in Figure 5 is qualitative ('highly correlated, appearing less blurry'). Please provide a quantitative similarity metric between matched-noise samples from the monolith and the DDM to support the claim that the DFM objective is correct.
  4. [§4.7 and Abstract] The abstract says the model was trained with 'just eight individual GPU nodes,' while Section 4.7 states each expert uses 16 GPUs, for a total of 128 GPUs. Please reconcile these statements or clarify the meaning of a 'node' in each context.
  5. [Table 1] The threshold-based inference strategies report no GFLOPs value. Please specify the effective inference cost of these strategies so the efficiency comparison in Table 1 is complete.
  6. [Introduction] There is a typo in the Introduction: 'distributing the modeling burdening of diffusion' should be 'distributing the modeling burden of diffusion.'
  7. [Figure 4e] The x-axis of Figure 4e appears to have a duplicated tick label '2e+20' (the text reads '0e+00 5e+19 1e+20 2e+20 2e+20'). Please correct the axis tick labels.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the core equality is an exact Bayes-rule identity with independently trained router and experts.

full rationale

The derivation chain is not circular. Equation (4) is an exact algebraic identity: the global marginal flow equals the sum over clusters of the cluster posterior weight p_{t,S_k}(x_t)/p_t(x_t) times the cluster-conditional marginal flow. This is Bayes' rule applied to the flow-matching integral in Eq. (2); it contains no fitted parameters, no hidden constants, and no dependence on the experimental results. The router is trained with a supervised cross-entropy loss (Algorithm 1) to predict cluster labels of noisy latents, and each expert is trained with the standard flow-matching loss Eq. (6) on its own cluster; neither training target is constructed from the ensemble output. The paper's central claim—that a perfect router and converged experts would recover the monolithic marginal flow—is therefore a conditional mathematical statement rather than a definitional identity. The only concern is that the main FID comparisons use top-1 expert selection rather than the full ensemble of Eq. (7), and the exact ensemble underperforms top-1 in Table 1; this is an experimental-validity gap between theorem and benchmark, not circularity. There are no load-bearing self-citations: reference [18], cited as the prior derivation this work generalizes, has disjoint authors from the present paper. Score 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or theoretical entities; its load-bearing assumptions are the union-of-manifolds data hypothesis, the accuracy of a classification-trained router as a density-ratio estimator, and the beneficial effect of DINOv2 clustering. The key free hyperparameter is the number of experts K, selected by validation FID.

free parameters (2)
  • Number of experts K = 8
    Chosen after ablating 4, 8, and 16 experts (Figures 4a and 4b); 16 degrades FID, 8 is the reported sweet spot. All main results use K=8.
  • Fine-grained cluster count and consolidation to K coarse centroids = 1024 -> K
    The clustering pipeline from Ma et al. [29] has its own granularity choices that affect partition quality; the paper adopts them without ablating them.
assumptions (4)
  • standard math Flow matching loss minimizer equals the marginal flow of the training distribution (standard CFM result, Lipman et al. [27]).
    Invoked in Sections 3.1 to 3.4 to claim each expert learns its subset's marginal flow.
  • domain assumption Image data lies on a disjoint union of manifolds, so semantically clustered partitions are learnable sub-distributions.
    Invoked in Section 3.2 with references [5,25,49]; if false, expert specialization may not materialize.
  • ad hoc to paper A router trained with cross-entropy on cluster labels accurately estimates p(k|x_t,t) for all timesteps, including the noisy latents used in sampling.
    Assumed in Section 3.3 and Eq. (5); no error analysis is provided, and top-1 selection relies on the router's ranking being correct.
  • domain assumption DINOv2 feature clustering yields higher mutual information within clusters than random assignment, improving expert efficiency.
    Hypothesized in Section 4.5; Figure 4c supports it empirically on ImageNet but the mechanism is not proven.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Decentralized Diffusion Models." pith.science (2026). https://pith.science/paper/SC6VBVKS

@misc{pith2026250105450,
  author       = {Pith},
  title        = {Pith review of: Decentralized Diffusion Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SC6VBVKS}},
  note         = {Machine review of arXiv:2501.05450}
}
read the original abstract

Large-scale AI model training divides work across thousands of GPUs, then synchronizes gradients across them at each step. This incurs a significant network burden that only centralized, monolithic clusters can support, driving up infrastructure costs and straining power systems. We propose Decentralized Diffusion Models, a scalable framework for distributing diffusion model training across independent clusters or datacenters by eliminating the dependence on a centralized, high-bandwidth networking fabric. Our method trains a set of expert diffusion models over partitions of the dataset, each in full isolation from one another. At inference time, the experts ensemble through a lightweight router. We show that the ensemble collectively optimizes the same objective as a single model trained over the whole dataset. This means we can divide the training burden among a number of "compute islands," lowering infrastructure costs and improving resilience to localized GPU failures. Decentralized diffusion models empower researchers to take advantage of smaller, more cost-effective and more readily available compute like on-demand GPU nodes rather than central integrated systems. We conduct extensive experiments on ImageNet and LAION Aesthetics, showing that decentralized diffusion models FLOP-for-FLOP outperform standard diffusion models. We finally scale our approach to 24 billion parameters, demonstrating that high-quality diffusion models can now be trained with just eight individual GPU nodes in less than a week.

Figures

Figures reproduced from arXiv: 2501.05450 by the authors.

Figure 1
Figure 1. Decentralized Diffusion Models (DDM). Left: Existing diffusion models (monolithic) require synchronized, centralized training across thousands of GPUs, making high-quality training systems expensive and inaccessible. Right: DDM divides a diffu￾sion model into an ensemble of expert models, each trained on its own data cluster in complete isolation. This ensemble collectively optimizes the same diffusion objective as … view at source ↗
Figure 2
Figure 2. Decentralized diffusion models train on readily-available hardware and generate high quality, diverse images. We present selected samples from our 8x3B parameter model. 2 [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Decentralized Diffusion Model (DDM) Training Overview. DDMs follow a three-step training process. We first cluster the dataset using off-the-shelf representation extraction models. We train a diffusion model over each of these clusters and a router that associates any input xt with its most likely clusters. At test-time, given a noisy sample, each expert (in red and green) predict their own flows, which combine line… view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: Ablations at the DiT XL model scale. Eight-expert DDMs display the best consistent performance on ImageNet (a) and LAION Aesthetics (b). We show the importance of image-based clustering on ImageNet compared to random clustering (c). Finally, FLOP-for-FLOP, decentralize…
Figure 5
Figure 5. Figure 5: DDMs optimize the global diffusion objective. We average samples from the monolithic and DDM ImageNet models using a deterministic sampler with matching random seeds (left) and compare them to outputs generated with random noise samples (right). The left samples are hi…
Figure 6
Figure 6. Figure 6: Decentralized diffusion models scale gracefully to billions of parameters. Throughout training, we plot the FID over LAION Aesthetics as a function of training compute. We find that increasing expert model capacity and training compute predictably improves performance.…
Figure 7
Figure 7. Figure 7 [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Additional Quantitative Analysis. We sweep CFG scales across decentralized and monolith diffusion models trained on LAION Aesthetics (a, b, c, d), finding that optimal CFG scales are consistent across models. Distillation matches the performance of training a monolith …
Figure 9
Figure 9. Figure 9: Additional Selected Samples. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Random Samples, Fixed Prompt. a photo of the dolomites 16 [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: Random Samples, Fixed Prompt. 1969 Polaris Colt, restored to showroom, static display in snow, winter sunrise 17 [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: Random Samples, Fixed Prompt. weather research station in extreme conditions, monitoring equipment, natural elements 18 [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Random Samples, Fixed Prompt. ancient bristlecone pine forest, twisted trees, high-altitude light, rugged mountain backdrop 19 [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: Random Samples, Fixed Prompt. deep desert slot canyon, sandstone textures, shaft of light, natural color gradients 20 [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]
Figure 15
Figure 15. Figure 15: Random Samples, Fixed Prompt. gothic cathedral spires piercing morning mist, ancient European city roofscape 21 [PITH_FULL_IMAGE:figures/full_fig_p021_15.png]
Figure 16
Figure 16. Figure 16: Random Samples, Fixed Prompt. historic textile mill interior, preserved machinery, sunbeams through industrial windows 22 [PITH_FULL_IMAGE:figures/full_fig_p022_16.png]
Figure 17
Figure 17. Figure 17: Random Samples, Fixed Prompt. symphony orchestra during rehearsal, conductor’s perspective, historic concert hall 23 [PITH_FULL_IMAGE:figures/full_fig_p023_17.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Heterogeneous Decentralized Diffusion Models

    cs.LG 2026-03 conditional novelty 6.0 of 10

    Isolated experts trained with mixed DDPM and Flow Matching objectives can be fused at inference via deterministic conversion, cutting reported DDM compute ~16× while slightly improving FID and diversity over a homogen...

  2. Decentralized Autoregressive Generation

    cs.LG 2026-01 reject novelty 4.0 of 10

    A theoretical claim that decentralized autoregressive training equals centralized training is undercut by a derivation that reduces to a conditional-mixture identity and by experiments lacking error bars.

Reference graph

Works this paper leans on

56 extracted references · 25 canonical work pages · cited by 2 Pith papers

  1. [1]

    Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning

    Wei An, Xiao Bi, Guanting Chen, Shanhuang Chen, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Wenjun Gao, Kang Guan, et al. Fire-flyer ai-hpc: A cost-effective software-hardware co-design for deep learning.arXiv preprint arXiv:2408.14158, 2024. 1

    work page Pith review arXiv 2024

  2. [2]

    Diffusion soup: Model merging for text-to-image diffusion models

    Benjamin Biggs, Arjun Seshadri, Yang Zou, Achin Jain, Aditya Golatkar, Yusheng Xie, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. Diffusion soup: Model merging for text-to-image diffusion models. arXiv preprint arXiv:2406.08431, 2024. 4

    arXiv 2024

  3. [3]

    On the opportunities and risks of foundation models

    Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Alt- man, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021. 3

    arXiv 2021

  4. [4]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, et al. Video generation models as world simulators

    work page

  5. [5]

    Verifying the union of manifolds hypothesis for image data

    Bradley CA Brown, Anthony L Caterini, Brendan Leigh Ross, Jesse C Cresswell, and Gabriel Loaiza-Ganem. Verifying the union of manifolds hypothesis for image data. arXiv preprint arXiv:2207.02862, 2022. 5

    arXiv 2022

  6. [6]

    Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and Sand- hini Agarwal et al

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and Sand- hini Agarwal et al. Language models are few-shot learners,

    work page

  7. [7]

    Pixart- α: Fast training of diffusion transformer for photorealistic text-to-image synthe- sis, 2023

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart- α: Fast training of diffusion transformer for photorealistic text-to-image synthe- sis, 2023. 1, 3, 6

    work page 2023

  8. [8]

    Neural ordinary differential equations

    Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018. 3

    work page 2018

Show all 56 references
  1. [9]

    Reproducible scal- ing laws for contrastive language-image learning

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuh- mann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scal- ing laws for contrastive language-image learning. In Proceed- ings of the IEEE/CVF Conference on Computer ...

    2023

  2. [10]

    Diffusion policy: Visuomotor policy learning via action diffu- sion

    Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffu- sion. The International Journal of Robotics Research, page 02783649241273668, 2023. 1

    2023

  3. [11]

    Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, 9 Hui Li, Hui Qu, J

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei L...

  4. [12]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. 5

    2018 arXiv

  5. [13]

    Diloco: Distributed low-communication training of language models

    Arthur Douillard, Qixuan Feng, Andrei A Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Marc’Aurelio Ranzato, Arthur Szlam, and Jiajun Shen. Diloco: Distributed low-communication training of language models. arXiv preprint arXiv:2311.08105, 2023. 4

    2023 arXiv

  6. [14]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  7. [15]

    Scaling rectified flow trans- formers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim En- tezari, Jonas M¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow trans- formers for high-resolution image synthesis. In Forty-first International Conference on Mac...

    2024

  8. [16]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022. 3, 5

    2022

  9. [17]

    Megablocks: Efficient sparse training with mixture- of-experts

    Trevor Gale, Deepak Narayanan, Cliff Young, and Matei Zaharia. Megablocks: Efficient sparse training with mixture- of-experts. Proceedings of Machine Learning and Systems, 5: 288–304, 2023. 3

    2023

  10. [18]

    Training data protection with composi- tional diffusion models, 2024

    Aditya Golatkar, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. Training data protection with composi- tional diffusion models, 2024. 4

    2024

  11. [19]

    Distilling the knowledge in a neural network

    Geoffrey Hinton. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. 6

    2015 arXiv

  12. [20]

    Classifier-free diffusion guidance, 2022

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance, 2022. 12

    2022

  13. [21]

    Denoising diffu- sion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 3

    2020

  14. [22]

    The curious case of neural text degeneration, 2020

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration, 2020. 7

    2020

  15. [23]

    Sim- ple diffusion: End-to-end diffusion for high resolution images,

    Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. Sim- ple diffusion: End-to-end diffusion for high resolution images,

  16. [24]

    Mixtral of experts

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Deven- dra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. 3

    2024 arXiv

  17. [25]

    A geometric view of data complexity: Efficient local intrinsic dimension estimation with diffusion models

    Hamidreza Kamkari, Brendan Leigh Ross, Rasa Hossein- zadeh, Jesse C Cresswell, and Gabriel Loaiza-Ganem. A geometric view of data complexity: Efficient local intrinsic dimension estimation with diffusion models. arXiv preprint arXiv:2406.03537, 2024. 5

    2024 arXiv

  18. [26]

    Branch- train-merge: Embarrassingly parallel training of expert lan- guage models

    Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A Smith, and Luke Zettlemoyer. Branch- train-merge: Embarrassingly parallel training of expert lan- guage models. arXiv preprint arXiv:2208.03306, 2022. 4

    2022 arXiv

  19. [27]

    Flow matching for generative modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022. 3, 4, 12

    2022 arXiv

  20. [28]

    Flow straight and fast: Learning to generate and transfer data with rectified flow

    Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003, 2022. 3

    2022 arXiv

  21. [29]

    Mode: Clip data experts via clustering

    Jiawei Ma, Po-Yao Huang, Saining Xie, Shang-Wen Li, Luke Zettlemoyer, Shih-Fu Chang, Wen-Tau Yih, and Hu Xu. Mode: Clip data experts via clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26354–26363, 2024. 6

    2024

  22. [30]

    Communication- efficient learning of deep networks from decentralized data

    Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication- efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273–1282. PMLR,

  23. [31]

    Data center owners turn to nuclear as potential electricity source - u.s

    Mark Morey. Data center owners turn to nuclear as potential electricity source - u.s. energy information administration (eia), 2024. 3

    2024

  24. [32]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth´ee Darcet, Th´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel 10 Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 6

    2023 arXiv

  25. [33]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 4195–4205,

  26. [34]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 1

    2023 arXiv

  27. [35]

    Movie gen: A cast of media foundation models

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma, Ching-Yao Chuang, et al. Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410.13720,

    arXiv

  28. [36]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  29. [37]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020. 9

    2020

  30. [38]

    Adaptive federated optimization.arXiv preprint arXiv:2003.00295, 2020

    Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Kone ˇcn`y, Sanjiv Kumar, and H Brendan McMahan. Adaptive federated optimization.arXiv preprint arXiv:2003.00295, 2020. 4

    2003 arXiv

  31. [39]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1

    2022

  32. [40]

    Pho- torealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Pho- torealistic text-to-image diffusion models with deep language understanding. Advances in neural informati...

    2022

  33. [41]

    Progressive distillation for fast sampling of diffusion models

    Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512, 2022. 6

    2022 arXiv

  34. [42]

    Laion-5b: An open large-scale dataset for training next gen- eration image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next gen- eration image-text models. Advances in Neural Infor...

    2022

  35. [43]

    Stretching each dollar: Diffu- sion training from scratch on a micro-budget

    Vikash Sehwag, Xianghao Kong, Jingtao Li, Michael Spranger, and Lingjuan Lyu. Stretching each dollar: Diffu- sion training from scratch on a micro-budget. arXiv preprint arXiv:2407.15811, 2024. 3

    2024 arXiv

  36. [44]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International confer- ence on machine learning, pages 2256–2265. PMLR, 2015. 3

    2015

  37. [45]

    Generative modeling by estimating gradients of the data distribution

    Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019

    2019

  38. [46]

    Score-based generative modeling through stochastic differential equations

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Ab- hishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020. 3, 4, 12

    2011 arXiv

  39. [47]

    Consistency models

    Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01469, 2023. 6

    2023 arXiv

  40. [48]

    Decentralized feder- ated averaging

    Tao Sun, Dongsheng Li, and Bao Wang. Decentralized feder- ated averaging. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4289–4301, 2022. 4

    2022

  41. [49]

    Diffusion models learn low-dimensional distributions via subspace clustering

    Peng Wang, Huijie Zhang, Zekai Zhang, Siyi Chen, Yi Ma, and Qing Qu. Diffusion models learn low-dimensional distributions via subspace clustering. arXiv preprint arXiv:2409.02426, 2024. 5

    2024 arXiv

  42. [50]

    Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing infer- ence time

    Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Re- becca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Ko- rnblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing ...

    2022

  43. [51]

    One-step diffusion with distribution matching distillation

    Tianwei Yin, Micha ¨el Gharbi, Richard Zhang, Eli Shecht- man, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 6613–6623...

    2024

  44. [52]

    Representa- tion alignment for generation: Training diffusion transformers is easier than you think

    Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representa- tion alignment for generation: Training diffusion transformers is easier than you think. arXiv preprint arXiv:2410.06940,

    arXiv

  45. [53]

    Pytorch fsdp: experiences on scaling fully sharded data parallel

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien- Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.11277,

    arXiv

  46. [54]

    Mixture-of-experts with expert choice routing.Advances in Neural Information Processing Systems , 35:7103–7114,

    Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. Mixture-of-experts with expert choice routing.Advances in Neural Information Processing Systems , 35:7103–7114,

  47. [2022]

    Score Matching Derivation We provide an alternative derivation of Decentralized Flow Matching based on score matching [ 46] rather than flow matching [27]

    3 11 A. Score Matching Derivation We provide an alternative derivation of Decentralized Flow Matching based on score matching [ 46] rather than flow matching [27]. We begin with the score, which is the gradient of the log likelihood, pt(xt). ∇xt log pt(xt) (9) By applying the ...

  48. [2024]

    com/research/video-generation- models-as-world-simulators, 3, 2024

    URL https://openai. com/research/video-generation- models-as-world-simulators, 3, 2024. 1

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.