Pith. sign in

REVIEW 4 major objections 6 minor 51 references

Diffusion transformers improve faster and cheaper when they import LLM-style expert efficiency, not just bigger sparse pools.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 08:25 UTC pith:P73RXVIS

load-bearing objection Solid controlled ablation of LLM MoE pieces inside SiT under a real single-node budget; the per-step FID path is real, but the quality-cost slogan overreaches because cost is not matched. the 4 major comments →

arxiv 2607.24665 v1 pith:P73RXVIS submitted 2026-07-27 cs.CV cs.GRcs.LG

MMOE: Modernizing Diffusion Transformers with Efficient Expert Design

Yanhao Jia , Jiepeng Wang , Haibin Huang , Chi Zhang , Erik Cambria , Xuelong Li This is my paper
classification cs.CV cs.GRcs.LG
keywords diffusion transformersmixture of expertsAIGC foundation modelsefficient generative modelsexpert routingattention residualsSiTImageNet generation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that AIGC foundation models, especially diffusion transformers, have been scaling MoE the wrong way: by enlarging total parameters and sparsity ratios without the efficiency mechanisms that made LLM MoE practical. It modernizes a SiT-style backbone into MMOE by composing routed experts, shared and lightweight (copy/zero/constant) experts, gate-residual routing, and attention-residual reuse inside each block, while leaving the generative objective and sampler unchanged. Every run fits a single eight-GPU H100 node at batch 256 for 400k steps. Under that matched budget, MMOE reaches lower FID at every checkpoint than dense SiT and intermediate sparse variants, and among the sparse designs it gives the best quality–cost balance. Routing stays specialized by depth, uses cheap routes heavily early on, and changes only modestly step to step during denoising. The sympathetic takeaway is that AFMs can follow the balanced LLM scaling path by importing proven efficiency designs rather than simply growing the expert pool.

Core claim

Under matched training and sampling at an accessible single-node budget, composing modern LLM MoE efficiency components inside a SiT-style diffusion transformer yields faster FID convergence and a better quality–cost tradeoff than dense MLPs or sparse routing alone. MMOE is not claimed as a new state-of-the-art against much longer, larger-batch prior systems; the claim is that the modernization path itself improves performance and cost balance within that controlled budget.

What carries the argument

The MMOE block: a SiT block whose MLP is a MoE++ layer (heavy MLPs plus copy, zero, and constant experts with gate-residual top-2 routing) and whose attention and expert sub-layers optionally aggregate prior completed block states via attention residuals. It selectively activates expensive MLPs, offers cheap routes when a full update is unnecessary, and reuses cross-depth information without changing the denoising interface.

Load-bearing premise

That a controlled internal path from dense SiT through intermediate sparse designs at a fixed small budget is enough to show AFMs should scale like efficient LLMs rather than by growing total experts and sparsity—while matched-budget contests against published large diffusion-MoE systems are still unfinished.

What would settle it

Train dense SiT, the intermediate sparse variants, and MMOE to the same larger matched budget and protocol as a strong published diffusion-MoE baseline (same batch, steps, sampler, and sample count): if MMOE no longer leads on FID-per-cost or FID-per-step, the modernization claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • AFM scaling recipes should report activated compute, memory, and wall-clock cost alongside total parameters and FID.
  • Lightweight zero-computation experts and gate residuals can cut activation memory and training time relative to attention-residual MoE alone while keeping most of the quality gain.
  • Stable block-wise expert specialization and low step-to-step routing churn are usable design targets for sparse denoisers, not only language models.
  • The same modernization stack can be tried at other scales and resolutions without changing the diffusion objective or sampler.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If communication, not expert FLOPs, dominates sparse AFM training—as the paper’s profiling hints—then systems work on dispatch and all-reduce may unlock more of the architectural savings than further expert-pool growth.
  • Depth-dependent lightweight routing (more cheap routes early, more heavy MLPs late) suggests curriculum-like compute allocation across denoising depth that text-to-image and video backbones could exploit under multimodal conditioning.
  • A fair next stress test is whether the same component order still wins when the backbone is conditioned on text rather than class labels at matched single-node budgets.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces MMOE, a modernization of SiT-style diffusion transformers that imports four efficiency mechanisms from LLM MoE research — routed experts, shared experts, MoE++-style lightweight (copy/zero/constant) experts with gate-residual routing, and attention-residual aggregation — into a diffusion backbone while keeping the objective, conditioning, VAE, and sampler fixed. A controlled modernization path (dense SiT → standard MoE → SMOE → MoE++ → AMOE → MMOE) is evaluated on class-conditional ImageNet-256 under a deliberately accessible budget (single 8×H100 node, batch 256, 400k steps). At matched steps, MMOE achieves the lowest FID at every recorded checkpoint (3.75 vs. 5.20 dense at 400k) and trains in 67h versus AMOE's 120h. Ablations, seed-robustness (3 seeds), an ImageNet-512 spot check, model-scale results (S/B/L), sampler/guidance sweeps, a memory profile, and a routing analysis (block-wise specialization, depth-dependent lightweight-route use, low step-to-step switch rates) support the component-wise story. The authors are candid that all comparisons are internal and that matched-budget comparison to published diffusion-MoE systems is deferred.

Significance. If the results hold, this is a useful ConvNeXt-style modernization study for diffusion transformers: it cleanly separates the contributions of several recent LLM-MoE components under a fixed training recipe, which prior diffusion-MoE work (typically a monolithic architecture at much larger budgets) does not. Strengths worth naming: the internal comparisons hold dataset, VAE, objective, sampler family, and budget fixed; the single-node 8×H100/batch-256/400k-step protocol is genuinely reproducible by most labs; FID, wall-clock, memory, and routing statistics are external measurements rather than quantities defined by the architecture; seed variance is measured for the main ranking (Table V); and the limitations section is unusually honest (communication-bound profiling, single-run FIDs, deferred external comparison). The paper does not claim SOTA FID, and Table II is appropriately labeled non-comparable. The per-step convergence claim is well supported; the broader "quality-cost balance" and "balanced scaling path" claims rest on a cost axis that is not yet normalized, which limits the strength of the headline conclusion as written.

major comments (4)
  1. [Table I, §IV.B] The abstract and §I claim MMOE 'attains the best quality-cost balance', but Table I holds training steps fixed (400k) while wall-clock cost varies 5x across variants (23h–120h). A quality-cost claim requires the cost axis to be normalized. Dense SiT-XL/2 is still improving steeply at 400k (9.80 → 6.50 → 5.20) and its per-step cost is ~3x lower than MMOE's; the missing control is dense SiT trained to matched wall-clock (~1.2M steps ≈ 69h vs. MMOE's 67h). If SiT at matched hours approaches 3.75, the claim reduces to 'sparse experts add capacity', which the paper explicitly disavows as the interesting finding. A FID-vs-wall-clock (or FID-vs-FLOP) curve for at least SiT, MoE++, and MMOE would settle this. Note also that MoE++ at 46h/4.64 FID is a point on the cost-quality frontier that MMOE does not dominate; the 'best balance' phrasing should be justified against intermediate operating poin
  2. [Table II, §IV.B] MMOE has 1.57B total parameters versus 676M for the dense SiT baseline (Table II), a 2.3x capacity increase, while the abstract and §VI argue AFMs should not scale 'by simply increasing total parameters and sparsity ratios'. Activated parameters (≤770M) are close to the dense 676M, which mitigates but does not resolve the confound, because the attention-residual aggregation and router parameters also add capacity, and total parameters affect memory and deployment cost — the very axes the paper says it is optimizing. The manuscript should (a) report activated FLOPs per training step and per denoising step for each variant, and (b) either add an iso-total-parameter dense control (e.g., a ~1.5B dense SiT, which is trainable within the stated budget) or explicitly scope the claim to iso-activated-compute comparisons.
  3. [§IV.C (profiling paragraph)] The paper's own profiling shows training is communication-bound: ncclAllReduce dominates and cudaStreamSynchronize is ~half of profiled time. This cuts both directions of the cost comparison and is currently under-analyzed: the sparse variants' wall-clock disadvantage partly reflects an unoptimized dispatch/synchronization implementation, while their quality advantage partly reflects capacity. Since wall-clock hours on an unoptimized implementation are the paper's primary cost metric, the quality-cost conclusions are implementation-sensitive. At minimum, report a hardware-independent cost measure (per-step FLOPs, activated FLOPs per token, tokens/sec) alongside hours, and state explicitly which conclusions survive a perfectly overlapped MoE implementation.
  4. [Tables II, IV, VIII; §IV.B–IV.C] The sampling-protocol results are internally inconsistent in a way that affects headline numbers. Table VIII (10k samples) shows ODE FID flat at 6.62–6.66 from 50 to 250 steps for MMOE-XL/2, yet Table II reports 3.60 for MMOE with 250-step ODE vs. 3.75 at 50 steps (Table I, 50k samples), and Table IV reports 5.81 → 5.47 for MoE++ when moving from 50 to 250 steps. The text attributes the flatness to the 10k-sample estimate and guidance confound and defers a controlled re-run; given that the 50k-sample, fixed-guidance re-run is cheap relative to training, it should be done rather than deferred, since the choice of 50-step vs. 250-step protocol changes the headline FID by ~0.15–0.35 and currently the protocol differs between Table I and Table II rows.
minor comments (6)
  1. [Table I] Table I: SMOE (45h) is faster than Standard MoE (54h) despite adding an always-active shared expert; a one-line explanation (e.g., fewer routed experts or better load balance reducing dispatch imbalance) would help readers trust the wall-clock row.
  2. [Table V, §IV.C] Table V measures seed robustness only for SiT, SMOE, and MMOE; MoE++ and AMOE are single-run. Since the MoE++ vs. SMOE gap at 400k is 0.02 FID (4.64 vs. 4.62), the 'progression is not strictly monotonic' caveat is appropriate, but the tie should be explicitly flagged as within noise rather than implied.
  3. [Table X, §V] Table X: load Gini ≈ 0.72 with normalized entropy ≈ 0.43 corresponds to ~2.5 effective experts out of 8 per block. The text frames this as 'strong block-wise specialization', but an equally consistent reading is partial expert under-utilization; please discuss whether raising the load-balancing coefficient (fixed at 0.01 throughout) trades quality for utilization, or at least report the aux-loss sensitivity.
  4. [Table III] Table III: the MMOE-over-SiT improvement grows dramatically as scale shrinks (S/2: 60.0→51.0; L/2: 22.3→14.7). This is a plausible and interesting pattern (sparse capacity helps under-parameterized models most) but is neither commented on nor given seed variance; a sentence of interpretation and a variance estimate would strengthen it.
  5. [§III.A, footnote 2] Footnote 2: the statement that dispatched features are pre-scaled by the top-k factor (each expert evaluated at 2h_i) is surprising and appears only in a footnote; please clarify whether this is intended, whether it interacts with the adaLN modulation, and whether baselines use the same convention.
  6. [Throughout] Typos/formatting: 'top-kexpert' (missing space, §I and §II.B), 'V AE' spacing throughout, Figure 2's x-axis covers only 200–400k steps while the text discusses the full trajectory (Figure 3 covers 0–400k; consider aligning). Reference [42] (Forge, gradient elimination) is cited alongside [8] as 'large sparse diffusion transformers' but appears to be an LLM-training systems paper; please check the citation context.

Circularity Check

0 steps flagged

No circularity: empirical architecture comparison with external FID/wall-clock metrics; no derivation that reduces to its inputs by construction.

full rationale

MMOE is an empirical systems/architecture paper. Its load-bearing claims are measured outcomes (FID on ImageNet under fixed VAE/sampler/protocol, wall-clock on one 8×H100 node, routing load/Gini/switch rates from forward hooks), not quantities derived from equations that encode those same quantities. The modernization path (dense SiT → Standard MoE → SMOE → MoE++ → AMOE → MMOE) is a controlled ablation of imported LLM MoE components; each step is justified by measured FID and cost, not by a self-definitional identity or a fitted parameter renamed as a prediction. Citations to MoE++, SiT, DiT, Attention Residuals, etc. supply prior designs and baselines; none is a uniqueness theorem by overlapping authors that forces the result. Concerns about matched-steps vs matched-wall-clock quality–cost framing are experimental-design/correctness issues, not circularity. No circular steps identified.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 3 invented entities

The central empirical claim rests on standard generative-modeling assumptions (latent diffusion/flow matching via SiT, FID as quality proxy, classifier-free guidance) plus several design and budget choices fixed by the authors. No new physical entities are postulated; ‘invented’ items are architectural modules assembled from prior work. Free parameters are ordinary deep-learning hyperparameters and MoE design knobs that affect the reported tradeoff.

free parameters (5)
  • load-balancing auxiliary loss coefficient = 0.01
    Set to 0.01 for MoE variants that expose the loss; directly affects expert utilization and thus FID/time.
  • top-k routing (k=2) and expert pool layout = k=2; 4 light + (E-4) MLP
    Default top-2 with E−4 MLP + 2 constant + 1 copy + 1 zero experts; chosen design knobs that define MMOE capacity and activated cost.
  • attention-residual block-group boundary = block_size/2
    Completed states appended every block_size/2; controls how much cross-depth reuse occurs.
  • training budget (steps, batch, LR, EMA) = 400k / 256 / 1e-4 / 0.9999
    400k steps, batch 256, AdamW 1e-4, EMA 0.9999 fixed for all main runs; the quality-cost claims are conditional on this budget.
  • classifier-free guidance and ODE step defaults = CFG 1.5; 50 ODE steps
    Main table uses CFG 1.5 and 50-step ODE; absolute FID is sensitive to these choices (Tables VIII–IX).
axioms (4)
  • domain assumption SiT latent interpolant training with velocity prediction and a frozen SD-style VAE is a valid fixed interface for comparing backbone block designs.
    Stated in §III and §IV.A; all variants keep objective, conditioning, and sampler family unchanged.
  • domain assumption FID on 50k ImageNet validations under fixed sampling is a sufficient primary quality metric for the architectural claim.
    Used throughout §IV; paper acknowledges sample-count sensitivity (Table VII) but still ranks models by FID.
  • ad hoc to paper Single-node 8×H100, batch-256, 400k-step budget is an appropriate regime in which to judge ‘balanced scaling’ for AFMs.
    Core framing in abstract and §I; contrasts with prior MoE works at batch 1024 and millions of iters without matched re-runs.
  • domain assumption Imported LLM MoE mechanisms (zero/copy/constant experts, gate residual, attention residual) transfer to iterative denoising without new generative theory.
    Method §III treats them as drop-in block changes; validity is empirical only inside the reported setting.
invented entities (3)
  • MMOE block (MoE++ experts + gate-residual + attention-residual aggregation on SiT) no independent evidence
    purpose: Unified sparse diffusion-transformer block that selectively activates heavy MLPs, cheap routes, and cross-depth state reuse.
    Composition is the paper’s architectural product; components are cited from MoE++ and Attention Residuals, not micro-invented here.
  • Gate residual passed across expert layers no independent evidence
    purpose: Make current routing logits depend on previous expert-layer routing tendency.
    Described in §III.A eqs. (1)–(2); borrowed from MoE++-style design and evaluated only inside this paper’s runs.
  • Grouped attention-residual state list C with partial state reset no independent evidence
    purpose: Reuse completed block-group representations before attention and before MoE without attending every layer.
    Algorithm 1 and §III.C; implementation choice specific to this modernization path.

pith-pipeline@v1.2.0-grok45-kimik3 · 23301 in / 3921 out tokens · 60867 ms · 2026-07-31T08:25:23.380010+00:00 · methodology

0 comments
read the original abstract

Modern large language models scale successfully by pairing capacity growth with efficiency, keeping per-token and deployment costs under control as capacity grows. AIGC Foundation Models (AFMs), especially diffusion-transformer backbones, have begun to adopt sparse experts, but recent efforts mostly enlarge total parameter counts and sparsity ratios without importing the efficiency mechanisms that made LLM scaling practical, so generation quality is seldom balanced against training and deployment cost. This raises a natural question: can the architectural principles behind efficient LLM scaling be adapted to AFMs in a more balanced way? We introduce ModernMOE (MMOE), a modernization of SiT-style diffusion transformers that systematically adapts routed experts, shared and lightweight experts, gate-residual routing, and attention-residual information reuse to AIGC generation. Rather than treating MoE as a single plug-in replacement, MMOE studies how different modern expert components affect convergence, efficiency, and generation quality when composed inside a diffusion transformer. Every experiment in this paper is trained on a single eight-GPU H100 node with batch size 256 for 400k steps, an accessible single-machine budget. Under matched training and sampling protocols and at this budget, MMOE reaches lower FID at every recorded checkpoint, that is, it converges faster per training step, than dense and intermediate sparse-expert baselines, and among the sparse variants it attains the best quality-cost balance. Routing analysis further shows stable expert specialization across depth, substantial use of lightweight routes, and modest step-to-step routing changes during denoising. These results suggest that AFMs can follow the balanced scaling path of LLMs by importing proven efficiency designs, rather than by simply increasing total parameters and sparsity ratios.

Figures

Figures reproduced from arXiv: 2607.24665 by Chi Zhang, Erik Cambria, Haibin Huang, Jiepeng Wang, Xuelong Li, Yanhao Jia.

Figure 1
Figure 1. Figure 1: Architecture of MMOE. Left: a vanilla MoE block that routes each token to a small pool of feed-forward experts. Right: the MMOE block, which [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: FID convergence on ImageNet-256 for XL/2 models along the [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Per-block activation memory (MB) for an XL/2 block: attention [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative comparison along the modernization path (top to bottom: Dense SiT, SMOE, MoE++, AMOE, MMOE) on five ImageNet classes. All [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

51 extracted references · 4 linked inside Pith

  1. [1]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,

    N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” inInternational Conference on Learning Representations (ICLR), 2017

    2017

  2. [2]

    GShard: Scaling giant models with condi- tional computation and automatic sharding,

    D. Lepikhin, H. Lee, Y . Xu, D. Chen, O. Firat, Y . Huang, M. Krikun, N. Shazeer, and Z. Chen, “GShard: Scaling giant models with condi- tional computation and automatic sharding,” inInternational Conference on Learning Representations (ICLR), 2021

    2021

  3. [3]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,

    W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,”Journal of Machine Learning Research, vol. 23, no. 120, pp. 1–39, 2022

    2022

  4. [4]

    Mixtral of experts,

    A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bam- ford, D. Singh Chaplot, D. de las Casaset al., “Mixtral of experts,” 2024

    2024

  5. [5]

    MoE++: Accelerating mixture- of-experts methods with zero-computation experts,

    P. Jin, B. Zhu, L. Yuan, and S. Yan, “MoE++: Accelerating mixture- of-experts methods with zero-computation experts,” inInternational Conference on Learning Representations (ICLR), 2025

    2025

  6. [6]

    Scalable diffusion models with transformers,

    W. Peebles and S. Xie, “Scalable diffusion models with transformers,” inProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 4172–4182

    2023

  7. [7]

    SiT: Exploring flow and diffusion-based generative models with scalable interpolant transformers,

    N. Ma, M. Goldstein, M. S. Albergo, N. M. Boffi, E. Vanden-Eijnden, and S. Xie, “SiT: Exploring flow and diffusion-based generative models with scalable interpolant transformers,” inEuropean Conference on Computer Vision (ECCV), 2024

    2024

  8. [8]

    Scaling diffusion transformers to 16 billion parameters,

    Z. Fei, M. Fan, C. Yu, D. Li, and J. Huang, “Scaling diffusion transformers to 16 billion parameters,” 2024

    2024

  9. [9]

    EC-DIT: Scaling diffusion transformers with adaptive expert- choice routing,

    H. Sun, T. Lei, B. Zhang, Y . Li, H. Huang, R. Pang, B. Dai, and N. Du, “EC-DIT: Scaling diffusion transformers with adaptive expert- choice routing,” inInternational Conference on Learning Representa- tions (ICLR), 2025

    2025

  10. [10]

    Diff-MoE: Diffusion transformer with time-aware and space- adaptive experts,

    K. Cheng, X. He, L. Yu, Z. Tu, M. Zhu, N. Wang, X. Gao, and J. Hu, “Diff-MoE: Diffusion transformer with time-aware and space- adaptive experts,” inProceedings of the 42nd International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 267, 2025, pp. 10 010–10 024

    2025

  11. [11]

    Expert race: A flexible routing strategy for scaling diffusion transformer with mixture of experts,

    Y . Yuan, Z. Wang, Z. Huang, D. Zhu, X. Zhou, J. Yu, and Q. Min, “Expert race: A flexible routing strategy for scaling diffusion transformer with mixture of experts,” 2025

    2025

  12. [12]

    Routing matters in MoE: Scaling diffusion transformers with explicit routing guidance,

    Y . Wei, S. Zhang, H. Yuan, Y . Han, Z. Chen, J. Wang, D. Zou, X. Liu, Y . Zhang, Y . Liu, and H. Shan, “Routing matters in MoE: Scaling diffusion transformers with explicit routing guidance,” 2025

    2025

  13. [13]

    Efficient training of diffusion mixture-of-experts models: A practical recipe,

    Y . Liu, Y . Yue, J. Zhang, C. Sun, Y . Zhou, W. Zeng, R. Tang, and G. Zhou, “Efficient training of diffusion mixture-of-experts models: A practical recipe,” 2025

    2025

  14. [14]

    A convnet for the 2020s,

    Z. Liu, H. Mao, C.-Y . Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A convnet for the 2020s,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 11 976– 11 986

    2022

  15. [15]

    Denoising diffusion probabilistic models,

    J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 33, 2020, pp. 6840–6851

    2020

  16. [16]

    Score-based generative modeling through stochastic differen- tial equations,

    Y . Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, “Score-based generative modeling through stochastic differen- tial equations,” inInternational Conference on Learning Representations (ICLR), 2021

    2021

  17. [17]

    Elucidating the design space of diffusion-based generative models,

    T. Karras, M. Aittala, T. Aila, and S. Laine, “Elucidating the design space of diffusion-based generative models,” inAdvances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  18. [18]

    High- resolution image synthesis with latent diffusion models,

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High- resolution image synthesis with latent diffusion models,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR), 2022, pp. 10 684–10 695

    2022

  19. [19]

    Flow matching for generative modeling,

    Y . Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le, “Flow matching for generative modeling,” inInternational Conference on Learning Representations (ICLR), 2023

    2023

  20. [20]

    Stochastic in- terpolants: A unifying framework for flows and diffusions,

    M. S. Albergo, N. M. Boffi, and E. Vanden-Eijnden, “Stochastic in- terpolants: A unifying framework for flows and diffusions,”Journal of Machine Learning Research, vol. 26, no. 209, pp. 1–80, 2025

    2025

  21. [21]

    All are worth words: A ViT backbone for diffusion models,

    F. Bao, S. Nie, K. Xue, Y . Cao, C. Li, H. Su, and J. Zhu, “All are worth words: A ViT backbone for diffusion models,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023

    2023

  22. [22]

    Representation alignment for generation: Training diffusion transform- ers is easier than you think,

    S. Yu, S. Kwak, H. Jang, J. Jeong, J. Huang, J. Shin, and S. Xie, “Representation alignment for generation: Training diffusion transform- ers is easier than you think,” inInternational Conference on Learning Representations (ICLR), 2025

    2025

  23. [23]

    Mmgen: Unified multi-modal image generation and understanding in one go,

    J. Wang, Z. Wang, H. Pan, Y . Liu, D. Yu, C. Wang, and W. Wang, “Mmgen: Unified multi-modal image generation and understanding in one go,”arXiv preprint arXiv:2503.20644, 2025

    Pith/arXiv arXiv 2025

  24. [24]

    Omnivdiff: Omni controllable video diffusion for generation and understanding,

    D. Xi, J. Wang, Y . Liang, X. Qiu, Y . Huo, R. Wang, C. Zhang, and X. Li, “Omnivdiff: Omni controllable video diffusion for generation and understanding,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 40, no. 13, 2026, pp. 10 915–10 923

    2026

  25. [25]

    Ctrlvdiff: Controllable video generation via unified multimodal video diffusion,

    D. Xi, J. Wang, Y . Liang, X. Qiu, J. Liu, H. Pan, Y . Huo, R. Wang, H. Huang, C. Zhanget al., “Ctrlvdiff: Controllable video generation via unified multimodal video diffusion,”arXiv preprint arXiv:2511.21129, 2025

    arXiv 2025

  26. [26]

    Edustory: A unified framework for pedagogically-consistent multi-shot stem instructional video generation,

    X. Wu, J. Teotia, S. Zhao, and E. Cambria, “Edustory: A unified framework for pedagogically-consistent multi-shot stem instructional video generation,”arXiv preprint arXiv:2605.09378, 2026

    Pith/arXiv arXiv 2026

  27. [27]

    Nero: Neural geometry and brdf reconstruction of reflective objects from multiview images,

    Y . Liu, P. Wang, C. Lin, X. Long, J. Wang, L. Liu, T. Komura, and W. Wang, “Nero: Neural geometry and brdf reconstruction of reflective objects from multiview images,”ACM Transactions on Graphics (ToG), vol. 42, no. 4, pp. 1–22, 2023

    2023

  28. [28]

    Uni-retrieval: A multi-style retrieval framework for stem’s education,

    Y . Jia, X. Wu, L. Hao, Z. Qinglin, Y . Hu, S. Zhao, and W. Fan, “Uni-retrieval: A multi-style retrieval framework for stem’s education,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025, pp. 10 182– 10 197

    2025

  29. [29]

    From query to explanation: Uni-rag for multi-modal retrieval-augmented learning in stem,

    X. Wu, Y . Jia, L. Xiao, S. Zhao, F. Chiang, and E. Cambria, “From query to explanation: Uni-rag for multi-modal retrieval-augmented learning in stem,”arXiv preprint arXiv:2507.03868, 2025

    Pith/arXiv arXiv 2025

  30. [30]

    Towards affective evaluation of stem education: Leveraging mllms in project- based learning,

    X. Wu, Y . Jia, Q. Zhang, Y . Qin, L. Xiao, and S. Zhao, “Towards affective evaluation of stem education: Leveraging mllms in project- based learning,”IEEE Transactions on Affective Computing, 2026

    2026

  31. [31]

    Seeing sound, hearing sight: Uncovering modality bias and conflict of ai mod- els in sound localization,

    Y . Jia, J. Xie, S. Jivaganesh, L. Hao, X. Wu, and M. Zhang, “Seeing sound, hearing sight: Uncovering modality bias and conflict of ai mod- els in sound localization,”Advances in neural information processing systems, vol. 38, pp. 148 468–148 499, 2026

    2026

  32. [32]

    Senticnet 9: Generative commonsense for emotion ai via conceptual primitive discovery and time shift mechanism,

    E. Cambria, R. Mao, X. Zhang, L. Xiao, T. Shen, and A. Anand, “Senticnet 9: Generative commonsense for emotion ai via conceptual primitive discovery and time shift mechanism,”IEEE Transactions on Computational Social Systems, 2026

    2026

  33. [33]

    Towards spatial reasoning and understanding via modeling modality conflict, bias and alignment,

    Y . Jia, X. Wu, J. Teotia, J. Dong, S. Zhao, P. Koniusz, and E. Cambria, “Towards spatial reasoning and understanding via modeling modality conflict, bias and alignment,” 2026

    2026

  34. [34]

    GLaM: Efficient scaling of language models with mixture-of- experts,

    N. Du, Y . Huang, A. M. Dai, S. Tong, D. Lepikhin, Y . Xu, M. Krikun, Y . Zhou, A. W. Yu, O. Firat, B. Zoph, L. Fedus, M. Bosma, Z. Zhou, T. Wang, Y . E. Wang, K. Webster, M. Pellat, K. Robinson, K. Meier- Hellstern, T. Duke, L. Dixon, K. Zhang, Q. V . Le, Y . Wu, Z. Chen, and C. Cui, “GLaM: Efficient scaling of language models with mixture-of- experts,” ...

    2022

  35. [35]

    ST-MoE: Designing stable and transferable sparse expert models,

    B. Zoph, I. Bello, S. Kumar, N. Du, Y . Huang, J. Dean, N. Shazeer, and W. Fedus, “ST-MoE: Designing stable and transferable sparse expert models,” 2022

    2022

  36. [36]

    DeepSeekMoE: Towards ultimate expert specialization in mixture-of-experts language models,

    D. Dai, C. Deng, C. Zhao, R. X. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y . Wu, Z. Xie, Y . K. Li, P. Huang, F. Luo, C. Ruan, Z. Sui, and W. Liang, “DeepSeekMoE: Towards ultimate expert specialization in mixture-of-experts language models,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024, pp. 1280–1297

    2024

  37. [37]

    Mixture-of-experts with expert choice routing,

    Y . Zhou, T. Lei, H. Liu, N. Du, Y . Huang, V . Zhao, A. M. Dai, Z. Chen, Q. V . Le, and J. Laudon, “Mixture-of-experts with expert choice routing,” inAdvances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  38. [38]

    DeepSeek-V3 technical report,

    DeepSeek-AI, “DeepSeek-V3 technical report,” 2024

    2024

  39. [39]

    Mixture-of-depths: Dynamically allocating compute in transformer-based language models,

    D. Raposo, S. Ritter, B. Richards, T. Lillicrap, P. C. Humphreys, and A. Santoro, “Mixture-of-depths: Dynamically allocating compute in transformer-based language models,” 2024

    2024

  40. [40]

    Scaling vision with sparse mixture of experts,

    C. Riquelme, J. Puigcerver, B. Mustafa, M. Neumann, R. Jenatton, A. Susano Pinto, D. Keysers, and N. Houlsby, “Scaling vision with sparse mixture of experts,” inAdvances in Neural Information Process- ing Systems (NeurIPS), 2021

    2021

  41. [41]

    Attention residuals,

    Kimi Team, G. Chen, Y . Zhang, J. Su, W. Xu, S. Panet al., “Attention residuals,” 2026

    2026

  42. [42]

    Forge: Fused on-register gradient elimination for memory-efficient llm training,

    D. Kukreja, K. Prasad, A. Anand, Z. Wang, E. Cambria, T. Liu, A. B. Ng, S. See, and B. Chatterjee, “Forge: Fused on-register gradient elimination for memory-efficient llm training,”arXiv preprint arXiv:2606.22932, 2026. IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. XX, NO. XX, MONTH YEAR 13

    Pith/arXiv arXiv 2026

  43. [43]

    DiffMoE: Dynamic token selection for scalable diffusion transformers,

    M. Shi, Z. Yuan, H. Yang, X. Wang, M. Zheng, X. Tao, W. Zhao, W. Zheng, J. Zhou, J. Lu, P. Wan, D. Zhang, and K. Gai, “DiffMoE: Dynamic token selection for scalable diffusion transformers,” 2025

    2025

  44. [44]

    Ai flow at the network edge,

    J. Shao and X. Li, “Ai flow at the network edge,”IEEE Network, 2025

    2025

  45. [45]

    Ai flow: Perspectives, scenarios, and approaches,

    H. An, W. Hu, S. Huang, S. Huang, R. Li, Y . Liang, J. Shao, Y . Song, Z. Wang, C. Yuanet al., “Ai flow: Perspectives, scenarios, and approaches,”Vicinagearth, vol. 3, no. 1, p. 1, 2026

    2026

  46. [46]

    Genera- tive transmission: Rethinking computation, bandwidth, and memory in communication,

    X. Chen, J. Luo, Y . Fan, H. Huang, C. Zhang, and X. Li, “Genera- tive transmission: Rethinking computation, bandwidth, and memory in communication,” 2026

    2026

  47. [47]

    ImageNet: A large-scale hierarchical image database,

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” inProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009, pp. 248–255

    2009

  48. [48]

    GANs trained by a two time-scale update rule converge to a local nash equilibrium,

    M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “GANs trained by a two time-scale update rule converge to a local nash equilibrium,” inAdvances in Neural Information Processing Systems (NeurIPS), 2017

    2017

  49. [49]

    Classifier-free diffusion guidance,

    J. Ho and T. Salimans, “Classifier-free diffusion guidance,” 2022

    2022

  50. [50]

    Dynamic diffusion transformer,

    W. Zhao, Y . Han, J. Tang, K. Wang, Y . Song, G. Huang, F. Wang, and Y . You, “Dynamic diffusion transformer,” inInternational Conference on Learning Representations, vol. 2025, 2025, pp. 65 520–65 552

    2025

  51. [51]

    Back to basics: Let denoising generative models denoise,

    T. Li and K. He, “Back to basics: Let denoising generative models denoise,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2026, pp. 36 115–36 125

    2026