Pith. sign in

REVIEW 4 major objections 4 minor 56 references

Latent Swap Joint Diffusion for 2D Long-Form Latent Generation

T0 review · 4 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that the averaging step used to stitch overlapping diffusion outputs destroys high-frequency detail in long-form audio, and that a binary frame-level latent swap preserves that detail while keeping distant views…

desk verdict Solid, well-motivated methods paper: the binary swap trick is simple and seems to work, but the key assumption about trajectory distance is never measured, and the evaluation has a few uncontrolled choices. read the letter →

arxiv 2502.05130 v3 pith:CW5K4MBK submitted 2025-02-07 cs.SD cs.AIcs.CVcs.MMeess.AS

Yusheng Dai , Chenxi Wang , Chang Li , Chen Wang , Jun Du , Kewei Li , Ruoyu Wang , Jiefeng Ma
show 2 more authors
Lei Sun Jianqing Gao
This is my paper · ORCID
classification cs.SDcs.AIcs.CVcs.MMeess.AS
keywords latentswapjointdiffusionlong-formaudiogenerationpanoramaspectrumaliasingtraining-freemodelcross-viewconsistency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Existing joint-diffusion methods that generate long outputs by denoising several overlapping subviews stitch the overlap regions by averaging their latents at every step. The paper argues that for mel-spectrogram latents this averaging acts as a low-pass filter: it progressively suppresses high-frequency components and produces blurred, aliased spectra and distorted audio. It proposes Swap Forward (SaFa), which replaces averaging in the overlap with a binary frame-level swap between the two neighboring subview trajectories, and adds a unidirectional Reference-Guided swap that borrows frames from a single reference trajectory during early denoising steps to keep distant regions consistent. The paper reports that this two-operator, feed-forward recipe outperforms existing joint-diffusion baselines on long-form audio and matches or exceeds them on panoramas while running 2 to 20 times faster, using both U-Net and DiT backbones. The payoff if true is that fixed-length diffusion models can be extended to arbitrarily long, coherent outputs without retraining or test-time optimization.

What carries the argument

The load-bearing object is the binary latent swap operator $W_{\mathrm{swap}} = \mathbf{1}_n \otimes v_m$, where $v_m$ alternates between 0 and 1 with period $w$, applied as a Hadamard selection between adjacent subview latents in the overlap (Eq. 9), together with its unidirectional counterpart $W_{\mathrm{refer}}$ for reference-guided consistency (Eq. 11). It does the work that averaging did in prior joint diffusion, but instead of blending the two denoising trajectories it selects whole frames from one or the other, preserving each trajectory’s independent high-frequency content. The paper also introduces Connectivity Inheritance: a channel-wise linear map relating a VAE latent to a downsampled original feature explains why spectrum latents, inheriting the sparse high-frequency structure of mel-spectrograms, are particularly damaged by averaging.

What would settle it

Compute the step-wise distance $d(\mathrm{Right}(\Phi(x_t^i \cup I_{i,i+1}(J_{t+1}), y_i)), \mathrm{Left}(\Phi(x_t^{i+1} \cup I_{i,i+1}(J_{t+1}), y_{i+1})))$ in Eq. (7) during DDIM sampling of a spectrally rich prompt. If for some prompt the median distance falls outside the assumed interval and SaFa’s output nevertheless shows no seam or artifact, the claimed mechanism is not doing the work; conversely, if the distance exceeds the upper bound and the output develops a narrow-band discontinuity in the overlap, the hard swap is the cause.

Watch

Extended reading notes

Core claim

At the center of the paper is the claim that the averaging operator used to merge overlapping subview latents is the cause of spectrum aliasing in long-form audio. Because a mel-spectrogram’s VAE latent inherits the high-frequency, low-connectivity structure of the spectrogram through a channel-wise linear map, step-wise averaging progressively attenuates high-frequency components in overlap regions. SaFa replaces this averaging with a binary swap operator, so the merged overlap is $I_{i,i+1}(J_t) = W_{\mathrm{swap}} \odot \mathrm{Right}(X_t^i) + (1-W_{\mathrm{swap}}) \odot \mathrm{Left}(X_t^{i+1})$, alternating frame-by-frame between the two subview trajectories. A second unidirectional operator $M_i(J_t) = W_{\mathrm{refer}} \odot \mathrm{Mid}(X_t^0) + (1-W_{\mathrm{refer}}) \odot \mathrm{Mid}(X_t^i)$ borrows from one reference trajectory in early steps to synchronize non-overlap regions. The authors support this with Fourier analysis of overlap latents and with quantitative gains in FD, FAD, KL, and CLAP for audio and FID, KID, and CLIP for panoramas.

Load-bearing premise

Swap works only if, at each denoising step, the two subviews’ denoised versions of the overlap region are neither too similar nor too different, so that swapping frames between them blends two coherent views instead of splicing two incompatible ones.

Editorial extensions

If this is right

  • A fixed-length text-to-audio diffusion model can be extended to 24–96 second audio by joint denoising with overlapping subviews, with stable quality metrics as the target length grows from 24s to 72s.
  • The same two swap operators transfer across modalities: they improve panorama generation with both U-Net and DiT backbones, at overlap rates as low as 0.2 and with 2 to 20 times faster inference than gradient-based joint diffusion.
  • Existing joint diffusion pipelines can replace their averaging operator with this binary swap without additional computation, since the operator is feed-forward and requires no training or backpropagation.
  • Reference-Guided Swap provides a single dial, the guidance rate, that trades cross-view similarity against local diversity, with 20–40% guidance reported as a good balance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One extension the authors leave implicit is that the binary swap acts like an adaptive filter whose pass-band is set by the difference between adjacent trajectories; this predicts the same operator should help other over-smoothing failure modes, such as long video or tiled high-resolution generation, wherever overlap averaging blurs fine structure.
  • Because the swap interval controls the spatial granularity of selection, the method suggests a general recipe: choose the swap unit to match the correlation length of the modality’s VAE tokens (frames for spectra, rows for images), which could be estimated from the latent’s autocorrelation before applying the method.
  • The paper’s Fourier diagnosis is testable without running full generation: measure the high-frequency energy of overlap versus non-overlap latents after averaging; audio latents should show a monotone gap that grows with denoising step, and if a dataset does not show that gap, SaFa’s advantage should shrink.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces Swap Forward (SaFa), a training-free joint diffusion framework for generating long-form 2D latents (mel-spectrograms and panoramic images). The core idea is to replace the averaging operator used in overlap regions by MultiDiffusion-style methods with two binary latent swap operators: Self-Loop Latent Swap (Eq. 9), which performs frame-level bidirectional swaps in overlapping regions to preserve high-frequency details, and Reference-Guided Latent Swap (Eq. 11), which aligns non-overlapping regions to a reference trajectory in early denoising steps. The authors motivate the approach through a Fourier analysis of spectrum aliasing and validate it on text-to-audio generation with U-Net and DiT models and on panorama generation with SD 2.0 and SD 3.5, reporting improved fidelity metrics, user-study preference, and 2-20x runtime speedups over baselines.

Significance. If the reported results hold, SaFa is a simple, architecture-agnostic, and computationally efficient alternative to existing joint diffusion methods for length extrapolation in audio and image generation. Its main strengths are the breadth of evaluation (two modalities, two architectures, multiple baselines, user studies), the concrete diagnostic of spectrum aliasing via Fourier analysis, the clear formulation of the swap operators, and the substantial efficiency gains. The paper also includes an appendix with additional experiments and a theoretical analysis of the reference-guided swap, though the theory is best viewed as a supporting explanation rather than a derivation of the method. The method is empirical and incremental relative to prior joint diffusion work, but the application to long-form audio and the swap mechanism are novel enough to be of interest to the community.

major comments (4)
  1. [§5, Eq. (7)] The robustness claim for the hard binary swap rests on the assumed bound εl ≤ d(Right(Φ(...)), Left(Φ(...))) ≤ εu in Eq. (7), but this bound is never measured or estimated anywhere in the paper. The authors do not report d(·) across denoising steps, prompts, models, or modalities, nor do they provide evidence that εl/εu hold. If trajectories diverge beyond εu, the binary swap in Eq. (9) can inject hard discontinuities; if they are too similar, the anti-aliasing argument loses its basis. Please either measure d(·) and exhibit the bounds, or explicitly re-frame Eq. (7) as a heuristic and add an ablation or failure analysis that characterizes when the hard swap breaks down.
  2. [§6.2, Table 4] The panorama comparison is confounded by different overlap rates: the baselines (MD, SyncD, MAD) are run at an overlap rate of 0.8, while SaFa uses 0.2. This makes the reported FID/KID and runtime differences difficult to attribute to the method itself rather than to the operating point. Please report baseline results at 0.2 overlap (or SaFa at 0.8), or otherwise justify why the overlap rate is a method-specific optimization choice and show that the qualitative conclusions are unchanged under matched overlap settings.
  3. [§6.1 and Appendix §8.3] The two key hyperparameters w=1 (self-loop swap interval) and rguide=0.3 (reference guidance rate) are selected after qualitative inspection of a few examples; no quantitative sensitivity analysis is provided. Since these parameters directly control the claimed similarity-diversity trade-off and the method's robustness, please include a quantitative sweep over w and rguide in both audio and panorama tasks, and specify the validation procedure used to select the final values.
  4. [§6.1, §6.2, Tables 2-5, 8-9] All quantitative metrics are reported as single point estimates without error bars, confidence intervals, or statistical significance tests. The phrase "significantly outperforms" is therefore not supported statistically. Please report standard errors or confidence intervals across seeds/prompts, or perform paired significance tests (e.g., bootstrap or permutation tests) on the main comparisons.
minor comments (4)
  1. [Appendix §8.2, Table 6] The text introduces SaFa+ as an improved variant for Make-An-Audio2 that resolves latent sensitivity via a five-token forward shift and mel-spectrogram concatenation, but Table 6 reports results only for SaFa. Define SaFa+ explicitly and include its quantitative results so the reader can verify the improvement.
  2. [Eq. (10)] The definition of the swap mask v_m uses floor((i−1)/w) but does not specify the index range of i or the relationship between m and the spatial dimensions; please clarify the indexing so the mask construction is unambiguous.
  3. [Fig. 2(c)] The Fourier analysis in Fig. 2(c) is shown for selected latents only; please state how many samples were averaged or whether the curves are representative, and consider reporting a quantitative high-frequency energy ratio over a larger set.
  4. [Throughout] There are several typos and inconsistencies: "Meanwile" in Section 1, "aroverlap" in Section 6.2, "SynD" for SyncDiffusion in the text near Table 4, and "specta" in Section 4. Also, the notation SaFa* and SaFa+ is introduced without a consistent explanation; please define both terms at first use.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; SaFa's swap operators are newly proposed mechanisms and the Appendix analysis is a post-hoc bound, not a fitted prediction.

full rationale

The paper's central contribution is the latent swap operator defined in Eq. 9 and the reference-guided variant in Eq. 11. These are new combination rules proposed and then empirically compared against external baselines, not quantities fitted to data and then renamed as predictions. Eq. 7 posits a bounded distance between adjacent step-wise trajectories as motivation for the swap, but the paper does not estimate epsilon_l or epsilon_u; this is an unmeasured premise and a robustness concern, not a circular step, because the swap operator is not derived from Eq. 7 by algebraic construction. The Fourier analysis in Fig. 2(c) is an illustration of the operator's effect, not a hidden target used to define the operator. The Appendix theoretical analysis in Sec. 10 proves a contraction bound for reference-guided swap; the corollary's tighter bound follows directly from the swap definition, which replaces part of each subview with the same reference slice. That is a valid theorem about the designed operator, not an independent 'prediction' secretly equivalent to its input. Evaluation metrics use reference sets generated by the same reference model, which is standard distributional evaluation for generative models; no metric is optimized during training, and the reported gains come from comparing fixed methods. The hyperparameters w=1 and rguide=0.3 are selected through ablations in the appendix, and while they are tuned on the same benchmark, this does not make the central derivation circular. No load-bearing self-citation chain appears: internal citations such as [35] are related-work mentions, and the external baseline MD* [42] is not authored by the present paper's authors. Overall, the derivation chain is self-contained, and no step reduces by construction to its own inputs.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical or chemical entities. The two swap operators are algorithmic constructs, not invented entities with independent evidence requirements. The free parameters are hyperparameters tuned on the evaluation tasks, and the main assumptions are about latent-space behavior of pretrained VAEs and the closeness of diffusion trajectories.

free parameters (2)
  • Swap interval w = 1
    Selected based on experimental results in Appendix 8.3; w=1 gives frame-level swapping and is described as optimal for spectral smoothness (Section 5).
  • Reference guidance rate rguide = 0.3
    Chosen as a similarity-diversity balance from Figure 6 (Section 5).
assumptions (3)
  • domain assumption Downsample(X) ≈ Wc·Z for a learnable channel-wise linear map Wc (Eq. 5)
    Used to argue that VAE latents inherit the connectivity/high-frequency variability of the original features. The paper provides only an illustrative diagram (Fig. 2a) and does not verify this approximation quantitatively across the VAEs it claims.
  • domain assumption Adjacent subview trajectories satisfy εl ≤ d(...) ≤ εu (Eq. 7)
    The swap operator requires that denoising paths in overlap regions are neither identical nor too divergent. The bounds are never measured or justified.
  • standard math Score network is bounded: ‖sθ(x,t,y)‖≤C (Appendix Sec. 10)
    Needed for the theoretical inequality on trajectory closeness; standard in SDE analysis but not verified for the specific models used.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Latent Swap Joint Diffusion for 2D Long-Form Latent Generation." pith.science (2026). https://pith.science/paper/CW5K4MBK

@misc{pith2026250205130,
  author       = {Pith},
  title        = {Pith review of: Latent Swap Joint Diffusion for 2D Long-Form Latent Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CW5K4MBK}},
  note         = {Machine review of arXiv:2502.05130}
}
abstract

This paper introduces Swap Forward (SaFa), a modality-agnostic and efficient method to generate seamless and coherence long spectrum and panorama through latent swap joint diffusion across multi-views. We first investigate the spectrum aliasing problem in spectrum-based audio generation caused by existing joint diffusion methods. Through a comparative analysis of the VAE latent representation of Mel-spectra and RGB images, we identify that the failure arises from excessive suppression of high-frequency components during the spectrum denoising process due to the averaging operator. To address this issue, we propose Self-Loop Latent Swap, a frame-level bidirectional swap applied to the overlapping region of adjacent views. Leveraging stepwise differentiated trajectories of adjacent subviews, this swap operator adaptively enhances high-frequency components and avoid spectrum distortion. Furthermore, to improve global cross-view consistency in non-overlapping regions, we introduce Reference-Guided Latent Swap, a unidirectional latent swap operator that provides a centralized reference trajectory to synchronize subview diffusions. By refining swap timing and intervals, we can achieve a cross-view similarity-diversity balance in a forward-only manner. Quantitative and qualitative experiments demonstrate that SaFa significantly outperforms existing joint diffusion methods and even training-based methods in audio generation using both U-Net and DiT models, along with effective longer length adaptation. It also adapts well to panorama generation, achieving comparable performance with 2 $\sim$ 20 $\times$ faster speed and greater model generalizability. More generation demos are available at https://swapforward.github.io/

Figures

Figures reproduced from arXiv: 2502.05130 by the authors.

Figure 1
Figure 1. Our SaFa preserves more spectral details and avoids aliasing or structural misalignment in both long-form spectrum and panorama [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. (a) The channel-wise linear approximation between original feature downsampling and its VAE latent ensures their Connectivity [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Our latent swap joint diffusion pipeline. As the core, the Self-Loop Latent Swap operator performs bidirectional frame-level swaps [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (26 more)
Figure 4
Figure 4. Figure 4: Compared to unguided results in the last row of subfig [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Qualitative comparisons. SaFa achieves better generation quality in both long audio generation and panoramas with [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: The effect of the trajectory guidance rate [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: The long-form spectrum generated by various joint diffu [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: The effect of the swap interval w (Eq. 10 ) of Self-Loop Latent Swap on spectrum generation. Better transition is achieved with lower values of w, 1 or 2, which indicate a high swap frequency between two step-wise differential trajectories to enhance the high-frequency…
Figure 9
Figure 9. Figure 9: User study results on audio generation. to the early denoising stages. To further explore the effects of the swap interval w (in Eq. 10), we apply the Self-Loop Latent Swap with various w values in spectrum generation, as shown in [PITH_FULL_IMAGE:figures/full_fig_p01…
Figure 11
Figure 11. Figure 11: Qualitative comparison on soundscape generation. MD* represent an enhanced MD method with triangular windows. [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: Qualitative comparison on soundscape generation. MD* represent an enhanced MD method with triangular windows. [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: Qualitative comparison on soundscape generation. MD* represent an enhanced MD method with triangular windows. [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]
Figure 14
Figure 14. Figure 14: Qualitative comparison on music generation. MD* represent an enhanced MD method with triangular windows. [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]
Figure 15
Figure 15. Figure 15: Qualitative comparison on music generation. MD* represent an enhanced MD method with triangular windows. [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]
Figure 16
Figure 16. Figure 16: Qualitative comparison on music generation. MD* represent an enhanced MD method with triangular windows. [PITH_FULL_IMAGE:figures/full_fig_p021_16.png]
Figure 17
Figure 17. Figure 17: Qualitative comparison on audio effect generation. MD* represent an enhanced MD method with triangular windows. [PITH_FULL_IMAGE:figures/full_fig_p022_17.png]
Figure 18
Figure 18. Figure 18: Qualitative comparison on audio effect generation. MD* represent an enhanced MD method with triangular windows. [PITH_FULL_IMAGE:figures/full_fig_p023_18.png]
Figure 19
Figure 19. Figure 19: Qualitative comparison on audio effect generation. MD* represent an enhanced MD method with triangular windows. [PITH_FULL_IMAGE:figures/full_fig_p024_19.png]
Figure 20
Figure 20. Figure 20: Qualitative comparison on panorama image generation. MD* represent an enhanced MD method with triangular windows. [PITH_FULL_IMAGE:figures/full_fig_p025_20.png]
Figure 21
Figure 21. Figure 21: Qualitative comparison on panorama image generation. MD* represent an enhanced MD method with triangular windows. [PITH_FULL_IMAGE:figures/full_fig_p026_21.png]
Figure 22
Figure 22. Figure 22: Qualitative comparison on panorama image generation. MD* represent an enhanced MD method with triangular windows. [PITH_FULL_IMAGE:figures/full_fig_p027_22.png]
Figure 23
Figure 23. Figure 23: Qualitative comparison on panorama image generation. MD* represent an enhanced MD method with triangular windows. [PITH_FULL_IMAGE:figures/full_fig_p028_23.png]
Figure 24
Figure 24. Figure 24: Qualitative comparison on panorama image generation. MD* represent an enhanced MD method with triangular windows. [PITH_FULL_IMAGE:figures/full_fig_p029_24.png]
Figure 25
Figure 25. Figure 25: Qualitative comparison on panorama image generation. MD* represent an enhanced MD method with triangular windows. [PITH_FULL_IMAGE:figures/full_fig_p030_25.png]
Figure 26
Figure 26. Figure 26: Qualitative comparison on panorama image generation. MD* represent an enhanced MD method with triangular windows. [PITH_FULL_IMAGE:figures/full_fig_p031_26.png]
Figure 27
Figure 27. Figure 27: Qualitative comparison on panorama image generation. MD* represent an enhanced MD method with triangular windows. [PITH_FULL_IMAGE:figures/full_fig_p032_27.png]
Figure 28
Figure 28. Figure 28: Qualitative comparison on panorama image generation. MD* represent an enhanced MD method with triangular windows. [PITH_FULL_IMAGE:figures/full_fig_p033_28.png]
Figure 29
Figure 29. Figure 29: Qualitative comparison on panorama image generation. MD* represent an enhanced MD method with triangular windows. [PITH_FULL_IMAGE:figures/full_fig_p034_29.png]
Figure 30
Figure 30. Figure 30: Qualitative comparison on panorama image generation. MD* represent an enhanced MD method with triangular windows. [PITH_FULL_IMAGE:figures/full_fig_p035_30.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 36 canonical work pages

  1. [1]

    Musiclm: Generating music from text

    Andrea Agostinelli, Timo I Denk, Zal ´an Borsos, Jesse En- gel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. Musiclm: Generating music from text. arXiv:2301.11325, 2023. 3

    arXiv 2023

  2. [2]

    Blended diffusion for text-driven editing of natural images

    Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18187–18197, 2021. 3

    work page 2022

  3. [3]

    Blended latent diffusion

    Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. ACM Transactions on Graphics (TOG), 42:1 – 11, 2022. 2, 3

    work page 2022

  4. [4]

    Blended diffusion for text-driven editing of natural images

    Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18208–18218, 2022. 6

    work page 2022

  5. [5]

    Multidiffusion: Fusing diffusion paths for controlled image generation

    Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusion: Fusing diffusion paths for controlled image generation. In International Conference on Machine Learn- ing, 2023. 2, 3, 5, 6, 7, 8

    work page 2023

  6. [6]

    Demystifying mmd gans

    Mikołaj Bi´nkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv:1801.01401,

    arXiv

  7. [7]

    Audiolm: a language modeling approach to audio genera- tion

    Zal´an Borsos, Rapha ¨el Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. Audiolm: a language modeling approach to audio genera- tion. IEEE/ACM transactions on audio, speech, and language processing, 31:2523–2533, 2023. 3

    work page 2023

  8. [8]

    Scaling instruction- finetuned language models

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction- finetuned language models. Journal of Machine Learning Research, 25(70):1–53, 2024. 6

    work page 2024

Show all 56 references
  1. [9]

    Sim- ple and controllable music generation

    Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre D´efossez. Sim- ple and controllable music generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. 3

    2023

  2. [10]

    Demofusion: Democratising high- resolution image generation with no $

    Ruoyi Du, Dongliang Chang, Timothy Hospedales, Yi-Zhe Song, and Zhanyu Ma. Demofusion: Democratising high- resolution image generation with no $. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6159–6168, 2024. 3

    2024

  3. [11]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021. 3

    2021

  4. [12]

    Scaling rectified flow trans- formers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim En- tezari, Jonas M¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow trans- formers for high-resolution image synthesis. In Forty-first international conference on mac...

    2024

  5. [13]

    Hawley, and Jordi Pons

    Zach Evans, CJ Carr, Josiah Taylor, Scott H. Hawley, and Jordi Pons. Fast timing-conditioned latent audio diffusion. In ICML 2024, 2024. 1, 3, 7, 11, 12

    2024

  6. [14]

    Long-form music generation with latent diffusion

    Zach Evans, Julian D Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. Long-form music generation with latent diffusion. arXiv:2404.10301, 2024. 2, 3

    2024 arXiv

  7. [15]

    Masked diffusion transformer is a strong image synthe- sizer

    Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Masked diffusion transformer is a strong image synthe- sizer. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pages 23164–23173, 2023. 6

    2023

  8. [16]

    Image style transfer using convolutional neural networks

    Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2414–2423, 2016. 8

    2016

  9. [18]

    Text-to-audio generation using instruction- tuned llm and latent diffusion model

    Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, and Soujanya Poria. Text-to-audio generation using instruction- tuned llm and latent diffusion model. arXiv:2304.13731,

    arXiv

  10. [19]

    Latent video diffusion models for high-fidelity long video generation

    Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation. arXiv:2211.13221, 2022. 1

    2022 arXiv

  11. [20]

    Scalecrafter: Tuning-free higher- resolution visual generation with diffusion models

    Yingqing He, Shaoshu Yang, Haoxin Chen, Xiaodong Cun, Menghan Xia, Yong Zhang, Xintao Wang, Ran He, Qifeng Chen, and Ying Shan. Scalecrafter: Tuning-free higher- resolution visual generation with diffusion models. In The Twelfth International Conference on Learning Representa-...

    2023

  12. [21]

    Clipscore: A reference-free evaluation metric for image captioning

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv:2104.08718, 2021. 8

    2021 arXiv

  13. [22]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bern- hard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017. 8

    2017

  14. [23]

    Denoising diffu- sion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 1, 3

    2020

  15. [24]

    Make-an-audio 2: Temporal-enhanced text- to-audio generation

    Jia-Bin Huang, Yi Ren, Rongjie Huang, Dongchao Yang, Zhenhui Ye, Chen Zhang, Jinglin Liu, Xiang Yin, Zejun Ma, and Zhou Zhao. Make-an-audio 2: Temporal-enhanced text- to-audio generation. ArXiv, 2023. 1, 3, 4, 6, 8, 11, 12

    2023

  16. [25]

    Park, Tao Wang, Timo I

    Qingqing Huang, Daniel S. Park, Tao Wang, Timo I. Denk, Andy Ly, Nanxin Chen, Zhengdong Zhang, Zhishuai Zhang, Jiahui Yu, Christian Havnø Frank, Jesse Engel, Quoc V . Le, William Chan, and Wei Han. Noise2music: Text- conditioned music generation with diffusion models. ArXiv, a...

    2023 arXiv

  17. [26]

    Mixture of diffusers for scene composition and high resolution image generation

    ´Alvaro Barbero Jim ´enez. Mixture of diffusers for scene composition and high resolution image generation. arXiv:2302.02412, 2023. 2, 3

    2023 arXiv

  18. [27]

    Audiocaps: Generating captions for audios in the wild

    Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume 1...

    2019

  19. [29]

    Fifo-diffusion: Generating infinite videos from text without training

    Jihwan Kim, Junoh Kang, Jinyoung Choi, and Bohyung Han. Fifo-diffusion: Generating infinite videos from text without training. arXiv:2405.11473, 2024. 1

    2024 arXiv

  20. [30]

    Synctweedies: A general generative framework based on synchronized diffusions

    Jaihoon Kim, Juil Koo, Kyeongmin Yeo, and Minhyuk Sung. Synctweedies: A general generative framework based on synchronized diffusions. arXiv:2403.14370, 2024. 3

    2024 arXiv

  21. [31]

    Leveraging early-stage robustness in diffusion models for efficient and high-quality image synthesis

    Yulhwa Kim, Dongwon Jo, Hyesung Jeon, Taesu Kim, Dae- hyun Ahn, Hyungjun Kim, et al. Leveraging early-stage robustness in diffusion models for efficient and high-quality image synthesis. Advances in Neural Information Processing Systems, 36, 2024. 7

    2024

  22. [32]

    Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis

    Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in neural information processing systems, 33:17022–17033, 2020. 6

    2020

  23. [33]

    Audiogen: Textually guided audio generation

    Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre D´efossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. Audiogen: Textually guided audio generation. arXiv:2209.15352, 2022. 2, 3, 7, 11

    2022 arXiv

  24. [34]

    Syncdiffusion: Coherent montage via synchronized joint dif- fusions

    Yuseung Lee, Kunho Kim, Hyunjin Kim, and Minhyuk Sung. Syncdiffusion: Coherent montage via synchronized joint dif- fusions. Advances in Neural Information Processing Systems, 36:50648–50660, 2023. 2, 3, 5, 6, 7, 8

    2023

  25. [35]

    Quality-aware masked diffusion transformer for enhanced music generation

    Chang Li, Ruoyu Wang, Lijuan Liu, Jun Du, Yixuan Sun, Zilu Guo, Zhenrong Zhang, and Yuan Jiang. Quality-aware masked diffusion transformer for enhanced music generation. arXiv:2405.15863, 2024. 1

    2024 arXiv

  26. [36]

    Panofree: Tuning-free holistic multi-view image generation with cross-view self-guidance

    Aoming Liu, Zhong Li, Zhang Chen, Nannan Li, Yi Xu, and Bryan A Plummer. Panofree: Tuning-free holistic multi-view image generation with cross-view self-guidance. In European Conference on Computer Vision, pages 146–164. Springer,

  27. [37]

    Audi- oldm: Text-to-audio generation with latent diffusion models

    Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audi- oldm: Text-to-audio generation with latent diffusion models. arXiv:2301.12503, 2023. 3, 4, 6, 7, 11, 12

    2023 arXiv

  28. [38]

    Mandic, Wenwu Wang, and MarkD

    Haohe Liu, Zehua Chen, Yiitan Yuan, Xinhao Mei, Xubo Liu, Danilo P. Mandic, Wenwu Wang, and MarkD . Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. In International Conference on Machine Learning,

  29. [39]

    Audioldm 2: Learning holistic audio gen- eration with self-supervised pretraining

    Haohe Liu, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Qiao Tian, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D Plumbley. Audioldm 2: Learning holistic audio gen- eration with self-supervised pretraining. IEEE/ACM Trans- actions on Audio, Speech, and Language Processing, 2024. 1

    2024

  30. [40]

    Autoregressive omni-aware outpainting for open- vocabulary 360-degree image generation

    Zhuqiang Lu, Kun Hu, Chaoyue Wang, Lei Bai, and Zhiy- ong Wang. Autoregressive omni-aware outpainting for open- vocabulary 360-degree image generation. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 14211– 14219, 2024. 2

    2024

  31. [41]

    Tango 2: Aligning diffusion-based text-to-audio generations through direct preference optimization

    Navonil Majumder, Chia-Yu Hung, Deepanway Ghosal, Wei- Ning Hsu, Rada Mihalcea, and Soujanya Poria. Tango 2: Aligning diffusion-based text-to-audio generations through direct preference optimization. In Proceedings of the 32nd ACM International Conference on Multimedia, pages ...

    2024

  32. [42]

    Movie gen: A cast of media foundation models

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma, Ching-Yao Chuang, et al. Movie gen: A cast of media foundation models. arXiv:2410.13720, 2024. 2, 3, 6, 7

    2024 arXiv

  33. [43]

    Freenoise: Tuning-free longer video diffusion via noise rescheduling

    Haonan Qiu, Menghan Xia, Yong Zhang, Yingqing He, Xintao Wang, Ying Shan, and Ziwei Liu. Freenoise: Tuning-free longer video diffusion via noise rescheduling. arXiv:2310.15169, 2023. 1

    2023 arXiv

  34. [44]

    Merging and splitting diffusion paths for semanti- cally coherent panoramas

    Fabio Quattrini, Vittorio Pippi, Silvia Cascianelli, and Rita Cucchiara. Merging and splitting diffusion paths for semanti- cally coherent panoramas. In ECCV, 2024. 2, 3, 6, 7, 8

    2024

  35. [45]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 3, 4, 8, 14

    2022

  36. [46]

    Freeu: Free lunch in diffusion u-net

    Chenyang Si, Ziqi Huang, Yuming Jiang, and Ziwei Liu. Freeu: Free lunch in diffusion u-net. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4733–4743, 2024. 4

    2024

  37. [47]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv:2010.02502, 2020. 1, 6

    2010 arXiv

  38. [48]

    Score-based generative modeling through stochastic differential equations

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Ab- hishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv:2011.13456, 2020. 1, 14

    2011 arXiv

  39. [49]

    Litefocus: Accelerated diffusion inference for long audio synthesis

    Zhenxiong Tan, Xinyin Ma, Gongfan Fang, and Xinchao Wang. Litefocus: Accelerated diffusion inference for long audio synthesis. arXiv:2407.10468, 2024. 3

    2024 arXiv

  40. [50]

    Gen-l-video: Multi-text to long video generation via temporal co-denoising

    Fu-Yun Wang, Wenshuo Chen, Guanglu Song, Han-Jia Ye, Yu Liu, and Hongsheng Li. Gen-l-video: Multi-text to long video generation via temporal co-denoising. arXiv:2305.18264,

    arXiv

  41. [51]

    Large-scale con- trastive language-audio pretraining with feature fusion and keyword-to-caption augmentation

    Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale con- trastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Si...

    2023

  42. [52]

    Stochsync: Stochastic diffusion synchronization for image generation in arbitrary spaces

    Kyeongmin Yeo, Jaihoon Kim, and Minhyuk Sung. Stochsync: Stochastic diffusion synchronization for image generation in arbitrary spaces. arXiv:2501.15445, 2025. 3

    2025 arXiv

  43. [53]

    Diffcollage: Parallel generation of large content with diffusion models

    Qinsheng Zhang, Jiaming Song, Xun Huang, Yongxin Chen, and Ming-Yu Liu. Diffcollage: Parallel generation of large content with diffusion models. In2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10188–10198. IEEE, 2023. 2

    2023

  44. [54]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018. 2, 8

    2018

  45. [55]

    Comparison with Training-Based Methods We further compare our method with more training-based long audio generation models, including both diffusion mod- els and language models

    More Quantitive Experiments 8.1. Comparison with Training-Based Methods We further compare our method with more training-based long audio generation models, including both diffusion mod- els and language models. Although strictly speaking, the absolute performance between mode...

  46. [56]

    We use the same notation as in Section 6.1

    User Study For subjective evaluation, we randomly select samples from the qualitative results of the top four methods in audio and panorama generation for user studies. We use the same notation as in Section 6.1. Specifically, SaFa is compared with MD, MD*, and MAD for audio g...

  47. [57]

    Theoretical Analysis of Refer-Guided Swap for Cross-View Similarity-Diversity Balance Reference-Guided Latent Swap improves cross-view consistency comparing with independent denoising process with reference model directly. When SD-2.0 [45] is employed as the reference model Φ,...

  48. [58]

    11 to 19 and panorama generation are in Fig

    Further Qualitative Comparison More qualitative results on the audio generation are in Fig. 11 to 19 and panorama generation are in Fig. 20 to 30. Casino Ambience, electronic slot machines SaFa MD MD* MAD SaFa MD MD* MAD Figure 11. Qualitative comparison on soundscape generati...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.