EditGen: Harnessing Cross-Attention Control for Instruction-Based Auto-Regressive Audio Editing

Alexandros Potamianos; Giorgos Paraskevopoulos; Vassilis Sioros

REVIEW 3 major objections 6 minor 53 references

EditGen: Harnessing Cross-Attention Control for Instruction-Based Auto-Regressive Audio Editing

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that re-injecting cross-attention maps from a source generation into a frozen autoregressive music model lets users edit audio by changing the text prompt, preserving structure while the edit takes effect, and beats a…

desk verdict A useful first adaptation of Prompt-to-Prompt to autoregressive music editing, but the headline comparison never isolates the edit mechanism from the base generator. read the letter →

arxiv 2507.11096 v1 pith:2FRBDBLJ submitted 2025-07-15 cs.SD cs.AIeess.AS

Vassilis Sioros , Alexandros Potamianos , Giorgos Paraskevopoulos This is my paper

classification cs.SDcs.AIeess.AS

keywords audioeditingprompt-to-promptcross-attentioncontrolautoregressivemusicgenerationGeninstruction-basedattentioninjectionsoft-blending

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper adapts Prompt-to-Prompt, an image-editing technique that steers generation by re-injecting cross-attention maps, to the autoregressive music model MusicGen. The aim is instruction-based audio editing—swapping instruments, adding or removing musical elements, changing genre or mood—without retraining or extra data. The authors introduce three editing operators (attention replacement, token reweighting, and refinement) plus a soft-blending schedule, and compare against Auffusion, a diffusion baseline. They report that the autoregressive version beats the diffusion baseline across melody, dynamics, tempo, and textual alignment metrics, and that human listeners judged its outputs more natural and faithful.

What carries the argument

The central object is the cross-attention map in MusicGen's decoder, the matrix that weights how each audio token position attends to each text token. The paper's three editing operators—Replace, Refine, Reweight—edit this matrix: Replace substitutes the source map before a threshold τ, Refine reuses source maps only for tokens common to both prompts via an alignment function, and Reweight scales one token's attention by a factor c. Soft-blending ($Z_t = \alpha X_t + (1-\alpha) Y_t$) then merges generated and injected features with a layer-indexed α, simulating the diffusion-style early-edit schedule. These components carry the argument because they make a frozen, pre-trained decoder behave as an editor without any weight updates.

What would settle it

Record the cross-attention maps while generating a clip from a prompt such as 'a jazz piano ballad,' then regenerate with the prompt 'a rock guitar ballad' while injecting scrambled or shuffled attention maps instead of the true source maps. If the output still changes the instrument and preserves the melody as cleanly as true injection does, then the specific attention values are not what carries the edit, and the paper's mechanism is not the active ingredient. A complementary check is to corrupt only the attention columns of the word being changed and measure whether the corresponding instrument's spectral content moves in the predicted direction.

Watch

Extended reading notes

Core claim

EditGen's central claim is that the attention-map injection scheme from Prompt-to-Prompt transfers to autoregressive decoding: by recording the cross- and self-attention maps produced while generating a source audio sample from a text prompt, then injecting those maps into a second generation whose prompt has been edited, the model produces an edited clip that keeps the original's global structure—melody outline, rhythm, dynamics—while changing only the tokens touched by the prompt edit. For replacement the source maps are injected until a timestep threshold; for refinement only the maps of shared tokens are re-used; for reweighting the selected token's map is scaled by a strength constant. Because MusicGen is sequence-to-sequence rather than iterative like diffusion, the authors apply injection at all decoding steps and add soft-blending, a layer-wise weighted average between generated and injected features, to approximate the original diffusion timing schedule. The evaluation, including a 24-listener study, is the basis for the paper's claim that this combination significantly outperforms the diffusion-based Auffusion baseline on melody, dynamics, tempo, and alignment.

Load-bearing premise

The central premise is that the attention maps inside the music generator reflect which parts of the audio are governed by which words in the prompt, exactly as they do in image diffusion models; if those maps do not carry that semantic structure, the three edit operators cannot produce controlled edits.

Editorial extensions

If this is right

Editing music by natural-language instructions becomes possible with a frozen autoregressive model, so no fine-tuning or paired edit data is needed to change instruments, mood, genre, melody, or structure.
The three attention operators give users direct control over edit locality: replace tokens, add or remove tokens, or amplify and attenuate a token's influence.
Soft-blending improves both text-to-audio and audio-to-audio similarity over hard-blending, so the layer-wise schedule is a viable adaptation of Prompt-to-Prompt's diffusion timing to sequence decoding.
Against the diffusion baseline, the autoregressive approach is claimed to improve melody accuracy, dynamics correlation, rhythm F1, and human-rated naturalness, with t-test p-values reported as significant.
The paper marks what it calls the first successful use of prompt-to-prompt in autoregressive audio editing, opening a direction for editing with large pre-trained generative audio models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Inference: The same injection scheme should generalize to other frozen autoregressive audio models built on different codecs, provided their cross-attention maps retain token-to-content localization; a direct test would rerun the three operators on a second decoder.
Inference: If autoregressive attention maps behave like diffusion pixel-token alignment, the method could support segment-level edits, such as changing only the instrument in one time window, by restricting attention-map injection to the corresponding decoding steps.
Inference: The paper's acknowledged limitation—MusicGen aggregates all sequence information into a single attention value—suggests that the current operators may over-inject; replacing that single-value cross-attention with a multi-token formulation could sharpen edit precision.
Inference: The reported margins rest on a small dataset of hand-curated and ChatGPT-generated prompt pairs with five seeds per pair, so a larger-scale replication across more prompts and genres would test whether the advantage over the baseline is stable.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The genuinely new thing here is the transfer: adapting Prompt-to-Prompt cross-attention control to a frozen autoregressive model (MusicGen) rather than to a diffusion model. The three operators—Replace, Refine, Reweight—follow the image-domain recipes closely, and soft-blending is a small heuristic, but the autoregressive adaptation is a real first. The paper also ships code and evaluates on music-specific metrics plus a human study, which is more than most short papers do.

The main problem is that the central comparison does not support the central claim. Every reported result is MusicGen-with-injection versus Auffusion-with-injection. There is no control that removes the injection while holding the base model fixed. Since MusicGen is a much stronger generator than Auffusion, the higher melody accuracy, dynamics correlation, rhythm F1, CLAP/A2A similarity, and MOS could all come from the base model alone. The human naturalness results (Table 3) are especially telling: participants compared naturally generated clips from the same prompt, so those preferences are about the generators, not the editing method. Without a null-edit or no-injection condition, the paper cannot attribute its gains to cross-attention injection.

The paper is also somewhat self-referential on the similarity metrics: the operators are designed to preserve source attention maps, so high Audio-to-Audio similarity partly reflects the construction rather than editing quality. The soft-blending gain (Table 1) is tiny and within noise, and the statistical reporting is thin—MOS distributions, t-tests, and p-values are given, but some figures lack error bars and the dataset of 22 prompt pairs per category is small.

That said, the paper is honest about its own limitation: the conclusion notes that MusicGen aggregates sequence information into a single attention value, which undercuts the premise that injected maps carry semantic structure. The authors do not hide this. The adaptation is plausible, the writing is clear, and the code availability is a plus. I would not call this a strong result, but it is a legitimate first step and a useful baseline for future work.

For peer review: yes, send it out. The missing controls are fixable, and the topic is timely. The referee should ask for a no-injection MusicGen baseline, a null-edit condition, and more careful treatment of the similarity metrics. The authors have the pieces—they just need to separate the edit mechanism from the generator.

Referee Report

3 major / 6 minor

Summary. The paper adapts Prompt-to-Prompt cross-attention control to MusicGen, a frozen autoregressive music generation model, proposing three editing operators (Replace, Refine, Reweight) and a soft-blending variant of attention injection. The authors compare this approach against Auffusion, a diffusion-based text-to-audio model, using automatic music metrics (Melody Accuracy, Dynamics Correlation, Rhythm F1, CLAP similarity) and a human study with MOS ratings. They report that MusicGen with attention injection outperforms Auffusion across all metrics and editing categories, and they release code.

Significance. If the central claim were properly established, this would be a useful zero-training audio editing method that extends image-domain Prompt-to-Prompt to autoregressive music models. The paper has notable strengths: it releases code, evaluates on several standard music metrics, includes a human study, and proposes a concrete soft-blending mechanism for autoregressive attention control. However, the current evidence does not isolate the effect of the editing mechanism from the choice of base generator, so the headline claim of superiority over the diffusion baseline is not yet supported. The central idea is plausible and the deficiencies are addressable with additional controls and re-analysis.

major comments (3)

[§5.2, Tables 2–4] The headline comparison confounds the editing mechanism with the base model. Every reported comparison is MusicGen-with-injection versus Auffusion-with-injection; there is no ablation of MusicGen without attention injection, no null-edit control (injecting source attention while keeping the prompt unchanged), and no comparable Auffusion control that removes the injection. Since MusicGen is a substantially stronger text-to-music generator than Auffusion, the higher Melody Accuracy, Dynamics Correlation, Rhythm F1, CLAP/A2A similarity, and MOS in Tables 2–4 could be due entirely to base-generation quality. This is especially clear in Table 3, where participants compared naturally generated Auffusion and MusicGen clips from the same prompt; the 60–82% naturalness preference is a base-model result, not evidence about editing quality. The p-values in Table 4 therefore do not establish that the cross-attention injection defined by Eqs. (1)–(3) contributes anything. I request ablations that fix the base model: report automatic metrics and human ratings for (a) MusicGen without injection, (b) MusicGen with null-edit injection, and (c) Auffusion without injection, in addition to the current conditions.
[§4.2, §5.2] Several of the automatic metrics are partly self-referential for an injection method. Melody Accuracy, Dynamics Correlation, Rhythm F1, and Audio-to-Audio cosine similarity all measure similarity between the source audio and the edited audio. Because Replace, Refine, and Reweight are explicitly designed to copy source attention maps, high preservation scores may reflect the degree of injection rather than successful editing. The paper does not report a systematic prompt-adherence measure for the edited prompt per operation; the only T2A similarity statement in Section 5 is about the prompt-strength sweep and is not broken down by edit operator and model. To support the editing claim, the authors should report CLAP T2A similarity to the edited prompt for each operator and each condition, and ideally show the trade-off between source preservation and prompt adherence, e.g., by varying the injection strength and plotting both quantities.
[§5.3, Table 2, Figure 3] The reported results lack uncertainty information and sufficient statistical detail. Table 2 gives mean MOS values with no confidence intervals or standard deviations; Figure 3 shows average metric bars without error bars; and Section 5.3 does not state how many ratings contributed to each cell in Table 2, nor whether the 24 evaluators and 16 sampled pairs are per condition. The p-values in Table 4 come from unpaired t-tests on four criteria without multiple-comparison correction. I ask for confidence intervals or error bars on all reported means, a statement of rating counts, inter-rater agreement (e.g., ICC or Krippendorff's alpha), and adjusted p-values or a note on the multiple-comparison issue.

minor comments (6)

[Title and Abstract] The paper describes the task as 'instruction-based' audio editing, but the experiments use prompt pairs (e.g., replacing tokens or changing the prompt) rather than free-form instructions. Consider renaming the task description or clarifying that the instructions are prompt edits.
[§3.2, Eq. (1)] The Replace formula defines a timestamp parameter τ, but the text immediately afterward says injection is applied at all timesteps for MusicGen. Please clarify whether τ is used at all in the autoregressive setting and how it interacts with the soft-blending formulation in Eq. (4).
[§3.2, Eq. (3)] The Reweight formula scales the source attention map M_t for token j* but leaves all other entries as M_t as well; it is unclear whether the target-prompt attention M*_t is used for the other tokens. Also, since cross-attention maps are normally non-negative (post-softmax), the allowable range c ∈ [−2, 2] with negative values needs a brief justification or a note on how negative attention is handled.
[§4.1] The sentence 'we generated 22 samples per edit category ... with 5 random seeds per prompt pair' is ambiguous. Please specify the number of prompt pairs per category and the total number of samples per model and condition, so that the reported total of 660 is transparent.
[Table 2] The last row is formatted as 'ReweightMUSICGEN' without a space; fix the alignment for readability.
[Figure 3] The figure would benefit from error bars, a legend, and explicit axis labels for the metrics, as the current caption does not state which bars correspond to which condition.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the headline MusicGen-vs-Auffusion comparison is an empirical whole-system comparison, and the missing no-injection control is an ablation/validity concern, not a circular derivation.

full rationale

The paper's central claims are empirical comparisons between two editing systems, MusicGen-with-injection and Auffusion-with-injection. The three edit operators in Eqs. (1)-(3) are explicitly adapted from Prompt-to-Prompt [1] and are not derived from the evaluation metrics. No parameter is fitted to the target metrics, no 'prediction' is computed from fitted inputs, and no uniqueness theorem is invoked. The only self-citation, ref. [30], appears in related work and is not load-bearing. The evaluation confound noted by the skeptic — that MusicGen is a stronger base generator, so whole-system superiority may reflect the base model rather than the attention-injection mechanism — is a real experimental-design limitation but not circularity: the paper never claims that a no-injection control isolates the mechanism, and its conclusion is about the combination. The paper's own limitation about MusicGen aggregating sequence information into a single attention value weakens the mechanism premise, but again that is a correctness/validity issue, not a circular reduction. No equation or defined quantity is equivalent to another by construction, so the circularity score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central comparison relies on pre-trained models (MusicGen, Auffusion, CLAP) and on the transfer of attention semantics from image diffusion to autoregressive audio decoding. No new physical or mathematical entities are introduced; the main burdens are the unablated baseline comparison and the unproven transferability of attention-map control.

free parameters (2)

Diffusion injection timestep tau (Auffusion baseline) = not reported
Prompt-to-Prompt for the diffusion baseline requires a cutoff timestep tau controlling how many denoising steps use injected attention maps; the paper does not report the value used, yet it directly affects structure preservation and edit strength.
Reweight scale c = range -2 to 2
User-chosen scalar multiplying the attention map of a target token in Eq. 3; no principled selection criterion or per-task value is given.

assumptions (3)

domain assumption Cross-attention maps in MUSICGEN align text tokens to audio tokens in a way that can be swapped or reweighted to control generation semantics.
Adopted from Prompt-to-Prompt [1] for diffusion image models and applied directly to autoregressive token decoding. The paper acknowledges in the conclusion that MusicGen aggregates sequence information into a single attention value, weakening the premise.
domain assumption Applying attention injection at all autoregressive timesteps preserves global structure of the source audio.
Section 3.2 justifies this as an adjustment to autoregressive generation, but no empirical or theoretical evidence is provided that all-timestep injection is equivalent to the diffusion tau schedule.
domain assumption Auffusion with Prompt-to-Prompt is a representative diffusion-based audio editing baseline.
Used as the sole baseline in Section 5.2. No comparison to other diffusion audio editors (AUDIT, InstructME) or to autoregressive generation without editing, so the baseline repertoire is narrow.

how reviews work

0 comments

Cite this review

Pith. "Pith review of EditGen: Harnessing Cross-Attention Control for Instruction-Based Auto-Regressive Audio Editing." pith.science (2026). https://pith.science/paper/2FRBDBLJ

@misc{pith2026250711096,
  author       = {Pith},
  title        = {Pith review of: EditGen: Harnessing Cross-Attention Control for Instruction-Based Auto-Regressive Audio Editing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2FRBDBLJ}},
  note         = {Machine review of arXiv:2507.11096}
}

read the original abstract

In this study, we investigate leveraging cross-attention control for efficient audio editing within auto-regressive models. Inspired by image editing methodologies, we develop a Prompt-to-Prompt-like approach that guides edits through cross and self-attention mechanisms. Integrating a diffusion-based strategy, influenced by Auffusion, we extend the model's functionality to support refinement edits, establishing a baseline for prompt-guided audio editing. Additionally, we introduce an alternative approach by incorporating MUSICGEN, a pre-trained frozen auto-regressive model, and propose three editing mechanisms, based on Replacement, Reweighting, and Refinement of the attention scores. We employ commonly-used music-specific evaluation metrics and a human study, to gauge time-varying controllability, adherence to global text cues, and overall audio realism. The automatic and human evaluations indicate that the proposed combination of prompt-to-prompt guidance with autoregressive generation models significantly outperforms the diffusion-based baseline in terms of melody, dynamics, and tempo of the generated audio. Our code is available at https://github.com/billsioros/EditGen

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 41 canonical work pages

[1]

EditGen: Harness- ing Cross Attention Control for Instruction-Based auto-regressive Audio Editing

INTRODUCTION Obtaining satisfactory outcomes in audio manipulation tasks has typically required large datasets with intricate an- notations, a process that is often labor-intensive and costly. Moreover, crafting effective model architectures tailored to the nuances of audio processing requires substantial exper- tise and experimentation. Fine-tuning exist...

work page 2024
[2]

We explore the utilization of a pre-trained frozen auto-regressive transformer model, initially de- signed for generating high-quality music samples from a given text prompt, for audio editing

work page
[3]

arXiv:2507.11096v1 [cs.SD] 15 Jul 2025

We design and implement three distinct audio edit- ing mechanisms, inspired by those presented in [1]. arXiv:2507.11096v1 [cs.SD] 15 Jul 2025

arXiv 2025
[4]

We evaluate our auto-regressive transformer model- based approach against existing diffusion-based methods using automatic music-related metrics and feedback from users

work page
[5]

Yang et al

RELATED WORK Diffusion-based audio generation: Diffusion-based models have been widely explored for generation tasks in the audio and music domains. Yang et al. [2] employ a VQ- V AE model trained on mel-spectrograms to convert them into discrete codes. These codes are then fed into a diffu- sion model to generate audio signals. Make-An-Audio [3] employs ...

work page
[6]

We began with Auffusion, leveraging its existing capabilities for prompt- based editing

CONCLUSION AND FUTURE WORKS In conclusion, we explored using two models for au- dio editing: Auffusion and MUSICGEN. We began with Auffusion, leveraging its existing capabilities for prompt- based editing. Furthermore, we introduced an alterna- tive approach by incorporating MUSICGEN, a pre-trained auto-regressive model known for its advanced capabili- ti...

work page
[7]

Masked autoencoders that listen,

P.-Y . Huang, H. Xu, J. Li, A. Baevski, M. Auli, W. Galuba, F. Metze, and C. Feichtenhofer, “Masked autoencoders that listen,” in Advances in Neural Infor- mation Processing Systems , S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. Curran Associates, Inc., 2022, pp. 28 708– 28 720

work page 2022
[8]

The input, a reference audio random variableX, is encoded into a continuous ten- sor with a lower frame rate ( fr) compared to the sample rate (fs)

METHODOLOGY 3.1 MUSICGEN MUSICGEN [23] employs EnCodec [24], a convolu- tional auto-encoder utilizing Residual Vector Quantization (RVQ) for latent space quantization. The input, a reference audio random variableX, is encoded into a continuous ten- sor with a lower frame rate ( fr) compared to the sample rate (fs). The continuous representation is then qu...

work page

Show all 53 references

[9]

prompt strength

EXPERIMENTAL SETUP 4.1 Dataset construction To evaluate our method, we initially create a dataset con- taining prompt pairs for each editing mechanism: Replace, Refine, and Reweight. Each pair consists of original and edited text prompts. Creating our dataset consists of two s...
[10]

prompt strength

RESULTS We perform an initial experiment where we systemati- cally vary the impact - "prompt strength"- of injected at- tention maps in audio generation, gradually increasing the influence of textual cues on editing. We calculate the average cosine similarity between original ...
[11]

HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis,

J. Kong, J. Kim, and J. Bae, “HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis,” in Advances in Neural Information Processing Systems , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc...

2020
[12]

Prompt-to- prompt image editing with cross-attention con- trol,

A. Hertz, R. Mokady, J. Tenenbaum, K. Aber- man, Y . Pritch, and D. Cohen-or, “Prompt-to- prompt image editing with cross-attention con- trol,” in The Eleventh International Conference on Learning Representations , 2023. [Online]. Available: https://openreview.net/forum?id=_CDixzkzeyb

2023
[13]

Auto-regressive audio generation: As an alternative to diffusion-based models for audio and music generation, autoregressive models have shown promise in recent years

leverages a tailored latent diffusion model, facilitating tasks such as adding, removing, and remixing musical el- ements while preserving harmonic integrity via chord pro- gression matrices. Auto-regressive audio generation: As an alternative to diffusion-based models for aud...
[14]

Diffsound: Discrete diffusion model for text-to-sound generation,

D. Yang, J. Yu, H. Wang, W. Wang, C. Weng, Y . Zou, and D. Yu, “Diffsound: Discrete diffusion model for text-to-sound generation,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 31, pp. 1720–1733, 2023

2023
[15]

Make- an-audio: Text-to-audio generation with prompt- enhanced diffusion models,

R. Huang, J. Huang, D. Yang, Y . Ren, L. Liu, M. Li, Z. Ye, J. Liu, X. Yin, and Z. Zhao, “Make- an-audio: Text-to-audio generation with prompt- enhanced diffusion models,” in Proceedings of the 40th International Conference on Machine Learning , ser. Proceedings of Machine Lea...

2023
[16]

CLAP: Learning audio concepts from natural lan- guage supervision,

B. Elizalde, S. Deshmukh, M. A. Ismail, and H. Wang, “CLAP: Learning audio concepts from natural lan- guage supervision,” in ICASSP 2023 - 2023 IEEE In- ternational Conference on Acoustics, Speech and Sig- nal Processing (ICASSP), 2023, pp. 1–5

2023
[17]

Audi- oLM [18] utilizes tokens generated by a SoundStream [19] neural codec [20, 21] as targets for a sequence modeling task

to compress raw audio into discrete codes, which are then modeled using auto-regressive transformers. Audi- oLM [18] utilizes tokens generated by a SoundStream [19] neural codec [20, 21] as targets for a sequence modeling task. In [22], Agostinelli et al. introduce MusicLM, fo...
[18]

AudioLDM: Text- to-audio generation with latent diffusion models,

H. Liu, Z. Chen, Y . Yuan, X. Mei, X. Liu, D. Mandic, W. Wang, and M. D. Plumbley, “AudioLDM: Text- to-audio generation with latent diffusion models,” in Proceedings of the 40th International Conference on Machine Learning , ser. Proceedings of Machine Learning Research, A. Kr...

2023
[19]

AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining,

H. Liu, Q. Tian, Y . Yuan, X. Liu, X. Mei, Q. Kong, Y . Wang, W. Wang, Y . Wang, and M. D. Plumbley, “AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining,” Sep. 2023, arXiv:2308.05734 [cs, eess]. [Online]. Available: http://arxiv.org/abs/2308.05734

2023 arXiv
[20]

AUDIT: Audio editing by following in- structions with latent diffusion models,

Y . Wang, Z. Ju, X. Tan, L. He, Z. Wu, J. Bian, and s. zhao, “AUDIT: Audio editing by following in- structions with latent diffusion models,” in Advances in Neural Information Processing Systems , A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. ...

2023
[21]

Text-to-audio generation using instruction guided latent diffusion model,

D. Ghosal, N. Majumder, A. Mehrish, and S. Poria, “Text-to-audio generation using instruction guided latent diffusion model,” in Proceedings of the 31st ACM International Conference on Multimedia , ser. MM ’23. New York, NY , USA: Association for Computing Machinery, 2023, p. ...

2023
[22]

Auffusion: Lever- aging the Power of Diffusion and Large Language Models for Text-to-Audio Generation,

J. Xue, Y . Deng, Y . Gao, and Y . Li, “Auffusion: Lever- aging the Power of Diffusion and Large Language Models for Text-to-Audio Generation,” Jan. 2024, arXiv:2401.01044 [cs, eess]. [Online]. Available: http://arxiv.org/abs/2401.01044

2024 arXiv
[23]

MusicLDM: Enhanc- ing novelty in text-to-music generation using beat- synchronous mixup strategies,

K. Chen, Y . Wu, H. Liu, M. Nezhurina, T. Berg- Kirkpatrick, and S. Dubnov, “MusicLDM: Enhanc- ing novelty in text-to-music generation using beat- synchronous mixup strategies,” inICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICAS...

2024
[24]

InstructME: An Instruction Guided Music Edit And Remix Framework with Latent Diffusion Models,

B. Han, J. Dai, X. Song, W. Hao, X. He, D. Guo, J. Chen, Y . Wang, and Y . Qian, “InstructME: An Instruction Guided Music Edit And Remix Framework with Latent Diffusion Models,” Sep. 2023, arXiv:2308.14360 [cs, eess]. [Online]. Available: http://arxiv.org/abs/2308.14360

2023 arXiv
[25]

WaveNet: A Generative Model for Raw Audio,

A. v. d. Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, “WaveNet: A Generative Model for Raw Audio,” Sep. 2016, arXiv:1609.03499 [cs]. [Online]. Available: http://arxiv.org/abs/1609.03499

2016 arXiv
[26]

Audiogen: Textually guided audio genera- tion,

F. Kreuk, G. Synnaeve, A. Polyak, U. Singer, A. Défossez, J. Copet, D. Parikh, Y . Taigman, and Y . Adi, “Audiogen: Textually guided audio genera- tion,” in The Eleventh International Conference on Learning Representations , 2023. [Online]. Available: https://openreview.net/fo...

2023
[27]

Jukebox: A Generative Model for Music,

P. Dhariwal, H. Jun, C. Payne, J. W. Kim, A. Radford, and I. Sutskever, “Jukebox: A Generative Model for Music,” Apr. 2020, arXiv:2005.00341 [cs, eess, stat]. [Online]. Available: http://arxiv.org/abs/2005.00341

2020 arXiv
[28]

Neural discrete representation learning,

A. van den Oord, O. Vinyals, and k. kavukcuoglu, “Neural discrete representation learning,” in Advances in Neural Information Processing Systems , I. Guyon, U. V . Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc...

2017
[29]

AudioLM: a Language Modeling Approach to Audio Generation,

Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, D. Grangier, M. Tagliasacchi, and N. Zeghidour, “AudioLM: a Language Modeling Approach to Audio Generation,” Jul. 2023, arXiv:2209.03143 [cs, eess]. [Online]. Available: http://ar...

2023 arXiv
[30]

Soundstream: An end-to-end neu- ral audio codec,

N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, “Soundstream: An end-to-end neu- ral audio codec,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 30, pp. 495– 507, 2022

2022
[31]

End-to-end optimized speech cod- ing with deep neural networks,

S. Kankanahalli, “End-to-end optimized speech cod- ing with deep neural networks,” in 2018 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018, pp. 2521–2525

2018
[32]

Harp- net: Hyper-autoencoded reconstruction propagation for scalable neural audio coding,

D. Petermann, S. Beack, and M. Kim, “Harp- net: Hyper-autoencoded reconstruction propagation for scalable neural audio coding,” in 2021 IEEE Work- shop on Applications of Signal Processing to Audio and Acoustics (WASPAA), 2021, pp. 316–320

2021
[33]

MusicLM: Generating Music From Text,

A. Agostinelli, T. I. Denk, Z. Borsos, J. Engel, M. Verzetti, A. Caillon, Q. Huang, A. Jansen, A. Roberts, M. Tagliasacchi, M. Sharifi, N. Zeghidour, and C. Frank, “MusicLM: Generating Music From Text,” Jan. 2023, arXiv:2301.11325 [cs, eess]. [Online]. Available: http://arxiv....

2023 arXiv
[34]

Simple and control- lable music generation,

J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant, G. Syn- naeve, Y . Adi, and A. Defossez, “Simple and control- lable music generation,” in Advances in Neural In- formation Processing Systems , A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36. Cu...

2023
[35]

High fidelity neural audio compression,

A. Défossez, J. Copet, G. Synnaeve, and Y . Adi, “High fidelity neural audio compression,” Transactions on Machine Learning Research, 2023, featured Certifica- tion, Reproducibility Certification. [Online]. Available: https://openreview.net/forum?id=ivCd8z8zR2

2023
[36]

Null-text inversion for editing real im- ages using guided diffusion models,

R. Mokady, A. Hertz, K. Aberman, Y . Pritch, and D. Cohen-Or, “Null-text inversion for editing real im- ages using guided diffusion models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2023, pp. 6038– 6047

2023
[37]

Dreambooth: Fine tuning text-to- image diffusion models for subject-driven generation,

N. Ruiz, Y . Li, V . Jampani, Y . Pritch, M. Rubinstein, and K. Aberman, “Dreambooth: Fine tuning text-to- image diffusion models for subject-driven generation,” in Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR) , June 2023, pp. 22 5...

2023
[38]

Multi-concept customization of text-to-image diffusion,

N. Kumari, B. Zhang, R. Zhang, E. Shechtman, and J.- Y . Zhu, “Multi-concept customization of text-to-image diffusion,” in Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), June 2023, pp. 1931–1941

2023
[39]

SVDiff: Compact parameter space for dif- fusion fine-tuning,

L. Han, Y . Li, H. Zhang, P. Milanfar, D. Metaxas, and F. Yang, “SVDiff: Compact parameter space for dif- fusion fine-tuning,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , October 2023, pp. 7323–7334

2023
[40]

Countering language drift via visual grounding,

J. Lee, K. Cho, and D. Kiela, “Countering language drift via visual grounding,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), K. Inui, J. Jiang...

2019
[41]

Likert scale: Explored and explained,

A. Joshi, S. Kale, S. Chandel, and D. K. Pal, “Likert scale: Explored and explained,” British journal of ap- plied science & technology, vol. 7, no. 4, pp. 396–403, 2015

2015
[42]

Investigating personaliza- tion methods in text to music generation,

M. Plitsis, T. Kouzelis, G. Paraskevopoulos, V . Kat- souros, and Y . Panagakis, “Investigating personaliza- tion methods in text to music generation,” in ICASSP 2024 - 2024 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP) , 2024, pp. 1081–1085

2024
[43]

Zero-Shot Unsuper- vised and Text-Based Audio Editing Using DDPM Inversion,

H. Manor and T. Michaeli, “Zero-Shot Unsuper- vised and Text-Based Audio Editing Using DDPM Inversion,” Feb. 2024, arXiv:2402.10009 [cs, eess]. [Online]. Available: http://arxiv.org/abs/2402.10009

2024 arXiv
[44]

An Edit Friendly DDPM Noise Space: Inversion and Manipulations,

I. Huberman-Spiegelglas, V . Kulikov, and T. Michaeli, “An Edit Friendly DDPM Noise Space: Inversion and Manipulations,” Apr. 2024, arXiv:2304.06140 [cs]. [Online]. Available: http://arxiv.org/abs/2304.06140

2024 arXiv
[45]

Photorealistic text-to-image diffu- sion models with deep language understanding,

C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans, J. Ho, D. J. Fleet, and M. Norouzi, “Photorealistic text-to-image diffu- sion models with deep language understanding,” in Advances in Neural Informat...

2022
[46]

Music controlnet: Multiple time-varying controls for music generation,

S.-L. Wu, C. Donahue, S. Watanabe, and N. J. Bryan, “Music controlnet: Multiple time-varying controls for music generation,” arXiv preprint arXiv:2311.07069 , 2023

2023 arXiv
[47]

Evaluation of Audio Beat Tracking and Music Tempo Extraction Algorithms,

M. F. McKinney, D. Moelants, M. E. P. Davies, and A. Klapuri, “Evaluation of Audio Beat Tracking and Music Tempo Extraction Algorithms,” Journal of New Music Research, vol. 36, no. 1, pp. 1–16, Mar. 2007. [Online]. Available: http://www.tandfonline.com/doi/ abs/10.1080/0929821...

2007 doi
[48]

mir_eval: A transparent implementation of common mir metrics,

C. Raffel, B. Mcfee, E. Humphrey, J. Salamon, O. Ni- eto, D. Liang, and D. Ellis, “mir_eval: A transparent implementation of common mir metrics,” 10 2014

2014
[49]

An efficient state- space model for joint tempo and meter tracking

F. Krebs, S. Böck, and G. Widmer, “An efficient state- space model for joint tempo and meter tracking.” inIS- MIR, 2015, pp. 72–78

2015
[50]

Large-scale contrastive language- audio pretraining with feature fusion and keyword-to- caption augmentation,

Y . Wu, K. Chen, T. Zhang, Y . Hui, T. Berg-Kirkpatrick, and S. Dubnov, “Large-scale contrastive language- audio pretraining with feature fusion and keyword-to- caption augmentation,” in ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processin...

2023
[51]

HTS-AT: A hierarchical token- semantic audio transformer for sound classification and detection,

K. Chen, X. Du, B. Zhu, Z. Ma, T. Berg-Kirkpatrick, and S. Dubnov, “HTS-AT: A hierarchical token- semantic audio transformer for sound classification and detection,” in ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), 2022...

2022
[52]

Representation Learning with Contrastive Predictive Coding,

A. v. d. Oord, Y . Li, and O. Vinyals, “Representation Learning with Contrastive Predictive Coding,” Jan. 2019, arXiv:1807.03748 [cs, stat]. [Online]. Available: http://arxiv.org/abs/1807.03748

2019 arXiv
[4395]

Available: https://aclanthology.org/ D19-1447

[Online]. Available: https://aclanthology.org/ D19-1447

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

EditGen: Harness- ing Cross Attention Control for Instruction-Based auto-regressive Audio Editing

INTRODUCTION Obtaining satisfactory outcomes in audio manipulation tasks has typically required large datasets with intricate an- notations, a process that is often labor-intensive and costly. Moreover, crafting effective model architectures tailored to the nuances of audio processing requires substantial exper- tise and experimentation. Fine-tuning exist...

work page 2024

[2] [2]

We explore the utilization of a pre-trained frozen auto-regressive transformer model, initially de- signed for generating high-quality music samples from a given text prompt, for audio editing

work page

[3] [3]

arXiv:2507.11096v1 [cs.SD] 15 Jul 2025

We design and implement three distinct audio edit- ing mechanisms, inspired by those presented in [1]. arXiv:2507.11096v1 [cs.SD] 15 Jul 2025

arXiv 2025

[4] [4]

We evaluate our auto-regressive transformer model- based approach against existing diffusion-based methods using automatic music-related metrics and feedback from users

work page

[5] [5]

Yang et al

RELATED WORK Diffusion-based audio generation: Diffusion-based models have been widely explored for generation tasks in the audio and music domains. Yang et al. [2] employ a VQ- V AE model trained on mel-spectrograms to convert them into discrete codes. These codes are then fed into a diffu- sion model to generate audio signals. Make-An-Audio [3] employs ...

work page

[6] [6]

We began with Auffusion, leveraging its existing capabilities for prompt- based editing

CONCLUSION AND FUTURE WORKS In conclusion, we explored using two models for au- dio editing: Auffusion and MUSICGEN. We began with Auffusion, leveraging its existing capabilities for prompt- based editing. Furthermore, we introduced an alterna- tive approach by incorporating MUSICGEN, a pre-trained auto-regressive model known for its advanced capabili- ti...

work page

[7] [7]

Masked autoencoders that listen,

P.-Y . Huang, H. Xu, J. Li, A. Baevski, M. Auli, W. Galuba, F. Metze, and C. Feichtenhofer, “Masked autoencoders that listen,” in Advances in Neural Infor- mation Processing Systems , S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. Curran Associates, Inc., 2022, pp. 28 708– 28 720

work page 2022

[8] [8]

The input, a reference audio random variableX, is encoded into a continuous ten- sor with a lower frame rate ( fr) compared to the sample rate (fs)

METHODOLOGY 3.1 MUSICGEN MUSICGEN [23] employs EnCodec [24], a convolu- tional auto-encoder utilizing Residual Vector Quantization (RVQ) for latent space quantization. The input, a reference audio random variableX, is encoded into a continuous ten- sor with a lower frame rate ( fr) compared to the sample rate (fs). The continuous representation is then qu...

work page