{"id":"aecadf59-efde-427f-9a22-1d1e39be4a32","arxiv_id":"2501.12385","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"An exemplar-based analogy model that manipulates audio textures by learning transformations from paired before-and-after clips, trained self-supervised on a synthetic quadruplet dataset.","lead":"This paper trains a latent diffusion model to edit audio by example: given a pair of before-and-after sound clips, it applies the same change to a new clip. The method, tested on adding, removing, and replacing ambient sounds, beats a text-prompt baseline in human listening tests and generalizes to non-speech audio.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The model may be performing class-conditional BBC-texture transfer via CLAP labels rather than learning content-invariant exemplar analogies; current evidence does not rule this out, so the central OOD generalization claim is unsupported.","rationale":"The reader's weakest assumption (content invariance) is the right general area, but the sharper formulation is representational: CLAP embeddings are semantic, and the training distribution only requires a categorical mapping. The paper's own Sec. II-A assertion that the model 'tailors its output according to the transformation implied by the exemplar pair' is an empirical claim for which the held-out-class test is the minimal decisive experiment. Table I's in-distribution metrics cannot distinguish analogy from label transfer, and Fig. 4 is not a statistical test. I would keep the paper's contribution as plausible but mark the central claim unverified until this experiment is reported; if it passes, the concern is resolved and the paper deserves acceptance.","tokens_in":9430,"tokens_out":9439,"duration_ms":106782,"concrete_test":"Leave-one-class-out experiment: hold out all BBC SFX segments belonging to several classes (e.g., bird, car, electrical hum, street); train Fθ without them. Build test quadruplets using only held-out classes, with oracle targets formed by mixing the same held-out texture into speech. Compare FAD/LSD/SI-SDR between Fθ outputs and oracle targets for held-out versus in-distribution classes. If held-out performance collapses while in-distribution remains strong, the model has memorized BBC texture classes rather than inferring transformations from exemplar pairs; if held-out performance is comparable, the content-invariance assumption survives. Also run the same protocol on at least 50 non-speech query inputs with objective oracle metrics instead of a single qualitative figure.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"Training quadruplets (Sec. III-A) are built exclusively from the BBC SFX texture inventory: the exemplar output a2_e is always speech plus one of a fixed set of ambient classes. Since the exemplar pair is encoded with a CLAP audio encoder that is pretrained on labeled sound events and fine-tuned here, Fθ can in principle solve the pretext task by reading the texture class from Ea(a2_e) and applying that class to zq, without ever computing a transformation from the pair. All positive evidence is consistent with this shortcut: Table I uses only BBC SFX textures, Table III uses CityWalk with familiar sound types, and the non-speech claim rests on one qualitative example (Fig. 4). The load-bearing assumption—that the exemplar pair communicates a content-invariant transformation beyond a semantic label—is therefore untested. This is not an internal inconsistency, but a correctness risk: if the model is only a class-conditional editor, the headline claims about exemplar analogy and OOD generalization fail.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an exemplar-based audio texture manipulation model. Given a pair of exemplar audio clips (a1_e, a2_e) that define a transformation such as adding, removing, or replacing an ambient texture, the model Fθ(a_q, a1_e, a2_e) is trained with a latent diffusion model to apply the same transformation to a new input audio a_q. Training uses self-supervised quadruplets built from LibriSpeech/VCTK speech and BBC SFX ambient textures. The authors compare against a re-implemented AUDIT baseline and MP-SENet, evaluate with objective metrics (FAD, FD, KL, LSD, IS, PESQ, STOI) and a human rating study, and report ablations of a learnable positional encoding plus qualitative generalization to real-world and non-speech inputs.","tokens_in":9644,"tokens_out":6723,"duration_ms":61278,"significance":"If the central claim holds, the paper offers a text-free interface for audio editing and a self-supervised training scheme that avoids manual annotation. The method is clearly described, the architecture is sensible, and the human evaluation with confidence intervals is a strength, as is the honest limitations section. However, the current experimental design does not rule out a class-conditional shortcut in which the model reads only the texture label from the exemplar output, so the main 'analogy' claim and the out-of-distribution generalization claims need direct verification before the contribution is established.","major_comments":[{"comment":"The self-supervised task admits a shortcut: because the exemplar output a2_e and the target ao always contain the same ambient texture class, and because the condition is encoded with a fine-tuned CLAP encoder pretrained on labeled sound events, the model can minimize the denoising loss by extracting a texture-class embedding from E_a(a2_e) and applying it to z_q while ignoring E_a(a1_e). This would make Fθ a class-conditional texture editor rather than an exemplar-analogy model, and the claim in Section II-A that 'the model learns to tailor its output according to the transformation implied by the exemplar pair' would not be established. Please add (i) an ablation that conditions only on a2_e, (ii) a mismatch test where a1_e and a2_e are unrelated textures, and (iii) an evaluation on held-out texture classes not present in BBC SFX. If the model performs equally well without a1_e, the analogy framing should be revised.","section":"II-A and III-A, Eq. (1)"},{"comment":"The objective results are reported without error bars, confidence intervals, or significance tests, and the pattern is not uniformly in the proposed model's favor (e.g., Addition 0→1 FAD is 5.58 for Ours vs 4.77 for AUDIT, while the +PE variant is 3.83; Replacement 1→1 FAD is 6.58 for Ours vs 4.94 for AUDIT). The statement in Section III-B that the model 'consistently outperforms AUDIT' in addition and removal is therefore stronger than the table supports. Please report standard errors or confidence intervals across seeds/test splits and state the number of evaluation samples.","section":"III-B, Table I"},{"comment":"Because AUDIT is not open source, the comparison rests on a re-implementation, but the paper provides no fidelity check against the original model, such as reproducing AUDIT's published results on a standard prompt-based editing benchmark. If the re-implementation is much weaker than the original, the comparative claims in Tables I and III would be misleading. Please provide a sanity check, for example by comparing the re-implementation's performance on a public benchmark with the published numbers, or making the re-implementation available with evaluation code.","section":"III-A (Baselines)"},{"comment":"The claimed generalization to non-speech inputs is supported by a single qualitative example (dog barking), and the City Walk evaluation in Table III uses familiar sound types that overlap with the BBC SFX texture inventory. Since out-of-distribution robustness is one of the paper's headline claims, this evidence is thin. Please add a quantitative evaluation on non-speech inputs and/or on texture classes that were not seen during training, with objective metrics analogous to Table I.","section":"III-E and Fig. 4"}],"minor_comments":[{"comment":"The sentence 'The model takes as input the encoded latent of the target audio, z0 = Enc(ao)' should clarify that this is the training target used to compute the loss in Eq. (1); at test time no target is available, so the wording is confusing.","section":"II-B"},{"comment":"The expectation subscript 'Ez0,a1e,a2e,ϵ∼N (0,I),t' should specify the distribution of the timestep t and be typeset properly, e.g., E_{z0,a1_e,a2_e,ϵ∼N(0,I),t∼U{1,...,N}}.","section":"Eq. (1)"},{"comment":"There is a typo 'PSEQ' (should be 'PESQ'), and the text should clarify whether the noisy and MP-SENet baselines in Table II were also processed through the HiFi-GAN vocoder so that the comparison is consistent.","section":"III-D"},{"comment":"The motivating sparrow/crow example is not evaluated in the experiments; if it is meant to illustrate a capability, please connect it to the tested tasks or acknowledge that such fine-grained source separation is outside the current scope.","section":"I"},{"comment":"It would help to show the exemplar input as well as the exemplar output for the non-speech example, so the reader can verify the transformation direction.","section":"Fig. 4"},{"comment":"The 'Ours + PE' row is the full proposed model; the caption should state this explicitly and describe what 'Ours' (without PE) represents.","section":"Table I"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to be of interest to the audio editing community, but the core claim of exemplar analogy needs a direct test of whether the exemplar input is actually used. If the class-conditional shortcut is confirmed, the contribution reduces to class-conditional editing, which would weaken the novelty. I recommend the authors run the ablations proposed in my major comments before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on arXiv:2501.12385. The paper has a fresh idea: specify audio texture edits with before/after exemplar pairs, learned via a self-supervised quadruplet task in a latent diffusion model. That's genuinely new, and the learnable positional encoding to disambiguate 'before' from 'after' is a nice touch. I also give them credit for running a 30-person perceptual study with confidence intervals and for openly reporting where they underperform MP-SENet on speech enhancement.\n\nThe soft spots are real. Table 1 reports objective metrics with no error bars or significance tests, so I can't tell if the differences over AUDIT are meaningful. AUDIT is re-implemented with no fidelity check. No code or data is released, and the vocoder processing for the main experiments isn't documented. Those are fixable reproducibility issues.\n\nThe larger concern is the one the stress-test raises: the model may be doing class-conditional texture transfer via CLAP embeddings rather than true exemplar analogy. Training textures all come from the BBC SFX inventory, and CLAP is pretrained on labeled sound events. So the model could solve the pretext task by reading the texture class from the exemplar embedding and applying that class to the input. All the positive evidence is consistent with that shortcut: Table I uses only BBC textures, Table III uses familiar urban sound types, and the non-speech claim rests on one qualitative example in Fig. 4. The paper's motivating case — remove the crow but keep the sparrow — is exactly what a class-conditional model cannot do, and it's never tested. So the central claim that the model learns a content-invariant transformation from the exemplar pair is unsupported by the current evidence.\n\nThat said, this is not a fatal incoherence. It's a load-bearing plausibility gap that better experiments can close: test with novel texture classes, test within-class distinctions, compare exemplar embeddings to actual transformation, release the code. The idea is good enough that I'd send it to peer review, but it needs a revision where the authors actively rule out the shortcut.","headline":"A genuinely novel exemplar-based audio editing approach, but the evidence doesn't yet rule out class-conditional shortcutting via CLAP, so the central analogy claim hangs on untested assumptions.","tokens_in":10145,"tokens_out":4373,"would_cite":true,"duration_ms":42819,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a pair of before-and-after audio clips can define an edit—adding, removing, or replacing a sound texture—and that a latent diffusion model trained only on synthetic quadruplets learns to apply that same edit to any…","keywords":["audio texture manipulation","exemplar-based analogy","latent diffusion models","self-supervised learning","audio editing","sound texture","CLAP embeddings","speech enhancement"],"falsifier":"Give the model an exemplar pair whose before-and-after change is only a texture (for instance, removing traffic noise from speech) but apply it to a query whose texture class was not seen in training, such as music, and objectively measure whether the target texture is removed; if the model fails on such out-of-domain content, the content-invariance claim is false.","tokens_in":9204,"feed_emoji":"🎧","tokens_out":5048,"duration_ms":48623,"temperature":0.7,"pith_summary":"The paper tries to establish that audio editing operations can be specified by example rather than by text. A model is given two exemplar clips—one before the edit and one after—and learns to apply the same transformation to a new input sound. If true, users could say 'make my recording sound like this pair' instead of struggling to describe edits in words, avoiding ambiguous prompts that cannot distinguish, say, removing a crow from removing all birds. The authors train a latent diffusion model self-supervised on synthetic quadruplets of speech plus ambient textures, and report that it outperforms a text-conditioned baseline and generalizes to non-speech and real-world audio.","feed_headline":"Two audio clips teach a model any sound edit","feed_subtitle":"Before-and-after audio pairs train a diffusion model to add, remove, or replace sounds without text.","key_machinery":"The load-bearing object is the self-supervised quadruplet $(a^1_e, a^2_e, a_q, a_o)$ together with the conditional latent diffusion model $F_\\theta$. Training simulates edits by mixing ambient sounds from BBC SFX into clean speech from LibriSpeech and VCTK, so the model never sees a human label and must recover the operation from the exemplar pair alone. The exemplar pair is represented by CLAP embeddings with a learnable positional encoding to mark order, and classifier-free guidance sharpens the conditional output. A pretrained VAE and HiFi-GAN vocoder from AudioLDM encode and reconstruct mel-spectrograms, with the diffusion model operating in the compressed latent space.","core_discovery":"The central claim is that a function $F_\\theta(a_q, a^1_e, a^2_e)$, trained solely on unlabeled quadruplets, learns to infer the transformation implied by an exemplar pair $(a^1_e, a^2_e)$ and apply it to a new input $a_q$. The exemplar pair is encoded with a CLAP audio encoder, marked with a learnable positional encoding to distinguish the 'before' from the 'after' clip, and fed into a latent diffusion model through cross-attention. At test time a completely different exemplar pair can be substituted, and the model tailors its output to that pair. The paper reports quantitative and perceptual results supporting this, including better performance than the text-conditioned AUDIT baseline on addition and removal tasks, and qualitative generalization to dog barking and CityWalk recordings.","pith_inferences":["The method could be extended to learn from unlabeled internet audio by mining natural before-and-after pairs, such as a recording with and without a door closing, removing the need for synthetic mixing.","Because conditioning relies on CLAP embeddings, which align audio with language, the model may be limited to textures that have semantic labels; low-level or unnamed textures might not transfer as reliably.","A natural next test is whether the model truly extracts an operation from the exemplar pair or instead copies the exemplar output's texture onto the input; ablating with mismatched content in the pair would separate these possibilities.","Exemplar-based analogy could combine with text prompts, letting users refine an ambiguous instruction by demonstrating the intended edit."],"forward_implications":["Users can specify edits such as 'remove only the crow, keep the sparrow' by supplying one before-and-after clip pair, without writing a text prompt.","The same training scheme covers three operation types—adding, removing, and replacing sound textures—so a single model handles multiple editing tasks.","Because the model is self-supervised on synthetic quadruplets, it avoids reliance on human annotations, which can be noisy or mislabeled.","The learnable positional encoding is necessary: without it the model cannot reliably tell which exemplar is 'before' and which is 'after', and performance drops.","Reported generalization to non-speech and real-world recordings suggests the learned operation is not tied to the synthetic training domain."],"supporting_citations":[{"why":"AUDIT is the text-conditioned latent diffusion baseline that the paper must match or beat across editing tasks.","marker":"[4]"},{"why":"AudioLDM supplies the pretrained VAE encoder-decoder and HiFi-GAN vocoder used to compress and reconstruct mel-spectrograms.","marker":"[24]"},{"why":"CLAP audio encoders extract the exemplar pair embeddings that condition the diffusion model.","marker":"[25]"},{"why":"Latent diffusion models provide the generative framework for manipulating audio in the compressed latent space.","marker":"[18]"},{"why":"Classifier-free guidance is applied to trade off diversity and quality in the transformed output.","marker":"[28]"},{"why":"LibriSpeech supplies a 100-hour subset of clean speech used for training and evaluation quadruplets.","marker":"[16]"},{"why":"VCTK supplies 50 hours of accented English speech used as the other source of clean speech for quadruplet construction.","marker":"[17]"},{"why":"BBC SFX provides the ambient sound textures mixed into speech to simulate add, remove, and replace operations.","marker":"[6]"},{"why":"The CityWalk dataset is used to test generalization to real-world, in-the-wild audio.","marker":"[12]"}],"fun_headline_variants":["Sound edits from example audio pairs, no text","Before-and-after clips teach AI to modify any sound","Exemplar audio pairs replace text for sound editing","Audio analogy: learn edits from paired examples","Diffusion model applies audio transformations from pairs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the transformation implied by an exemplar pair transfers to any new input regardless of content, so the model need not separate the texture change from the speech content of the exemplars.","fun_headline_variants_meta":{"raw":{"variants":["Sound edits from example audio pairs, no text","Before-and-after clips teach AI to modify any sound","Exemplar audio pairs replace text for sound editing","Audio analogy: learn edits from paired examples","Diffusion model applies audio transformations from pairs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00017,"raw_usage":{"total_tokens":1219,"prompt_tokens":847,"completion_tokens":372,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":463,"completion_tokens_details":{"reasoning_tokens":301}},"tokens_in":463,"tokens_out":372,"duration_ms":4133,"temperature":1.0,"reasoning_tokens":301,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:12:21.569732+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give the model an exemplar pair whose before-and-after change is only a texture (for instance, removing traffic noise from speech) but apply it to a query whose texture class was not seen in training, such as music, and objectively measure whether the target texture is removed; if the model fails on such out-of-domain content, the content-invariance claim is false.","supporting_citations":[{"cited_title":"Audit: Audio editing by following instructions with latent diffusion models,","cited_arxiv_id":null,"evidence_quote":"AUDIT is the text-conditioned latent diffusion baseline that the paper must match or beat across editing tasks."},{"cited_title":"Clap learning audio concepts from natural language supervision,","cited_arxiv_id":null,"evidence_quote":"CLAP audio encoders extract the exemplar pair embeddings that condition the diffusion model."},{"cited_title":"Bbc sound effects,","cited_arxiv_id":null,"evidence_quote":"BBC SFX provides the ambient sound textures mixed into speech to simulate add, remove, and replace operations."},{"cited_title":"Self- supervised audio-visual soundscape stylization,","cited_arxiv_id":null,"evidence_quote":"The CityWalk dataset is used to test generalization to real-world, in-the-wild audio."}],"review_version":1}