Pith. sign in

REVIEW 3 major objections 6 minor 20 references

RAG-Audio: Retrieval-Augmented Generation for Faithful Brain-to-Audio Reconstruction

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Brain-to-audio reconstruction fails when a frozen generator's prior overwhelms the weak fMRI-derived condition, and starting that generator from a retrieved real-audio exemplar restores stimulus identification to retrieval level while…

desk verdict Quantifies a real failure mode with a plausible fix, but the CLAP-based identification metric leaves the 'faithful' claim under-supported until same-genre and acoustic checks are added. read the letter →

arxiv 2608.09331 v1 pith:IKR4CUCX submitted 2026-08-10 cs.SD cs.AI

Ambuj Mehrish , Sebastiano Vascon This is my paper · ORCID
classification cs.SDcs.AI
keywords brain-to-audioreconstructionpriordominationretrieval-augmentedgenerationexemplaranchoringlatentdiffusionrectifiedflowfMRIdecodingCLAPsemanticaudioembedding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Brain-to-audio reconstruction usually works in two steps: a decoder turns fMRI into a semantic audio embedding, and a frozen pretrained generator turns that embedding into sound. This paper identifies a failure mode it calls prior domination: the generator's learned distribution over plausible audio swamps the weak, noisy brain-derived condition, so the output sounds realistic but the heard clip is almost unrecognizable, around 14–18% 10-way identification against a 10% chance level. The proposed fix, RAG-Audio, retrieves the real audio clip whose embedding is nearest to the decoded one, partially perturbs that exemplar's latent to an intermediate point in the generator's sampling trajectory, and then lets the same frozen generator finish the sample conditioned on the decoded embedding. Across five subjects this restores identification to 40–43%, matching a pure-retrieval baseline, while cutting Fréchet Audio Distance by roughly an order of magnitude for the diffusion and flow generators and keeping the output newly generated rather than a replay. The point matters because it turns a qualitative failure seen in brain-to-image decoding into a quantitative, generator-general account, and offers a tunable faithfulness-versus-novelty knob for any generator that exposes a continuous latent trajectory.

What carries the argument

The central object is exemplar anchoring: a procedure that starts a frozen latent generator not from noise but from the latent code of a retrieved real-audio clip, partially corrupted to an intermediate time. Concretely, for a latent-diffusion generator the retrieved exemplar's latent $z_0$ is noised according to the forward process up to step $t_0 = \lfloor sT \rfloor$ (the SDEdit recipe), and for a rectified-flow generator the latent is interpolated to $\tau_0 = s$ along the flow trajectory; in both cases the reverse or flow-integration process then runs from that point conditioned on the brain-decoded CLAP embedding $\hat{z}$. The anchoring strength $s$ is the knob that trades faithfulness against novelty: small $s$ keeps the exemplar nearly intact and approaches retrieval, while $s \to 1$ erases the exemplar and decays to direct, prior-dominated generation. This continuous-trajectory mechanism is what distinguishes the latent generators, where the intervention works, from the autoregressive MusicGen control, where the exemplar can only enter as melody conditioning and the token prior still dominates.

What would settle it

Run the anchored reconstructions through a forced-choice identification by human listeners, or by an audio model whose training data are disjoint from the stimulus genres, and compare with the CLAP 10-way scores: if human identification of the heard clip does not rise with the CLAP-based gain, the central claim of restored faithful reconstruction fails. Alternatively, anchor from a randomly chosen exemplar at the same strength; if identification stays near retrieval level, the nearest-neighbour content is not carrying the effect.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that stimulus information survives fMRI decoding and is then thrown away by the generator: the contrastive fMRI-to-CLAP decoder reaches 0.43 in 10-way identification, but feeding the same embedding to frozen AudioLDM, TangoFlux, or MusicGen drops identification to 0.14–0.18, near the 0.10 chance level. RAG-Audio counters this prior domination by retrieving the nearest real training exemplar to the decoded embedding, encoding it into the generator's latent space, adding noise or flow interpolation up to an intermediate time $t_0 = sT$ (equivalently $\tau_0 = s$ for the flow generator), and running the frozen reverse process from there with the decoded embedding still as conditioning. With anchoring strength $s \approx 0.25$ to $0.40$, 10-way identification rises to 0.43 for AudioLDM and 0.40 for TangoFlux, matching the 0.40 retrieval baseline; FAD falls from 13.50 to 1.25 for AudioLDM and from 7.89 to 2.36 for TangoFlux. The MusicGen control, which cannot be initialized at an intermediate latent time, gains almost nothing (0.18 to 0.20), which the authors take to show that trajectory initialization, not the presence of a retrieved clip, is the operative mechanism.

Load-bearing premise

The central claim rests on treating 10-way identification in CLAP space as the measure of stimulus faithfulness; if that embedding space, which is also the retrieval bank's geometry and the decoder's training objective, does not track the perceptual or acoustic properties of the heard clip, the reconstruction gains are not established.

Editorial extensions

If this is right

  • At $s \approx 0.25$ to $0.40$, RAG-Audio matches nearest-neighbour retrieval on 10-way identification while emitting a newly generated sample, with novelty about 0.18 versus about 0.06 for verbatim retrieval; the output is an edit of the exemplar, not a copy.
  • Prior domination is shown to be a property of the decode-then-generate recipe rather than of one generator: latent diffusion, rectified flow, and autoregressive generators all drop decoded identification from 0.43 to 0.14–0.18 in direct conditioning.
  • The mechanism only transfers to generators that expose an initializable continuous latent trajectory; autoregressive models need a different intervention, because supplying the same retrieved exemplar as melody conditioning leaves identification essentially unchanged.
  • The anchoring strength controls a monotone faithfulness-novelty trade-off, so downstream uses can select an operating point: low $s$ for fidelity, higher $s$ for diversity, with FAD roughly stable across the working range.
  • Faithfulness is restored only within the support of the memory bank; content outside the training stimuli cannot be anchored, so the method inherits the retrieval bank's coverage limits.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implied test of the paper's framing is whether human listeners, who do not share CLAP's embedding geometry, also identify the heard clip at the higher rate; if they do not, the measured gain is a property of the evaluation space rather than of reconstructed sound.
  • The same trajectory-anchoring logic should apply to brain-to-image latent diffusion decoders, where realistic-but-drifting reconstructions have been reported qualitatively; a 10-way identification metric in an image embedding space would turn that qualitative pattern into the same quantitative prior-domination curve.
  • The memory-bank ablation reported in the paper is an honest negative result: anchoring's benefit does not widen when the bank shrinks, which suggests the gain tracks retrieval quality rather than scarcity. A stronger test of generalization would hold out entire genres and check whether moderate $s$ still reconstructs them from less similar exemplars.
  • Because CLAP is a joint audio-language embedding, one could add a textual description of the decoded embedding as a second condition and use exemplar anchoring to reconcile a weak semantic condition with a strong prior; this would test the mechanism outside brain decoding, in ordinary text-to-audio generation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces RAG-Audio, a retrieval-augmented pipeline for fMRI-to-audio reconstruction. A contrastive decoder maps fMRI to a CLAP embedding; the nearest training exemplar is retrieved in CLAP space; and a frozen diffusion or flow generator is initialized at an intermediate time from that exemplar while conditioned on the decoded embedding. On the Brain2Music dataset, the authors report that 10-way stimulus identification rises from 0.14–0.18 for direct generation to 0.40–0.43 for RAG-Audio, matching the retrieval baseline of about 0.40, and that FAD drops by roughly an order of magnitude for AudioLDM and TangoFlux. An autoregressive MusicGen control shows only a small gain, which the authors attribute to the absence of an initializable latent trajectory. The paper concludes that retrieval-guided trajectory initialization mitigates prior domination.

Significance. If the central claim holds, the paper makes a useful contribution: it provides a quantitative, generator-general characterization of a failure mode that has largely been qualitative in brain-to-image work, and it proposes a simple, well-motivated intervention that trades faithfulness against novelty in a controllable way. The single-harness comparison across three generator families, the per-subject decoding analysis, the auditory-cortex localization check, and the honest memory-bank negative result are strengths. The main risk is that the evaluation metric and the retrieval mechanism share the same CLAP cosine geometry, so the headline identification gains may partly reflect inherited genre-level or retrieval-level similarity rather than stimulus-level reconstruction. This concern is acknowledged in the Limitations but is not resolved by the current experiments.

major comments (3)
  1. [§3, Eq. (4); Appendix G] The identification metric is defined as CLAP cosine similarity, and every stage of the pipeline uses the same geometry: the decoder is trained with InfoNCE on CLAP embeddings (Eq. 6), retrieval selects the nearest CLAP embedding (Eq. 7), and the evaluation is CLAP cosine. Because RAG-Audio initializes from the retrieved exemplar, its output inherits the exemplar's CLAP neighborhood. The paper's own Appendix G reports top-1 genre accuracy of only 0.33 (3.3× chance) for the RAG arm, so a system that recovers genre-level information could already score well above chance on a 10-way identification task if distractors are drawn across genres. The authors state in Limitations that Eq. (4) measures semantic agreement in CLAP space rather than sample-accurate waveform recovery, but this does not address the more specific risk that the reported 0.40–0.43 identification is carried by coarse genre/timbre similarity. I ask the authors to report identification with same-genre distractors (e.g., distractor sets matched on predicted or true genre) and to add at least one acoustic-level measure (spectrogram correlation, onset alignment, or similar) that does not depend on CLAP cosine, in order to substantiate the claim of faithful reconstruction rather than genre-level matching.
  2. [§5, Implementation; §6, Table 7] The anchoring strength s is a free parameter, and the reported headline numbers use the working range s≈0.25–0.40, with the best AudioLDM result at s=0.2 (0.49 identification) and TangoFlux at s=0.4. It is not stated whether s was selected on the held-out test set or on a separate validation split. If s was chosen on the same 300 test generations used to report accuracy, the reported values are optimistic and the comparison to retrieval (which has no such tuned parameter) is not on equal footing. In addition, no confidence intervals or significance tests are provided; with n=300 generations, the difference between 0.14 and 0.43 is large, but the difference between 0.40 (RAG TangoFlux) and 0.43 (AudioLDM RAG) and the difference between RAG and retrieval (0.40 vs 0.40–0.43) need uncertainty quantification to support the claim that RAG 'matches' retrieval rather than merely being statistically indistinguishable under high variance. Please clarify the selection procedure and report per-subject or bootstrap intervals for the central comparisons.
  3. [§4.3, §6, Mechanism Control] The conclusion that trajectory initialization, rather than retrieval alone, is the operative mechanism rests entirely on the MusicGen negative control. MusicGen is autoregressive, so the retrieved exemplar can only enter through melody conditioning, which is a different conditioning interface from the latent-space initialization used for AudioLDM and TangoFlux. The comparison therefore varies two things at once: the generator family and the way the exemplar is injected. A cleaner control would use a latent-diffusion or flow generator and ablate the initialization: condition on the retrieved audio's embedding (or concatenate it to the decoded embedding) without starting the trajectory from the exemplar. Without such an ablation, the claim that the improvement is not attributable to the mere availability of the retrieved exemplar is under-supported, although the current evidence is suggestive.
minor comments (6)
  1. [Abstract, Table 1, Table 5] The FAD value for direct AudioLDM is reported as 13.49 in the abstract and Table 5 but 13.5 in Table 1; please make these consistent.
  2. [Appendix A] The reproducibility section states that the full codebase is released in a public repository, but the repository field is 'TBA'; please provide the actual URL or state the reason for the delay.
  3. [Appendix F and Appendix H] There are two objects numbered 'Table 8' with different content: one for memory-bank size ablation and one for the SDEdit strength sweep. Please renumber the tables consistently and ensure the in-text references point to the correct table.
  4. [Table 6] The novelty definition contains a typo: '1−costo the nearest training clip' should read '1 − cosine similarity to the nearest training clip'.
  5. [§2, Related Work] The second and third paragraphs of Section 2 contain substantially overlapping text about brain-to-image reconstruction and prior domination; please consolidate to avoid redundancy.
  6. [Figure 7] The caption says 'cyan lines are stimulus onsets (shared per row)', but it is not clear from the figure how the shared onsets are computed or why they are shared; please clarify in the caption or the figure itself.

Circularity Check

0 steps flagged · score 2.0 of 10

No load-bearing circularity; the CLAP-geometry overlap is an acknowledged evaluation limitation, and the only author-overlapping citation (TangoFlux) is not load-bearing.

full rationale

The paper's derivation is self-contained: the decoder is trained with InfoNCE (Eq. 6), retrieval is defined by CLAP cosine (Eq. 7), the generator is initialized from the retrieved exemplar via SDEdit/flow interpolation (Eqs. 9 and 11), and identification is evaluated by CLAP cosine (Eq. 4). These are explicit design choices rather than hidden reductions. The author-overlapping citation, TangoFlux (Hung et al., 2024), is used only as a frozen public generator checkpoint and is not load-bearing for the core argument. The closest thing to a circularity concern is that the training objective, retrieval mechanism, and evaluation metric all live in the same CLAP cosine geometry, so the identification gain of RAG-Audio is partly inherited from the retrieval exemplar by construction. The paper acknowledges this in Limitations: 'Our N-way identification measures semantic agreement in CLAP space rather than sample-accurate waveform recovery.' Appendix G's top-1 genre accuracy of 0.33 further indicates that CLAP identification may reflect genre-level similarity rather than full stimulus recovery. However, this is a limitation on what the metric proves about faithfulness, not a circular derivation: the headline comparisons (direct versus RAG, the MusicGen negative control, and the FAD reductions) are empirical results on held-out test stimuli against external benchmarks. Under the required standard of a specific equation-level reduction or a fitted parameter renamed as a prediction, no circular step is established.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on domain assumptions about CLAP-space evaluation, voxel selection, and latent-space preservation, plus one ad hoc assumption specific to the MusicGen control. The anchoring strength s is the main free parameter and is selected using test data. The paper introduces no new physical or neural entities.

free parameters (5)
  • Anchoring strength s = 0.25 for AudioLDM headline, 0.4 for TangoFlux, grid swept from 0.15 to 1.0
    Controls how much exemplar structure is preserved at the start of generation; the working range s≈0.25-0.40 is selected using test-set accuracy in Table 7 without a described validation split.
  • Number of selected voxels K = 2000
    Chosen by hand in the voxel-selection step (Eq. 5); determines the decoder input dimensionality and the localization map.
  • Hemodynamic lag l = 3 TRs (4.5 s)
    Chosen by hand in preprocessing to align fMRI responses to the audio stimulus; affects which temporal slice of BOLD activity is decoded.
  • Temporal window w = 8 TRs
    Chosen by hand in preprocessing; the window over which voxel responses are aggregated before ranking and decoding.
  • InfoNCE temperature tau = 0.07
    Standard contrastive temperature used in the decoder training objective (Eq. 6); assumed rather than tuned against a validation set.
assumptions (5)
  • domain assumption CLAP cosine similarity is a valid proxy for semantic stimulus identity in the N-way identification metric (Eq. 4).
    The headline faithfulness result is measured entirely in CLAP space, and the paper concedes in Limitations that this is not sample-accurate waveform recovery.
  • domain assumption BOLD responses at lag 3 TRs, window 8 TRs, and the top 2000 voxels by max correlation contain usable stimulus information.
    The entire decoding pipeline relies on these preprocessing choices (Section 4.1, Eq. 5); there is no ablation showing they are optimal or sufficient.
  • domain assumption SDEdit-style partial noising and rectified-flow interpolation preserve the retrieved exemplar's stimulus-relevant structure in the frozen generator's latent space.
    The method's central mechanism (Eq. 9 and Eq. 11) assumes that initializing from a partially corrupted exemplar latent keeps enough structure to guide the output toward the stimulus.
  • domain assumption Pretrained generators AudioLDM, TangoFlux, and MusicGen encode strong priors that dominate weak brain-derived conditions.
    The prior-domination framing in Section 3 is the motivation for the method; the paper provides evidence for it, but the framing is assumed as the correct model of the failure.
  • ad hoc to paper MusicGen's melody conditioning is a fair way to inject the retrieved exemplar, making it a valid negative control for trajectory initialization.
    The mechanism attribution depends on comparing MusicGen's conditioning pathway with diffusion and flow trajectory initialization, but the two injection mechanisms are not matched in strength or architecture.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RAG-Audio: Retrieval-Augmented Generation for Faithful Brain-to-Audio Reconstruction." pith.science (2026). https://pith.science/paper/IKR4CUCX

@misc{pith2026260809331,
  author       = {Pith},
  title        = {Pith review of: RAG-Audio: Retrieval-Augmented Generation for Faithful Brain-to-Audio Reconstruction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IKR4CUCX}},
  note         = {Machine review of arXiv:2608.09331}
}
abstract

Brain-to-audio reconstruction is limited by \emph{prior domination}: when a pretrained generator is conditioned on a weak neural signal, it produces realistic but stimulus-inaccurate audio. We introduce RAG-Audio, which decodes fMRI into a semantic audio embedding, retrieves a matching real-audio exemplar, and initializes the frozen generator's sampling trajectory from that exemplar while retaining the decoded embedding as conditioning. On Brain2Music, RAG-Audio improves 10-way stimulus identification from $0.14$--$0.18$ for direct generation, near the $0.10$ chance level, to $0.40$--$0.43$, comparable to retrieval. It also reduces Fr\'echet Audio Distance by roughly an order of magnitude, from $13.49$ to $1.25$ for AudioLDM. RAG-Audio approaches nearest-neighbor retrieval in identification while remaining generative; its higher FAD is expected because retrieval directly replays real audio. An autoregressive negative control, which lacks an initializable latent trajectory, shows no comparable gain, attributing the improvement to trajectory initialization. These results suggest that retrieval-guided initialization can mitigate prior domination in brain-to-audio generation.

Figures

Figures reproduced from arXiv: 2608.09331 by the authors.

Figure 1
Figure 1. Overview of RAG-Audio and two baselines. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The RAG-Audio pipeline. The decoder 𝑓𝜃 maps an fMRI response 𝑥 to a CLAP embedding 𝑧ˆ; the nearest exemplar 𝑎 ∗ is retrieved and encoded to 𝑧0, perturbed to an intermediate time 𝑡0 = 𝑠𝑇 (𝑧𝑡0 ), and the frozen generator denoises from 𝑧𝑡0 conditioned on 𝑧ˆ to produce 𝑎ˆ. The strength 𝑠 trades faithfulness against novelty. 3 Problem Setup Let 𝑎 denote an audio clip heard by a subject while functional magnetic resonance… view at source ↗
Figure 3
Figure 3. Decoder localization: voxel relevance scores (red) overlap the Harvard–Oxford auditory-cortex mask (blue), with 78% of top voxels in bilateral superior temporal gyrus. drawn from the ten GTZAN genres (Tzanetakis and Cook, 2002), with a repetition time of TR = 1.5,s. We follow the official subject-wise split of 480 training clips and 60 held-out test clips. All metrics are averaged across subjects, corresponding to a… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Ten-way identification versus anchoring strength 𝑠: low 𝑠 stays near the retrieval bound, high 𝑠 decays to the 0.10 chance level; working range 𝑠≈0.25–0.40. temperature 𝜏 = 0.07. Complete preprocessing, optimization, and generation settings are provided in Appendix C. …
Figure 5
Figure 5. Figure 5: Ten-way identification and FAD per generator: anchoring restores AudioLDM and TangoFlux to retrieval level but not autoregressive MusicGen. 9 [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Faithfulness (identification) versus novelty: direct is novel but unfaithful, retrieval faithful but verbatim, RAG-Audio faithful and generative. scores concentrate in the bilateral superior temporal gyrus, a region associated with auditory processing. Under the Harvar…
Figure 7
Figure 7. Figure 7: Per-genre reconstructions: stimulus, RAG-Audio, and direct generation (log-mel spectrograms). Cyan lines are stimulus onsets (shared per row); the corner box is CLAP similarity to the stimulus (green/orange/red). Direct generation is realistic but off-grid and low-simi…
Figure 8
Figure 8. Figure 8: Decoder 10-way identification (5-subj mean) [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Genre confusion for the RAG arm (TangoFlux). H SDEdit Strength Sweep [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 8 canonical work pages

  1. [4]

    R&B -- Rhythm and Brain: Cross-subject Decoding of Music from Human Brain Activity

    Matteo Ferrante, Matteo Ciferri, and Nicola Toschi. R&b–rhythm and brain: Cross-subject decoding of music from human brain activity.arXiv preprint arXiv:2406.15537,

    work page Pith review arXiv

  2. [5]

    All runs use a single GPU per job; we release seeds, code, and configurations

    Auditory-cortex overlap is computed against the Harvard-Oxford atlas (Desikan et al., 2006). All runs use a single GPU per job; we release seeds, code, and configurations. 22

    work page 2006

  3. [10]

    Flow matching for generative modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. InThe eleventh international conference on learning representations. Che Liu, Changde Du, Xiaoyu Chen, and Huiguang He. Reverse the auditory processing pathway: Coarse-to-fine audio reconstruction from fmri.arXiv preprint arXiv:2405.18726,

    arXiv

  4. [12]

    Sound reconstruction from human brain activity via a generative model with brain-like auditory features.arXiv preprint arXiv:2306.11629,

    Jong-Yun Park, Mitsuaki Tsukamoto, Misato Tanaka, and Yukiyasu Kamitani. Sound reconstruction from human brain activity via a generative model with brain-like auditory features.arXiv preprint arXiv:2306.11629,

    arXiv

  5. [13]

    Naturalistic music decoding from eeg data via latent diffusion models

    16 RAG-Audio: Faithful Brain-to-Audio Reconstruction Preprint Emilian Postolache, Natalia Polouliakh, Hiroaki Kitano, Akima Connelly, Emanuele Rodolà, Luca Cosmo, and Taketo Akama. Naturalistic music decoding from eeg data via latent diffusion models. InICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pag...

    work page 2025

  6. [14]

    Semantic image inversion and editing using rectified stochastic differential equations.arXiv preprint arXiv:2410.10792,

    Litu Rout, Yujia Chen, Nataniel Ruiz, Constantine Caramanis, Sanjay Shakkottai, and Wen-Sheng Chu. Semantic image inversion and editing using rectified stochastic differential equations.arXiv preprint arXiv:2410.10792,

    arXiv

  7. [15]

    Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502, 2020a

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502, 2020a. Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations.arXiv preprint arXiv:2011.13456, 2020b. YuTakagiandShinjiNish...

    arXiv 2010

  8. [17]

    InICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 581–585

    YiYuan,HaoheLiu,XuboLiu,QiushiHuang,MarkDPlumbley,andWenwuWang.Retrieval-augmented text-to-audio generation. InICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 581–585. IEEE,

    work page 2024

Show all 20 references
  1. [18]

    Voxels are selected by the relevance score of Eq

    at temperature𝜏=0.07 , using the Adam optimizer with weight decay; full layer widths, learning rate, and batch size are given in our released configuration. Voxels are selected by the relevance score of Eq. 5 (𝐾=2000 ), with a haemodynamic lag ofℓ=3 TRs and a temporal window o...

    2000

  2. [19]

    genres (metal, rock, blues) that are confusable even for audio-only classifiers. The confusions are musically coherent (e.g., disco↔pop↔reggae, rock↔disco), indicating that reconstruction errors preserve broad timbral and rhythmic structure rather than being random. Figure 9:G...

    2023

  3. [2002]

    Large- scalecontrastivelanguage-audiopretrainingwithfeaturefusionandkeyword-to-captionaugmentation

    Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large- scalecontrastivelanguage-audiopretrainingwithfeaturefusionandkeyword-to-captionaugmentation. InICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Proces...

    2023

  4. [2015]

    Representation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748,

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748,

    arXiv

  5. [2017]

    Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598,

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598,

    arXiv

  6. [2018]

    Generalization through memorization: Nearest neighbor language models.arXiv preprint arXiv:1911.00172,

    Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models.arXiv preprint arXiv:1911.00172,

    1911 arXiv

  7. [2019]

    A reproducibleevaluationofantssimilaritymetricperformanceinbrainimageregistration.Neuroimage, 54(3):2033–2044,

    Brian B Avants, Nicholas J Tustison, Gang Song, Philip A Cook, Arno Klein, and James C Gee. A reproducibleevaluationofantssimilaritymetricperformanceinbrainimageregistration.Neuroimage, 54(3):2033–2044,

  8. [2020]

    Mulan: A joint embedding of music audio and natural language.arXiv preprint arXiv:2208.12415,

    Qingqing Huang, Aren Jansen, Joonseok Lee, Ravi Ganti, Judith Yue Li, and Daniel PW Ellis. Mulan: A joint embedding of music audio and natural language.arXiv preprint arXiv:2208.12415,

    arXiv

  9. [2021]

    Matteo Ciferri, Matteo Ferrante, and Nicola Toschi

    arXiv:2108.02938. Matteo Ciferri, Matteo Ferrante, and Nicola Toschi. Reconstructing music perception from brain activity using a prior guided diffusion model.Scientific Reports, 15(1):42108,

    arXiv

  10. [2022]

    Tangoflux: Super fast and faithful text to audio generation with flow matching and clap-ranked preference optimization.arXiv preprint arXiv:2412.21037,

    Chia-Yu Hung, Navonil Majumder, Zhifeng Kong, Ambuj Mehrish, Amir Ali Bagherzadeh, Chuan Li, Rafael Valle, Bryan Catanzaro, and Soujanya Poria. Tangoflux: Super fast and faithful text to audio generation with flow matching and clap-ranked preference optimization.arXiv preprint...

    arXiv

  11. [2023]

    Semi-parametric neural image synthesis.arXiv preprint arXiv:2204.11824,

    Andreas Blattmann, Robin Rombach, Kaan Oktay, Jonas Müller, and Björn Ommer. Semi-parametric neural image synthesis.arXiv preprint arXiv:2204.11824,

    arXiv

  12. [2024]

    Neuropictor: Refining fmri-to-image reconstruction via multi-individual pretraining and multi-level modulation.arXiv preprint arXiv:2403.18211,

    Jingyang Huo, Yikai Wang, Xuelin Qian, Yun Wang, Chong Li, Jianfeng Feng, and Yanwei Fu. Neuropictor: Refining fmri-to-image reconstruction via multi-individual pretraining and multi-level modulation.arXiv preprint arXiv:2403.18211,

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.