Pith. sign in

REVIEW 10 cited by

Taming Visually Guided Sound Generation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2110.08791 v1 pith:DFH6ZZ5E submitted 2021-10-17 cs.CV cs.AIcs.LGcs.SDeess.AS

Vladimir Iashin , Esa Rahtu This is my paper · ORCID
classification cs.CVcs.AIcs.LGcs.SDeess.AS
keywords generatedgenerationmetricsmodelsamplessamplingspectrogramaudio
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Recent advances in visually-induced audio generation are based on sampling short, low-fidelity, and one-class sounds. Moreover, sampling 1 second of audio from the state-of-the-art model takes minutes on a high-end GPU. In this work, we propose a single model capable of generating visually relevant, high-fidelity sounds prompted with a set of frames from open-domain videos in less time than it takes to play it on a single GPU. We train a transformer to sample a new spectrogram from the pre-trained spectrogram codebook given the set of video features. The codebook is obtained using a variant of VQGAN trained to produce a compact sampling space with a novel spectrogram-based perceptual loss. The generated spectrogram is transformed into a waveform using a window-based GAN that significantly speeds up generation. Considering the lack of metrics for automatic evaluation of generated spectrograms, we also build a family of metrics called FID and MKL. These metrics are based on a novel sound classifier, called Melception, and designed to evaluate the fidelity and relevance of open-domain samples. Both qualitative and quantitative studies are conducted on small- and large-scale datasets to evaluate the fidelity and relevance of generated samples. We also compare our model to the state-of-the-art and observe a substantial improvement in quality, size, and computation time. Code, demo, and samples: v-iashin.github.io/SpecVQGAN

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FoleySpace: Vision-Aligned Binaural Spatial Audio Generation

    cs.SD 2025-08 conditional novelty 7.0 of 10

    FoleySpace generates binaural audio from silent video by estimating a 3D sound-source trajectory from object detection and depth and conditioning a diffusion model on that trajectory plus monaural audio.

  2. HarmoniDPO: Video-guided Audio Generation via Preference-Optimized Diffusion

    cs.CV 2026-08 reject novelty 6.0 of 10

    HarmoniDPO pairs global and frame-level video features with preference-style optimization to generate audio from silent video, reporting improved synchronization and quality metrics over prior V2A baselines.

  3. Hear-Your-Click: Interactive Object-Specific Video-to-Audio Generation

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A new interactive method lets users click on an object in a video and generates audio for just that object, using mask-conditioned contrastive fine-tuning and latent diffusion.

  4. Hearing Hands: Generating Sounds from Physical Interactions in 3D Scenes

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A rectified flow model conditioned on 3D hand trajectories and rendered scene video generates realistic hand-scene interaction sounds, with a human study finding near-chance discrimination (47% misclassified).

  5. SpecMaskFoley: Steering Pretrained Spectral Masked Generative Transformer Toward Synchronized Video-to-audio Synthesis via ControlNet

    cs.SD 2025-05 conditional novelty 6.0 of 10

    A ControlNet branch plus a frequency-aware feature aligner lets a pretrained masked generative TTA model produce video-synchronized foley, beating several from-scratch models on VGGSound.

  6. OmniAudio: Generating Spatial Audio from 360-Degree Video

    eess.AS 2025-04 conditional novelty 6.0 of 10

    OmniAudio generates First-order Ambisonics audio directly from 360-degree video using dual-branch video encoding and flow-matching pre-training, and it introduces the Sphere360 dataset and benchmark.

  7. UniForm: A Unified Multi-Task Diffusion Transformer for Audio-Video Generation

    cs.MM 2025-02 conditional novelty 6.0 of 10

    UniForm trains one diffusion transformer on a shared audio-video latent space to handle video-to-audio, audio-to-video, and text-to-audio-video generation with competitive results.

  8. AGAV-Rater: Adapting Large Multimodal Model for AI-Generated Audio-Visual Quality Assessment

    cs.MM 2025-01 conditional novelty 6.0 of 10

    AGAV-Rater, an LMM fine-tuned in two stages, achieves state-of-the-art quality scores for AI-generated audio-visual content, text-to-audio, and text-to-music.

  9. Tri-Ergon: Fine-grained Video-to-Audio Generation with Multi-modal Conditions and LUFS Control

    cs.CV 2024-12 conditional novelty 6.0 of 10

    A diffusion Transformer generates up to 60 seconds of 44.1 kHz stereo audio from video, text, and audio prompts, with a learned loudness envelope for fine-grained control.

  10. YingSound: Video-Guided Sound Effects Generation with Multi-modal Chain-of-Thought Controls

    cs.SD 2024-12 reject novelty 4.0 of 10

    A video-guided sound effects model with a learnable audio-visual aggregator and multi-modal chain-of-thought module reports strong VGGSound benchmark scores, but its few-shot claim rests on three qualitative samples.

Pith tools