{"work":{"id":"15978118-0cb9-48fb-8c66-d8f1ea7e79fc","openalex_id":null,"doi":null,"arxiv_id":"2407.01494","raw_key":null,"title":"FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Sounds","authors":null,"authors_text":"Y","year":2024,"venue":"cs.CV","abstract":"We study Neural Foley, the automatic generation of high-quality sound effects synchronizing with videos, enabling an immersive audio-visual experience. Despite its wide range of applications, existing approaches encounter limitations when it comes to simultaneously synthesizing high-quality and video-aligned (i.e.,, semantic relevant and temporal synchronized) sounds. To overcome these limitations, we propose FoleyCrafter, a novel framework that leverages a pre-trained text-to-audio model to ensure high-quality audio generation. FoleyCrafter comprises two key components: the semantic adapter for semantic alignment and the temporal controller for precise audio-video synchronization. The semantic adapter utilizes parallel cross-attention layers to condition audio generation on video features, producing realistic sound effects that are semantically relevant to the visual content. Meanwhile, the temporal controller incorporates an onset detector and a timestampbased adapter to achieve precise audio-video alignment. One notable advantage of FoleyCrafter is its compatibility with text prompts, enabling the use of text descriptions to achieve controllable and diverse video-to-audio generation according to user intents. We conduct extensive quantitative and qualitative experiments on standard benchmarks to verify the effectiveness of FoleyCrafter. Models and codes are available at https://github.com/open-mmlab/FoleyCrafter.","external_url":"https://arxiv.org/abs/2407.01494","cited_by_count":null,"metadata_source":"pith","metadata_fetched_at":"2026-07-08T07:14:45.319897+00:00","pith_arxiv_id":"2407.01494","created_at":"2026-05-10T09:59:03.507816+00:00","updated_at":"2026-07-08T07:14:45.319897+00:00","title_quality_ok":true,"display_title":"Foleycrafter: Bring silent videos to life with lifelike and synchronized sounds","render_title":"Foleycrafter: Bring silent videos to life with lifelike and synchronized sounds"},"hub":{"state":{"work_id":"15978118-0cb9-48fb-8c66-d8f1ea7e79fc","tier":"hub","tier_reason":"10+ Pith inbound or 1,000+ external citations","pith_inbound_count":17,"external_cited_by_count":null,"distinct_field_count":3,"first_pith_cited_at":"2025-09-22T18:00:54+00:00","last_pith_cited_at":"2026-07-07T15:34:43+00:00","author_build_status":"not_needed","summary_status":"needed","contexts_status":"needed","graph_status":"needed","ask_index_status":"not_needed","reader_status":"not_needed","recognition_status":"not_needed","updated_at":"2026-08-23T16:20:01.714485+00:00","tier_text":"hub"},"tier":"hub","role_counts":[{"context_role":"background","n":1}],"polarity_counts":[{"context_polarity":"background","n":1}],"runs":{},"summary":{},"graph":{},"authors":[]}}