{"id":"487b5945-d830-48fa-b883-1cbb324634f3","arxiv_id":"2505.22053","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A training-free multi-agent framework that decomposes multimodal inputs into audio events, selects specialized generators, and self-corrects outputs to produce multiple audio types.","lead":"AudioGenie is a new system that uses multiple AI agents to turn videos, text, or images into sound effects, speech, music, and songs without training. It also introduces a benchmark of 198 videos to measure how well such systems work.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim rests on an asymmetric comparison in Table 3: AudioGenie outputs a mixed multi-event/multi-type track while baselines emit a single audio event, so large gains on MOS-Accuracy and AES complexity may reflect output plurality rather than the proposed mechanisms.","rationale":"The reader's conditional verdict is appropriate: the central claim needs code/data release and stronger evaluation before it can be accepted at face value. I disagree with the precise weak point: the more load-bearing problem is not only whether the supervisor loop converges, but that Table 3's comparison is structurally unequal, so even a perfect self-correction loop would not establish the headline claim from those numbers. The paper does have partial independent support: SISO results on VGGSound/V2M (Appendix B), MLLM robustness (Appendix C), and ablations showing each module contributes. These prevent rejection, but they do not resolve the asymmetric Table 3 comparison or the lack of blinded subjective ratings. Given the accepted-paper status and available appendices, CONDITIONAL/UNCHANGED remains the right call, with additional conditions: release MA-Bench, prompts, and code; include a decomposed mixed-baseline; and provide blind MOS with agreement metrics.","tokens_in":18221,"tokens_out":7978,"duration_ms":95758,"concrete_test":"Take a random 50-video subset of MA-Bench and generate per-event prompts from AudioGenie's Stage 1. Without using AudioGenie's MoE, supervisor, or trial-and-error modules, generate each event with the strongest single-type baseline (MMAudio for sound effects, CosyVoice 2 for speech, MusicGen/InspireMusic for music, DiffRhythm for song), mix the outputs, and recompute the nine Table 3 metrics. If this decomposed-mixture baseline matches or exceeds AudioGenie, the headline SOTA advantage is explained by output multiplicity rather than by the proposed framework.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.4.1 (Table 3) compares under 'identical-input, unconstrained-output': AudioGenie is allowed to decompose the input into multiple events of different audio types and synthesize them separately, while each baseline produces one audio category (AudioX produces two). This makes the comparison structurally unequal. The subjective MOS-Accuracy item explicitly asks whether all audio events in the input are produced, and the AES 'Production Complexity' dimension rewards richness, so a system that simply outputs more tracks is favored irrespective of per-event quality. The claimed SOTA margins in Table 3 are therefore not evidence that the agent, supervisor, MoE, or trial-and-error modules outperform the baselines; they are at least partly evidence that multiple outputs were generated. In the more controlled SISO setting (Table 4) the advantages shrink: for Audio Effect Generation, AudioGenie is below AudioX on PAM and CU, and in Song Generation it is below DiffRhythm on PQ. Section 5 lists automatic MM2MA evaluation metrics as future work, conceding that the current AES/PAM metrics are not established for this setting, and the subjective MOS protocol (Section 4.3.2) does not report blinding, rater independence, or inter-rater agreement. MA-Bench is neither released nor accompanied by code or exact prompts, so the central 9-metric/8-task SOTA claim is not independently checkable. Appendix results on VGGSound/V2M and with different MLLMs partially offset generalization concerns, but they do not repair the asymmetric comparison on the headline benchmark.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes AudioGenie, a training-free multi-agent framework for multimodal-to-multiaudio (MM2MA) generation, i.e., producing sound effects, speech, music, and songs from video, text, image, or combinations of these inputs. The system consists of a generation team that decomposes the input into audio events, selects domain-specific experts and models from a tool library, and applies a tree-of-thought trial-and-error refinement loop, plus a supervisor team that evaluates plans and generated audio and provides feedback. The authors also introduce MA-Bench, a benchmark of 198 curated videos with multi-type audio annotations and comprehensive event-level annotations. Experiments on MA-Bench report state-of-the-art or comparable performance across nine metrics in eight tasks, with ablations and additional evaluations on VGGSound, V2M, and with different MLLMs in the appendices.","tokens_in":18533,"tokens_out":8195,"duration_ms":81079,"significance":"The proposed system is relevant to the multimedia community and, if the results are validated, would be a useful demonstration that a training-free orchestration of existing audio generators can handle multi-type audio generation in a unified framework. The paper has several strengths: it is training-free and builds on existing models; it includes a new benchmark that could facilitate future research; it provides ablations for each major component; and it reports generalization experiments on public datasets and with different MLLMs, which partially address the 'self-built benchmark' concern. However, the central SOTA claim currently rests on an asymmetric comparison in Table 3, on objective metrics that the authors themselves identify as not yet established for MM2MA (Section 5), and on a subjective study that lacks statistical rigor. These issues need to be addressed before the claimed superiority can be accepted.","major_comments":[{"comment":"The comparison in Table 3 is structurally unequal. AudioGenie is allowed to produce a mixed output with multiple audio events of different types, while each baseline produces a single audio category (AudioX produces two). Since MOS-Accuracy explicitly asks whether all audio events in the input are produced, and AES Production Complexity rewards richness and complexity, the large gains on these metrics for AudioGenie (e.g., MOS-Acc 3.54 vs 2.90 for video input, PC 3.99 vs 3.02 for AudioX) may reflect the greater number of output tracks rather than the quality of the proposed decomposition, MoE selection, or self-correction mechanisms. To support the SOTA claim, the authors should report an additional comparison in which the baselines are given the same decomposed event list or in which the evaluation is done per audio event, so that track count is not a confounding factor.","section":"Section 4.4.1, Table 3"},{"comment":"In the SISO setting, which is more comparable to the baselines, the advantages of AudioGenie are smaller and in some cases negative. For audio effect generation, AudioGenie is below AudioX on PAM (0.750 vs 0.786) and CU (6.043 vs 6.209); for song generation, it is below DiffRhythm on PQ (8.212 vs 8.328). The paper describes these results as 'SOTA or comparable', but these cases below SOTA should be discussed explicitly in the text, and the abstract's universal 9-metric superiority claim should be qualified accordingly.","section":"Table 4, Section 4.4.2"},{"comment":"The paper states that establishing automatic MM2MA evaluation metrics is future work, which is an explicit admission that the objective metrics used in Tables 3 and 4 (PAM and AES) are not established for the MM2MA setting. Since these metrics are the quantitative basis of the SOTA claim, the authors should validate them for this new setting (e.g., by reporting correlation with human judgments on MA-Bench) or explicitly label the results as preliminary. Without such validation, the claimed SOTA on 9 objective/subjective metrics is not adequately supported.","section":"Section 5"},{"comment":"The subjective MOS protocol is under-specified. The paper reports mean scores and standard deviations from 'at least 20 participants', but does not state whether raters were blind to the system identity, how many ratings were obtained per audio clip, whether raters rated independently, or any inter-rater agreement statistic (e.g., ICC or Krippendorff's alpha). In addition, no significance tests are reported, so differences such as the 3.54 vs 2.90 MOS-Accuracy gap cannot be assessed for reliability. The authors should provide these details and apply appropriate statistical tests to support the subjective claims.","section":"Section 4.3.2"},{"comment":"The central evaluation is conducted on the authors' self-built MA-Bench, but the benchmark annotations, the implementation code, and the exact agent prompts are not released. The paper only mentions a project website with audio samples. Since the framework is training-free and composed of existing models, releasing the prompts and the benchmark would allow the results to be independently reproduced and would substantially strengthen the SOTA claim. I recommend making these resources available.","section":"Reproducibility"}],"minor_comments":[{"comment":"There is a typo in 'Each aduio event' which should be 'Each audio event'.","section":"Section 3.2"},{"comment":"The abstract mentions '9 metrics in 8 tasks', but the paper does not explicitly enumerate these 8 tasks; Table 3 has four input configurations and Table 4 has four generation tasks. Please clarify what the 8 tasks are.","section":"Abstract"},{"comment":"The term 'adaptive Mixture-of-Experts' is used, but the implementation is a discrete model selection from a fixed tool library based on expert knowledge. Since 'MoE' normally implies a learned or trainable routing, a brief clarification or a different term would avoid confusion.","section":"Section 3.3"},{"comment":"In the Song Generation row, AudioGenie is below DiffRhythm on PQ; the paper should add a sentence explaining this specific case, as it is the only objective metric where AudioGenie is not best among the SOTA baselines.","section":"Table 4"},{"comment":"The ablations in Table 5 report differences (e.g., PAM 0.966 vs 0.888 for w/o supervisor team) without significance tests; given the small size of MA-Bench, confidence intervals or at least per-sample variability should be reported.","section":"Section 4.5"},{"comment":"The conclusion says 'SOTA or comparable performance across several metrics' while the abstract claims '9 metrics in 8 tasks'. These statements should be aligned.","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of ACM Multimedia and introduces a useful benchmark. The self-citations in References [41] and [42] are not load-bearing, but the authors should ensure that citation is appropriate. Given that the main evaluation is on a self-built benchmark, I strongly encourage the editor to ask for code and data release as part of the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the honest read. The paper is not a dud, but its headline claim is bigger than its evidence. The task it frames — generating multiple audio types (sound effects, speech, music, song) from multiple input modalities in one training-free agent pipeline — is genuinely new. AudioX covers only effects and music; LVAS-Agent only effects. That framing alone has value, and the system is coherent: decompose the input into typed audio events, route each to a domain expert, generate, then run a tree-of-thought retry loop supervised by a Qwen2.5-VL and Audio-Reasoner team. The ablations in Table 5 show each component helping, which is more than many agent papers manage.\n\nThe soft spots, in size order. First, the stress-test note is right about Table 3. The comparison is structurally unequal: AudioGenie may output a mixed multi-event, multi-type track while each baseline emits a single event or category. Two metrics there — MOS-Accuracy, which asks whether all events are produced, and AES Production Complexity, which rewards richness — bias toward more output. So the large SOTA margins partly reflect output plurality, not the merit of the proposed mechanisms. The controlled single-output setting (Table 4) reads more honestly: the advantages shrink, and AudioGenie sits below AudioX on PAM and CU for effects and below DiffRhythm on PQ for songs. The authors don't hide those numbers, but the abstract and intro lean on the inflated ones.\n\nSecond, nothing is released: no code, no MA-Bench, no exact prompts. The MOS protocol lacks blinding, rater independence, inter-rater agreement, and significance tests. Section 5 itself concedes that automatic MM2MA metrics are future work. The Appendix results on VGGSound and V2M partly offset the self-built benchmark worry — the system works on public data and with several MLLMs — but they don't repair the headline comparison.\n\nThird, the weakest assumption is the LLM supervisor itself: the trial-and-error module in Section 3.4 trusts the supervisor's judgments of quality, alignment, and aesthetics. Noisy feedback could plateau or degrade output. The ablation shows removing the ToT loop hurts, which is internal evidence the loop helps, but there is no analysis of supervisor error. Minor-to-moderate gap, not fatal.\n\nWho it's for: people working on audio generation agents or multimodal generation pipelines, and anyone who wants a clean case study in how evaluation design inflates SOTA claims. It deserves serious refereeing. My recommendation: engage with it, but treat the Table 3 SOTA claim as conditional until the authors release MA-Bench, code, and prompts and re-run the comparison where baselines get equal output scope. The central idea holds; the proof is incomplete.","headline":"A genuinely new task framing and a coherent training-free multi-agent system, but the headline SOTA claim is inflated by an asymmetric comparison and missing releases.","tokens_in":19049,"tokens_out":7110,"would_cite":true,"duration_ms":70105,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AudioGenie claims that multimodality-to-multiaudio generation can be solved without training by coordinating off-the-shelf models in a plan-and-verify multi-agent loop.","keywords":["multimodality-to-multiaudio generation","multi-agent system","training-free framework","self-correction","tree-of-thought","video-to-audio generation","audio generation benchmark","mixture-of-experts"],"falsifier":"Run MA-Bench twice through AudioGenie, once with the real supervisor team and once with random or inverted supervisor verdicts while keeping the same tree-of-thought search, and compare the mean opinion scores. If random feedback matches the real supervisor, the reported gains come from retrying rather than from evaluation quality; if real feedback clearly wins, the supervisor's judgment is the load-bearing component.","tokens_in":18043,"feed_emoji":"🎧","tokens_out":8633,"duration_ms":80309,"temperature":0.7,"pith_summary":"AudioGenie is a training-free multi-agent system that claims to generate multiple audio types—sound effects, speech, music, and songs—from video, text, image, or combined inputs in a single unified pipeline. Instead of training one model, it coordinates a generation team and a supervisor team: the generation team decomposes a scene into timestamped audio sub-events, assigns each event to a specialist model, and mixes the results, while the supervisor team verifies every plan and output and drives retries. To measure this, the paper introduces MA-Bench, a benchmark of 198 annotated videos in which each video contains several audio events with detailed timestamps and descriptions. Across eight generation tasks and nine metrics, the paper reports state-of-the-art or comparable results, including higher human ratings on accuracy and alignment than single-pass baselines. The reason this matters is that it suggests complex audio production can be assembled from off-the-shelf components rather than custom-trained systems.","feed_headline":"Zero-training agents make speech, music, and effects from one video","feed_subtitle":"LLM agents decompose scenes into events, dispatch specialists, and self-correct until the mix passes review.","key_machinery":"The load-bearing mechanism is the 'plan-and-verify' loop built around three modules. Fine-grained task decomposition turns the input into a structured JSON list of sub-audio events, which is what makes multiple simultaneous events visible and separately generatable. The adaptive Mixture-of-Experts collaborative entity gives each audio type a dedicated expert that selects two candidate models from a tool library and refines plans through intra-expert self-reflection and inter-expert collaborative refinement. The trial-and-error iterative refinement module organizes each event's generation as a tree-of-thought with initial, generation, and refinement nodes, using the supervisor team's verdicts to decide between targeted post-processing and full retries. The supervisor team is the arbiter that connects all stages, evaluating plans for timing and content and evaluating audio for quality, alignment, and aesthetics.","core_discovery":"The paper's central claim is that multimodality-to-multiaudio generation does not require a single unified trained model. AudioGenie is a dual-layer system: the generation team runs three stages—fine-grained task decomposition, adaptive Mixture-of-Experts selection, and trial-and-error iterative refinement—while the supervisor team checks each stage by acting as an audio planning expert, a domain expert, and a professional audio evaluator. Each audio event is planned as a JSON entry with type, object, timestamps, description, and volume; the expert layer picks two candidate generators from a tool library, refines the event plan, and generates the event; the supervisor scores output on quality, alignment, and aesthetics. If the output is fixable, refinement nodes post-process it; if it fails core requirements, sibling generation nodes retry with adjusted prompts or another model. On MA-Bench, the authors report state-of-the-art or comparable performance on 9 metrics in 8 tasks, and ablations show that removing the supervisor team and the tree-of-thought self-correction causes the largest drops.","pith_inferences":["If the supervisor's verdicts are the real engine of quality, then swapping in a stronger audio-language evaluator or a learned audio-quality metric should lift results further; this is directly testable.","The architecture suggests a scaling pattern for tool libraries: adding more candidate models per audio type should improve the best-branch outcome at the cost of more trial-and-error calls, which could be measured as a compute-versus-quality curve.","A natural follow-up the paper does not explore is distilling the accepted branches into a single trained model, which would trade the inference-time search for a cheaper one-pass generator with similar event coverage."],"forward_implications":["Complex videos with several simultaneous sounds should yield complete, mixed audio tracks instead of a single dominant event, because each sub-event is generated and mixed separately.","New input modalities or audio types can be added without retraining by placing a new model in the tool library and giving a domain expert its prior knowledge.","Unreliable single-pass outputs can be improved by spending more compute: the tree-of-thought loop retries failed events with adjusted prompts or alternative models.","The MA-Bench dataset of 198 annotated multi-audio videos provides a common test bed for future multimodality-to-multiaudio systems, since no comparable benchmark existed before."],"supporting_citations":[{"why":"supplies the main video-to-audio and text-to-audio sound-effect generator used in the tool library and the primary state-of-the-art comparator in the evaluation tables.","marker":"[7]"},{"why":"is the closest prior unified anything-to-audio model and the main baseline that accepts multimodal inputs; the paper compares against it across video, text, and combined inputs.","marker":"[44]"},{"why":"provides the text-to-audio baseline and a second sound-effect generation model in the tool library.","marker":"[55]"},{"why":"underlies the whole agent: this video-language model acts as the generation-team brain and as the supervisor for the first two stages.","marker":"[1]"},{"why":"is the audio-language model used as the stage-three supervisor for evaluating, reflecting on, and refining generated audio.","marker":"[53]"},{"why":"provides the trial-and-error tree-of-thought pattern that the iterative refinement module adapts for audio events.","marker":"[50]"},{"why":"defines the AES metric with its PQ, PC, CE, and CU dimensions used for objective evaluation of mixed audio types.","marker":"[46]"},{"why":"supplies the PAM overall audio-quality score used as the second objective metric.","marker":"[14]"},{"why":"represents the prior agent-based video-to-sound-effect system that the paper extends from single-input single-output to multi-input multi-audio with self-correction.","marker":"[60]"}],"fun_headline_variants":["Training-free agents turn video into speech, music, sound effects","Multi-agent system generates diverse audio from video without training","AudioGenie: zero-training agents craft audio from video, text, images","Agents split scenes, pick tools, self-correct to make audio from video","No training needed: agents craft soundscapes from multimodal inputs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The self-correction loop assumes the LLM-based supervisor team can reliably judge generated audio on quality, alignment, and aesthetics, and that each retry or refinement moves the output closer to the supervisor's verdict.","fun_headline_variants_meta":{"raw":{"variants":["Training-free agents turn video into speech, music, sound effects","Multi-agent system generates diverse audio from video without training","AudioGenie: zero-training agents craft audio from video, text, images","Agents split scenes, pick tools, self-correct to make audio from video","No training needed: agents craft soundscapes from multimodal inputs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000838,"raw_usage":{"total_tokens":3721,"prompt_tokens":1083,"completion_tokens":2638,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":699,"completion_tokens_details":{"reasoning_tokens":2548}},"tokens_in":699,"tokens_out":2638,"duration_ms":17470,"temperature":1.0,"reasoning_tokens":2548,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:15:04.455333+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run MA-Bench twice through AudioGenie, once with the real supervisor team and once with random or inverted supervisor verdicts while keeping the same tree-of-thought search, and compare the mean opinion scores. If random feedback matches the real supervisor, the reported gains come from retrying rather than from evaluation quality; if real feedback clearly wins, the supervisor's judgment is the load-bearing component.","supporting_citations":[{"cited_title":"FilmComposer: LLM-Driven Music Production for Silent Film Clips","cited_arxiv_id":"2503.08147","evidence_quote":"provides the text-to-audio baseline and a second sound-effect generation model in the tool library."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"defines the AES metric with its PQ, PC, CE, and CU dimensions used for objective evaluation of mixed audio types."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the PAM overall audio-quality score used as the second objective metric."}],"review_version":1}