{"id":"4de9dc8f-d6a5-4bb4-a9f5-3fa2742d1e21","arxiv_id":"2602.03762","paper_version":4,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Conditional flow matching with a rollout loss and early audio-visual fusion achieves state-of-the-art results on visually-guided acoustic highlighting.","lead":"This paper reframes audio highlighting — boosting the sounds that match what's on screen — as a generative problem, using conditional flow matching with a new rollout loss. On the Muddy Mix benchmark it beats the previous discriminative model, suggesting generative models are better suited to this inherently ambiguous remixing task.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 3's 'Rollout' ablation shows a model trained without the CFM loss nearly matches the full system, so the reported gains may be due to iterative refinement rather than generative modeling, undermining the central claim.","rationale":"The paper's headline claim is the categorical conclusion that generative modeling is the best approach for visually-guided audio remixing. For this to hold, the experiments must isolate the generative (CFM) component. They do not. Table 3's 'Rollout' row is a non-generative control trained with only the rollout loss; it achieves KLD 9.92 / LDif 7.71 with standard CLIP conditioning, versus 11.37 / 9.66 for the discriminative VisAH CLIP baseline. The full model with the same conditioning (FM+Rollout) is 9.79 / 7.87, only slightly better. This shows the improvement over the discriminative baseline is driven by the rollout loss (iterative refinement) rather than by distribution matching. The paper's own discussion of the 'Rollout' row concedes this, yet the Conclusion retains the 'generative modeling' claim without the missing ablation. The reader's verdict focused on external validity of the metrics; we agree that is a separate problem, but the internal attribution flaw is more directly load-bearing: even if the metrics are accepted, the headline conclusion does not follow from the reported numbers. A single experiment—training VisAH-FM with the CFM term zeroed out—would settle whether the generative objective is necessary. We recommend keeping the verdict at CONDITIONAL (i.e., UNCHANGED), but the revision requirements should include either this ablation or a softening of the 'best addressed through generative modeling' claim to reflect that the observed gains are primarily attributable to rollout-based iterative refinement and the cross-modal adapter.","tokens_in":17388,"tokens_out":9922,"duration_ms":107208,"concrete_test":"Retrain the full VisAH-FM configuration (CLIP-CLAP adapter, λ=0.3, T=4) with the CFM term (Eq. 4) removed—i.e., optimize only the rollout loss E[||xhat_T - x_T||^2]—and report all Table 1 metrics on the Muddy Mix test set. If the resulting IB/KLD/LDif are within roughly 0.1/0.5/0.5 of VisAH-FM's 29.12/9.70/7.77, then the generative objective contributes negligibly and the headline claim must be revised to credit the rollout loss.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that visually-guided audio remixing is 'best addressed through generative modeling'—requires that the generative CFM objective is the source of the improvement over the discriminative baseline. The paper's own Table 3 contradicts this attribution. The 'Rollout' row trains the same architecture with only the rollout loss (Eq. 5) and no flow-matching loss (Eq. 4). With standard CLIP conditioning, it achieves KLD 9.92 and LDif 7.71, already far better than the discriminative VisAH CLIP baseline (KLD 11.37, LDif 9.66) and close to the full CFM+Rollout model with the same conditioning (KLD 9.79, LDif 7.87). The rollout-only model is deterministic iterative refinement: it optimizes a final MSE after T recurrent passes and does not learn a velocity field or match distributions. Thus the experiment cannot distinguish 'generative modeling' from 'self-correcting iterative refinement.' The paper even concedes this in Section 3.2.1 ('even without the intermediate supervision and linearity prior imposed by flow matching, the model achieves strong performance'), yet the Conclusion still credits generative modeling. To support the claim, the authors must demonstrate that the CFM term contributes beyond the rollout term in the full system including the adapter.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces VisAH-FM, a conditional flow matching framework for visually-guided acoustic highlighting. It builds on the discriminative VisAH/DEMUCS backbone, adds a flow-matching objective with time-step conditioning, a rollout loss that backpropagates through T=4 integration steps to penalize endpoint drift, and a cross-modal adapter that injects CLAP audio features into intermediate CLIP layers. Experiments on the Muddy Mix dataset report consistent improvements over the VisAH baselines (CLIP and T5) across semantic, signal, and remixing metrics, with ablations of the rollout loss, conditioning module, and hyperparameters, plus a small subjective test and trajectory analyses.","tokens_in":17711,"tokens_out":5252,"duration_ms":55556,"significance":"If the reported gains are robust, the paper makes a useful empirical contribution: it shows that iterative self-correcting refinement is highly effective for audio remixing, and it provides a practical early-fusion adapter for audio-visual conditioning. The trajectory analyses in Fig. 4 and Table 12 are informative and go beyond a simple 'it works' evaluation. However, the headline conclusion—that visually-guided audio remixing is 'best addressed through generative modeling'—is not supported by the paper's own ablation, because a model trained with only the rollout loss (no flow-matching objective) nearly matches the full system. The contribution is better characterized as an effective iterative-refinement architecture with a useful conditioning module; this is still a valid contribution, but it is distinct from the generative-modeling claim made in the Abstract and Conclusion.","major_comments":[{"comment":"The 'Rollout' row is the critical control for the paper's central claim. This model is trained with only the rollout loss (Eq. 5) and no CFM loss (Eq. 4), so it is a deterministic recurrent refinement model, not a generative distribution-matching model. On standard CLIP conditioning it attains KLD 9.92 and LDif 7.71, which are within a small margin of the full FM+Rollout model (KLD 9.79, LDif 7.87) and far better than the discriminative VisAH CLIP baseline (KLD 11.37, LDif 9.66). The paper itself acknowledges in §3.2.1 that 'even without the intermediate supervision and linearity prior imposed by flow matching, the model achieves strong performance,' yet the Abstract and Conclusion attribute the gains to generative modeling. Without error bars or significance tests, the small difference between Rollout and FM+Rollout cannot be distinguished from noise. To sustain the generative-modeling","section":"§3.2.1 / Table 3"},{"comment":"The main results in Table 1 compare VisAH-FM trained with on-the-fly random degraded mixes against VisAH baselines trained on the fixed Muddy Mix dataset. This confounds the choice of model class with a change in training data distribution. The fixed-dataset experiment in Table 5 is an attempt to control for this, but it only reports the 'VisAH Text' baseline, not the 'VisAH CLIP' baseline used in Table 1, and the gains are smaller. In that controlled comparison, the 'Rollout' row (no CFM loss) is worse than the discriminative baseline on KLD (11.38 vs 11.02) and only marginally better on LDif (8.87 vs 9.23), which is inconsistent with the main-paper story that rollout-only refinement is responsible for the improvement. A proper controlled comparison over the same fixed dataset, including both VisAH variants, is needed to support the central claim.","section":"§3.1 / Table 5"},{"comment":"The subjective test uses only 5 participants and 8 videos (60% vs 10% win rate), which is far too small to be conclusive. In addition, all quantitative tables report point estimates without error bars, confidence intervals, or repeated-seed variance. This is especially problematic for the small differences used to argue for the rollout loss and the adapter (e.g., KLD 9.79 vs 9.92 in Table 3; KLD 9.70 vs 9.79 in Table 2). Please add variability estimates and, if possible, a larger perceptual study.","section":"§3.3 and all tables"}],"minor_comments":[{"comment":"Typo: 'velocity field field' should be 'velocity field'. Also, Section 1 contains 'the the data manifold' and should be corrected.","section":"§2.2 / Eq. (2)"},{"comment":"Equation (7) uses 'A' in 'adapter(Fk, E, A)' but the right-hand side and Eq. (6) do not show how 'A' enters; clarify the notation.","section":"§2.2.2 / Eqs. (6)-(7)"},{"comment":"The Env value of 0.32 for the 3-step row appears to be an order-of-magnitude typo (other rows are around 2.8); please check and correct.","section":"Appendix Table 8"},{"comment":"Notation is inconsistent: 'PaSST' vs 'PASST', 'PMLR' vs 'PmLR'. Also, Tables 3 and 10 duplicate the same ablation; consider consolidating.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper's most defensible contribution is the combination of iterative refinement via the rollout loss and the CLAP-injected CLIP adapter. The current framing overstates the role of generative modeling, and the control ablation (Table 3) directly undermines the Abstract/Conclusion claim. If the authors can provide the missing control experiment in the full system or reposition the contribution, the paper could become acceptable. The lack of error bars and the very small subjective test are additional concerns for a journal-level empirical paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful core here is real: first conditional flow matching treatment of visually-guided acoustic highlighting, a sensible rollout loss that supervises final-step drift, and a CLAP-into-CLIP adapter that gives the conditioning module actual audio information. The ablations are thorough — conditioning modalities, rollout variants, hyperparameters, fixed versus on-the-fly training sets — and the trajectory analysis (cosine similarity, curvature, PCA) is a nice addition that most papers this size would skip. The gains over the discriminative VisAH baseline are consistent across metrics and survive on the fixed Muddy Mix dataset, which is the right sanity check. Credit where due: this is a careful, well-motivated engineering contribution, and the rollout loss is the kind of idea that could transfer to other iterative generation tasks.\n\nBut the stress-test hits a real soft spot, and it is not minor. Table 3's 'Rollout' row — same architecture, standard CLIP conditioning, only the final MSE loss, no flow matching loss — gets KLD 9.92 and LDif 7.71, beating the discriminative baseline by a wide margin and nearly matching the full FM+Rollout model (KLD 9.79, LDif 7.87). That means the experiment cannot separate 'generative distribution matching' from 'self-correcting iterative refinement.' The paper actually concedes this in Section 3.2.1, where it attributes the rollout-only variant's strength to repeated refinement, yet the abstract and conclusion still say generative modeling is 'best addressed.' That overreach matters, because the central claim is exactly the thing the ablation fails to isolate. The authors need either an experiment that shows the CFM term adds value beyond the rollout term in the full system (including the adapter), or a softened claim that limits the conclusion to 'flow matching with rollout and early cross-modal fusion.'\n\nOther issues are more standard: no error bars or seed variance anywhere; the subjective test is 5 participants and 8 videos — fine as a sanity check, not as evidence; no released code or data; the metrics lean on a third-party separator and a synthetic pseudo-data process. The limitations section acknowledges the synthetic-data and paired-data issues, which is honest.\n\nBottom line: this is a method paper with genuine components and reproducible-looking ablations, but the headline attribution is not established. A serious referee should see it, and conditional acceptance with a request to fix the attribution (or add the missing experiment) is the right outcome.","headline":"A solid method paper whose headline claim outruns its own ablations: the rollout-only model nearly matches the full system, so 'best addressed through generative modeling' is not actually supported.","tokens_in":18198,"tokens_out":1373,"would_cite":true,"duration_ms":17342,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Audio remixing is a many-to-many problem, so a flow-matching generative model learns it better than a discriminative one.","keywords":["acoustic highlighting","flow matching","generative audio","audio-visual alignment","audio remixing","rollout loss","cross-modal conditioning"],"falsifier":"A controlled listening test with a larger panel (e.g., 50+ participants) on real-world poorly-balanced videos—not synthetic Muddy Mix—where the discriminative baseline is preferred at or above chance would directly contradict the paper's conclusion; alternatively, a metric-inversion result where VisAH-FM improves KLD and ImageBind but lowers subjective alignment would show the metrics do not track perceptual quality.","tokens_in":17278,"feed_emoji":"🎬","tokens_out":6497,"duration_ms":56746,"temperature":0.7,"pith_summary":"The paper claims that visually-guided acoustic highlighting—rebalancing the loudness of speech, music, and effects in a video's audio to match what is shown—is inherently a many-to-many distribution-transport problem, not a one-to-one regression. To exploit that, the authors cast the task as conditional flow matching, in which a vector field transports a poorly-balanced mix toward the distribution of well-balanced mixes under visual guidance. Two additions make it work: a rollout loss that backpropagates through the full generated trajectory, penalizing drift at the final step and teaching the model to self-correct; and an adapter that injects audio features into the visual encoder so the conditioning pathway itself selects the source to highlight. On the Muddy Mix benchmark, their model (VisAH-FM) improves over the prior discriminative baseline on all reported metrics, supporting the paper's conclusion that generative modeling is the right framing for audio remixing.","feed_headline":"Generative flow matching beats discriminative audio highlighting","feed_subtitle":"Treating remixing as many-to-many transport, the model rebalances speech, music, and effects to match the visual scene.","key_machinery":"The load-bearing object is the conditional flow-matching vector field, defined by the interpolant x_t = (1−t)x_0 + t x_1 and the ODE dx_t/dt = (x_1 − x_t)/(1 − t). A U-Net adapted from the DEMUCS-based VisAH backbone estimates this field as v_θ(x_t, t, c) = x_0 − u_θ(x_t, t, c), where u_θ is the model's direct prediction, enabling warm-start from pretrained discriminative weights. Two mechanisms carry the improvement: (1) the rollout loss, which runs the full T-step flow during training and penalizes the squared error between the final prediction and ground truth, exposing the model to its own intermediate errors (exposure-bias mitigation); and (2) the conditioning adapter, which computes cr","core_discovery":"The central discovery is that a generative formulation with trajectory-level supervision beats the discriminative baseline for acoustic highlighting. Concretely, the model learns a time-dependent velocity field v_θ(x_t, t, c) that interpolates from a degraded mix x_0 to a well-balanced mix x_1, conditioned on visual features c. The rollout loss adds a term ||x̂_T − x_T||², where x̂_T is the output after T=4 recurrent integration steps, which stabilizes long-range trajectories and prevents early errors in source selection from compounding. Additionally, a cross-modal adapter injects CLAP audio embeddings into intermediate CLIP layers via cross-attention, letting the conditioning encoder ident","pith_inferences":["Because the model learns a continuous trajectory between mixes, a natural extension not explored in the paper is semantic interpolation: stopping the flow at an intermediate step could give users controllable 'amount of highlight,' which the linearity analysis in semantic space supports.","The on-the-fly random remixing during training effectively augments the data distribution; this suggests that training-time stochasticity in the conditioning source selection is a cheap regularizer that may transfer to other augmentations like random gain or equalization perturbations.","The ablation showing text features add nothing when audio is injected hints that a single shared audio-visual embedding (rather than separate text and audio branches) could simplify the conditioning stack; a testable extension is replacing CLIP/CLAP with a jointly trained audio-visual encoder.","The paper's own limitation—training on paired artificially-mixed data—points to a concrete next step: adapting the CFM framework to unpaired data via optimal-transport or adversarial alignment, which would unlock real-world recordings."],"forward_implications":["If the generative framing is right, the same distribution-matching objective should transfer to other ambiguous audio tasks, such as source separation and style transfer, where discriminative regression is known to struggle.","The rollout loss — backpropagating through the full flow with a final-step MSE — is presented as a general countermeasure to exposure bias; the paper shows it outperforms consistency losses and input-noise injection in this setting.","Injecting audio features into the visual encoder (CLAP into CLIP) is shown to be more effective than adding text captions, and the paper notes it avoids the cost of a large text encoder, making the method cheaper at inference.","With only four integration steps, the flow-based model adds minor compute beyond the one-time encoder passes, suggesting the approach is practical for video editing pipelines."],"fun_headline_variants":["Flow matching rebalances audio to match video","Generative model beats discriminative for audio highlight","Rollout loss stabilizes flow for audio-visual remix","CFM with rollout loss aids visually-guided audio"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The central claim rests on the assumption that the Muddy Mix pseudo-data generation (separation, adjustment, remixing) and the third-party metrics (PASST KL divergence, ImageBind score, and separator-based loudness difference) faithfully reflect what human listeners perceive as good acoustic highlighting.","fun_headline_variants_meta":{"raw":{"variants":["Flow matching rebalances audio to match video","Generative model beats discriminative for audio highlight","Rollout loss stabilizes flow for audio-visual remix","CFM with rollout loss aids visually-guided audio"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00015,"raw_usage":{"total_tokens":1032,"prompt_tokens":744,"completion_tokens":288,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":488,"completion_tokens_details":{"reasoning_tokens":236}},"tokens_in":488,"tokens_out":288,"duration_ms":3298,"temperature":1.0,"reasoning_tokens":236,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T04:50:02.187673+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled listening test with a larger panel (e.g., 50+ participants) on real-world poorly-balanced videos—not synthetic Muddy Mix—where the discriminative baseline is preferred at or above chance would directly contradict the paper's conclusion; alternatively, a metric-inversion result where VisAH-FM improves KLD and ImageBind but lowers subjective alignment would show the metrics do not track perceptual quality.","supporting_citations":[],"review_version":1}