{"id":"9aa02127-f9c4-4ad2-aaf8-fcdf5085cc82","arxiv_id":"2608.05816","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Selective convergence in contrastive audio-visual learning is exploited in a two-stage framework to localize dual sound sources without labels, with a new segmentation-mask evaluation benchmark.","lead":"A self-supervised method localizes two simultaneous sound sources by first letting a contrastive model lock onto the more salient source, then using that as a spatial prior to reveal the second source. The paper also introduces a pixel-level mask benchmark and argues that bounding-box evaluation systematically favors imprecise predictions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Stage 1's 'selective convergence' is defined as convergence to whichever source has the largest initial audio-visual similarity, but no evidence shows that this largest-similarity source is a true sound producer; acknowledged false activations mean M_dom can be corrupted, and Eq.","rationale":"The reader's weakest assumption identifies exactly the load-bearing point: Stage 1's selective convergence is not shown to select a true sound source, and a false activation corrupts the spatial prior before Stage 2 begins. My reading strengthens this by noting that Table 4's post hoc dominant-source labeling hides cases where neither ground truth is selected, and that Appendix E assumes g1 > g2 rather than deriving it from sound-producing properties. This is not a rejection of the paper: the empirical performance gains and the Stage-2 ablation suggest the framework can work when the prior is correct. The right remedy is to measure the prior's fidelity directly, which is feasible with the existing test set, and to report the failure fraction. Since the reader's conditional verdict already accounts for this uncertainty, no verdict change is needed.","tokens_in":21295,"tokens_out":8835,"duration_ms":97557,"concrete_test":"Run the trained Stage-1 model on the full VGGSound-Duet test set (3,951 pairs). For each sample compute CIoU(M_dom, GT1) and CIoU(M_dom, GT2) under the frame-wise protocol with threshold 0.3, and classify selection as reliable only when max(CIoU1, CIoU2) >= 0.3. Report the fraction of samples where Stage-1 fails to reach 0.3 IoU against either true source, along with the distribution of max IoU. If the failure fraction is non-negligible (e.g. > 5-10%) or the low-end mass is heavy, M_dom is frequently not a true source and the spatial prior is corrupted. As a complement, replace M_dom with a visual-saliency-only prior (e.g. a frozen ImageNet Grad-CAM map) in the same two-stage pipeline; if performance is largely unchanged, the fidelity of the learned prior is not what drives the result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that selective convergence yields a usable spatial prior depends on an unverified identification: the source the contrastive model converges to is a sound-producing object. In Sec. 3.2 the dominant source is defined as 'whichever source the model converges to', and Appendix E only shows that an assumed initial gap g1 > g2 is amplified into winner-take-all selection (Theorem 1); it never explains why the largest initial audio-visual similarity should correspond to a true source. The paper's own failure analysis (Appendix A.2) documents false activations on visually salient but non-sounding regions, e.g. a performer's mouth instead of the instrument. If such a false activation becomes M_dom, the Stage 2 partition V2 = V * (1 - M_dom) suppresses the region that actually contains the true subdominant source, and there is no self-supervised signal in Stage 2 to recover it. Table 4 does not close this gap: it labels the dominant source post hoc as the ground-truth mask with the higher IoU, so samples where both IoUs are near zero (false selection) are silently absorbed into the 'dominant source' statistics. Thus the load-bearing assumption that selective convergence selects a true source is asserted, its failure modes are acknowledged, but it is never directly measured on the dual-source benchmarks.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SCAV, a fully self-supervised two-stage framework for dual-source audio-visual localisation built on an observed phenomenon the authors call selective convergence: contrastive audio-visual models trained on mixed audio naturally focus on one dominant source rather than representing all sources equally. Stage 1 trains encoders with a contrastive loss and differentiable thresholding to produce a dominant-source map M_dom; Stage 2 uses M_dom as a spatial prior to partition visual features, decouples audio and visual features via cross-attention, and trains source-specific contrastive losses to localise the subdominant source. The paper also argues that the established bounding-box evaluation protocol inflates scores, introduces a new pixel-level mask benchmark, VGGSound-DuetMask, and reports state-of-the-art results among self-supervised methods. A theoretical analysis of selective convergence is provided in Appendix E, and a reproduction of the NoPrior baseline under a frame-wise protocol is documented in Appendix F.","tokens_in":21534,"tokens_out":6894,"duration_ms":73785,"significance":"If the selective-convergence claim holds, it is a genuinely interesting observation that reframes a known limitation of contrastive learning as a usable inductive bias, and the two-stage framework is a plausible way to break the circular dependency of multi-source localisation. The paper also makes two useful community contributions: a formal analysis of source-wise versus frame-wise evaluation protocols (Appendix D, Eq. (24)-(25)) and a new large-scale mask-annotated benchmark, VGGSound-DuetMask. The authors are commendably explicit about the scope of their theoretical result, stating in Appendix E that their analysis amplifies an assumed gap g1 > g2 rather than predicting which source is selected, and they provide a detailed reproduction of a baseline in Appendix F. However, the load-bearing empirical claim—that selective convergence selects a true sound source—is not directly measured, and the headline comparison in Table 1 mixes evaluation protocols, so the current evidence does not fully support the reported performance claims.","major_comments":[{"comment":"The headline comparison in Table 1 mixes evaluation protocols. The table note states that Mix-and-Localize is re-run under the paper's frame-wise evaluation for VGGSound-Duet, while AVGN and OA-SSL are taken as originally reported. By the paper's own account in Appendix D, the original VGGSound-Duet evaluation is source-wise: activations on the opposite half-frame are discarded before IoU is computed. Eq. (25) shows that source-wise IoU is always at least frame-wise IoU for the same predictions. Therefore SCAV's frame-wise numbers are not directly comparable to AVGN and OA-SSL's original source-wise numbers, and the claims of 'best performance among self-supervised methods' and 'surpasses some weakly-supervised approaches' are not supported by Table 1 as presented. Please re-run AVGN and OA-SSL under the same frame-wise protocol (as was done for NoPrior), or alternatively report SCAV under the source-wise protocol, so that all entries in Table 1 are apples-to-apples.","section":"Sec. 4.3, Table 1, Appendix D"},{"comment":"The paper does not directly measure whether the Stage 1 model selects a true sound-producing source. In Sec. 4.5, the dominant source is defined post hoc as the ground-truth mask with the higher IoU (Eq. (18)); samples where both IoUs are near zero are absorbed into the 'dominant source' statistics, so Table 4 cannot distinguish true selection from a model that simply biases toward one arbitrary region. Appendix E is explicit that its analysis assumes an initial gap g1 > g2 and proves amplification, not selection of a true source. Since Sec. 3.2 defines dominance as 'whichever source the model converges to,' the central premise that M_dom is a usable spatial prior remains unverified. Please add a direct selection-accuracy measurement, e.g., the fraction of test samples where M_dom's peak or binarised region has IoU above a meaningful threshold with either ground-truth mask, and report the false-selection rate where both IoUs are near zero. Such a measurement would close the gap between the observed selective-convergence phenomenon and the claim that it identifies a true sound source.","section":"Sec. 4.5, Eq. (18), Table 4; Appendix E"},{"comment":"Stage 2's hard partition V2 = V ⊙ (1 - M_dom) assumes that M_dom does not suppress the true subdominant source. The paper's own failure analysis in Appendix A.2 documents false activations on visually salient but non-sounding regions, e.g., a performer's mouth instead of the instrument. If such a false region becomes M_dom, the true subdominant source is either discarded if it overlaps with the false region, or it is forced into the same branch as the remaining foreground and background; there is no self-supervised signal in Stage 2 that recovers a source lost at the partition. The failure-case discussion acknowledges this qualitatively, but no quantitative analysis is provided. Please add an ablation that corrupts M_dom with controlled false regions or noise and measures Stage 2's sensitivity, so the reader can assess how often false selection propagates and whether the framework is robust to imperfect priors.","section":"Sec. 3.3, Eq. (9), Appendix A.2"},{"comment":"The theoretical analysis rests on conditions that are not verified for the actual training setup. Lemma 2 requires the threshold to satisfy ar{S}_1 - δ > ε_p > ar{S}_2 + δ, but the thresholds ε_p = 0.65 and ε_n = 0.4 in Eq. (3) are fixed while the similarity values S(p) depend on the learned feature scales. Assumptions 1-3 also postulate orthonormal audio source features and very low visual noise, which are strong idealisations. As stated, Theorem 1 characterises an idealised amplification mechanism and does not establish that the Stage 1 training operates in this regime. Please either verify the threshold condition empirically (e.g., by inspecting the distributions of S(p) on source and background regions during training) or clarify in the main text that the theory is an illustrative analysis rather than a proof that selective convergence occurs under the exact training conditions.","section":"Appendix E, Eq. (3), Lemma 2"}],"minor_comments":[{"comment":"The definition of CAP uses AP_k, which is never defined; please clarify whether it is the average precision for class k and how the binary decision is made for each source pair.","section":"Sec. 4.2, Eq. (15)"},{"comment":"Please specify explicitly whether all methods on VGGSound-DuetMask were re-run by the authors under the same frame-wise, mask-based protocol, or whether some numbers are taken from original papers. The current table caption does not state this, and the protocol is critical for interpreting the comparison.","section":"Table 2"},{"comment":"The reproduction changes the concatenation dimension in datasets_flow.py from dim=1 to dim=2 to obtain a horizontal layout. Since convolutional and positional-encoding layers may be sensitive to the spatial arrangement, please verify that the original model was intended for horizontal concatenation or discuss whether the dimension change could affect the reproduced results.","section":"Appendix F"},{"comment":"In the definition of the background embedding vneg, the weight N(i,j) is multiplied by S(i,j); please state explicitly whether S(i,j) is the raw similarity score or some normalised version, since the notation is not defined at that point.","section":"Algorithm 1"},{"comment":"The method is referred to as both 'Mix-and-Localize' and 'Mix-and-Localise' in different places; please standardise the spelling.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"This is a solid paper with a useful phenomenon and a valuable benchmark contribution, but the performance claims need to be re-evaluated under a uniform protocol and the selective-convergence claim needs a direct measurement of selection accuracy. The theoretical appendix is honest about its scope but is currently weaker than the main text suggests; the authors should either add the empirical verification or temper the language. The benchmark itself is a meaningful contribution that could benefit the community if the baseline evaluation protocol is documented precisely."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this if you care about multi-source audio-visual localization. The paper's real content is an observation and a method: contrastive models trained on mixed audio lock onto one dominant source, and a two-stage design uses that heatmap as a spatial prior to find the second source. That works on the benchmarks they run. The frame-wise vs source-wise evaluation analysis is the most valuable part; the Appendix D argument that source-wise evaluation inflates IoU by discarding cross-source activations is correct and should push the subfield to standardize. The VGGSound-DuetMask benchmark is a reasonable next step, though it is built by the same group.\n\nNow the soft spots. Table 1 mixes protocols: SCAV is evaluated frame-wise, while AVGN and OA-SSL are taken as originally reported, which the paper itself shows uses the more lenient source-wise protocol. So the headline comparison is not apples-to-apples. Reproducing NoPrior under frame-wise in Appendix F is good but does not fix the rest of the table. The new benchmark has no released data or code, which limits independent verification. More substantively, the load-bearing claim—that Stage 1 selects a true sound-producing source—is not directly measured. The Appendix E theory labels the eventual winner as g1 > g2 and then shows the gap is amplified; it explains concentration, not correctness. Table 4 does the same: it post hoc calls the ground-truth mask with higher IoU the dominant source, so samples where both IoUs are near zero are absorbed into the dominant-source statistics. The acknowledged false activations (mouth instead of instrument, Appendix A.2) are exactly the failure mode that corrupts M_dom and can make Stage 2 discard the true subdominant source. Also, audio separation quality is never measured, which is why \"breaks the circular dependency\" overstates the evidence.\n\nNone of this kills the paper. The empirical gap in Table 4 is real evidence of concentration, the ablation shows progressive training matters, and the protocol analysis is a genuine contribution. What the paper needs: protocol-consistent numbers for all baselines, a direct measurement of how often the selected dominant region is a true sound producer, and released benchmark/code. With those, it is a solid venue-level paper. Send it to review.","headline":"A useful two-stage method and a needed evaluation fix, held back by protocol-mixing in the main table and an unmeasured identification step at the core.","tokens_in":22093,"tokens_out":3648,"would_cite":true,"duration_ms":34745,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A contrastive model trained on mixed audio naturally locks onto one dominant sound source, and the paper turns that bias into a two-stage method that also finds the quieter source.","keywords":["audio-visual localisation","self-supervised learning","contrastive learning","selective convergence","dual-source sound localisation","spatial prior","segmentation-mask evaluation","sound source separation"],"falsifier":"Build a synthetic dual-source test set in which one object is small and visually inconspicuous but clearly audible, while a large, motion-rich, silent object dominates the frame. Train only Stage 1 with the paper's setup; if the heatmap concentrates on the silent object, or if Stage 2 then fails to recover the true subdominant source, the claim that selective convergence supplies a reliable spatial prior fails. At the theory level, measuring the initial per-region similarity gap $g_1-g_2$ across random initialisations would decide whether that gap is a pre-existing premise or an artefact of training.","tokens_in":21041,"feed_emoji":"🎵","tokens_out":8856,"duration_ms":84024,"temperature":0.7,"pith_summary":"The paper tries to establish that the circular dependency of multi-source audio-visual localisation—separating audio requires knowing where sources are, localising requires separated audio—can be broken by an emergent bias of contrastive learning. When a contrastive model is trained on mixed audio, it naturally locks onto the single most salient sound-source correspondence, a behaviour the authors call selective convergence. The paper builds a two-stage framework that first harvests that bias into a dominant-source prior and then uses the prior to unmask the remaining source, achieving the best numbers among self-supervised methods on dual-source benchmarks without any manual labels. It also argues that the standard bounding-box evaluation is systematically biased and contributes a large pixel-level mask benchmark, VGGSound-DuetMask, so that localisation quality is measured against actual object boundaries. If true, the practical payoff is a scalable, annotation-free route to multi-source localisation that treats a known failure mode of contrastive learning as a useful spatial prior.","feed_headline":"Selective convergence localizes two sound sources, no labels needed","feed_subtitle":"A two-stage contrastive model lets the dominant source win first, then uses it to unmask the quieter one.","key_machinery":"Selective convergence is the paper's name for the tendency of contrastive audio-visual training on mixed audio to concentrate the learned similarity map on a single, most salient source correspondence instead of spreading across all sources. Stage 1 obtains a heatmap from the dot product $\\langle V_i, a_i \\rangle$ and a contrastive loss with differentiable thresholding; this heatmap is post-processed into a clean spatial prior $M_{\\mathrm{dom}}$. Stage 2 partitions visual features as $V_1 = V \\odot M_{\\mathrm{dom}}$ and $V_2 = V \\odot (1 - M_{\\mathrm{dom}})$, then a cross-attention audio decoupler uses each visual partition as keys and values to extract source-specific audio features, and source-specific contrastive losses produce the final maps. The supporting theorem shows that the gradient update strengthens only the dominant audio component $c_1$ while leaving $c_2$ unchanged, so any initial gap $g_1 > g_2$ is amplified into winner-take-all selection.","core_discovery":"The paper's central claim is that contrastive audio-visual training on mixed audio does not attempt to represent all sound sources equally: it selectively converges to the most salient audio-visual correspondence, and this bias can be exploited rather than suppressed. The paper formalises this as SCAV, a two-stage framework in which Stage 1 learns a heatmap for the dominant source and Stage 2 uses that heatmap as a spatial prior to partition visual features and cross-attend the mixed audio, yielding two independent localisation maps. The paper further claims that this approach outperforms all self-supervised methods on dual-source benchmarks and even surpasses some weakly supervised methods on certain metrics, and that bounding-box evaluation systematically overrewards coarse activation, motivating a new segmentation-mask benchmark.","pith_inferences":["A natural extension the paper does not test is iterating the same select-then-unmask recipe beyond two sources: after Stage 2 produces two maps, a third stage could mask both learned sources and re-run the decoupler.","Selective convergence may be a general explanation for why many contrastive audio-visual models fail on multi-source scenes; prior methods that engineer against this bias with hand-tuned thresholds may be fighting the learning dynamics instead of using them.","Stage 2's cross-attention audio decoupler effectively performs audio separation from a visual prior, so the same architecture could be turned into a separation model by reading out the decoupled audio features, a testable extension.","Using SAM-generated masks as training pseudo-labels rather than only as a test benchmark could further remove the need for bounding boxes, but that goes beyond what the paper claims."],"forward_implications":["Because selective convergence is a property of the training objective rather than a hand-crafted rule, any contrastive audio-visual model trained on mixed audio can supply the dominant-source prior for free, with no annotations.","Progressive two-stage training outperforms joint end-to-end training: an unstable early prior misleads the decoupler, so letting Stage 1 converge first is a necessary condition for Stage 2's gains.","The source-wise protocol used by earlier benchmarks hides cross-source false positives, so published IoU numbers for some multi-source methods are inflated; the derivation in the paper shows that source-wise IoU always upper-bounds frame-wise IoU for the same predictions.","Box-based metrics reward activating background pixels inside the rectangle, whereas SCAV's gains are larger under the new segmentation-mask evaluation, indicating that the improvement is genuine localisation rather than coarse rectangle-filling.","The performance gap over self-supervised baselines widens on the larger and more diverse VGGSound-Duet set, suggesting the benefit of leveraging selectivity grows with data scale."],"supporting_citations":[{"why":"It supplies the contrastive objective with differentiable thresholding that Stage 1 trains with, and the VGGSound-Source test set that the new benchmark builds on.","marker":"[6]"},{"why":"It defines the dual-source formulation and the MUSIC-Duet benchmark with bounding-box annotations that SCAV is tested on.","marker":"[15]"},{"why":"It provides the VGGSound-Instruments segmentation-mask benchmark and the frame-wise evaluation convention adopted in this paper.","marker":"[17]"},{"why":"It contributes the VGGSound-Duet benchmark and the source-wise evaluation code whose inflation the paper analyzes, and is a weakly-supervised baseline.","marker":"[26]"},{"why":"Its simplicity-bias theory of feature suppression in contrastive learning is what Theorem 1 adapts to explain selective convergence.","marker":"[38]"},{"why":"It generates the segmentation masks used to construct the new VGGSound-DuetMask evaluation set.","marker":"[22]"},{"why":"It is the iterative curriculum baseline that the paper reproduces under frame-wise evaluation, requiring code changes for the dual-source layout.","marker":"[19]"}],"fun_headline_variants":["Selective convergence unlocks dual-source audio-visual localization","Two-stage self-supervision finds both sound sources, no labels","Pixel-level masks reveal evaluation bias in audio-visual localization","Contrastive models pick the loudest source first, then find the quiet one","Dominant source first: self-supervised two-source localization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes that whatever object Stage 1 selectively converges to is actually producing the dominant sound; because the paper defines dominance as whatever the model picks, a wrong pick corrupts the spatial prior and can cause Stage 2 to discard the true subdominant source.","fun_headline_variants_meta":{"raw":{"variants":["Selective convergence unlocks dual-source audio-visual localization","Two-stage self-supervision finds both sound sources, no labels","Pixel-level masks reveal evaluation bias in audio-visual localization","Contrastive models pick the loudest source first, then find the quiet one","Dominant source first: self-supervised two-source localization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000651,"raw_usage":{"total_tokens":2986,"prompt_tokens":943,"completion_tokens":2043,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":1956}},"tokens_in":559,"tokens_out":2043,"duration_ms":14960,"temperature":1.0,"reasoning_tokens":1956,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T22:58:06.370936+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a synthetic dual-source test set in which one object is small and visually inconspicuous but clearly audible, while a large, motion-rich, silent object dominates the frame. Train only Stage 1 with the paper's setup; if the heatmap concentrates on the silent object, or if Stage 2 then fails to recover the true subdominant source, the claim that selective convergence supplies a reliable spatial prior fails. At the theory level, measuring the initial per-region similarity gap $g_1-g_2$ across random initialisations would decide whether that gap is a pre-existing premise or an artefact of training.","supporting_citations":[{"cited_title":"In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","cited_arxiv_id":null,"evidence_quote":"It supplies the contrastive objective with differentiable thresholding that Stage 1 trains with, and the VGGSound-Source test set that the new benchmark builds on."},{"cited_title":"Advances in Neural Information Processing Systems33, 10077–10087 (2020)","cited_arxiv_id":null,"evidence_quote":"It defines the dual-source formulation and the MUSIC-Duet benchmark with bounding-box annotations that SCAV is tested on."},{"cited_title":"In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (2022) Whence the Voice? 33","cited_arxiv_id":null,"evidence_quote":"It provides the VGGSound-Instruments segmentation-mask benchmark and the frame-wise evaluation convention adopted in this paper."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It contributes the VGGSound-Duet benchmark and the source-wise evaluation code whose inflation the paper analyzes, and is a weakly-supervised baseline."},{"cited_title":"In: International Conference on Machine Learning (2023)","cited_arxiv_id":null,"evidence_quote":"Its simplicity-bias theory of feature suppression in contrastive learning is what Theorem 1 adapts to explain selective convergence."},{"cited_title":"In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","cited_arxiv_id":null,"evidence_quote":"It is the iterative curriculum baseline that the paper reproduces under frame-wise evaluation, requiring code changes for the dual-source layout."}],"review_version":1}