{"id":"710f5c59-7756-4d4f-92bb-5ba95ef70142","arxiv_id":"2608.03264","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A new audio-visual instance segmentation architecture, using audio separation and an audio-modulated Mamba, reaches 48.54 mAP on AVISeg with a COCO-pretrained ResNet50.","lead":"This paper presents a model that separates overlapping sounds and uses them to identify and track the objects that make them in video. It reports the new best score on the AVISeg audio-visual instance segmentation benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"HCM's cross-modal similarity is dimensionally undefined and uses unaligned feature spaces; the ASP's claimed causal gain is not established.","rationale":"The reader's conditional verdict is appropriate. My stress-test converges on the same weakest assumption: the ASP's HCM lacks any demonstrated shared semantic space. I add that Eq. (3) as written is dimensionally inconsistent (outer product, not scalar correlation), which makes the mechanism unimplementable without an unstated correction. If the authors intended a dot product, the absence of a learned alignment remains. The +3.83 mAP ablation (Table 3) is the only direct evidence that the HCM works, so this is load-bearing: without a valid correspondence metric, the causal story for the ASP collapses, even if the final mAP number is reproducible. The abstract's 'surpassing previous by 7.8%' is also inconsistent with Table 2 (ACVIS COCO is 46.68; 48.54/46.68−1 = 3.98%), but that is an easier reporting fix. I do not see grounds to reject outright: the ADM ablation and the zero-shot AVSS results provide independent support, and the HCM could be repaired with a learned projection. The verdict should remain conditional, contingent on the authors correcting Eq. (3), demonstrating or adding feature alignment, releasing code, and fixing the comparison statement.","tokens_in":16664,"tokens_out":15248,"duration_ms":154855,"concrete_test":"Re-implement Eq. (3) with the exact tensor shapes from Sec. 3.3.1/3.3.2: if C_ij is D×D, the softmax in P cannot be evaluated, confirming the formula as written is unimplementable. Then run the HCM with (a) raw VGGish/pixel-decoder centroids (current) vs (b) a learned linear projection on audio features before k-means/dot product, on AVISeg with the same training protocol. If (b) changes mAP by more than ~1 point, the 'shared semantic space' assumption is load-bearing; if (b) is neutral, the model learns the alignment implicitly and the conceptual concern is not practically damaging.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central causal claim rests on the Hierarchical Correspondence Mechanism. Eq. (3) defines C_ij = centroid(AC_i) × centroid(VC_j)^T / sqrt(D) as the 'correlation between cluster centroids.' As written, this is an outer product of two D-vectors, yielding a D×D matrix, so the subsequent softmax P = VC_j exp(C_ij) / sum_k VC_k exp(C_ik) is not defined. If the intended operation is a dot product, the paper still asserts in Sec. 3.3.2 that k-means clusters of frozen VGGish audio features and trainable pixel-decoder video features live in 'a shared semantic space' without any learned projection, metric alignment, or semantic supervision. VGGish is an AudioSet audio-tagging embedding; pixel-decoder features are optimized for segmentation; their raw coordinate systems are not commensurable. The ASP is credited with +3.83 mAP (Table 3), so if C_ij is arbitrary, the improvement may come from added sparse-attention capacity rather than from the claimed audio-visual correspondence. This is a load-bearing gap in the mechanism, independent of whether the ADM or the final SOTA number is correct.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes H2S (Hear to See), a two-module architecture for audio-visual instance segmentation (AVIS). An Acoustic-Semantic Projector (ASP) separates the mixed audio signal into independent streams with a frozen MixIT source-separation model and a frozen VGGish encoder, then performs a k-means-based Hierarchical Correspondence Mechanism (HCM) to associate audio and visual clusters. An Asynchronous Dynamics Modulator (ADM) adapts the Mamba state-transition parameter Δ using audio gating, intended to improve tracking through asynchronous audio-visual state changes. Experiments on AVISeg report 48.54 mAP with a COCO-pretrained ResNet50, which the abstract and introduction describe as surpassing the previous best by 7.8%. Ablations evaluate ASP, ADM, ASD, HCM hyperparameters, source-separation pretraining, and zero-shot generalization to AVSS.","tokens_in":17007,"tokens_out":7146,"duration_ms":69551,"significance":"If the reported numbers are correct, H2S would be a meaningful advance on AVISeg, and the design is original: disentangling audio sources with an explicit separation model and modulating Mamba's Δ with audio information are both fresh ideas in this task. The empirical study is unusually careful in several places: Table 7 shows that naively concatenating MixIT features to AVISM hurts performance, Table 8 attempts to rule out YFCC100M pretraining as the sole source of gain, and the asynchronous subset targets the paper's central difficulty. However, two load-bearing issues currently prevent acceptance: the formal definition of the HCM correlation is ill-posed, and the '7.8% over previous best' claim is numerically inaccurate with respect to the actual previous best (ACVIS). The hyperparameter narrative is also internally inconsistent with the authors' own Table 5. These issues are substantial but appear fixable within the scope of a major revision.","major_comments":[{"comment":"The claim of 'surpassing the previous best approach by 7.8%' is not correct against the strongest prior. Table 2 lists ACVIS with COCO-pretrained ResNet50 at 46.68 mAP; H2S reaches 48.54 mAP, a relative improvement of about 3.99%, not 7.8%. The 7.8% figure is obtained only when comparing to AVISM at 45.04 mAP. The abstract and introduction should be corrected so the comparator is explicitly ACVIS and the percentage is computed relative to the actual previous best.","section":"Abstract, §1, §4.3, Table 2"},{"comment":"The HCM correlation is not dimensionally defined. If centroid(AC_i) and centroid(VC_j) are both D-dimensional vectors, then centroid(AC_i) × centroid(VC_j)^T is a D×D outer product, and the subsequent expression P = VC_j exp(C_ij) / Σ_k VC_k exp(C_ik) is ill-formed. If '×' is intended to be a dot product, the result is a scalar only under that reading, but the text still does not justify why dot products between VGGish audio centroids and pixel-decoder video centroids are semantically meaningful. The paper asserts these live in 'a shared semantic space,' yet no learned projection, metric alignment, or semantic supervision is described. VGGish is trained for AudioSet tagging, while pixel-decoder features are trained for segmentation; their raw geometries are not commensurable. Because Table 3 attributes +3.83 mAP to the ASP, this is a load-bearing issue: the observed gain could come from","section":"Eq. (3), §3.3.2, Table 3"},{"comment":"The textual interpretation of the hyperparameter ablation is contradicted by the table. The text says 'Setting the cluster number to 8 achieves the best performance,' but Table 5 (left) shows mAP = 48.87 for C_ka = C_kv = 10 versus 48.54 for 8. Similarly, the text states that the hierarchical Top-P configuration achieves optimal performance, but the uniform Top-P = 0.8 row has mAP = 48.67 versus 48.54 for the hierarchical [0.7, 0.8, 0.9] configuration. The authors should either correct the text, report which metric determines 'best,' or explain why the selected hyperparameters are preferred despite the higher mAP of the alternative settings.","section":"Appendix A.1, Table 5"},{"comment":"The asynchronous-subset experiment is central to the ADM claim, but the subset is not specified: there is no description of how 'asynchronous' is defined, how many videos/frames it contains, or whether the subset is fixed and used identically for both methods. Moreover, the comparison is only against AVISM, not against ACVIS, the actual previous best. Without this information, the +14.09 mAP result is difficult to interpret, and the claim that ADM is responsible for the robustness gain is not fully established. Please provide the subset construction details and, if possible, include ACVIS as a comparison.","section":"§4.6, Figure 8"}],"minor_comments":[{"comment":"There are several typos and grammatical issues: 'multimodel' should be 'multimodal' (§2.2, §3), 'workes' should be 'works' (§3.2, §3.4), 'focuse' should be 'focused' (§2.2), and 'Intuitively, a larger Δ forces... NaN' has an extra period. Table 1 has '32.2229.83' (missing space) in the AVISM row.","section":"Throughout"},{"comment":"The symbol D in the denominator sqrt(D) is not defined in Section 3.3.2. Also, the Top-P algorithm is never formally described; only the probability p is listed in Implementation Details. Please define the procedure or cite a source.","section":"Eq. (3) and §3.3.2"},{"comment":"The sentence 'outperforms previous approaches, improving the mAP by 7.8%' should explicitly name ACVIS as the actual previous best, otherwise the comparison is misleading. The GPT-4-style claim 'state-of-the-art' is fine, but the numeric basis must be transparent.","section":"§4.3"},{"comment":"The zero-shot AVSS evaluation is only against AVISM. Since ACVIS is also an AVIS method, it would strengthen the generalization claim to include it. In addition, the two bar charts use different y-axis scales, which makes visual comparison of J and F gains less intuitive, though the numbers are stated in the text.","section":"Fig. 9 and Appendix A.2"},{"comment":"The limitation about offline operation is well stated. However, the hard cap of N_a = 8 separated audio streams is not discussed as a limitation; a scene with more than eight simultaneous sources cannot be represented by the architecture. This should be acknowledged, or justified with dataset statistics.","section":"§5 and Appendix B"}],"recommendation":"major_revision","confidential_remarks":"This is a technically interesting paper with a strong empirical setup, but the two main gates are the ill-posed HCM definition/alignment and the inaccurate '7.8%' comparison. Both are fixable in revision. I would also ask the editor to ensure the authors provide the asynchronous-subset details and correct the Table 5 narrative; these are not fatal but are necessary for a trustworthy final version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a credible SOTA paper on AVISeg with two genuinely new modules, but the headline number is inflated by comparing to the wrong baseline, and Eq. (3) has a shape error that needs to be fixed before the mechanism can be fully trusted.\n\nThe genuinely new stuff is the combination of a MixIT-based sound source separation with a k-means cluster-to-cluster sparse attention (the ASP), and the idea of modulating Mamba's delta parameter with audio dynamics (the ADM). The ablation study is more careful than most in this area: they show that naive concatenation of separated audio actually hurts, that modulating delta beats modulating B or C, and that the MixIT pre-training dataset doesn't explain the gains. That is honest, reproducible-sounding engineering, and I believe the 48.54 mAP result is real.\n\nThe soft spots are real but fixable. First, Eq. (3) defines C_ij as a D×D outer product, then feeds it into a softmax over cluster indices. That can't be right; it should be a scalar dot product. The authors need to correct the notation. Second, the 'shared semantic space' claim is hand-wavy. VGGish features are frozen and pixel-decoder features are trained; the paper doesn't describe any projection or metric alignment between them. Since ASP accounts for +3.83 mAP, the authors need to convince us this isn't just added sparse attention capacity. They could show, for example, that the cluster assignments align with known sound classes. Third, the abstract's '7.8% over the previous best' only holds against AVISM, not ACVIS (46.68 → 48.54 is about 4.0%). That is misleading and should be fixed. There are also minor reporting issues (the text in A.1 says hierarchical Top-p is best, but Table 5 shows fixed 0.8 gives 48.67), and the asynchronous subset is never specified.\n\nOverall, the core idea is worthwhile and the empirical work is above average for the subfield. A reviewer should get a chance to push on the correspondence mechanism and ask for code and error bars. That's a normal revision path, not a desk reject.","headline":"Plausible new SOTA on AVISeg with two genuinely new mechanisms, but the headline 7.8% compares against the wrong baseline and Eq. (3) has a shape error that needs fixing.","tokens_in":17488,"tokens_out":3667,"would_cite":true,"duration_ms":35493,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that disentangling mixed audio into separate streams before matching them to video pixels, then using audio dynamics to modulate the state-transition parameter of a Mamba tracker, enables state-of-the-art audio-visual insta","keywords":["audio-visual instance segmentation","acoustic disentanglement","source separation","asynchronous dynamics","state space models","Mamba","cross-modal correspondence","video instance segmentation"],"falsifier":"Replace the frozen VGGish audio encoder with a different pretrained audio network (or with random projections) and retrain H2S on AVISeg; if the mAP gain over the no-ASP baseline largely persists, the cluster-correspondence mechanism does not depend on the shared semantic space. Separately, evaluate on a constructed test set with more than eight concurrently sounding objects: a sharp mAP drop would confirm the fixed N_a=8 streams as the binding limit.","tokens_in":16576,"feed_emoji":"🎧","tokens_out":9788,"duration_ms":85795,"temperature":0.7,"pith_summary":"The paper tackles audio-visual instance segmentation: producing a pixel-level mask for each sounding object and holding its identity across time. It argues that the two bottlenecks—mixed audio that superimposes several sources in one channel, and audio and video states that change at different moments—can be solved by separating the audio before matching, and by letting audio dynamics control a state-space tracker's transition speed. The proposed H2S combines the Acoustic-Semantic Projector (ASP), which uses a frozen MixIT separator and VGGish to create independent audio streams and matches them to video via k-means cluster attention, with the Asynchronous Dynamics Modulator (ADM), which injects audio-derived gating into Mamba's Δ parameter. On the AVISeg benchmark, with a COCO-pretrained ResNet-50, H2S reports 48.54 mAP, 65.70 HOTA, and 47.58 FSLA, 7.8% above the previous best, and a 14.09 mAP gain over the fixed-window baseline on an asynchronous subset. If the claims hold, the work establishes a reusable recipe for audio-visual instance-level understanding: disentangle first, then let audio steer the temporal dynamics.","feed_headline":"Disentangle the audio, then match: 48.54 mAP on AVISeg","feed_subtitle":"Separating overlapping sound sources and letting audio steer the tracker's state transitions handles sudden on/off events.","key_machinery":"The two load-bearing mechanisms are the Acoustic-Semantic Projector (ASP) and the Asynchronous Dynamics Modulator (ADM). ASP consists of Audio Source Disentanglement (ASD), a frozen MixIT separation model plus VGGish that converts one mixed audio channel into N_a=8 independent feature streams, and a Hierarchical Correspondence Mechanism (HCM) that k-means clusters audio and video tokens, scores cluster-centroid similarities in a shared semantic space, and uses top-p filtered sparse matrix multiplication to attend from audio streams to visual tokens. ADM wraps an Audio-Dynamically Modulated Mamba (AMM), where the SSM's Δ (the zero-order-hold timescale) is augmented by audio features passed th","core_discovery":"H2S is an audio-visual instance segmentation architecture whose central claim is that precise audio-visual correspondence and robust tracking under asynchronous state changes are both achievable by treating the audio track as a set of separable sources and by making the temporal model state-aware. The ASP first runs MixIT source separation and VGGish to produce N_a=8 independent audio streams (frozen), then k-means clusters audio and video features, computes a cluster-centroid similarity weighted by cluster size, filters it with top-p, and projects the result back to tokens to produce sparse cross-modal attention. The ADM feeds video object queries into a Mamba block and modifies the Δ param","pith_inferences":["A natural testable extension is to check whether the cluster-level correspondence remains discriminative under a learned projection or contrastive loss; if the k-means centroids already work without alignment, training such a projection might push mAP further, and if they do not, the current 48.54 would be tied to the specific VGGish geometry.","The fixed N_a=8 separation streams and cluster counts (8) look tuned to the AVISeg distribution; in denser or more varied scenes an adaptive stream count would be needed, and one quick check is to measure mAP on synthetic scenes with more than eight simultaneous sources.","The authors scope the claim to offline inference; extending ADM to causal/streaming settings would require replacing full-context audio separation with a streaming separator and causal scanning, so the SOTA result should not be read as an online tracking result.","If the recipe generalizes, the same 'disentangle then cluster-match, then modulate timescale' pattern could apply to other many-to-one multimodal alignment problems, such as multi-speaker localization or audio-guided video object tracking, where one modality is a mixed superposition and the other is spatially structured."],"forward_implications":["On AVISeg, H2S establishes a new reported state of the art: 48.54 mAP with a COCO-pretrained ResNet-50, beating the AVISM baseline by 7.8% relative mAP and the strongest prior AVIS result by roughly 4%.","Explicit source separation before matching is the effective ingredient: naively concatenating MixIT-separated features to the AVISM baseline drops mAP by 0.62, while the full ASP adds +3.50 mAP, so the clustering-and-attention step is what converts separated audio into a gain.","The ADM's Δ modulation primarily buys tracking accuracy: switching from modulating Δ to modulating B or C reduces HOTA, and on the asynchronous subset H2S gains 14.09 mAP over the fixed-window AVISM baseline.","The performance gain does not come from the external YFCC100M pretraining of MixIT: retraining the separator on AVISeg alone gives 48.62 mAP, essentially identical to 48.54, which the paper reads as evidence that the architecture, not external data, drives the result.","Zero-shot transfer to the AVSS dataset improves class-agnostic J and F scores by 1.13 and 1.85 over AVISM, suggesting the learned audio-visual alignment transfers beyond the 26 AVISeg labels."],"supporting_citations":[{"why":"Defines the AVIS task and AVISeg dataset and provides the AVISM baseline architecture and loss setup that H2S extends and compares against; the headline 48.54 mAP is measured on this dataset.","marker":"[14]"},{"why":"Supplies the MixIT unsupervised source-separation model used in ASD to disentangle the mixed audio into streams; the paper's core claim depends on separated audio streams.","marker":"[36]"},{"why":"Supplies the VGGish audio feature extractor (pretrained on AudioSet) that turns each separated source into the audio features entering the HCM and ADM.","marker":"[18]"},{"why":"Supplies the Mamba selective state-space model and its input-dependent Δ parameter, which the ADM modulates with audio gating; the asynchronous-dynamics mechanism is built directly on this formulation.","marker":"[12]"},{"why":"Supplies the Mask2Former-style frame-level mask transformer and loss formulation used to supervise frame-level queries and decode masks.","marker":"[5]"},{"why":"Supplies the inter-frame communication transformer formulation used for video-level query matching and temporal consistency loss in the tracker.","marker":"[21]"},{"why":"Supplies the VITA query-association paradigm and the similarity loss aligning frame and video queries, which the tracker adopts for identity consistency.","marker":"[17]"}],"fun_headline_variants":["Hear to See: 48.54 mAP, 7.8% gain on AVISeg","Stateful listening lifts AVISeg to 48.54 mAP","Disentangle audio, sync states: 48.54 mAP on AVISeg","Audio-guided Mamba state transitions: 7.8% AVISeg mAP jump"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The load-bearing premise is that the frozen VGGish audio streams and the trainable video features are already comparable in a shared semantic space, so that the k-means centroid dot product in Eq. (3) measures genuine audio-visual correspondence; the paper describes no learned projection or semantic alignment, and the fixed eight-stream limit also caps the number of simultaneous sources the model can represent.","fun_headline_variants_meta":{"raw":{"variants":["Hear to See: 48.54 mAP, 7.8% gain on AVISeg","Stateful listening lifts AVISeg to 48.54 mAP","Disentangle audio, sync states: 48.54 mAP on AVISeg","Audio-guided Mamba state transitions: 7.8% AVISeg mAP jump"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001339,"raw_usage":{"total_tokens":5278,"prompt_tokens":738,"completion_tokens":4540,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":482,"completion_tokens_details":{"reasoning_tokens":4445}},"tokens_in":482,"tokens_out":4540,"duration_ms":35861,"temperature":1.0,"reasoning_tokens":4445,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T22:13:44.209334+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the frozen VGGish audio encoder with a different pretrained audio network (or with random projections) and retrain H2S on AVISeg; if the mAP gain over the no-ASP baseline largely persists, the cluster-correspondence mechanism does not depend on the shared semantic space. Separately, evaluate on a constructed test set with more than eight concurrently sounding objects: a sharp mAP drop would confirm the fixed N_a=8 streams as the binding limit.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the AVIS task and AVISeg dataset and provides the AVISM baseline architecture and loss setup that H2S extends and compares against; the headline 48.54 mAP is measured on this dataset."},{"cited_title":"Unsupervised Sound Separation Using Mixture Invariant Training","cited_arxiv_id":"2006.12701","evidence_quote":"Supplies the MixIT unsupervised source-separation model used in ASD to disentangle the mixed audio into streams; the paper's core claim depends on separated audio streams."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Mask2Former-style frame-level mask transformer and loss formulation used to supervise frame-level queries and decode masks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the inter-frame communication transformer formulation used for video-level query matching and temporal consistency loss in the tracker."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the VITA query-association paradigm and the similarity loss aligning frame and video queries, which the tracker adopts for identity consistency."}],"review_version":1}