{"id":"db903124-df40-4850-8f55-a0dd9679d8d3","arxiv_id":"2608.08147","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"SAMOT uses adaptive Mamba step modulation and optimal transport matching to set a new state-of-the-art on the AVISeg audio-visual instance segmentation benchmark.","lead":"This paper presents a system for audio-visual instance segmentation: locating, segmenting, and tracking every object that is making a sound in a video. It combines adaptive state-space model step sizes with optimal transport matching to handle changing sound sources and audio-visual mismatches.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The core ADSM mechanism is asserted but never written: Eq. (6) defines Delta = Delta_0 + d and says Delta is directly injected into the Mamba discretization, yet no discrete state-space update or A_bar/B_bar formula appears; the implementation could be a gating heuristic rather than true step…","rationale":"I read the paper as making two coupled claims: (1) SAMOT is the current SOTA on AVISeg, and (2) the gains come from ADSM and OT-MM. The benchmark numbers are plausible internally, with consistent ablation trends and a coherent pipeline, so I do not dispute the empirical headline as reported. The least secure load-bearing point is ADSM: the entire mechanism rests on a scalar offset being 'directly injected' into the Mamba discretization, but the manuscript never writes the resulting state update. The reader's weakest-assumption analysis identifies this same gap, and I agree it is the right place to probe. A real implementation could easily differ from the prose, and the difference is not cosmetic: a multiplicative gate on the hidden state would still improve accuracy while not being a step-size modulation. The proposed check is straightforward because the code is promised; if the code confirms the matrix-exponential discretization, the concern closes, and if not, the mechanism claim needs revision. I also noted the disclosed mismatch between the official AVISM baseline in Table 1 and the reproduced ablation baseline, and the missing error bars, but those are secondary to the unverified core mechanism. Keeping the CONDITIONAL verdict is appropriate: the method may well work, but the central mechanism must be specified and checked before the SOTA-and-causation claim is accepted.","tokens_in":16739,"tokens_out":6172,"duration_ms":60603,"concrete_test":"Inspect the released code at https://github.com/happylife-pk/SAMOT at the ADSM forward pass and write the exact discretized update: confirm whether Delta_t = Delta_0 + d_t enters A_bar = exp(Delta_t A) and B_bar = (Delta_t A)^{-1}(exp(Delta_t A)-I) Delta_t B. If the code realizes those matrix exponentials with the modulated step, the concern is resolved; if Delta_t only multiplies a gate or an update scale outside the discretization, the paper's mechanism description is inaccurate and the ablation-based attribution should be downgraded. Either way, the resulting equation should be added to Sec. 3.2.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that ADSM drives the reported gains requires that the scalar offset d in Eq. (6) actually changes the Mamba discretization step. The paper never writes the discrete-time state-space recursion. In Mamba, the step size enters through A_bar = exp(Delta A) and B_bar = (Delta A)^{-1}(exp(Delta A)-I) Delta B, so to substantiate 'directly injected into the discretization step' the paper must specify h_t = A_bar(Delta_t) h_{t-1} + B_bar(Delta_t) x_t, the relationship of Delta_0 to the per-channel Mamba time-scale parameter, and the behavior when Delta_0 + d is nonpositive. Without this, an implementation could apply d as a gate on h_{t-1} or as a scaling factor on the update, which is not state-space step modulation. Then the +3.76 FSLA gain could still be obtained, but it would not validate the mechanism claimed in Sec. 3.2, and the attribution of the gain to a truly adaptive discretization step is unsupported. This is an internal gap between Eqs. (1)-(6) and the assertion that follows them, not a disagreement with consensus.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SAMOT, a framework for audio-visual instance segmentation (AVIS) that combines Adaptive Dynamic Step Modulation (ADSM) for Mamba-based temporal modeling and Optimal Transport-based Matching Modulation (OT-MM) with MMD regularization for cross-modal instance-level association. The method is evaluated on the AVISeg benchmark and claims state-of-the-art results, with reported gains of +3.76 FSLA, +2.75 HOTA, and +2.58 mAP over AVISM under MS-COCO pretraining, supported by module ablations, qualitative visualizations, and an efficiency analysis. The code and model are made publicly available.","tokens_in":17013,"tokens_out":3864,"duration_ms":37815,"significance":"If the reported results are reproducible, SAMOT would be the current best performer on AVISeg, and the two proposed mechanisms address real limitations of fixed-step Transformers and implicit matching in long-sequence audio-visual modeling. The paper's strengths include a public code release, a systematic set of ablations for the main components (Tables 2-5), and an efficiency comparison (Fig. 7). However, the central mechanistic claim about ADSM is asserted but not formally derived, the ablation baseline is numerically different from the official baseline used in the headline comparison, and all results come from single runs without error bars. These issues currently leave the attribution of the reported gains to the proposed mechanisms insufficiently supported.","major_comments":[{"comment":"The statement that the adjusted delta is 'directly injected into the discretization step of the selective state-space mechanism' is not substantiated by any equation in the main text. The paper never specifies how delta enters the discrete-time recursion (e.g., A_bar_t = exp(delta_t A) and B_bar_t = (delta_t A)^{-1}(exp(delta_t A) - I) delta_t B), nor does it define the relationship between Delta_0 and the Mamba per-channel time-scale parameter, nor the behavior when Delta_0 + d is non-positive. Without this specification, the implementation could act as a gating or scaling heuristic rather than true step-size modulation, and the attribution of the +3.76 FSLA gain to genuinely adaptive discretization is unsupported. This is a load-bearing gap between Eq. (6) and the claim that follows it.","section":"Sec. 3.2, Eq. (6)"},{"comment":"The headline comparison in Table 1 uses the official AVISM baseline, while the module ablation in Table 2 uses a reproduced baseline that includes the hierarchical temporal tracker but omits ADSM and OT-MM. The paper explicitly notes that these baselines are not numerically identical, but it does not report where the reproduced baseline sits relative to the official AVISM number. Consequently, the incremental improvements in Table 2 (e.g., +2.62 FSLA from adding ADSM and OT-MM to the reproduced baseline) cannot be reconciled with the +3.76 FSLA gain over official AVISM in Table 1. The authors should either include the reproduced baseline in Table 1 or provide the numerical relationship between the two baselines; otherwise the attribution of the headline gains to the individual modules is not directly verifiable.","section":"Sec. 4.1, Tables 1 and 2"},{"comment":"All experimental results are reported from single runs with no standard deviations, confidence intervals, or significance tests. This is particularly concerning for the standard-setting comparison, where the HOTA gain over AVISM is only +0.85; such a difference is within plausible run-to-run variation for instance segmentation methods trained with batch size 1. The authors should report results over multiple seeds (or at least state the number of runs) to support the state-of-the-art claim.","section":"Sec. 4.1, Table 1"},{"comment":"The log-domain Sinkhorn updates are essential for OT-MM, but the 'detailed derivations' are deferred to a supplementary file that is not included in the preprint. The reader therefore cannot verify that the temporal prior in Eq. (8) preserves the desired marginal constraints, how M^{t-1} is initialized at the start of a sequence, or why L=10 Sinkhorn iterations are sufficient. Please include these derivations in the manuscript or provide a public version of the supplementary that is referenced with a stable link.","section":"Sec. 3.3, Eqs. (8)-(9)"}],"minor_comments":[{"comment":"The phrase 'methods performing implicit matching ignore the inherent distributional inconsistencies' is overly broad; implicit matching methods such as cross-attention may reduce semantic gaps without explicitly modeling distributional discrepancies, and the sentence could be rephrased to make that distinction.","section":"Abstract and Intro"},{"comment":"The sentence 'AV-Mamba [25] integrate temporal and cross-modal selective mechanisms' has a subject-verb agreement error; it should be 'integrates'.","section":"Sec. 2.2"},{"comment":"The numeric labels inside the bar chart are very small and hard to read; a table or larger font would make the comparison of alternative distance formulations clearer.","section":"Fig. 6"},{"comment":"Reference [3] cites the arXiv version of Mask2Former for Video Instance Segmentation; if a peer-reviewed version exists, it should be cited instead.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is a credible benchmark contribution with a clear and addressable gap: the ADSM mechanism is described only at an architectural level, and the missing state-space equations are essential for validating the central claim. I recommend major revision rather than reject because the missing derivations and baseline clarification can be added without changing the overall method. Note also that the preprint references a supplementary file that is not included; if the camera-ready submission relies on that supplement, it must be made available. The large reported FSLAn gaps in Table 1 (e.g., Ours* 36.73 vs. ACVIS* 10.74) are striking and would benefit from an explicit explanation in the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look. AVIS is a small subfield and this is one of a handful of methods on the benchmark, so the SOTA claim means something locally. The two ideas—adaptive step-size for Mamba and OT-based cross-modal matching—are not brand new individually, but the combination and the instance-level framing are new. The paper does its homework: clear problem statement, sensible ablations that mostly isolate the components, and qualitative evidence that the dynamics do what is claimed. Code and models are promised, which raises the bar for verification.\n\nThe main soft spot is exactly the one in the stress test: the core ADSM mechanism is described in prose but the discrete state-space equation never appears. The paper says Δ is 'directly injected into the discretization step,' but no h_t = A_bar(Δ_t)h_{t-1} + B_bar(Δ_t)x_t, no relation of Δ_0 to the Mamba time-scale parameter, and no discussion of nonpositive Δ. That is a real gap, because the reader cannot tell whether they actually changed the SSM discretization or just applied a gate or scaling on the hidden state. The ablation shows a gain, and the visualization of effective Δ is suggestive, but the mechanism is under-specified. This is fixable with one equation and a sentence or two; it is not evidence of a bad method.\n\nAnother soft spot is the ablation baseline. They are upfront that it is their own reproduced framework, not the official AVISM, so the incremental numbers are internally consistent but not directly comparable to the Table 1 gains. That is acceptable but should be flagged more prominently; also, there are no error bars or multiple seeds, which is common in this literature but still limits confidence.\n\nThe missing supplementary derivations (Sinkhorn updates, MMD details) are an artifact of the arXiv version, not a flaw in the work itself, but they make verification harder.\n\nBottom line: a plausible, well-engineered contribution to AVIS. It will not reshape a major field, but for a niche benchmark it is a clear advance. It deserves a serious referee. I would send it to review with a request for the missing ADSM equation, the supplementary, and ideally code or at least seed-variance numbers.","headline":"Solid, honest AVIS paper with a real mechanism gap: the ADSM-to-Mamba equation is asserted but never written; everything else points to a legitimate SOTA result that deserves a normal peer review.","tokens_in":17488,"tokens_out":2210,"would_cite":true,"duration_ms":23120,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A framework with adaptive Mamba step sizes and optimal-transport matching achieves state-of-the-art results on the AVISeg benchmark, beating the prior best by +3.76 FSLA, +2.75 HOTA, and +2.58 mAP under MS-COCO pretraining.","keywords":["Audio-visual instance segmentation","State space model","Mamba","Optimal transport","Sinkhorn iterations","Multimodal matching","Video instance segmentation","Temporal modeling"],"falsifier":"If a variant that keeps the base Mamba step size fixed and instead multiplies the audio-conditioned visual features by a per-frame learned gate reproduces the same FSLA, HOTA, and mAP gains on AVISeg, then the step-size injection is not the operative mechanism and the ADSM claim collapses.","tokens_in":16574,"feed_emoji":"🎬","tokens_out":6264,"duration_ms":56206,"temperature":0.7,"pith_summary":"This paper addresses audio-visual instance segmentation, the task of classifying, segmenting, and tracking every sounding object in a video. It argues that existing fixed-step Transformers and Mamba models cannot handle modality-state changes in long sequences, and that implicit cross-attention cannot resolve structural and distributional differences between audio and video. To fix this, the paper proposes SAMOT with Adaptive Dynamic Step Modulation, which adjusts the Mamba state-update step size based on temporal and cross-modal signals, and Optimal Transport-based Matching Modulation, which aligns audio and video queries by solving an entropy-regularized optimal transport problem with spatial and temporal priors plus an MMD regularizer. On the AVISeg benchmark the framework reports state-of-the-art results, beating the previous best by +3.76 FSLA, +2.75 HOTA, and +2.58 mAP under MS-COCO pretraining.","feed_headline":"Adaptive Mamba steps and optimal transport lift AVISeg scores","feed_subtitle":"A state-aware step-size controller plus explicit audio-visual matching beats the prior best on the AVISeg benchmark.","key_machinery":"The machinery is a pair of modules inserted into a Mamba-based audio-visual segmentation pipeline. ADSM computes two scalar indicators of the current modality state: $s_{\\text{modality}}$ (max-norm temporal change of audio-conditioned and raw visual features) and $s_{\\text{diff}}$ (cosine distance between them), then uses a $\\Delta$ Gating Router to combine three tanh branches—current audio-conditioned visual, current audio, and historical context—into a step-size offset $d$ that is added to the fixed base step $\\Delta_0$ and injected into the selective state-space discretization. OT-MM builds a transport cost $C_t$ from normalized feature distance plus a relative positional prior $\\beta |i-j| / \\max(N_a, N_v)$, warms the log-kernel with the previous transport plan as a temporal consistency prior, solves the entropy-regularized problem in the log domain with Sinkhorn iterations, projects audio queries through the transport plan, and regularizes the resulting distributions with a multi-kernel MMD loss.","core_discovery":"The central claim is that the two bottlenecks in audio-visual instance segmentation—complex modality-state changes and cross-modal distribution mismatch—can be addressed by making the temporal state-space model input-adaptive and by making cross-modal association an explicit matching problem. ADSM estimates a temporal variation score and a cross-modal discrepancy score, feeds them through a $\\Delta$ Gating Router that weights visual, audio, and historical branches, and adds the resulting offset to the base Mamba step size, so the model updates faster at modality transitions and stays stable otherwise. OT-MM defines a cost matrix combining cosine feature distance with a relative positional prior, reuses the previous time step's transport plan as a temporal consistency prior, solves the entropy-regularized problem with log-domain Sinkhorn iterations, and adds an MMD loss to align the matched audio and video distributions. The paper reports that this combination achieves state-of-the-art performance on AVISeg, with FSLA 48.18, HOTA 67.27, and mAP 47.62 under MS-COCO pretraining, surpassing AVISM by +3.76, +2.75, and +2.58 respectively.","pith_inferences":["The same ADSM recipe—per-step state indicators routed into a step-size offset—could be applied to other state-space-model multimodal tasks such as audio-visual event localization or video question answering, where modality-state changes also break long-range modeling.","The log-domain Sinkhorn matching with a temporal prior and MMD regularizer could serve as a generic alignment layer for any tracking-by-matching problem with heterogeneous query sets, not just sound-video pairs.","A testable extension: replacing the manually fixed priors $\\beta$, $\\gamma$, and $\\epsilon$ with learned or scheduled values could further improve the transport plan when instance counts vary widely across frames.","The paper's Fig. 9 suggests a causal variant; an explicit benchmark of streaming latency versus offline accuracy would clarify deployment potential."],"forward_implications":["If ADSM works as claimed, state-space models for long-video audio-visual understanding can remain stable during steady segments while reacting quickly to sound-source switches or audio dropouts.","If OT-MM works as claimed, explicit optimal-transport matching with positional and temporal priors yields more reliable instance-level audio-visual correspondence than implicit cross-attention or contrastive learning.","The reported gains on FSLAs and FSLAm suggest the framework is especially stronger in silent and multi-source scenarios, which are the hard cases for AVIS.","The online (causal) variant retaining competitive performance suggests the same principles could transfer to streaming inference."],"supporting_citations":[{"why":"Supplies the selective state-space model whose step size ADSM modulates.","marker":"[16]"},{"why":"Defines the AVIS task and benchmark, and is the primary baseline AVISM that SAMOT compares against.","marker":"[19]"},{"why":"Provides the Mamba settings and SS2D scan used in the implementation and baseline comparisons.","marker":"[39]"},{"why":"Earlier Mamba-based audio-visual segmentation work that ADSM extends from fixed-step to adaptive-step.","marker":"[12]"},{"why":"Provides the OT-LA alignment baseline that OT-MM is compared against in Table 5.","marker":"[30]"},{"why":"Competitive AVIS baseline ACVIS that SAMOT surpasses in the experiments.","marker":"[45]"},{"why":"Supplies the query-learning protocol used to derive instance-level visual queries.","marker":"[4]"},{"why":"Supplies the query-learning and object-token association protocol and serves as a VIS baseline.","marker":"[22]"}],"fun_headline_variants":["State-adaptive Mamba step control wins AVISeg with OT matching","Adaptive step sizing and optimal transport boost instance segmentation","SAMOT improves audio-visual segmentation via adaptive Mamba and OT","Modality-adaptive Mamba steps and OT matching advance AVISeg","Optimal transport matching plus adaptive Mamba steps top AVISeg"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that adding the scalar offset $\\Delta$ to the base step size inside the Mamba discretization is what actually changes the state update in the trained model, and that the reported gains come from that mechanism rather than from the extra feature reweighting; the paper never writes the resulting recurrence, so this premise is unverified.","fun_headline_variants_meta":{"raw":{"variants":["State-adaptive Mamba step control wins AVISeg with OT matching","Adaptive step sizing and optimal transport boost instance segmentation","SAMOT improves audio-visual segmentation via adaptive Mamba and OT","Modality-adaptive Mamba steps and OT matching advance AVISeg","Optimal transport matching plus adaptive Mamba steps top AVISeg"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001429,"raw_usage":{"total_tokens":5811,"prompt_tokens":1037,"completion_tokens":4774,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":653,"completion_tokens_details":{"reasoning_tokens":4684}},"tokens_in":653,"tokens_out":4774,"duration_ms":32595,"temperature":1.0,"reasoning_tokens":4684,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:19:41.513343+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If a variant that keeps the base Mamba step size fixed and instead multiplies the audio-conditioned visual features by a per-frame learned gate reproduces the same FSLA, HOTA, and mAP gains on AVISeg, then the step-size injection is not the operative mechanism and the ADSM claim collapses.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Competitive AVIS baseline ACVIS that SAMOT surpasses in the experiments."}],"review_version":1}