{"id":"a1ffef41-c853-40e8-bff3-25b1eab3e4e0","arxiv_id":"2507.22886","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"OmniAVS is a new 2,104-video dataset with 8 multimodal expression types for referring audio-visual segmentation, and OISA is an MLLM baseline that outperforms prior methods on it.","lead":"This paper introduces OmniAVS, a dataset of 2,104 videos and 61,095 referring expressions that mix text, speech, sound, and images to point at objects in audio-visual scenes. It also presents OISA, a multimodal language model baseline that segments objects and explains its reasoning.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No-audio control is missing: OmniAVS's claim that its expressions demand audio-content understanding is untested because OISA is never evaluated with video audio ablated.","rationale":"The reader's weakest assumption was annotation quality and full-temporal mask validity. Those concerns are legitimate and should be addressed, but they are standard quality-control issues applicable to any new benchmark. I focused instead on a missing control that is specific to the paper's central premise: that OmniAVS expressions demand understanding of audio content. The abstract, introduction, and Section 3.2 all assert this property, and the OISA comparison is interpreted through it. However, all reported experiments feed real video audio to every method; there is no video-only baseline, so the dataset's audio-dependence is never demonstrated. The fusion ablation in Table 3 is insufficient because it varies only fusion architecture, not audio presence. A concrete test is feasible: OISA's architecture separates video audio content tokens from expression tokens, so zeroing the video audio tokens isolates whether audio content is needed. If the performance drop is small, the central claim should be revised from 'demands audio understanding' to 'encourages audio understanding'; if the drop is large, the claim is confirmed. This does not overturn the reader's conditional acceptance, but it should be added as an explicit condition, together with the annotation-agreement check the reader identified.","tokens_in":16791,"tokens_out":8679,"duration_ms":125436,"concrete_test":"Run OISA-1B on the OmniAVS test set with only the video audio content tokens zeroed or masked, keeping visual tokens, text/speech expression inputs, and image/sound expression references intact, under the exact training/inference protocol of Section 5.1. Compare per-split and overall J&F against the full-audio 41.1% result. If the no-video-audio model drops by less than roughly 2-3 J&F points on the audio-dependent splits (especially III, IV, VII, VIII), then audio-content understanding is not load-bearing and the benchmark claim should be weakened; if it drops substantially (e.g., more than 5 points), the claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central novelty is a benchmark whose expressions are claimed (Section 1; Section 3.2, rules 1-3) to demand understanding of audio content, not just detection of sound presence. Yet Section 5 contains no experiment that removes or masks the video audio stream. The fusion ablations in Table 3 vary only how audio tokens are combined; every condition still feeds real audio into the model. Because many expressions carry semantic redundancy in text and vision (e.g., 'Who is most likely to be sick?' can be inferred from visible coughing; 'The dog warning' is stated in the text), a strong vision-language model might reach much of the reported 41.1% J&F without understanding the audio content at all. If that is the case, the claimed 5.0-point advantage over LISA-13B and the inference that OmniAVS is qualitatively harder than Ref-AVS (58.0% vs 41.1%) overstate the role of audio-content understanding. This missing audio-ablation control is load-bearing for the paper's primary contribution, not a cosmetic baseline addition.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes OmniAVS, a new referring audio-visual segmentation dataset with 2,104 videos and 61,095 referring expressions spanning eight modality combinations (text/speech with sound and/or image). The authors argue that existing RAVS datasets such as Ref-AVS rely on surface-level acoustic properties, whereas OmniAVS expressions require understanding audio content and performing reasoning, such as inferring illness from coughing sounds. The paper also introduces OISA-1B, an MLLM-based baseline with two technical components: audio-visual interleaving for temporal alignment and query propagation for mask decoding. Experiments report state-of-the-art results on OmniAVS (41.1% J&F, outperforming LISA-13B by 5.0 points) and competitive results on Ref-AVS, referring image/video segmentation, and ReVOS.","tokens_in":17004,"tokens_out":6451,"duration_ms":72526,"significance":"If the dataset annotations are reliable, OmniAVS is a useful resource that moves referring audio-visual segmentation from sound-presence cues toward audio-content understanding and reasoning. The OISA design choices, audio-visual interleaving and query propagation, are simple, parameter-free, and show consistent gains in the in-model ablations. The evaluation is performed on a held-out test split and does not rely on fitted constants, so the main benchmark claim is not circular. However, the absence of an audio-ablated control leaves the core claim that OmniAVS expressions demand audio-content understanding empirically unsupported, and the benchmark's reliability is not yet established because annotation quality is not measured. For these reasons the contribution is promising but not fully established.","major_comments":[{"comment":"No experiment removes or masks the video audio stream. All fusion ablations in Table 3 vary only how real audio tokens are combined, and all benchmark conditions in Table 5 feed audio into the model. Because many OmniAVS expressions are semantically redundant with text and vision (e.g., 'Who is most likely to be sick?' is inferable from visible coughing, and 'The dog warning' names the sound in the text), a vision-language model without audio understanding could plausibly achieve much of the reported 41.1% J&F. An audio-ablated control (e.g., replacing audio tokens with silence or zeros, evaluated for both OISA and adapted LISA) is needed to support the paper's central claim that OmniAVS demands audio-content understanding beyond sound-presence detection.","section":"§5.2–5.3 (Tables 3 and 5)"},{"comment":"The reliability of OmniAVS as a benchmark depends on annotation quality, but Section 3.2 reports no inter-annotator agreement, no double-annotation rate, and no quality-control metric for the 61,095 expressions or the 206k mask labels. The expression rules (e.g., 'emphasize the sound's content') leave room for subjective judgment, and annotator use of SAM2 assistance does not by itself validate mask correctness. Please report agreement statistics (e.g., mask IoU between annotators and expression-validity agreement) on a sample, and state how ambiguous or failing annotations were resolved.","section":"§3.2"},{"comment":"The LISA baseline is enhanced only with the same audio-text alignment as OISA, while OISA additionally uses audio-visual interleaving and query propagation. The 5.0-point advantage over LISA-13B therefore conflates the proposed architectural components with the ability to reason about audio content. To support the claim that OISA outperforms existing methods on omnimodal reasoning, the comparison should give LISA the same interleaving and query-propagation benefits, or isolate each component's contribution on the OmniAVS test set.","section":"§5.3 (Table 5)"},{"comment":"All reported metrics are single-run point estimates with no variance or significance testing. This matters for claims such as the 5.0-point gain over LISA-13B and the 0.2-point improvement over VISA on ReVOS; without error bars or multiple seeds, small differences may not be reproducible. Please report standard deviations over multiple runs, or at least provide evidence that the main conclusions are stable under different random seeds.","section":"§5.3–5.4 (Tables 5–8)"},{"comment":"The discussion of the Ref-AVS Null split is speculative: the claim that EEMC's 0.7% S score 'likely' reflects overfitting to audio patterns rather than genuine null understanding is not tested. Because OISA-1B's S=9.8% is substantially worse on this split, the paper should either provide an error analysis supporting the overfitting explanation or qualify the claim of 'greatly surpassing' EEMC on Ref-AVS.","section":"§5.4 (Table 6)"}],"minor_comments":[{"comment":"The sentence 'splits VII (text+speech+image) and VIII (text+sound+image)' mislabels the expression types: according to Section 3.2, type VII is text+sound+image and type VIII is speech+sound+image.","section":"§5.3"},{"comment":"The terms 'dense frames' and 'sparse frames' are used without definition; please clarify how the dense and sparse frame subsets are selected and how they differ during training.","section":"§5.1"},{"comment":"For no-target expressions, J&F is set to 1 when the prediction is empty and 0 otherwise; this gives full credit for predicting an empty mask, which could inflate scores. Please report the proportion of no-target expressions in OmniAVS and the sensitivity of the overall J&F to this convention.","section":"§5 (Evaluation Metrics)"},{"comment":"Audio tokens are divided uniformly across the N sampled frames, but OmniAVS videos have annotation frame rates of 3–15 FPS; please clarify whether the audio segmentation uses actual timestamps or a simplified uniform division, and discuss the effect on audio-visual alignment for variable-FPS videos.","section":"§4.2"}],"recommendation":"major_revision","confidential_remarks":"This is a solid dataset-and-baseline submission from a strong group, and the proposed dataset could be valuable to the community. The key technical gap is the missing audio-ablation control, which is load-bearing for the paper's central claim that OmniAVS expressions require audio-content understanding. The annotation-quality question is also important for a benchmark paper and should be addressed with quantitative agreement metrics. If these points are resolved, the paper could be suitable for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before you read this one. First, the OmniAVS dataset is a genuine step forward: eight expression types mixing text, speech, sound, and image, with reasoning-oriented expressions and 34k explanations. That fills a real gap next to Ref-AVS, which only has text and stops at surface-level sound properties. Second, the paper never actually shows that its expressions require understanding audio content. No experiment masks or removes the audio stream. The fusion ablations only vary how audio tokens are combined; every condition feeds real audio in. Given that many expressions can be solved from vision and text alone (visible coughing for \"sick,\" text saying \"dog warning\"), the claimed 5.0-point gain over LISA-13B does not demonstrate audio-content reasoning as cleanly as the abstract suggests. That missing control is load-bearing for the paper's central novelty, not a cosmetic omission.\n\nWhat the paper does well: the dataset construction is careful on paper—videos from diverse sources, annotation protocol with SAM2 assistance, no-target and multi-target expressions. OISA is a reasonable baseline, and showing competitive results on Ref-AVS, MeViS, and ReVOS suggests the approach transfers. The query-propagation mask head is a modest but sensible improvement over one-token-seg-all.\n\nSoft spots beyond the audio ablation: no inter-annotator agreement or quality stats for the 61k expressions and 206k masks, single-run metrics without error bars, and the LISA baseline gets audio alignment but not the interleaving or query propagation. These are addressable, but they weaken the current reliability of the benchmark rankings. The Ref-AVS Null split discussion is speculative but not central.\n\nThe stress-test note holds up: without an audio-off control, the benchmark's claim to emphasize \"understanding audio content beyond detecting presence\" is untested. Still, the dataset is useful even if that claim turns out to be overstated—a multimodal referring expression benchmark with reasoning chains is valuable on its own.\n\nFor you: worth a serious referee. The right outcome is major revision—add an audio-ablated baseline, report agreement statistics, release code/data or state why not, and run the key numbers multiple times. I'd bring it to reading group once, and I'd likely cite the dataset in future audio-visual work.","headline":"New omnimodal referring dataset is a real contribution, but the missing audio-ablation control leaves its central audio-understanding claim unproven.","tokens_in":17433,"tokens_out":1700,"would_cite":true,"duration_ms":23050,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces OmniAVS, a benchmark of 61,095 omnimodal referring expressions for audio-visual segmentation, and OISA, a multimodal LLM that reaches 41.1% J&F on it, beating prior best by 5.0 points.","keywords":["referring audio-visual segmentation","omnimodal expressions","reasoning segmentation","multimodal large language model","audio-visual interleaving","query propagation","dataset benchmark","explanation generation"],"falsifier":"Re-annotate a random 5% of OmniAVS test expressions with an independent annotation team and measure inter-annotator agreement on the referred-object masks; if agreement falls near the 5.0-point J&F gap between OISA-1B and LISA-13B, the ranking would not be trustworthy. Alternatively, rerun OISA-1B with the audio and video tokens interleaved in random order instead of the frame-aligned order; if J&F does not drop substantially, the claimed benefit of Audio-Visual Interleaving is not load-bearing.","tokens_in":16629,"feed_emoji":"🎬","tokens_out":4693,"duration_ms":52206,"temperature":0.7,"pith_summary":"The paper argues that referring audio-visual segmentation should move beyond detecting obvious sound sources toward understanding the content of sounds and performing reasoning over multimodal cues. To make this concrete, it introduces OmniAVS, a dataset of 2,104 videos with 61,095 expressions that flexibly combine text, speech, sound, and images in eight modality types, requiring skills like inferring illness from coughing or locating an object by matching a sound and an image. It also presents OISA, a multimodal large language model that handles these omnimodal expressions and outputs both a segmentation mask and a natural-language explanation. On OmniAVS, OISA-1B achieves 41.1% average J&F, outperforming the strongest prior method LISA-13B by 5.0 points, and it also reports competitive results on existing referring segmentation benchmarks.","feed_headline":"New omnimodal benchmark pushes referring AV segmentation to 41.1 J&F","feed_subtitle":"OmniAVS adds 61,095 reason-rich expressions mixing text, speech, sound, and images. OISA-1B beats LISA-13B by 5.0 points.","key_machinery":"The load-bearing mechanisms are Audio-Visual Interleaving and Query Propagation. Audio-Visual Interleaving divides the audio token sequence into clips and places each clip immediately after its corresponding frame's vision tokens, forming a synchronized sequence of the form $\\{v_1, a_1, v_2, a_2, \\ldots, v_N, a_N\\}$ without adding parameters. Query Propagation updates the [SEG] token frame-by-frame in the mask decoder, rather than using one fixed token for all frames, so the query tracks object motion and avoids identity switches. The [SEG] token, produced by the MLLM from the interleaved multimodal context, is fed into the mask head for segmentation.","core_discovery":"The central claim is that a new benchmark, OmniAVS, can push referring audio-visual segmentation from surface-level acoustic attributes to semantic content understanding and reasoning, and that a multimodal LLM-based model can be adapted to this harder task. OISA-1B accomplishes this by interleaving audio and visual tokens for temporal synchronization and by propagating a single segmentation query across frames, achieving state-of-the-art results on OmniAVS and strong transfer to related referring and reasoning segmentation tasks.","pith_inferences":["Because speech expressions are generated by converting text via TTS, there may be a gap between these utterances and natural spontaneous speech; a follow-up could test whether OISA's gains persist with human-spoken expressions.","The full-temporal mask convention, inherited from MeViS, may be ill-matched to expressions that are only valid in part of a video; a temporally-gated evaluation variant could reveal whether models locate objects during the relevant segment.","The 5.0-point gain over LISA-13B may partly stem from the audio-text alignment pretraining stage rather than the interleaving mechanism itself; ablating that stage would clarify which contribution is decisive.","The benchmark's emphasis on sound-content reasoning connects naturally to audio-visual question answering, so a joint model fine-tuned on both OmniAVS and A-VQA may improve both tasks through shared reasoning supervision."],"forward_implications":["If OmniAVS becomes a standard benchmark, referring segmentation evaluation will include audio-content reasoning and explanation quality, not just acoustic event detection.","The eight-expression-type interface could push development of models that accept arbitrary combinations of text, speech, sound, and image as referring input, closer to human interaction.","Audio-Visual Interleaving and Query Propagation are architecture-agnostic enough to be incorporated into other MLLM-based segmentation systems, potentially improving video-level referring segmentation broadly.","The explanations provided for reasoning expressions enable quantifying a model's interpretability, a dimension absent from prior referring audio-visual benchmarks."],"supporting_citations":[{"why":"Ref-AVS is the prior audio-visual referring segmentation dataset that OmniAVS extends and compares against.","marker":"[68]"},{"why":"MeViS supplies the full-temporal mask annotation convention adopted by OmniAVS and serves as a training and evaluation dataset.","marker":"[17]"},{"why":"LISA provides the [SEG] token and mask head paradigm that OISA builds on and is the main comparison baseline.","marker":"[33]"},{"why":"VideoLISA is the source of the One-Token-Seg-All approach that OISA's Query Propagation is designed to improve.","marker":"[3]"},{"why":"InternVL2 is the MLLM backbone of OISA, providing the vision encoder and LLM that process the interleaved tokens.","marker":"[13]"},{"why":"Whisper-large-v3 is the audio encoder used in OISA to convert sound and speech into tokens.","marker":"[54]"},{"why":"SAM 2 powers the interactive annotation tool used to create OmniAVS mask labels.","marker":"[56]"},{"why":"Mask2Former supplies the pixel decoder and mask decoder design that forms OISA's mask head.","marker":"[14]"}],"fun_headline_variants":["OmniAVS: 61K expressions that make AV segmentation reason","OISA-1B beats LISA-13B by 5 points on OmniAVS benchmark","OmniAVS: audiovisual expressions that demand real reasoning","From sounds to semantics: OmniAVS teaches AV segmentation to reason"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's reliability rests on the assumption that its 61,095 expressions and the associated mask annotations are clean and unambiguous; the paper reports no inter-annotator agreement or quality checks, so if those labels are noisy, the reported rankings and difficulty comparisons could change.","fun_headline_variants_meta":{"raw":{"variants":["OmniAVS: 61K expressions that make AV segmentation reason","OISA-1B beats LISA-13B by 5 points on OmniAVS benchmark","OmniAVS: audiovisual expressions that demand real reasoning","From sounds to semantics: OmniAVS teaches AV segmentation to reason"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000996,"raw_usage":{"total_tokens":4173,"prompt_tokens":858,"completion_tokens":3315,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":474,"completion_tokens_details":{"reasoning_tokens":3231}},"tokens_in":474,"tokens_out":3315,"duration_ms":28844,"temperature":1.0,"reasoning_tokens":3231,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:10:41.079538+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random 5% of OmniAVS test expressions with an independent annotation team and measure inter-annotator agreement on the referred-object masks; if agreement falls near the 5.0-point J&F gap between OISA-1B and LISA-13B, the ranking would not be trustworthy. Alternatively, rerun OISA-1B with the audio and video tokens interleaved in random order instead of the frame-aligned order; if J&F does not drop substantially, the claimed benefit of Audio-Visual Interleaving is not load-bearing.","supporting_citations":[{"cited_title":"Ref-A VS: Refer and Segment Objects in Audio-Visual Scenes","cited_arxiv_id":null,"evidence_quote":"Ref-AVS is the prior audio-visual referring segmentation dataset that OmniAVS extends and compares against."},{"cited_title":"MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressions","cited_arxiv_id":null,"evidence_quote":"MeViS supplies the full-temporal mask annotation convention adopted by OmniAVS and serves as a training and evaluation dataset."},{"cited_title":"LISA: Reasoning Segmen- tation via Large Language Model","cited_arxiv_id":null,"evidence_quote":"LISA provides the [SEG] token and mask head paradigm that OISA builds on and is the main comparison baseline."},{"cited_title":"Robust Speech Recognition via Large-Scale Weak Supervision","cited_arxiv_id":null,"evidence_quote":"Whisper-large-v3 is the audio encoder used in OISA to convert sound and speech into tokens."},{"cited_title":"Masked-attention Mask Transformer for Universal Image Segmentation","cited_arxiv_id":null,"evidence_quote":"Mask2Former supplies the pixel decoder and mask decoder design that forms OISA's mask head."}],"review_version":1}