Think Before You Segment: An Object-aware Reasoning Agent for Referring Audio-Visual Segmentation

Hisham Cholakkal; Jinxing Zhou; Mingfei Han; Rao Muhammad Anwer; Tong Wang; Xiaojun Chang; Yanghao Zhou

REVIEW 4 major objections 6 minor 1 cited by

Think Before You Segment: An Object-aware Reasoning Agent for Referring Audio-Visual Segmentation

T0 review · 4 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TGS-Agent contends that referring audio-visual segmentation is best solved by an explicit Think-Ground-Segment pipeline: a reasoning multimodal language model names the object, an open-vocabulary detector boxes it, frozen SAM2 masks it, wit

desk verdict A genuinely new and mostly convincing agentic pipeline for Ref-AVS, but treat the R2-AVSBench SOTA with caution: the same Gemini model that writes the teacher chains also generates that benchmark. read the letter →

arxiv 2508.04418 v1 pith:L4UHJG72 submitted 2025-08-06 cs.MM cs.CVcs.MAcs.SDeess.AS

Jinxing Zhou , Yanghao Zhou , Mingfei Han , Tong Wang , Xiaojun Chang , Hisham Cholakkal , Rao Muhammad Anwer This is my paper

classification cs.MMcs.CVcs.MAcs.SDeess.AS

keywords referringaudio-visualsegmentationagenticreasoningmultimodallargelanguagemodelobject-awareinstructiontuningopen-vocabularygroundingSAM2reasoning-intensivebenchmarkinterpretable

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TGS-Agent tackles referring audio-visual segmentation—cutting out the object in a video that a natural-language expression points to, using sound and sight—by decomposing it into three explicit stages: Think, Ground, Segment. A fine-tuned multimodal language model, Ref-Thinker, first reads the reference alongside the video and audio and writes out what the referred object is; an open-vocabulary detector then draws its bounding box; a frozen SAM2 turns that box into a mask. The paper claims this agentic pipeline needs no pixel-level mask supervision, is explainable at every step, and beats end-to-end models that fuse multimodal features and tune a SAM/SAM2 decoder. To test generalization, the authors build R2-AVSBench, a 400-video benchmark whose references are rewritten to be linguistically diverse and reasoning-intensive, and report state-of-the-art results on both R2-AVSBench and the original Ref-AVSBench.

What carries the argument

The Think-Ground-Segment (TGS) workflow, with Ref-Thinker as the load-bearing component. Ref-Thinker is a multimodal LLM (LLaMA-2-7b backbone, CLIP-ViT visual encoder, BEATs audio encoder, Q-Former compression, LoRA tuning) trained on Gemini-1.5-Pro-generated instruction chains. Each chain contains a <think> passage analyzing the reference against visible and audible content and an <answer> with two object descriptions: a fine-grained <f object> and a category-only <s object>. This explicit text bottleneck—multimodal inputs collapsed into a natural-language object description—is what gets handed to Grounding-DINO and then to frozen SAM2, converting segmentation into reference → object text →

What would settle it

Have human annotators verify a random sample of the Gemini-generated think-answer chains and compute agreement with ground-truth object labels; then fine-tune Ref-Thinker only on chains judged correct and measure J&F on R2-AVSBench. If the verified-set accuracy is close to chance, or if cleaned training does not change downstream performance, the claim that explicit reasoning quality drives the reported gains is falsified.

Watch

Extended reading notes

Core claim

The central claim is that explicit reference understanding can replace implicit multimodal fusion. Once the referred object is named as text, generic frozen tools—Grounding-DINO for grounding, SAM2 for segmentation—suffice and outperform the previous state of the art, SAM2-LOVE, by 7.4 points in mean J&F on the mixed Ref-AVSBench test set (65.9 vs 58.5). Using the simplified object category as the detection prompt beats feeding the original reference or a fine-grained description on average, though fine-grained text helps when multiple instances of the same category appear. The companion R2-AVSBench rewrites references to hide direct object names and force external, comparative, functional,

Load-bearing premise

The load-bearing premise is that the Gemini-1.5-Pro-generated think-answer chains used to fine-tune Ref-Thinker are accurate enough to teach reliable object identification; the paper calls them high-quality without human verification, and later reports exact category match of only 55.1% on Seen and 49.3% on Unseen test splits.

Editorial extensions

If this is right

If the pipeline is correct, strong Ref-AVS performance does not require pixel-level supervision or tuning of segmentation decoders; frozen open-vocabulary detection plus SAM2 suffice given a good object description.
The explicit reasoning chain makes outputs auditable: a wrong mask can be traced to a wrong object name in Think, a wrong box in Ground, or a wrong mask given the box in Segment.
The simplified object category is competitive or better as a grounding prompt than the original natural-language reference, suggesting object identity rather than paraphrase fidelity is the load-bearing information for detection.
On R2-AVSBench, all models drop when references become reasoning-intensive, but TGS-Agent's drop is smaller, indicating that explicit reasoning helps cross-reference generalization.
Because Ref-Thinker is open-vocabulary, the pipeline should handle object categories unseen in training as long as the frozen detector recognizes the named object; the Unseen split results support this corollary.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper leaves implicit that the quality of its instruction set caps the whole pipeline: exact category match is only 55.1% on Seen and 49.3% on Unseen test splits, so cleaner or human-verified think-answer chains likely have large untapped headroom.
The decomposed design makes the failure modes separable: the appendix shows a temporal-audio reasoning error in the Think stage and a rare-object detection error in the Ground stage, implying that better temporal audio reasoning and broader detector coverage would improve results roughly additively.
R2-AVSBench could be used as a diagnostic that separates language robustness from perceptual grounding; methods that fuse latent embeddings may survive template-style references yet degrade sharply when the same object is described through function, comparison, or external knowledge.
A natural testable extension is to swap Ref-Thinker's teacher for a stronger or human-verified reasoner without changing the Ground and Segment tools; if gains track Think-stage accuracy, the agentic decomposition is validated as the source of the improvement.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know this one for two reasons. First, it works: a modular Think-Ground-Segment pipeline with a fine-tuned MLLM, frozen Grounding-DINO, and frozen SAM2 beats end-to-end tuned methods on the standard Ref-AVSBench without any pixel-level supervision. Second, the new R2-AVSBench result is shakier than the headline suggests, because Gemini-1.5-Pro is both the teacher that distills reasoning chains into Ref-Thinker and the generator of the benchmark references. That is a real distributional overlap, and it likely inflates the large margins in Table 2.

The genuinely new thing is the explicit object-decoupling: Ref-Thinker outputs a short object phrase (category or fine-grained description), which then drives an off-the-shelf detector and segmenter. The paper shows this beats tuned SAM2 decoders and is much more interpretable. The ablations on object description type are informative, and the failure-case appendix is honest—it admits factual errors in the Think stage and imprecise bboxes from Grounding-DINO. Credit where due: the core Ref-AVSBench win is independent of the Gemini-overlap concern and is the strongest evidence for the approach.

The soft spots, in proportion: (1) The R2-AVSBench claim is weakened as described above; human verification of the transformed references helps, but the teacher/test alignment remains. (2) Appendix D reports exact category match of only 55.1% on Seen and 49.3% on Unseen for the s_object output. The authors argue open-vocabulary and cite CLIPScore, but that number is low enough to warrant a real characterization of the instruction data quality, not just a one-line assertion. (3) Minor: threshold selection (tau_bbox, tau_text) is reported as a default but the split used for selection isn't stated, and code is promised but not yet released.

None of these are fatal. The standard-benchmark result stands, and the agentic framing is a legitimate contribution to the Ref-AVS subfield and referring segmentation more broadly. This paper deserves a serious referee and a conditional accept path: release code and data, add an independent evaluation on a non-Gemini-generated benchmark, and clarify the threshold selection procedure. I'd bring it to a reading group and would cite the Ref-AVSBench result in work on MLLM-guided segmentation.

Referee Report

4 major / 6 minor

Summary. The paper proposes TGS-Agent, a training-free-at-segmentation agentic framework for Referring Audio-Visual Segmentation (Ref-AVS). The pipeline decomposes the task into Think (Ref-Thinker, an audio-visual LLM fine-tuned on Gemini-generated reasoning chains), Ground (Grounding-DINO), and Segment (frozen SAM2), using only reference text, video, and audio without pixel-level mask supervision. The authors also introduce R2-AVSBench, a 400-video benchmark with Gemini-transformed, human-verified references intended to be more linguistically diverse and reasoning-intensive. They report state-of-the-art results on both the standard Ref-AVSBench and the proposed R2-AVSBench, with ablations on object-description granularity and detection thresholds.

Significance. If the central claims hold, the paper offers a conceptually different baseline for Ref-AVS: replacing end-to-end multimodal fusion with an explicit, inspectable Think-Ground-Segment chain using frozen foundation models. This is valuable because it decouples reasoning from mask supervision and provides a new stress-test benchmark. The paper is also transparent in reporting failure cases and quantitative object-category accuracy. However, the significance depends on two unresolved issues: the reliability of the Gemini-generated teacher chains used to train Ref-Thinker, and the overlap between the teacher model and the model used to construct R2-AVSBench. Both are fixable with additional experiments and analysis.

major comments (4)

[Sec. 3.2 and Appendix D] The instruction-tuning set is described as 'high-quality' without human verification or agreement statistics. Appendix D later reports exact category match of only 55.1% (Seen) and 49.3% (Unseen) for the simplified object output. Since this output is the sole prompt passed to Grounding-DINO, the reliability of the Think step directly caps downstream Ground and Segment performance. The CLIPScore argument is only qualitative ('majority in [0.8,1]') with no reported distribution. Please provide human verification of a sample of teacher chains, report exact and semantic accuracy quantitatively, and show whether the low exact-match rate bounds segmentation performance.
[Sec. 4 vs Sec. 3.2] The same model, Gemini-1.5-Pro, is used both to generate Ref-Thinker's training chains (Sec. 3.2) and to construct R2-AVSBench references (Sec. 4). This creates a distributional alignment: Ref-Thinker is fine-tuned to imitate Gemini's paraphrase style, and the test references come from the same generator. The large margin on R2-AVSBench in Table 2 may therefore reflect familiarity with Gemini's phrasing rather than generalizable multimodal reasoning. Please evaluate on human-written R2-style references, train with a different teacher model, or otherwise demonstrate that the R2 advantage is not an artifact of the shared generator.
[Sec. 4 benchmark validation] The R2-AVSBench construction states that human annotators revise or reject generated references, but no statistics are given: how many references were rejected, how many were revised, what the inter-annotator agreement was, or how ambiguity was measured. Without these numbers it is difficult to assess whether R2-AVSBench is genuinely unambiguous and reasoning-intensive. Please report these statistics, or at least a human-validation protocol with agreement rates.
[Sec. 5.4 and Table 3] The fine-grained object description T_f is the interpretable output emphasized in the paper's explainability narrative, yet Table 3 shows it consistently underperforms the simplified T_s (e.g., 49.0 vs 54.9 J&F on Seen, 73.7 vs 76.9 on Unseen). This suggests that the detailed reasoning chain is not what drives the state-of-the-art result, and that the practical benefit of the explicit Think step may be limited to category identification. Please discuss this discrepancy and provide quantitative evidence for when T_f helps beyond the single qualitative example in Fig. 4.

minor comments (6)

[Figures 4, 10-14] Several figure captions contain stray Chinese text (e.g., '选取12356 张' in Fig. 4 and similar artifacts in Figs. 10-14). These should be removed before submission.
[Sec. 3.1] The output tags are written as '<f object>' and '<s object>' with a space; this is inconsistent with standard XML/HTML tag conventions and could confuse readers implementing the format. Please define the exact literal tag syntax or use 'f_object' and 's_object' consistently.
[Table 2] The column heading 'Ref. Source' is unclear; consider renaming to 'Reference set' or 'Benchmark' to distinguish the original Ref-AVSBench references from the transformed R2-AVSBench references.
[Appendix D] The CLIPScore analysis should report quantitative aggregates (mean, percentiles, or a histogram) rather than the qualitative statement that 'the majority of CLIPScore values fall within [0.8, 1]'.
[Fig. 5] The y-axis label 'J&F' is ambiguous; specify whether it is the average of J and F over Seen+Unseen, and state the default operating point clearly on the plots.
[Sec. 5.4] The sentence 'the object detector is confused when recognizing the correct guitar at instance level' is grammatically awkward; rephrase for clarity.

Circularity Check

1 steps flagged · score 5.0 of 10

R2-AVSBench SOTA is partially confounded: the same Gemini-1.5-Pro model generates both the Ref-Thinker instruction-tuning chains and the R2-AVSBench test references, so the proposed benchmark does not independently establish cross-reference reasoning; the standard Ref-AVSBench result remains independent.

fitted input called prediction [Sec. 3.2 (Ref-Thinker training) and Sec. 4 (R2-AVSBench construction); Appendix D]
"Specifically, given a reference expression and a video from the training set of the official Ref-AVSBench dataset ... we leverage Gemini-1.5-Pro ... to analyze the video and generate an object-aware reasoning chain that strictly follows the think-answer format shown in Sec. 3.1. ... We perform the reference transformation with the aid of Gemini-1.5-Pro, and the corresponding prompt is provided in Appendix B."

Ref-Thinker is instruction-tuned to imitate Gemini-1.5-Pro's think-answer chains (Sec. 3.2), and the R2-AVSBench test references are generated by the same Gemini-1.5-Pro model (Sec. 4). The model is therefore fitted to the output distribution of the very generator that produces the proposed benchmark. Its large margin on R2-AVSBench (Table 2) can reflect distributional familiarity with Gemini's paraphrasing style rather than generalizable multimodal reasoning. The standard Ref-AVSBench comparison is not affected, but the R2-AVSBench 'prediction' is not an independent test of the claimed cross-reference generalization.

full rationale

The core Think-Ground-Segment derivation is not definitionally circular: Ref-Thinker outputs object descriptions, Grounding-DINO converts them to boxes, and frozen SAM2 produces masks. The Ref-AVSBench evaluation is independent, uses human-annotated masks and expressions, and shows SOTA against tuned baselines. However, the paper's second benchmark, R2-AVSBench, is built by prompting Gemini-1.5-Pro to transform references, while Ref-Thinker is trained on think-answer chains also produced by Gemini-1.5-Pro. Human verification of R2-AVSBench references reduces factual-error risk but does not remove the stylistic and reasoning-pattern overlap between training supervision and test generation. Appendix D further reports only 55.1% (Seen) and 49.3% (Unseen) exact category matches for Ref-Thinker's simplified object output, which undercuts the claim that the teacher data produces 'high-quality' object-aware reasoning. This is an evaluator-loop problem specific to the R2-AVSBench claim; the standard Ref-AVSBench result retains independent content, so the paper is only partially circular rather than fully forced by definition or self-citation.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No new physical or mathematical entities are introduced. The ledger captures the hand-set thresholds, the training recipe, and the unverified assumptions about the teacher-generated instruction data, the frozen foundation tools, and the quality of the transformed benchmark.

free parameters (3)

tau_bbox (Grounding-DINO box confidence threshold) = 0.1
Chosen via parameter study on Ref-AVSBench (Fig. 5); controls whether a detected box is used, so it directly gates the Segment step.
tau_text (Grounding-DINO text similarity threshold) = 0.25
Chosen together with tau_bbox; controls which textual descriptions produce candidate boxes for the detector.
Ref-Thinker fine-tuning hyperparameters = LoRA rank 8, scaling 16, lr 1e-4, batch 4, 6 epochs, 32 query tokens
Hand-set for instruction tuning; no sensitivity analysis is reported, and the resulting model is the load-bearing first stage of the pipeline.

assumptions (5)

ad hoc to paper Gemini-1.5-Pro generated think-answer chains are accurate and 'high-quality' without independent verification
Sec 3.2: the instruction set is produced by Gemini with no reported human check; Appendix D later shows only about 50% exact category match, so this assumption is load-bearing and partially contradicted by the paper's own appendix.
domain assumption Grounding-DINO can localize any object from short open-vocabulary text prompts in RGB frames
Sec 3.1 Eq (2): frozen Grounding-DINO is the only object localizer; the paper's own failure case (Fig. 14b) shows it can fail on rare categories such as marimba.
domain assumption SAM2 produces accurate segmentation masks from bounding-box prompts
Sec 3.1 Eq (3): frozen SAM2 is the segmentation engine; the method inherits all SAM2 failure modes.
domain assumption R2-AVSBench transformed references identify the same target object as the original Ref-AVSBench references
Sec 4: human verification is asserted but no inter-annotator agreement or error analysis is provided; benchmark labels are reused from Ref-AVSBench, so any reference drift affects all models.
domain assumption Ref-Thinker's architecture (LLaMA-2-7b-chat + CLIP + BEATs + Q-Former) can learn audio-visual-text reasoning from the constructed instruction set
Sec 3.2: assumed by the training design; the pretraining stage only uses caption datasets, and the instruction set is unverified.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Think Before You Segment: An Object-aware Reasoning Agent for Referring Audio-Visual Segmentation." pith.science (2026). https://pith.science/paper/L4UHJG72

@misc{pith2026250804418,
  author       = {Pith},
  title        = {Pith review of: Think Before You Segment: An Object-aware Reasoning Agent for Referring Audio-Visual Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L4UHJG72}},
  note         = {Machine review of arXiv:2508.04418}
}

read the original abstract

Referring Audio-Visual Segmentation (Ref-AVS) aims to segment target objects in audible videos based on given reference expressions. Prior works typically rely on learning latent embeddings via multimodal fusion to prompt a tunable SAM/SAM2 decoder for segmentation, which requires strong pixel-level supervision and lacks interpretability. From a novel perspective of explicit reference understanding, we propose TGS-Agent, which decomposes the task into a Think-Ground-Segment process, mimicking the human reasoning procedure by first identifying the referred object through multimodal analysis, followed by coarse-grained grounding and precise segmentation. To this end, we first propose Ref-Thinker, a multimodal language model capable of reasoning over textual, visual, and auditory cues. We construct an instruction-tuning dataset with explicit object-aware think-answer chains for Ref-Thinker fine-tuning. The object description inferred by Ref-Thinker is used as an explicit prompt for Grounding-DINO and SAM2, which perform grounding and segmentation without relying on pixel-level supervision. Additionally, we introduce R\textsuperscript{2}-AVSBench, a new benchmark with linguistically diverse and reasoning-intensive references for better evaluating model generalization. Our approach achieves state-of-the-art results on both standard Ref-AVSBench and proposed R\textsuperscript{2}-AVSBench. Code will be available at https://github.com/jasongief/TGS-Agent.

Figures

Figures reproduced from arXiv: 2508.04418 by the authors.

**Figure 2.** Workflow of our Think-Ground-Segment Agent. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Word clouds of different reference types. More [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (10 more)

**Figure 4.** Figure 4: Visualization results of using different object de [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Parameter study on the τbbox and τtext. 5.4 Ablation Study and Qualitative Results In this section, the experiments are conducted on the RefAVSBench dataset unless otherwise specified. Impact of Object Description Types. Our method designs Ref-Thinker to generate two …

**Figure 7.** Figure 7: Prompt used for constructing Ref-Thinker instruction tuning set. [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]

**Figure 8.** Figure 8: Prompt used for constructing R2 -AVSBench [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]

**Figure 9.** Figure 9: Examples of reference expressions from our R [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]

**Figure 10.** Figure 10: Comparison results on the Ref-AVSBench dataset: [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]

**Figure 11.** Figure 11: Comparison results on the Ref-AVSBench dataset: [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]

**Figure 12.** Figure 12: Comparison results on the R2 -AVSBench dataset: Seen test set samples [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]

**Figure 13.** Figure 13: Comparison results on the R2 -AVSBench dataset: Unseen test set samples [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]

**Figure 14.** Figure 14: Failure cases analysis. Samples are from the Ref-AVSBench dataset. [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

MAR3: Multi-Agent Recognition, Reasoning, and Reflection for Reference Audio-Visual Segmentation
cs.MM 2026-03 conditional novelty 6.0 of 10

A training-free multi-agent system (Delphi recognition, modality-adaptive reasoning, reflective mask correction) reaches 69.2% J&F on Ref-AVSBench, +3.4 over prior SOTA.

Reference graph

Works this paper leans on

15 extracted references · 1 canonical work pages · cited by 1 Pith paper

[1]

As illustrated in Fig

Factual errors at the Think stage. As illustrated in Fig. 14a, both the ukulele and the girl produce sound, and the reference specifies the object making the longest sound duration. Although Ref-Thinker successfully identifies the presence of both the girl’s voice and the ukulele sound, it incorrectly concludes that the ukulele has the longest continu- ou...
[2]

In CVPR, 18804–18814

Crab: A unified audio-visual scene understanding model with explicit cooperation. In CVPR, 18804–18814. Gao, S.; Chen, Z.; Chen, G.; Wang, W.; and Lu, T. 2024. Avsegformer: Audio-visual segmentation with transformer. In AAAI, volume 38, 12155–12163. Google. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. https://...

arXiv 2024
[3]

ABSOLUTELY CRITICAL RESTRICTION: A VOID DIRECT ATTRIBUTES AND SIMPLE MULTIMODAL COMBINATIONS. * DO NOT use explicit, easily identifiable attributes like color (”red car”, ”blue bird”), simple shape (”round ball”, ”square box”), obvious size (”large building”, ”tiny bug”), direct, dominant sounds (”loud alarm”, ”siren”, ”dog barking”, ”engine roaring”), or...
[4]

In AAAI, 5075–5083

Patch-level sounding object tracking for audio-visual question answering. In AAAI, 5075–5083. Lin, B.; Ye, Y .; Zhu, B.; Cui, J.; Ning, M.; Jin, P.; and Yuan, L. 2023. Video-llava: Learning united visual repre- sentation by alignment before projection. arXiv preprint arXiv:2311.10122. Ling, Y .; Li, Y .; Gan, Z.; Zhang, J.; Chi, M.; and Wang, Y

arXiv 2023
[5]

The sounding object near the woman.\

Examples (New Style of HIGHLY CHALLENGING, MULTIMODAL, Reasoning-focused Refs - Concise and Complex): - Target Object: ”hair-dryer” (Pixel mask shows it in hand, pointed at hair, generating sound and hot air movement.) Reasoning-intensive Ref: ”The handheld device creating localized heat and continuous ambient noise.” (11 words) - Target Object: ”emergenc...

work page
[8]

Imprecise bounding boxes despite correct reasoning. In Fig. 14b, Ref-Thinker correctly interprets the reference and identifies the target object as marimba. However, during the Ground phase, the object detector (Grounding-DINO) fails to generate accurate bounding boxes. This may stem from limited pre-training data involving rare objects like the marimba. ...
[9]

Do not alter the given ref in any way

In <think>, you must: - Start your <think> reasoning with: The referential expression is: ”<ref>”. Do not alter the given ref in any way. - First provide a brief description of the overall visual and audio context. If the audio has been explicitly indicated as silent, state ’The audio is silent.’ directly. Otherwise, analyze the provided audio file. If th...
[10]

If the object is unique in the scene, a slightly more detailed description than <s object> is sufficient

In <answer>, output must follow this strict format: <f object> A fine-grained description of the object (appearance, location, attributes, actions) </f object> <s object> The simplified category of the object (e.g., guitar, dog, car) </s object> - Keep the <f object> description concise (6-10 words). If the object is unique in the scene, a slightly more d...

Show all 15 references

[11]

This complex ref must unambiguously and accurately refer to the exact target object uniquely identified by the uid and its associated pixel mask in the video

To create a completely newcomplex ref for the target object (”{target object name}”). This complex ref must unambiguously and accurately refer to the exact target object uniquely identified by the uid and its associated pixel mask in the video
[12]

Crucially: avoid any simple, direct descrip- tions

Your complex ref must be ABSOLUTELY CONCISE (STRICTLY 5-15 WORDS, NO EXCEPTIONS) and strictly incorporate one or more of these challenging reasoning types. Crucially: avoid any simple, direct descrip- tions. If a direct description (like ”red car” or ”loud dog barking”) can id...
[14]

Output Format: Return a JSON object containing the result for the current uid
[2021]

We observe that the majority of CLIPScore values fall within the range of [0.8, 1], indicating strong se- mantic similarity

between the predicted descriptions and the ground- truth categories. We observe that the majority of CLIPScore values fall within the range of [0.8, 1], indicating strong se- mantic similarity. For instance, as shown in Fig. 13b, although the annotated label of the referred ob...
[2022]

In CVPR, 19108–19118

Learning to answer questions in dynamic audio-visual scenarios. In CVPR, 19108–19118. Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023a. Blip-2: Boot- strapping language-image pre-training with frozen image encoders and large language models. In ICML, 19730–19742. PMLR. Li, K.; ...

2024
[2024]

traditional East Asian melodies

TransA VS: End-to-End Audio-Visual Segmentation with Transformer. In ICASSP, 7845–7849. IEEE. Liu, J.; Wang, Y .; Ju, C.; Ma, C.; Zhang, Y .; and Xie, W. 2024a. Annotation-free audio-visual segmentation. In WACV, 5604–5614. Liu, S.; Zeng, Z.; Ren, T.; Li, F.; Zhang, H.; Yang, ...

2025 arXiv
[2025]

In AAAI, volume 39, 15567–15575

Unsupervised audio-visual segmentation with modality alignment. In AAAI, volume 39, 15567–15575. Chen, H.; Xie, W.; Afouras, T.; Nagrani, A.; Vedaldi, A.; and Zisserman, A. 2021. Localizing visual sounds the hard way. In CVPR, 16867–16876. Chen, S.; Wu, Y .; Wang, C.; Liu, S.;...

2021 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

As illustrated in Fig

Factual errors at the Think stage. As illustrated in Fig. 14a, both the ukulele and the girl produce sound, and the reference specifies the object making the longest sound duration. Although Ref-Thinker successfully identifies the presence of both the girl’s voice and the ukulele sound, it incorrectly concludes that the ukulele has the longest continu- ou...

[2] [2]

In CVPR, 18804–18814

Crab: A unified audio-visual scene understanding model with explicit cooperation. In CVPR, 18804–18814. Gao, S.; Chen, Z.; Chen, G.; Wang, W.; and Lu, T. 2024. Avsegformer: Audio-visual segmentation with transformer. In AAAI, volume 38, 12155–12163. Google. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. https://...

arXiv 2024

[3] [3]

ABSOLUTELY CRITICAL RESTRICTION: A VOID DIRECT ATTRIBUTES AND SIMPLE MULTIMODAL COMBINATIONS. * DO NOT use explicit, easily identifiable attributes like color (”red car”, ”blue bird”), simple shape (”round ball”, ”square box”), obvious size (”large building”, ”tiny bug”), direct, dominant sounds (”loud alarm”, ”siren”, ”dog barking”, ”engine roaring”), or...

[4] [4]

In AAAI, 5075–5083

Patch-level sounding object tracking for audio-visual question answering. In AAAI, 5075–5083. Lin, B.; Ye, Y .; Zhu, B.; Cui, J.; Ning, M.; Jin, P.; and Yuan, L. 2023. Video-llava: Learning united visual repre- sentation by alignment before projection. arXiv preprint arXiv:2311.10122. Ling, Y .; Li, Y .; Gan, Z.; Zhang, J.; Chi, M.; and Wang, Y

arXiv 2023

[5] [5]

The sounding object near the woman.\

Examples (New Style of HIGHLY CHALLENGING, MULTIMODAL, Reasoning-focused Refs - Concise and Complex): - Target Object: ”hair-dryer” (Pixel mask shows it in hand, pointed at hair, generating sound and hot air movement.) Reasoning-intensive Ref: ”The handheld device creating localized heat and continuous ambient noise.” (11 words) - Target Object: ”emergenc...

work page

[6] [8]

Imprecise bounding boxes despite correct reasoning. In Fig. 14b, Ref-Thinker correctly interprets the reference and identifies the target object as marimba. However, during the Ground phase, the object detector (Grounding-DINO) fails to generate accurate bounding boxes. This may stem from limited pre-training data involving rare objects like the marimba. ...

[7] [9]

Do not alter the given ref in any way

In <think>, you must: - Start your <think> reasoning with: The referential expression is: ”<ref>”. Do not alter the given ref in any way. - First provide a brief description of the overall visual and audio context. If the audio has been explicitly indicated as silent, state ’The audio is silent.’ directly. Otherwise, analyze the provided audio file. If th...

[8] [10]

If the object is unique in the scene, a slightly more detailed description than <s object> is sufficient

In <answer>, output must follow this strict format: <f object> A fine-grained description of the object (appearance, location, attributes, actions) </f object> <s object> The simplified category of the object (e.g., guitar, dog, car) </s object> - Keep the <f object> description concise (6-10 words). If the object is unique in the scene, a slightly more d...