Pixel-Level Reasoning Segmentation via Multi-turn Conversations

Daling Wang; Dexian Cai; Shi Feng; Soujanya Poria; Xiaocui Yang; Yifei Zhang; Yongkang Liu

REVIEW 1 major objections 6 minor 1 cited by

Pixel-Level Reasoning Segmentation via Multi-turn Conversations

T0 review · 1 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Pixel-level reasoning segmentation works through multi-turn dialogue, new dataset and model show.

desk verdict The PRIST dataset is a genuine, useful contribution to reasoning segmentation; the stress-test ambiguity worry is largely mitigated by the dialogue examples, but the reasoning evaluation is self-referential and the 'evolving intent' framing is oversold. read the letter →

arxiv 2502.09447 v1 pith:6AT4TEOR submitted 2025-02-13 cs.CV cs.CL

Dexian Cai , Xiaocui Yang , Yongkang Liu , Daling Wang , Shi Feng , Yifei Zhang , Soujanya Poria This is my paper · ORCID

classification cs.CVcs.CL

keywords pixel-levelreasoningsegmentationmulti-turnconversationreferringexpressionmultimodallargelanguagemodeltreefine-grainedinteractive

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Existing visual perception systems can segment objects from a single explicit instruction, but they cannot follow a conversation and narrow down what a user means as the dialogue unfolds. The paper claims that multi-turn dialogue can resolve vague, implicit requests into precise pixel-level targets, and it establishes this as a new task, Pixel-level Reasoning Segmentation. To make the task testable, it builds PRIST, a dataset of 8.3k multi-turn conversational scenarios with fine-grained segmentation masks, and proposes MIRAS, a model that couples a multimodal language model with a mask decoder to segment and explain in the same response. On PRIST, MIRAS outperforms both general multimodal models and segmentation-specific baselines, which matters because it offers a path from coarse instructions such as 'segment the ingredients' to fine-grained outputs like 'the oil bottle.'

What carries the argument

The load-bearing machinery is a hierarchical reasoning tree generated by an LLM, which decomposes each target object's complex question into progressively finer question-answer pairs, and the tree is then flattened into a multi-turn dialogue so that intent sharpens turn by turn. On the model side, MIRAS connects a multimodal LLM to a mask decoder through a special [SEG] token, with a semantic region alignment module that extracts the hidden states between [OBJ] and [SEG] as the segmentation prompt. The dual visual encoder, combining a 768×768 ConvNeXt-L stream with a 336×336 CLIP-L/14 stream via cross-attention, supplies the high-resolution detail that pixel-level masks require. The design work is to let language reasoning at the conversation level condition where, and how finely, the mask decoder cuts.

What would settle it

Recruit fresh annotators who see only the dialogue and the final generic instruction, without the recorded [OBJ]...[SEG] target, and compare their masks to the dataset's masks; if their agreement is no better than chance, the dialogue alone does not determine the intended segmentation.

Watch

Extended reading notes

Core claim

In the paper's own terms, the central claim is a new task definition and its first working system: given an image and a multi-turn dialogue, a model should jointly produce a textual reasoning chain and a pixel-level segmentation mask that reflects the intent reached at the end of the conversation. The paper argues that single-turn reasoning segmentation datasets stop at region-level masks and cannot track intent that changes or sharpens across turns, so it constructs PRIST with a three-step pipeline that extracts visible elements, expands them into a hierarchical reasoning tree, and linearizes tree paths into 4–8 turn dialogues whose final query is a generic segmentation instruction. The MIRAS framework is designed for this setting, fusing a high-resolution ConvNeXt feature stream with a low-resolution CLIP stream, injecting the description of the target into the mask decoder through a [OBJ]{CLASS}[SEG] template, and training end-to-end with text, BCE, and DICE losses. The experimental claim is that MIRAS outperforms existing baselines on PRIST across pixel-level segmentation, conversational response, and LLM-judged reasoning quality, setting the initial state of the art for the task.

Load-bearing premise

The dialogue and the final generic segmentation instruction uniquely determine which object's mask should be produced, so the same conversation reliably maps to the same target for different users.

Editorial extensions

If this is right

Fine-tuning on PRIST raises the CIoU and precision of every tested segmentation-specific multimodal model, with LISA's precision rising from 15.33 to 26.23.
MIRAS sets the current benchmark on PRIST with CIoU 14.72, F1 30.34, and a 42% reasoning Win Rate, approaching human expert reasoning scores.
Models fine-tuned on PRIST improve their LLM-judged reasoning quality by about 10 percentage points in Win Rate on average, across all tested architectures.
MIRAS remains competitive on the classic referring expression segmentation benchmarks refCOCO, refCOCO+, and refCOCOg, despite being optimized for the multi-turn task.
The precision-recall shift observed after fine-tuning means the dataset pushes models toward tighter boundaries rather than broad over-segmentation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same dialogue-generation machinery could be extended to multiple distinct objects per scene or to empty targets, turning the benchmark into a general interactive perception test rather than a single-object segmentation task.
Because the final query is a fixed generic instruction, PRIST's dialogues can be reused as a multi-turn visual question-answering benchmark when the segmentation instruction is dropped, giving a second task for free.
The LLM-as-judge metrics (Progressiveness, Logical Coherence, Content Consistency, Target Relevance) could transfer to any interactive perception task that must track evolving user intent.
The observed precision/recall trade-off suggests a controllable specificity knob: adjusting the PRIST fine-tuning mixture could let a single model dial between fine-grained segmentation and broader zero-shot generalization.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Zoe,

Quick take: the dataset is the contribution. PRIST gives the community 8.3k multi-turn conversational scenarios with pixel-level masks, extending reasoning segmentation from single-turn, region-level to multi-turn, pixel-level. The construction pipeline is described in detail, the human mask annotation has IoU >0.80 and Kappa >0.75, and the dataset statistics look reasonable. MIRAS is an engineering combination of known parts (LISA-style [SEG] token, dual encoders, SAM decoder), but it is a sensible instantiation and performs well on PRIST and on the RES benchmarks. The ablations are clean. So credit is earned for the benchmark and task framing.

The stress-test worry about ambiguity is weaker than it looks. In the published examples, the dialogue's final assistant turn explicitly names the target: "The result is [OBJ]bus front display[SEG]." So the mask is determined by the dialogue—the generic final user query is paired with an assistant answer that resolves the referent. The deeper issue is not ill-posedness but ecological validity: the 'evolving intent' is scripted by GPT-4o, not observed from real users, and the target is spelled out in the last assistant turn. That makes the task closer to referring segmentation with a long conversational context. Still a useful testbed, but it does not fully deliver the 'tracking evolving user intent' framing the paper advertises.

The other soft spots are real but minor. The reasoning-quality evaluation is self-referential—GPT-4 judges GPT-4o-generated dialogues—and the 100 human-scored samples are reported but not linked to the LLM judge by any agreement metric. They should report GPT-4/human correlation or kappa on those samples. There are also no error bars or significance tests on the main table, which is common in this line of work but worth flagging.

I would not cite the model as a methodological novelty, but I would cite the dataset as a benchmark. Recommendation: send it to review. The core resource is valuable, the experiments are adequate, and the weaknesses are addressable rather than fatal. Major revision requests: validate the LLM judge against human ratings, report target-selection agreement from fresh readers of the dialogue, and temper the 'evolving intent' claim to match what the data actually supports.

Referee Report

1 major / 6 minor

Summary. The paper proposes a new task, Pixel-level Reasoning Segmentation (Pixel-level RS), in which a model receives an image and a multi-turn dialogue and must output both a textual reasoning chain and a pixel-level segmentation mask. The authors construct PRIST, a dataset of 8,320 multi-turn conversational scenarios (24k utterances) with manually annotated masks, using a GPT-4o-based three-step pipeline that first extracts visible elements, builds a hierarchical reasoning tree, and then converts the tree into dialogues whose final query is a generic segmentation instruction. They also propose MIRAS, a framework combining a dual visual encoder, an MLLM with a [SEG] token, and a SAM-based mask decoder, trained in two stages. Experiments report that MIRAS achieves the best results on PRIST for CIoU (14.72), F1 (30.34), several response metrics, and a reasoning Win Rate of 42%, and that MIRAS is competitive on standard referring expression segmentation benchmarks.

Significance. If the PRIST benchmark is well-posed and the reported gains are reliable, the paper makes a useful contribution: it introduces a new interactive fine-grained segmentation task, releases a sizable dataset with high-quality manual masks, and provides an open framework (code and data are promised) that outperforms existing segmentation-specific MLLMs on the new benchmark. The consistency checks on mask annotation (IoU > 0.80, Kappa > 0.75) and the positive generalization results on refCOCO/refCOCO+/refCOCOg provide independent grounding that the model architecture is sound. The main risks are the unverified well-posedness of the dialogue-to-target mapping, the self-referential LLM-based reasoning evaluation, and the absence of an ablation isolating the multi-turn benefit.

major comments (1)

[§3.2.3, Eq. (1), §3.3, Figure 9] The comparison of MIRAS Stage-1 and Stage-2 in Table 5 is confounded by the base LLM version: MIRAS (Stage-1) with LLaVA-v1.6 is compared to MIRAS (Stage-2) with LLaVA-v1.6, but the "MIRAS (Stage-1) (v1)" row and "MIRAS (Stage-2) (v1.6)" differ in both the training stage and the base model. This makes it difficult to attribute the RES improvement to the two-stage procedure rather than to the stronger base model. The ablation of training stage should be done with the same base model.

minor comments (6)

[Abstract and throughout] The name "MIRSA" appears in the abstract and in the conclusion; the correct acronym is "MIRAS". Please standardize.
[§4.2] The text says "WarmipDecayLR"; this should be "WarmupDecayLR".
[Eq. (4)] The notation X'_H ∈ R^{H×W×3} is confusing because the output of ConvNext is a feature map with channel dimension C, not 3. Please use a channel dimension variable and state the actual number of features.
[References] The citation for GPT-4o (Achiam et al., 2023) points to the GPT-4 technical report, not to GPT-4o. Please cite the GPT-4o model or the specific version used (gpt-4o-2024-08-06) appropriately.
[Table 3] The table header contains "LLaV A" instead of "LLaVA". Please correct this in the table and in the prose.
[§C.4.2] The sentence "To ensure fairness and consistency, we carefully designed prompts for each metric" is repeated verbatim. Please remove the duplicate.

Circularity Check

2 steps flagged · score 4.0 of 10

PRIST's target masks are fixed by hidden pipeline-selected objects rather than by the published dialogue, and the reasoning-quality scores use an LLM judge from the same family that generated the gold dialogues; segmentation metrics and the external RES benchmark keep the central model claim partially independent.

self definitional [Eq. (1); Sec. 3.2.2-3.2.3; Fig. 9]
"(M, {a1, a2, . . . , aN }) =Model(I, D). ... This step expands a complex reasoning question Qi for each target oi, serving as the overall origin for next question decomposition and the theme for multi-turn dialogues in Step-3. ... the final query in each dialogue being a segmentation-related instruction (e.g., 'Please segment the core objects according to the above dialogue')."

The benchmark's task is defined by Eq. (1) as a mapping from (I, D) to M, but the dataset generation pipeline first selects a hidden target o_i and then constructs Qi and the dialogue Di around it. The final query is deliberately generic, so the published input does not uniquely identify which of the K candidate objects is the labelled one. The Figure 9 'Bus' example even shows parallel paths whose targets ('bus front display' vs. text 'CARISA-HOSPITAL') are both plausible answers to the same dialogue. The reported annotator IoU > 0.80 / Kappa > 0.75 checks only mask drawing once the target is already known from the generated text; it does not test whether a fresh reader can select the target from the dialogue.
other [Sec. 5.1; App. B.2; App. C.4.2]
"The pipeline, entirely based on GPT-4o ( gpt-4o-2024-08-06), incorporates CoT into a feedback loop to generate relevant multi-turn reasoning dialogues for various images. ... Meanwhile, we employ GPT-4o as a judge to assess dialogue reasoning quality. The model wins when its score surpasses that of the human response, as reflected by the Win Rate (%) metric."

The gold dialogues and their [OBJ]...[SEG] targets are produced by GPT-4o, and the reasoning-quality scores and Win Rate are assigned by GPT-4o/GPT-4 from the same model family. Because the fine-tuned models are trained to imitate those GPT-4o-generated dialogues, the LLM judge is evaluating outputs from the same distribution it was used to create. The PR/LC/CC/TR scores and the 42% Win Rate therefore measure stylistic agreement with the generator rather than an independently verified reasoning property. This is an evaluation loop rather than a fitted-parameter prediction, but it makes the 'approaching human expert' reasoning claim partially circular.

full rationale

The paper does not rely on a self-citation chain or an imported uniqueness theorem; no fitted parameter is renamed as a prediction, and the segmentation numbers (CIoU 14.72, F1 30.34) are computed against human-annotated masks rather than against the model's own outputs. MIRAS is also tested on the external refCOCO/refCOCO+/refCOCOg RES benchmark, which gives independent grounding for the architecture's segmentation capability. The circularity burden comes from two places. First, PRIST's ground-truth masks are selected by the pipeline's hidden target o_i before the dialogue is generated, and the final query is generic, so the dialogue alone may not determine the target; the reported annotator agreement tests mask drawing, not target selection. Second, the reasoning-quality evaluation uses GPT-4/GPT-4o as judge over dialogues and targets generated by GPT-4o, creating a same-family evaluation loop for the Win Rate and PR/LC/CC/TR claims. Neither issue collapses the entire derivation: the segmentation metrics and RES benchmark remain independent evidence. But the benchmark's well-posedness and the reasoning-quality claim are partially self-referential, so the paper is not free of circularity.

Assumptions & free parameters 6 free parameters · 5 assumptions · 1 invented entities

The central claims rest on assumptions about GPT-4o data quality, the well-posedness of the generic segmentation instruction, and the validity of GPT-4 judging. Standard ML hyperparameters are numerous but not scientifically load-bearing; key ones are listed as free parameters where they directly shape the dataset or model.

free parameters (6)

Loss weights (lambda_t, lambda_bce, lambda_dice) = 1.0, 2.0, 0.5
Set following LISA in Eq. (8); directly balance text and segmentation objectives during training.
LoRA rank r = 8
Chosen for efficient LLM fine-tuning; affects model capacity and final performance.
Target object count K per image = 2 to 4 (random)
Step 2-1 randomly selects K objects; controls dialogue count and complexity in PRIST.
Max child nodes per reasoning tree layer = 3
Step 2-2 caps tree width; shapes the structure of generated multi-turn dialogues.
Dual-encoder input resolutions = 768x768 and 336x336
Equations in Section 4.1; chosen to capture fine visual details and affect segmentation quality.
Two-stage training schedule = Stage1: 50k steps, lr 3e-4, batch 16; Stage2: 20k steps, lr 1e-5, batch 32
Appendix C.1 Table 9; empirical hyperparameters that influence reported results.

assumptions (5)

domain assumption GPT-4o reliably extracts visible objects and constructs coherent reasoning trees and dialogues from images.
The entire PRIST pipeline in Section 3.2 uses GPT-4o output as raw material, corrected only for commonsense errors by annotators.
ad hoc to paper The generic final instruction "Please segment the core objects according to the above dialogue" uniquely determines the target mask.
Section 3.2.3 defines the final query; if this instruction is ambiguous, the dataset's supervision signal is ill-posed.
domain assumption GPT-4 as a judge gives valid, unbiased reasoning-quality scores suitable for Win Rate comparison.
Section 5.1 and Appendix C.4.2 use GPT-4 to score all model and human responses; no validation against independent human preference is reported.
domain assumption Annotator agreement (IoU > 0.80, Kappa > 0.75) is sufficient evidence of ground-truth quality.
Section 3.3 uses these thresholds to certify masks, but annotators knew the intended target from generated text.
domain assumption The SAM mask decoder can be trained to output masks from MLLM [SEG] hidden states.
Section 4.1 builds on prior LISA/SAM work; this adaptation is inherited rather than derived in this paper.

invented entities (1)

[SEG] and [OBJ] special tokens
purpose: Connect the MLLM text output to the mask decoder and extract the target description subsequence.
Introduced in Section 4.1 as architectural tokens; they are implementation artifacts, not independently falsifiable entities.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Pixel-Level Reasoning Segmentation via Multi-turn Conversations." pith.science (2026). https://pith.science/paper/6AT4TEOR

@misc{pith2026250209447,
  author       = {Pith},
  title        = {Pith review of: Pixel-Level Reasoning Segmentation via Multi-turn Conversations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6AT4TEOR}},
  note         = {Machine review of arXiv:2502.09447}
}

read the original abstract

Existing visual perception systems focus on region-level segmentation in single-turn dialogues, relying on complex and explicit query instructions. Such systems cannot reason at the pixel level and comprehend dynamic user intent that changes over interaction. Our work tackles this issue by introducing a novel task, Pixel-level Reasoning Segmentation (Pixel-level RS) based on multi-turn conversations, tracking evolving user intent via multi-turn interactions for fine-grained segmentation. To establish a benchmark for this novel task, we build a Pixel-level ReasonIng Segmentation Dataset Based on Multi-Turn Conversations (PRIST), comprising 24k utterances from 8.3k multi-turn conversational scenarios with segmentation targets. Building on PRIST, we further propose MIRAS, a Multi-turn Interactive ReAsoning Segmentation framework, integrates pixel-level segmentation with robust multi-turn conversation understanding, generating pixel-grounded explanations aligned with user intent. The PRIST dataset and MIRSA framework fill the gap in pixel-level reasoning segmentation. Experimental results on the PRIST dataset demonstrate that our method outperforms current segmentation-specific baselines in terms of segmentation and LLM-based reasoning metrics. The code and data are available at: https://github.com/ccccai239/PixelRIST.

Figures

Figures reproduced from arXiv: 2502.09447 by the authors.

**Figure 1.** RS vs. Pixel-level RS. Pixel-level RS refines intent understanding and segmentation (e.g., "oil bottle") through multi-turn interactions, while RS produces rough segmentation (e.g., "all ingredients") and handles implicit single-turn queries poorly. 2023; You et al., 2023; Pi et al., 2023; Zhang et al., 2023a), they depend on user-provided regions to trigger reasoning. These perception systems still cannot actively… view at source ↗

**Figure 2.** The Generation Pipeline of PRIST Dataset. i) Step 1 extracts visible elements from images, establishing a semantic foundation for subsequent steps. ii) Step 2-1 generates complex reasoning questions from these elements, while Step 2-2 iteratively refines the questions through a reasoning tree, ensuring rigorous reasoning. iii) Step 3 organizes the nodes in reasoning tree into a multi-turn dialogue format. 3 PRIST Th… view at source ↗

**Figure 3.** The focus distribution of PRIST. We analyze focus objects across 3 dimensions: noun, adjective and preposition, which capture fine granularity, diversity, and close spatial relationships between objects. 1024 × 1024, PRIST meets the fine-grained requirements of pixel-level segmentation. We emphasizes exhibiting high diversity in categories and descriptions to enhance expressiveness. Illustrated the focus distribu… view at source ↗

Figures from the paper (7 more)

**Figure 4.** Figure 4: Overview Architecture of MIRAS. The model integrates MLLM and SAM modules by introducing a special token [SEG]. MIRAS can perform both (a) Multi-turn Response and (b) Segmentation tasks end-to-end. [OBJ]{CLASS}[SEG], where {CLASS} is the object description (e.g., [OBJ…

**Figure 5.** Figure 5: Wordcloud of the 200 popular focus-related [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]

**Figure 6.** Figure 6: Qualitative results of the performance of MI [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]

**Figure 7.** Figure 7: The prompts and output formats of our dataset annotation pipeline. [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]

**Figure 8.** Figure 8: Illustration of the LLM-based evaluation of reasoning quality. [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]

**Figure 9.** Figure 9: The samples of PRIST. The figure displays two examples from the PRIST dataset, created through the automated annotation pipeline. It offers a range of semantic labels and attributes for the identified objects, as well as contextualized multi-turn conversations. 18 [PI…

**Figure 10.** Figure 10: Pixel-level segmentation and multi-turn conversational interactions facilitated by MIRAS. [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Reasoning Segmentation for Images and Videos: A Survey
cs.CV 2025-05 conditional novelty 3.0 of 10

The paper organizes the field of reasoning segmentation into image and video tracks, cataloging 26 methods, 12 metrics, and 29 datasets.

Reference graph

Works this paper leans on

20 extracted references · 15 canonical work pages · cited by 1 Pith paper

[1]

identify and list the elements visible in an image. \

work page
[2]

elements

Provide detailed descriptions of each element, with text, numbers, and other information.3. List at least 3 elements visible in the image. ## Output Format ： {"elements": [{ "name": "The name of the element1", "text": "Text description of the element1", "number": (optional)"Number of the element", "caption": "Caption of the element"}, ...]} (a) Illustrati...

work page
[3]

root_node

Display a reasoning tree structure: Present the reasoning tree structure showing the progression of reasoning for each question. Highlight any overlapping nodes or steps among the three questions to reflect common reasoning pathways. 4. Output the corresponding reasoning questions for each of the three paths. ## Output Format ： {"root_node": "Root Node A"...

work page
[4]

## Evaluation Criteria:

Please focus on linking the dimensions between each round of conversation. ## Evaluation Criteria:

work page
[5]

Please seg- ment the apple in the image

Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36. Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. 2023. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704. Liche...

arXiv 2023
[8]

Design 3 reasoning questions based on the listed elements and details from the image and caption, with scores of 2-3 points

work page
[9]

Each question must involve reasoning and use common sense to analyze visible elements

work page
[11]

Each dialogue must consist of 4 to 8 turns of questions and answers. 2. Each question should focus on a single, specific object visible in the image, with reasoning directly tied to the elements listed in the title and the image details

work page

Show all 20 references

[12]

Question1

Expand each reasoning path into a multi-round dialogue format according to the provided reasoning tree, demonstrating the logical progression of thought based on the identified paths. ## Output Format ：{"Question1": { "Q1": "<First Question>", "A1": "<Answer to First Question>...
[13]

Please analyze the following multi-turn conversation and score it based on the following dimensions: - Progression: Does the conversation build logically from one turn to the next? Is the topic being developed effectively? - Logical Coherence: Is there a clear and natural logi...
[14]

Only output the reasoning scores in the above format
[15]

Please evaluate the entire process to the content of the image
[17]

2-3 points: Partially advances the dialogue, but the connection feels somewhat awkward

Progression Score: 4-5 points: Smoothly transitions to the next turn with sufficient setup for subsequent content. 2-3 points: Partially advances the dialogue, but the connection feels somewhat awkward. 0-1 points: Fails to advance the dialogue or is disconnected from the next turn
[18]

2-3 points: Generally clear logic, but with minor gaps or weak links

Logical Coherence Score: 4-5 points: Perfectly connected with rigorous logic. 2-3 points: Generally clear logic, but with minor gaps or weak links. 0-1 points: No logical connection or logically incoherent
[19]

2-3 points: Generally consistent, but with a slight deviation

Content Consistency Score: 4-5 points: Content is fully aligned and tightly focused on the goal. 2-3 points: Generally consistent, but with a slight deviation. 0-1 points: Deviates from the topic, losing the direction of the dialogue
[20]

scores": [ {

Focal Goal Relevance Score: 4-5 points: Highly relevant and continuously advances the goal. 2-3 points: Somewhat relevant, but with limited contribution. 0-1 points: Deviates from the goal, failing to advance it. ## Output in JSON Format: Your output should be a JSON object co...

1953
[2017]

International journal of computer vision, 123:32–73

Visual genome: Connecting language and vi- sion using crowdsourced dense image annotations. International journal of computer vision, 123:32–73. Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. 2023. Lisa: Reason- ing segmentation via large la...

2023 arXiv
[2018]

In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1209–1218

Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1209–1218. Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. 2023. Shikra: Unleashing multimodal llm’s referential d...

2023 arXiv
[2023]

arXiv preprint arXiv:2306.14824

Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824. Renjie Pi, Jiahui Gao, Shizhe Diao, Rui Pan, Hanze Dong, Jipeng Zhang, Lewei Yao, Jianhua Han, Hang Xu, Lingpeng Kong, et al. 2023. Detgpt: De- tect what you need via reasoning....

2023 arXiv
[2024]

In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 26374–26383

Pixellm: Pixel reasoning with large multi- modal model. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 26374–26383. Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. 2020. Textcaps: a dataset for image captioning...

2020 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

identify and list the elements visible in an image. \

work page

[2] [2]

elements

Provide detailed descriptions of each element, with text, numbers, and other information.3. List at least 3 elements visible in the image. ## Output Format ： {"elements": [{ "name": "The name of the element1", "text": "Text description of the element1", "number": (optional)"Number of the element", "caption": "Caption of the element"}, ...]} (a) Illustrati...

work page

[3] [3]

root_node

Display a reasoning tree structure: Present the reasoning tree structure showing the progression of reasoning for each question. Highlight any overlapping nodes or steps among the three questions to reflect common reasoning pathways. 4. Output the corresponding reasoning questions for each of the three paths. ## Output Format ： {"root_node": "Root Node A"...

work page

[4] [4]

## Evaluation Criteria:

Please focus on linking the dimensions between each round of conversation. ## Evaluation Criteria:

work page

[5] [5]

Please seg- ment the apple in the image

Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36. Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. 2023. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704. Liche...

arXiv 2023

[6] [8]

Design 3 reasoning questions based on the listed elements and details from the image and caption, with scores of 2-3 points

work page

[7] [9]

Each question must involve reasoning and use common sense to analyze visible elements

work page

[8] [11]

Each dialogue must consist of 4 to 8 turns of questions and answers. 2. Each question should focus on a single, specific object visible in the image, with reasoning directly tied to the elements listed in the title and the image details

work page