Pith. sign in

REVIEW 3 major objections 5 minor 76 references

Better, Stronger, Faster, and Broader: Structured All-Mask Prediction for MLLM-Based Segmentation

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims STAMPlus resolves the segmentation trilemma for multi-target segmentation with a single structured all-mask pass.

desk verdict STAMPlus is a genuine extension of STAMP with a broad benchmark sweep; the Phase 1 target-list bottleneck is real but disclosed, and the paper deserves peer review. read the letter →

arxiv 2608.02791 v1 pith:GWHYNDZP submitted 2026-08-03 cs.CV

Jiazhen Liu , Mingkuan Feng , Long Chen This is my paper · ORCID
classification cs.CV
keywords multimodallargelanguagemodelsimagesegmentationstructuredall-maskpredictionnon-autoregressivemaskreferringopen-vocabularysemanticinstanceremotesensingsmall-target
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multimodal large-language models that segment images face a three-way trade-off: accurate masks, natural dialogue, and fast inference. The paper claims that STAMPlus breaks this trilemma for multi-target segmentation by splitting the task into two phases: the model first writes a structured list of targets, each with an ID and an optional bounding box, then classifies every image patch against those IDs in a single parallel forward pass. Because mask prediction stays inside the token interface and does not use an external decoder, general dialogue ability is preserved, and because all masks are predicted at once, latency no longer grows with the number of targets. The paper reports state-of-the-art segmentation across referring, reasoning, open-vocabulary semantic, instance-aware, and remote-sensing small-target benchmarks using one unified checkpoint.

What carries the argument

The load-bearing mechanism is the two-phase all-mask pipeline with dynamic ID binding. Phase 1 is ordinary autoregressive text generation that produces a structured target list and a <SEG> trigger; Phase 2 prefills one mask token per image patch, fuses each token with its patch features and positional encoding, runs hybrid attention (causal on history, bidirectional among mask tokens), and classifies all tokens simultaneously. STAMPlus changes the prediction head from binary to a multi-class head with fixed capacity 200 and makes the label space instruction-defined by the Phase 1 list. KV-cache reuse is what makes Phase 2 cheap and what lets the mask tokens condition on the generated IDs.

What would settle it

Run STAMPlus on an image with two same-category instances, then edit the Phase 1 output so the two IDs' entries are swapped in the JSON list (or drop the last entry) while keeping the image fixed. If the predicted masks do not swap identities accordingly, or if the dropped target's mask persists, the claimed dynamic ID binding through cached KV states is not what drives Phase 2; the paper's own oracle-box result predicts that such a perturbation should substantially degrade cIoU.

Watch

Extended reading notes

Core claim

Structured All-Mask Prediction generalizes the authors' binary STAMP design to multi-target outputs. In Phase 1 the model autoregressively emits a JSON-style structured target list in which each entry assigns a numeric cls ID to a semantic category or object instance and may include a bounding box; the list ends with an in-vocabulary <SEG> trigger. In Phase 2, image-aligned [MASK] tokens, one per patch, are prefilled and processed in one non-autoregressive forward pass; a 200-way mask head labels each token as background or one of the active IDs, and predictions outside the Phase 1 list are discarded. Because Phase 2 reuses the Phase 1 KV cache, the mask tokens attend to the generated list, binding IDs to masks. The paper claims this resolves the trilemma beyond single targets: state-of-the-art cIoU and mIoU across referring, reasoning, open-vocabulary semantic, instance-aware, and remote-sensing small-target benchmarks; preserved general multimodal dialogue; and a 12-category mask latency reduction from 13.50 seconds for repeated STAMP inference to 5.16 seconds.

Load-bearing premise

The trilemma claim collapses if Phase 1's generated target list is not a reliable, correctly ordered, complete specification of the targets; the paper's own experiment shows that replacing the auto-generated boxes with ground-truth boxes lifts RefCOCO-family cIoU from 77.4 to 89.1, so the Phase 1 list, not the mask classifier, is the current bottleneck.

Editorial extensions

If this is right

  • Multi-category and multi-instance segmentation share a single Phase 2 forward pass, so inference latency no longer scales with the number of targets.
  • A single unified checkpoint, without task-specific fine-tuning, covers referring, reasoning, open-vocabulary semantic, instance-aware, and remote-sensing small-target segmentation.
  • Jointly scaling input resolution and mask-token budget from 1024-1280 to 2560-3200 tokens improves small-target gIoU by 5.5 points on RRSIS-D and 4.9 points on EarthReason.
  • Mixed training with visual-instruction data keeps general multimodal scores close to the backbone while adding segmentation, avoiding the dialogue collapse seen in embedding-prediction training.
  • The quality of the Phase 1 target specification gates Phase 2 accuracy: providing the target box raises RefCOCO-family average cIoU from 77.4 to 89.1.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The explicit, parseable Phase 1 list turns the mask head into a per-sample open lexicon; the same binding could extend to panoptic video or 3D scenes without altering Phase 2, a direction the paper shows only as a qualitative panoptic example.
  • The 77.4-to-89.1 oracle-box jump implies a cheap testable upgrade: couple Phase 1 generation with a detector or reinforcement-learning-trained box proposer and measure whether the cIoU gain transfers to other benchmarks.
  • The fixed 200-class head capacity is an unexamined ceiling; scenes with more than about 200 targets would need hierarchical IDs or repeated structured lists, and no experiment probes that boundary.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper extends the authors' prior STAMP (binary all-mask prediction) to STAMPlus, which performs structured all-mask prediction for multi-target segmentation in MLLMs. STAMPlus lets Phase 1 autoregressively generate a JSON-style target list with explicit IDs and optional bounding boxes before the <SEG> trigger; Phase 2 then prefills image-aligned [MASK] tokens and, in a single non-autoregressive forward pass, classifies each token against a fixed-capacity multi-class head whose active IDs are dynamically bound to the Phase 1 list. A single unified checkpoint is evaluated across referring segmentation (RefCOCO family, gRefCOCO, ReasonSeg), remote-sensing small-target segmentation (RRSIS-D, EarthReason), open-vocabulary semantic segmentation (ADE20K, Pascal Context, Pascal VOC), instance-aware multi-target segmentation (MUSE), general multimodal instruction following, and latency scaling. The central claim is that STAMPlus resolves the accuracy–dialogue-compatibility–efficiency trilemma beyond single-target prediction, achieving state-of-the-art segmentation in these broader settings while retaining general dialogue ability and reducing 12-category latency from 13.50 s (repeated STAMP) to 5.16 s.

Significance. If the central claims hold, STAMPlus would be a valuable general interface for MLLM-based dense prediction: it extends a non-autoregressive, token-only-supervised mask prediction paradigm from binary targets to multi-category and multi-instance settings without task-specific fine-tuning. The paper has notable strengths: a single unified checkpoint is evaluated across many benchmarks; the resolution-scaling study in Table XI is a controlled comparison; the multi-target latency scaling in Fig. 9 is reported with mean and standard deviation; and the diagnostic experiments in Table VIII explicitly probe the interaction between Phase 1 and Phase 2. These controlled experiments and the breadth of evaluation are genuinely useful. The main uncertainty is whether the demonstrated multi-target capability is attributable to the structured binding mechanism or to the autoregressive Phase 1 list generator, because the paper's own Table VIII(a) shows a very large sensitivity to Phase 1 target-cue quality. This issue is load-bearing for the 'state-of-the-art across these settings' claim and must be addressed before the central claim can be accepted.

major comments (3)
  1. [Sec. IV-D1, Table VIII(a), Sec. III-B2] The paper's own analysis shows that the structured binding mechanism is sharply sensitive to the quality of the Phase 1 target list: replacing auto-generated target cues with human-provided boxes raises RefCOCO-family cIoU from 77.4 to 89.1, an 11.7-point gain that is larger than the reported margins over prior methods. Since Phase 2's active ID set K is defined entirely by the Phase 1 list (Sec. III-B2), a missing, duplicated, or misordered entry in Phase 1 is unrecoverable in Phase 2. The multi-target SOTA claims on ADE20K, MUSE, RRSIS-D, and EarthReason are reported only end-to-end, and no Phase 1 parse-success rate, list-completeness metric, or conditional result with an oracle/verified Phase 1 list is provided. The text acknowledges that 'the quality of the Phase 1 target description has a substantial impact on Phase 2 segmentation' and proposes future RL/tool assistance, but this admission is not carried into the SOTA claim as a limitation. The paper should report Phase 1 list-quality statistics and, where feasible, results conditioned on a human-verified or oracle Phase 1 list, so that the contribution of the structured binding mechanism can be separated from the autoregressive list generator. Without this, the central claim that the trilemma is resolved in broader settings is not fully supported.
  2. [Abstract and Table VII(b)] The abstract claims that 'Across these settings, STAMPlus achieves state-of-the-art segmentation performance,' but the MUSE results do not support this on the reported average metric: Text4Seg++ averages 63.8 while STAMPlus-7B averages 63.5 in Table VII(b). The body text appropriately says STAMPlus is 'competitive with Text4Seg++ overall' and exceeds it on the test split, but the abstract and contributions overstate the result. This is a claim-safety issue: either the abstract should be qualified, or the MUSE comparison should be presented with a metric or analysis that justifies the SOTA statement (for example, a significance test or a clear primary-metric argument).
  3. [Tables II–VII] All accuracy numbers across the main comparison tables are reported without error bars, confidence intervals, or multiple-seed standard deviations; only the latency experiment in Fig. 9 includes mean and standard deviation. This is particularly important for the SOTA claims where margins are small (e.g., 63.5 vs. 63.8 on MUSE in Table VII(b), and 76.2 vs. 74.4 on RRSIS-D in Table V). Without any variance information, it is difficult for a reader to determine whether the reported differences are meaningful. The authors should report at least two or three seeds or, alternatively, explicitly state that results are single-run and avoid strong comparative wording where margins are within likely noise.
minor comments (5)
  1. [Table IV] In the READ row, '59.867.6' appears to be a typographical error for '59.8 67.6'; please fix the missing spacing.
  2. [Sec. III-B3] The high-resolution scaling description could be more precise about how the mask-token budget N is selected for different input resolutions and how the 1024–1280 and 2560–3200 ranges are determined; the controlled study in Table XI would be easier to interpret if the exact N values per benchmark were listed.
  3. [Sec. IV-D2, Table VIII(b)] The look-twice experiment uses attention, not predicted masks, to derive grounding regions; this is stated in the text, but the conclusion that 'segmentation-learned spatial grounding' supports downstream understanding is indirect. A one-sentence clarification that the mechanism tested is segmentation-induced attention, not mask-feature reuse, would reduce the risk of overreading.
  4. [Introduction, Codebase] The paper states that 'The complete codebase ... is included in the supplementary material' but does not provide a public repository URL or commit hash. If this is a journal submission, please provide a link or state clearly in the final version how reviewers and readers can obtain the code.
  5. [Fig. 8] The caption says 'marker size indicates model scale,' but the marker sizes are not otherwise defined; please add an explicit legend or note the correspondence between marker sizes and model scales (2B/7B/13B).

Circularity Check

0 steps flagged · score 2.0 of 10

No circularity; self-citations are not load-bearing and central claims are validated against external benchmarks.

full rationale

The paper's central claims are validated against external benchmarks with published baselines (Tables II-VII), using a single unified checkpoint without benchmark-specific fine-tuning; no parameter is fitted to a target metric and then renamed as a prediction. The only self-citation is to the authors' prior STAMP paper [1], which is the explicitly acknowledged foundation being extended, not an unverified premise used to force the STAMPlus conclusions; STAMPlus's structured Phase 1 list and dynamic ID binding are architectural mechanisms whose components are ablated (Table X) and whose Phase 1 bottleneck is openly measured with human-provided boxes (Table VIII(a)), i.e., an honest diagnostic rather than a hidden fit. The latency reduction (13.50s to 5.16s) is a direct measurement under fixed hardware and target counts. No equation or definition reduces a claimed prediction to its input: the loss (Eqs. 1-8) supervises text and masks from ground truth, and Phase 2's active ID set K is defined by the generated list, but this is the designed interface, not a circular derivation. The paper's own Table VIII(a) identifies a real limitation (Phase 1 list quality), but reporting that limitation as an analysis does not make the SOTA claim circular. Score 2 reflects only the presence of non-load-bearing self-citations.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The ledger is light: the paper introduces no new physical entities. The main assumptions are that the pretrained Qwen2-VL backbone remains intact under mask-token training, that the structured list conditions mask prediction through cached context, and that standard datasets and metrics are trustworthy. Free parameters are architectural choices: the 200-class capacity, the mask-token budgets, and input resolutions.

free parameters (3)
  • Multi-class head capacity C = 200
    Phase 2 predicts over a fixed 200-channel class space; samples with more than 199 target entries are excluded or truncated during preprocessing. This is a hard capacity limit on the structured mask space.
  • Mask-token budget N = 1024-1280 natural images; 2560-3200 remote sensing
    Number of image-aligned [MASK] placeholders is a hand-chosen trade-off between resolution and compute, and is jointly scaled with input resolution for small-target scenes.
  • Input resolution = 896x896 default; 726x726 and 504x504 speed variants
    Resolution is a selected hyperparameter with direct latency and accuracy impact, evaluated in the resolution study in Table X.
assumptions (3)
  • domain assumption Qwen2-VL backbone provides sufficient visual and dialogue priors and is not destroyed by adding mask-token objectives.
    All STAMP and STAMPlus variants build on Qwen2-VL; dialogue preservation claims are conditional on the backbone retaining its abilities under fine-tuning.
  • domain assumption Phase 2 mask tokens can condition on the Phase 1 structured target list via cached KV states and hybrid attention.
    This is the mechanism that binds IDs to classes, described in Section III-B2. It is validated empirically but not proven formally.
  • domain assumption Evaluation datasets, metrics, and published baseline numbers are correct and comparable.
    All state-of-the-art comparisons rely on external benchmark protocols and previously reported numbers, which are taken at face value.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Better, Stronger, Faster, and Broader: Structured All-Mask Prediction for MLLM-Based Segmentation." pith.science (2026). https://pith.science/paper/GWHYNDZP

@misc{pith2026260802791,
  author       = {Pith},
  title        = {Pith review of: Better, Stronger, Faster, and Broader: Structured All-Mask Prediction for MLLM-Based Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GWHYNDZP}},
  note         = {Machine review of arXiv:2608.02791}
}
read the original abstract

MLLM-based segmentation faces a core segmentation trilemma: high segmentation performance, preserved dialogue ability, and fast inference. Embedding-prediction methods may disrupt language modeling through pixel-level objectives, whereas next-token generation is inefficient for dense masks. We propose All-Mask Prediction, decoupling autoregressive dialogue from non-autoregressive mask prediction. Its binary instantiation, STAMP (Simultaneous Textual All-Mask Prediction), emits an in-vocabulary <SEG> trigger, fuses image-aligned mask tokens with corresponding patch features, and uses hybrid attention to classify all tokens as foreground or background in one pass. It thereby combines strong referring and reasoning segmentation with preserved multimodal ability and efficient inference. However, binary masks cannot retain multiple semantic or instance identities without repeated target-specific predictions. We therefore propose Structured All-Mask Prediction and develop STAMPlus. It generates a target list with explicit IDs and optional boxes, binds these IDs to a shared multi-class mask space, and jointly predicts all targets in one non-autoregressive pass. A single unified checkpoint retains STAMP's referring and reasoning capabilities while extending to open-vocabulary semantic, instance-aware, and remote-sensing small-target segmentation, where high-resolution mask-token scaling preserves finer spatial evidence. Across these settings, STAMPlus achieves state-of-the-art segmentation performance, preserves general multimodal instruction following, and reduces 12-category latency from 13.50s for repeated STAMP inference to 5.16s. Further analyses show that accurate target cues improve segmentation and learned spatial grounding benefits look-twice reasoning. Overall, STAMPlus resolves the trilemma beyond single-target prediction.

Figures

Figures reproduced from arXiv: 2608.02791 by the authors.

Figure 1
Figure 1. The trilemma of segmentation in MLLMs. Embedding prediction may harm dialogue abilities. Next-token prediction methods are either fast with poor segmentation performance or achieve superior performance at the cost of inference speed, particularly when generating rich content (e.g., chain￾of-thought or patch-wise classification). TABLE I COMPARISON OF MLLM SEGMENTATION PARADIGMS. “MASK STEPS” COUNTS THE MASK-GENERATI… view at source ↗
Figure 2
Figure 2. Comparison of MLLM-based segmentation paradigms. (a) Embedding Prediction: Pixel-level supervision for external mask decoding [12, 14] may degrade the MLLM’s general dialogue capabilities. (b) Next-Token Prediction: Autoregressively generates textual mask representations [11, 18, 23], forcing a trade-off between poor segmentation performance (for sparse outputs) and slow inference (for rich outputs). (c) Our All-Mas… view at source ↗
Figure 3
Figure 3. STAMP versus STAMPlus in multi-target segmentation. (a) STAMP must repeat its single-target prediction for each object, and ambiguous referring expressions may yield incorrect masks. (b) STAMPlus first generates a structured target list and then predicts all target IDs jointly in one shared structured mask map. II. RELATED WORK A. Multimodal Large Language Models (MLLMs) The advent of MLLMs, powered by the advanced … view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: The STAMP Pipeline. Phase 1 (Dialogue Generation): The MLLM autoregressively generates a conversational response, emitting a special <SEG> token to trigger Phase 2. Phase 2 (Binary All-Mask Prediction): Triggered by <SEG>, image-aligned [MASK] tokens are prefilled and …
Figure 5
Figure 5. Figure 5: The STAMPlus Pipeline. Phase 1 (Structured Target Generation): The MLLM generates a target list at the single-target, multi-category, or multi-instance granularity requested by the instruction; each entry receives an ID and may include a bounding box for instance disam…
Figure 6
Figure 6. Figure 6: Qualitative comparison on referring segmentation. Each column corresponds to one query; from top to bottom, the rows show the input image, ground truth, and predictions from STAMP, Text4Seg, and LISA. Prompt “The most suitable facility for a backhand stroke” Prompt “A …
Figure 7
Figure 7. Figure 7: Qualitative showcase of the capabilities added by STAMPlus beyond STAMP. STAMPlus retains the established single-target referring and reasoning segmentation capabilities of STAMP. Because [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 9
Figure 9. Figure 9: Inference-time scaling with the number of target categories. STAMP and LISA repeat a separate prediction of similar cost for each cate￾gory, resulting in a steep latency increase. The additional cost of STAMPlus is concentrated in the Phase 1 autoregressive generation …

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

76 extracted references · 53 canonical work pages

  1. [15]

    MMR: A large-scale benchmark dataset for multi-target and multi- granularity reasoning segmentation,

    D. Jang, Y . Cho, S. Lee, T. Kim, and D. Kim, “MMR: A large-scale benchmark dataset for multi-target and multi- granularity reasoning segmentation,” inICLR, 2025

    work page 2025

  2. [1]

    Better, stronger, faster: Tackling the trilemma in mllm-based segmentation with simultaneous textual mask prediction,

    J. Liu, M. Feng, and L. Chen, “Better, stronger, faster: Tackling the trilemma in mllm-based segmentation with simultaneous textual mask prediction,” inCVPR, 2026

    work page 2026

  3. [2]

    Qwen3 technical report,

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lvet al., “Qwen3 technical report,”arXiv preprint arXiv:2505.09388, 2025

    arXiv 2025

  4. [3]

    LION: Empowering multimodal large language model with dual-level visual knowledge,

    G. Chen, L. Shen, R. Shao, X. Deng, and L. Nie, “LION: Empowering multimodal large language model with dual-level visual knowledge,” inCVPR, 2024

    work page 2024

  5. [4]

    MLLMs know where to look: Training-free perception of small visual details with multimodal llms,

    J. Zhang, M. Khayatkhoei, P. Chhikara, and F. Ilievski, “MLLMs know where to look: Training-free perception of small visual details with multimodal llms,” inICLR, 2025

    work page 2025

  6. [5]

    Detect anything via next point prediction,

    Q. Jiang, J. Huo, X. Chen, Y . Xiong, Z. Zeng, Y . Chen, T. Ren, J. Yu, and L. Zhang, “Detect anything via next point prediction,”arXiv preprint arXiv:2510.12798, 2025

    arXiv 2025

  7. [6]

    InternVL3: Exploring advanced training and test-time recipes for open-source multimodal models,

    J. Zhu, W. Wang, Z. Chen, Z. Liu, S. Ye, L. Gu, H. Tian, Y . Duan, W. Su, J. Shaoet al., “InternVL3: Exploring advanced training and test-time recipes for open-source multimodal models,”arXiv preprint arXiv:2504.10479, 2025

    arXiv 2025

  8. [7]

    LLaV A- OneVision: Easy visual task transfer,

    B. Li, Y . Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang, Y . Li, Z. Liu, and C. Li, “LLaV A- OneVision: Easy visual task transfer,”arXiv preprint arXiv:2408.03326, 2024

    arXiv 2024

Show all 76 references
  1. [8]

    PhD: A ChatGPT-prompted visual halluci- nation evaluation dataset,

    J. Liu, Y . Fu, R. Xie, R. Xie, X. Sun, F. Lian, Z. Kang, and X. Li, “PhD: A ChatGPT-prompted visual halluci- nation evaluation dataset,” inCVPR, 2025

    2025

  2. [9]

    Qwen2.5- VL technical report,

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tanget al., “Qwen2.5- VL technical report,”arXiv preprint arXiv:2502.13923, 2025

    2025 arXiv

  3. [10]

    Segmentation as a plug-and-play capability for frozen multimodal LLMs,

    J. Liu and L. Chen, “Segmentation as a plug-and-play capability for frozen multimodal LLMs,”arXiv preprint arXiv:2510.16785, 2025

    2025

  4. [11]

    Text4Seg: Reimagining image segmentation as text generation,

    M. Lan, C. Chen, Y . Zhou, J. Xu, Y . Ke, X. Wang, L. Feng, and W. Zhang, “Text4Seg: Reimagining image segmentation as text generation,” inICLR, 2025

    2025

  5. [12]

    LISA: Reasoning segmentation via large language model,

    X. Lai, Z. Tian, Y . Chen, Y . Li, Y . Yuan, S. Liu, and J. Jia, “LISA: Reasoning segmentation via large language model,” inCVPR, 2024

    2024

  6. [13]

    GSV A: Generalized segmentation via multimodal large language models,

    Z. Xia, D. Han, Y . Han, X. Pan, S. Song, and G. Huang, “GSV A: Generalized segmentation via multimodal large language models,” inCVPR, 2024

    2024

  7. [14]

    PixelLM: Pixel reasoning with large multimodal model,

    Z. Ren, Z. Huang, Y . Wei, Y . Zhao, D. Fu, J. Feng, and X. Jin, “PixelLM: Pixel reasoning with large multimodal model,” inCVPR, 2024

    2024

  8. [16]

    Reasoning to attend: Try to understand how<SEG>token works,

    R. Qian, X. Yin, and D. Dou, “Reasoning to attend: Try to understand how<SEG>token works,” inCVPR, 2025

    2025

  9. [17]

    VisionLLM v2: An end-to-end generalist multimodal large language SUBMISSION 17 model for hundreds of vision-language tasks,

    J. Wu, M. Zhong, S. Xing, Z. Lai, Z. Liu, Z. Chen, W. Wang, X. Zhu, L. Lu, T. Luet al., “VisionLLM v2: An end-to-end generalist multimodal large language SUBMISSION 17 model for hundreds of vision-language tasks,”Advances in Neural Information Processing Systems, vol. 37, pp. ...

    2024

  10. [18]

    Seg-Zero: Reasoning-chain guided seg- mentation via cognitive reinforcement,

    Y . Liu, B. Peng, Z. Zhong, Z. Yue, F. Lu, B. Yu, and J. Jia, “Seg-Zero: Reasoning-chain guided seg- mentation via cognitive reinforcement,”arXiv preprint arXiv:2503.06520, 2025

    2025 arXiv

  11. [19]

    SegAgent: Exploring pixel understanding capabilities in mllms by imitating human annotator trajectories,

    M. Zhu, Y . Tian, H. Chen, C. Zhou, Q. Guo, Y . Liu, M. Yang, and C. Shen, “SegAgent: Exploring pixel understanding capabilities in mllms by imitating human annotator trajectories,” inCVPR, 2025

    2025

  12. [20]

    Text4Seg++: Advancing image segmentation via generative language modeling,

    M. Lan, C. Chen, J. Xu, Z. Li, Y . Ke, X. Jiang, Y . Yu, Y . Zhao, and S. Bai, “ Text4Seg++: Advancing image segmentation via generative language modeling,”IEEE Transactions on Pattern Analysis and Machine Intelli- gence, vol. 48, no. 08, pp. 9486–9501, 2026

    2026

  13. [21]

    GLaMM: Pixel grounding large multimodal model,

    H. Rasheed, M. Maaz, S. Shaji, A. Shaker, S. Khan, H. Cholakkal, R. M. Anwer, E. Xing, M.-H. Yang, and F. S. Khan, “GLaMM: Pixel grounding large multimodal model,” inCVPR, 2024

    2024

  14. [22]

    See say and segment: Teaching LMMs to overcome false premises,

    T.-H. Wu, G. Biamby, D. Chan, L. Dunlap, R. Gupta, X. Wang, J. E. Gonzalez, and T. Darrell, “See say and segment: Teaching LMMs to overcome false premises,” inCVPR, 2024

    2024

  15. [23]

    VisionLLM: Large language model is also an open-ended decoder for vision-centric tasks,

    W. Wang, Z. Chen, X. Chen, J. Wu, X. Zhu, G. Zeng, P. Luo, T. Lu, J. Zhou, Y . Qiaoet al., “VisionLLM: Large language model is also an open-ended decoder for vision-centric tasks,”Advances in Neural Information Processing Systems, vol. 36, pp. 61 501–61 513, 2023

    2023

  16. [24]

    ReferItGame: Referring to objects in photographs of natural scenes,

    S. Kazemzadeh, V . Ordonez, M. Matten, and T. Berg, “ReferItGame: Referring to objects in photographs of natural scenes,” inEMNLP, 2014

    2014

  17. [25]

    Generation and comprehension of un- ambiguous object descriptions,

    J. Mao, J. Huang, A. Toshev, O. Camburu, A. L. Yuille, and K. Murphy, “Generation and comprehension of un- ambiguous object descriptions,” inCVPR, 2016

    2016

  18. [26]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,

    P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Geet al., “Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,”arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  19. [27]

    Scaling laws for neural language models,

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,” arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  20. [28]

    Hello GPT-4o,

    OpenAI, “Hello GPT-4o,” 2024, https://openai.com/ index/hello-gpt-4o/. [Online]. Available: https://openai. com/index/hello-gpt-4o/

    2024

  21. [29]

    Google Gemini 2.5 Pro,

    Google, “Google Gemini 2.5 Pro,” 2025, https: //deepmind.google/technologies/gemini/pro/. [Online]. Available: https://deepmind.google/technologies/gemini/ pro/

    2025

  22. [30]

    Empowering small VLMs to think with dynamic memorization and exploration,

    J. Liu, Y . Deng, and L. Chen, “Empowering small VLMs to think with dynamic memorization and exploration,” in ICLR, 2026

    2026

  23. [31]

    Flamingo: a visual language model for few-shot learning,

    J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynoldset al., “Flamingo: a visual language model for few-shot learning,”Advances in neural information processing systems, vol. 35, pp. 23 716–23 736, 2022

    2022

  24. [32]

    InstructBLIP: Towards general- purpose vision-language models with instruction tuning,

    W. Dai, J. Li, D. Li, A. Tiong, J. Zhao, W. Wang, B. Li, P. N. Fung, and S. Hoi, “InstructBLIP: Towards general- purpose vision-language models with instruction tuning,” inNIPS, 2023

    2023

  25. [33]

    Visual instruc- tion tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruc- tion tuning,”Advances in neural information processing systems, vol. 36, pp. 34 892–34 916, 2023

    2023

  26. [34]

    Improved baselines with visual instruction tuning,

    H. Liu, C. Li, Y . Li, and Y . J. Lee, “Improved baselines with visual instruction tuning,” inCVPR, 2024

    2024

  27. [35]

    Qwen-VL: A versatile vision- language model for understanding, localization, text reading, and beyond,

    J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou, “Qwen-VL: A versatile vision- language model for understanding, localization, text reading, and beyond,”arXiv preprint arXiv:2308.12966, 2023

    2023 arXiv

  28. [36]

    Qwen3- VL technical report,

    S. Bai, Y . Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Geet al., “Qwen3- VL technical report,”arXiv preprint arXiv:2511.21631, 2025

    2025 arXiv

  29. [37]

    Latent visual reason- ing,

    B. Li, X. Sun, J. Liu, Z. Wang, J. Wu, X. Yu, H. Chen, E. Barsoum, M. Chen, and Z. Liu, “Latent visual reason- ing,”arXiv preprint arXiv:2509.24251, 2025

    2025 arXiv

  30. [38]

    Segment anything,

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Loet al., “Segment anything,” inICCV, 2023

    2023

  31. [39]

    SegLLM: Multi- round reasoning segmentation with large language mod- els,

    X. Wang, S. Zhang, S. Li, K. Li, K. Kallidromitis, Y . Kato, K. Kozuka, and T. Darrell, “SegLLM: Multi- round reasoning segmentation with large language mod- els,” inICLR, 2025

    2025

  32. [40]

    MLLM can see? dynamic correction decoding for hallucination mitigation,

    C. Wang, X. Chen, N. Zhang, B. Tian, H. Xu, S. Deng, and H. Chen, “MLLM can see? dynamic correction decoding for hallucination mitigation,” inICLR, 2025

    2025

  33. [41]

    Grounding multimodal large lan- guage models to the world,

    Z. Peng, W. Wang, L. Dong, Y . Hao, S. Huang, S. Ma, Q. Ye, and F. Wei, “Grounding multimodal large lan- guage models to the world,” inICLR, 2024

    2024

  34. [42]

    Shikra: Unleashing multimodal LLM’s refer- ential dialogue magic,

    K. Chen, Z. Zhang, W. Zeng, R. Zhang, F. Zhu, and R. Zhao, “Shikra: Unleashing multimodal LLM’s refer- ential dialogue magic,”arXiv preprint arXiv:2306.15195, 2023

    2023 arXiv

  35. [43]

    Perception tokens en- hance visual reasoning in multimodal language models,

    M. Bigverdi, Z. Luo, C.-Y . Hsieh, E. Shen, D. Chen, L. G. Shapiro, and R. Krishna, “Perception tokens en- hance visual reasoning in multimodal language models,” inCVPR, 2025

    2025

  36. [44]

    ViperGPT: Visual inference via python execution for reasoning,

    D. Sur ´ıs, S. Menon, and C. V ondrick, “ViperGPT: Visual inference via python execution for reasoning,” inICCV, 2023

    2023

  37. [45]

    Chameleon: Plug- and-play compositional reasoning with large language models,

    P. Lu, B. Peng, H. Cheng, M. Galley, K.-W. Chang, Y . N. Wu, S.-C. Zhu, and J. Gao, “Chameleon: Plug- and-play compositional reasoning with large language models,”Advances in Neural Information Processing Systems, vol. 36, pp. 43 447–43 478, 2023

    2023

  38. [46]

    MM-REACT: Prompting ChatGPT for multimodal reasoning and ac- tion,

    Z. Yang, L. Li, J. Wang, K. Lin, E. Azarnasab, F. Ahmed, Z. Liu, C. Liu, M. Zeng, and L. Wang, “MM-REACT: Prompting ChatGPT for multimodal reasoning and ac- tion,”arXiv preprint arXiv:2303.11381, 2023

    2023 arXiv

  39. [47]

    Visual sketchpad: Sketching as a visual chain of thought for multimodal language models,

    Y . Hu, W. Shi, X. Fu, D. Roth, M. Ostendorf, L. Zettle- moyer, N. A. Smith, and R. Krishna, “Visual sketchpad: Sketching as a visual chain of thought for multimodal language models,”Advances in Neural Information Pro- SUBMISSION 18 cessing Systems, vol. 37, pp. 139 348–139 379, 2024

    2024

  40. [48]

    Machine mental imagery: Empower multimodal reasoning with latent visual tokens,

    Z. Yang, X. Yu, D. Chen, M. Shen, and C. Gan, “Machine mental imagery: Empower multimodal reasoning with latent visual tokens,”arXiv preprint arXiv:2506.17218, 2025

    2025 arXiv

  41. [49]

    Multimodal chain of continu- ous thought for latent-space reasoning in vision-language models,

    T.-H. Pham and C. Ngo, “Multimodal chain of continu- ous thought for latent-space reasoning in vision-language models,”arXiv preprint arXiv:2508.12587, 2025

    2025

  42. [50]

    GRES: Generalized referring expression segmentation,

    C. Liu, H. Ding, and X. Jiang, “GRES: Generalized referring expression segmentation,” inCVPR, 2023

    2023

  43. [51]

    Rotated multi-scale interaction network for referring remote sensing image segmentation,

    S. Liu, Y . Ma, X. Zhang, H. Wang, J. Ji, X. Sun, and R. Ji, “Rotated multi-scale interaction network for referring remote sensing image segmentation,” inCVPR, 2024

    2024

  44. [52]

    SegEarth-R1: Geospatial pixel reasoning via large language model,

    K. Li, Z. Xin, L. Pang, C. Pang, Y . Deng, J. Yao, G. Xia, D. Meng, Z. Wang, and X. Cao, “SegEarth-R1: Geospatial pixel reasoning via large language model,” arXiv preprint arXiv:2504.09644, 2025

    2025 arXiv

  45. [53]

    COCO-Stuff: Thing and stuff classes in context,

    H. Caesar, J. Uijlings, and V . Ferrari, “COCO-Stuff: Thing and stuff classes in context,” inCVPR, 2018

    2018

  46. [54]

    Microsoft COCO: Common objects in context,

    T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll ´ar, and C. L. Zitnick, “Microsoft COCO: Common objects in context,” inECCV, 2014

    2014

  47. [55]

    Universal instance perception as object discovery and retrieval,

    B. Yan, Y . Jiang, J. Wu, D. Wang, P. Luo, Z. Yuan, and H. Lu, “Universal instance perception as object discovery and retrieval,” inCVPR, 2023

    2023

  48. [56]

    PolyFormer: Referring im- age segmentation as sequential polygon generation,

    J. Liu, H. Ding, Z. Cai, Y . Zhang, R. K. Satzoda, V . Ma- hadevan, and R. Manmatha, “PolyFormer: Referring im- age segmentation as sequential polygon generation,” in CVPR, 2023

    2023

  49. [57]

    Language-aware vision transformer for referring segmentation,

    Z. Yang, J. Wang, X. Ye, Y . Tang, K. Chen, H. Zhao, and P. H. S. Torr, “Language-aware vision transformer for referring segmentation,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 47, no. 7, pp. 5238–5255, 2025

    2025

  50. [58]

    Open-vocabulary semantic segmentation with mask-adapted clip,

    F. Liang, B. Wu, X. Dai, K. Li, Y . Zhao, H. Zhang, P. Zhang, P. Vajda, and D. Marculescu, “Open-vocabulary semantic segmentation with mask-adapted clip,” in CVPR, 2023

    2023

  51. [59]

    NExT- Chat: An LMM for chat, detection and segmentation,

    A. Zhang, Y . Yao, W. Ji, Z. Liu, and T.-S. Chua, “NExT- Chat: An LMM for chat, detection and segmentation,” in ICML, 2024

    2024

  52. [60]

    GeoGround: A unified large vision-language model for remote sensing visual grounding,

    Y . Zhou, M. Lan, X. Li, L. Feng, Y . Ke, X. Jiang, Q. Li, X. Yang, and W. Zhang, “GeoGround: A unified large vision-language model for remote sensing visual grounding,”arXiv preprint arXiv:2411.11904, 2024

    2024 arXiv

  53. [61]

    Semantic understanding of scenes through the ADE20K dataset,

    B. Zhou, H. Zhao, X. Puig, T. Xiao, S. Fidler, A. Bar- riuso, and A. Torralba, “Semantic understanding of scenes through the ADE20K dataset,”International Jour- nal of Computer Vision, vol. 127, no. 3, pp. 302–321, 2019

    2019

  54. [62]

    The role of context for object detection and semantic segmentation in the wild,

    R. Mottaghi, X. Chen, X. Liu, N.-G. Cho, S.-W. Lee, S. Fidler, R. Urtasun, and A. Yuille, “The role of context for object detection and semantic segmentation in the wild,” inCVPR, 2014

    2014

  55. [63]

    The PASCAL visual object classes (VOC) challenge,

    M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman, “The PASCAL visual object classes (VOC) challenge,”International Journal of Computer Vision, vol. 88, no. 2, pp. 303–338, 2010

    2010

  56. [64]

    ClearCLIP: Decomposing clip representa- tions for dense vision-language inference,

    M. Lan, C. Chen, Y . Ke, X. Wang, L. Feng, and W. Zhang, “ClearCLIP: Decomposing clip representa- tions for dense vision-language inference,” inECCV. Springer, 2024

    2024

  57. [65]

    ProxyCLIP: Proxy attention improves clip for open-vocabulary segmentation,

    ——, “ProxyCLIP: Proxy attention improves clip for open-vocabulary segmentation,” inECCV, 2024

    2024

  58. [66]

    Open-vocabulary universal image segmentation with maskclip,

    Z. Ding, J. Wang, and Z. Tu, “Open-vocabulary universal image segmentation with maskclip,” inICML, 2023

    2023

  59. [67]

    GroupViT: Semantic segmenta- tion emerges from text supervision,

    J. Xu, S. De Mello, S. Liu, W. Byeon, T. Breuel, J. Kautz, and X. Wang, “GroupViT: Semantic segmenta- tion emerges from text supervision,” inCVPR, 2022

    2022

  60. [68]

    SAN: Side adapter network for open-vocabulary semantic seg- mentation,

    M. Xu, Z. Zhang, F. Wei, H. Hu, and X. Bai, “SAN: Side adapter network for open-vocabulary semantic seg- mentation,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 12, pp. 15 546–15 561, 2023

    2023

  61. [69]

    LaSagnA: Language-based segmentation assistant for complex queries,

    C. Wei, H. Tan, Y . Zhong, Y . Yang, and L. Ma, “LaSagnA: Language-based segmentation assistant for complex queries,”arXiv preprint arXiv:2404.08506, 2024

    2024 arXiv

  62. [70]

    POPEN: Preference-based optimization and ensemble for LVLM-based reasoning segmentation,

    L. Zhu, T. Chen, Q. Xu, X. Liu, D. Ji, H. Wu, D. W. Soh, and J. Liu, “POPEN: Preference-based optimization and ensemble for LVLM-based reasoning segmentation,” in CVPR, 2025

    2025

  63. [71]

    MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi,

    X. Yue, Y . Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y . Sunet al., “MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi,” inCVPR, 2024

    2024

  64. [72]

    MMBench: Is your multi-modal model an all-around player?

    Y . Liu, H. Duan, Y . Zhang, B. Li, S. Zhang, W. Zhao, Y . Yuan, J. Wang, C. He, Z. Liuet al., “MMBench: Is your multi-modal model an all-around player?” inECCV, 2024

    2024

  65. [73]

    Are we on the right way for evaluating large vision-language models?

    L. Chen, J. Li, X. Dong, P. Zhang, Y . Zang, Z. Chen, H. Duan, J. Wang, Y . Qiao, D. Linet al., “Are we on the right way for evaluating large vision-language models?”Advances in Neural Information Processing Systems, vol. 37, pp. 27 056–27 087, 2024

    2024

  66. [74]

    Learn to explain: Multimodal reasoning via thought chains for science question answering,

    P. Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S.-C. Zhu, O. Tafjord, P. Clark, and A. Kalyan, “Learn to explain: Multimodal reasoning via thought chains for science question answering,” inNeurIPS, 2022

    2022

  67. [75]

    Towards VQA models that can read,

    A. Singh, V . Natarajan, M. Shah, Y . Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach, “Towards VQA models that can read,” inCVPR, 2019

    2019

  68. [76]

    VizWiz grand challenge: Answering visual questions from blind people,

    D. Gurari, Q. Li, A. J. Stangl, A. Guo, C. Lin, K. Grau- man, J. Luo, and J. P. Bigham, “VizWiz grand challenge: Answering visual questions from blind people,” inCVPR, 2018

    2018

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.