Credit the Right Box: Marginal Contribution Assignment for Structured Visual Perception

Feng Pan; Jianfei Wang; Shuai Li; Weixing Li; Xiang Wang; Xinheng Han; Yu Chen

REVIEW 3 major objections 3 minor 85 references

Credit the Right Box: Marginal Contribution Assignment for Structured Visual Perception

T0 review · 3 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that reinforcement learning for multimodal large language models should reward individual predicted boxes, not the whole answer, and that a leave-one-out marginal-contribution signal computed from a count-aware set score d

desk verdict Box-level leave-one-out credit for GRPO is a real and useful idea, and the formal analysis holds up, but the hand-tuned set evaluator is the load-bearing assumption and never gets a sensitivity check. read the letter →

arxiv 2608.01055 v1 pith:3MUCY7KZ submitted 2026-08-02 cs.CV cs.AI

Xinheng Han , Jianfei Wang , Yu Chen , Xiang Wang , Shuai Li , Weixing Li , Feng Pan This is my paper

classification cs.CVcs.AI

keywords reinforcementlearninggrouprelativepolicyoptimizationcreditassignmentstructuredvisualperceptionmultimodallargelanguagemodelsreferringexpressioncomprehensionimagesegmentationobjectcounting

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Structured perception tasks such as grounding, detection, segmentation, and counting demand that a model output a set of object records, yet standard GRPO reinforcement learning broadcasts a single response-level advantage to every token in the answer. This paper argues that such coarse credit cannot tell a correct box from a duplicate or a near-miss, and introduces MCR-GRPO, which derives box-level credit from each sampled response by leave-one-out evaluation: it removes each predicted box and measures how a continuous matched set value changes. After within-response normalization, helpful boxes receive positive credit, redundant or harmful boxes receive negative credit, and the credit is mapped onto the token spans that generated each box. The set value itself is a continuous, Hungarian-matched, count-aware evaluator, and the paper shows this continuity is necessary for stable attribution; without it, box-level credit alone degrades counting. If the claim holds, finer-grained reinforcement learning for structured visual perception is achievable at no extra sampling cost, and the method reports consistent gains across REC, DOD, segmentation, and counting benchmarks while preserving VQA ability.

What carries the argument

The load-bearing mechanism is the combination of a Continuous Matched Set Value Evaluator $V(S,G)$ with the Marginal Contribution Reward (MCR). $V$ is a permutation-invariant, count-aware, graded set score: pair scores combine IoU, a truncated exponential box-distance term with a 40-pixel cutoff, and a point-in-mask score; optimal one-to-one matching is solved by the Hungarian algorithm; and the matched sum is divided by $\max(K,N)$ so extra and missing records are penalized. MCR then computes each box's leave-one-out value difference $\Delta V_i = V(S,G) - V(S \setminus \{d_i\}, G)$, z-scores these differences within each response, and maps the normalized credits to the token spans that gen

What would settle it

Train MCR-GRPO with a variant evaluator that keeps the same pair scores but divides by $N$ instead of $\max(K,N)$, removing the count penalty; if counting accuracy does not fall back toward the baseline, the count-aware denominator is not the mechanism suppressing duplicates. Conversely, re-run the main benchmarks over a small grid of evaluator weights and the 40-pixel cutoff: if moderate reweighting (e.g., $\alpha=1$, $\beta=2$, $\gamma=1$) erases the gains over the response-supervised baseline, the reported improvements are artifacts of evaluator calibration rather than of leave-one-out cred

Watch

Extended reading notes

Core claim

The paper claims that response-level GRPO advantages are too coarse for multi-object structured outputs because they reinforce correct and harmful boxes together. Its central discovery is that a signed, box-specific credit can be derived from each sampled response by leave-one-out evaluation: $\Delta V_i = V(S,G) - V(S \setminus \{d_i\}, G)$, where $V$ is the continuous matched set value, i.e., the optimal one-to-one (Hungarian) matching sum divided by $\max(K,N)$. Within a response these differences are z-scored, making the credit zero-sum: under over-prediction ($K>N$) unmatched boxes provably get negative raw credit, while under $K \le N$ below-mean contributors get negative relative cred

Load-bearing premise

The paper's box-level credits inherit all the biases of its hand-designed continuous evaluator $V(S,G)$ — fixed weights $\alpha=2$, $\beta=\gamma=1$, a 40-pixel distance cutoff, and point scores 1.0/0.3/0 — which is assumed to faithfully rank structured perception quality; if that evaluator is miscalibrated, the leave-one-out signal rewards boxes that inflate $V$ rather than boxes that are truly correct.

Editorial extensions

If this is right

Structured perception models can be trained with box-local credit at no extra sampling cost: no revision passes or auxiliary trajectories are needed, and inference remains single-stage.
Duplicate and false-positive boxes are suppressed by the count-aware denominator $\max(K,N)$ rather than by hand-designed penalties, so over-prediction is corrected inside the response itself.
The continuous evaluator is load-bearing: its graded IoU/box/point scores make leave-one-out differences smooth enough to be informative, while the same MCR on a thresholded evaluator degrades counting.
Because the composite advantage preserves the sign of the response-level advantage whenever $\lambda^{\mathrm{mcr}}\sqrt{K-1} < |A^{\mathrm{seq}}|$, between-response ranking is retained while box-level residuals refine it.
The method generalizes across REC, DOD, segmentation, and counting and preserves or improves six VQA benchmarks, suggesting box-level credit does not trade away general multimodal ability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the same leave-one-out credit scheme could be applied to other structured units with token spans, such as mask prompts, keypoints, or individual reasoning steps, as long as a set-level evaluator exists; the paper's mechanism may outlive its box-and-point instantiation.
Editorial inference: the fixed evaluator weights ($\alpha=2$, $\beta=\gamma=1$, 40-pixel cutoff, point scores 1.0/0.3/0) are a calibration bet; if the evaluator mis-scores near-miss boxes, MCR could amplify boxes that game $V$ rather than boxes that are semantically correct, and the paper does not include a sensitivity analysis.
Editorial inference: since MCR is zero-sum within a response, its value depends on diversity inside the sampled group; training with more rollouts per prompt or more variable box sets could make the leave-one-out comparisons sharper and the credits more informative.
Editorial inference: the box-level credits themselves could serve as a debugging tool; inspecting which boxes receive negative marginal credit after training reveals what the model systematically over-predicts, giving an interpretability lens on RL-trained perception models.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The core idea is worth taking seriously: instead of broadcasting one response-level advantage to every token, MCR-GRPO computes per-box credit by leave-one-out changes in a matched set value, then maps those credits to the token spans that produced each box. That is a genuine step beyond VisionReasoner and GroupRevision, and it preserves single-stage inference, which is practically valuable. The formal appendix is also solid — the regime-dependent sign result and the |A_mcr| <= sqrt(K-1) bound are correct, and the authors are honest that under K<=N the credit is relative, not absolute. The ablation in Table 4 is informative, especially the finding that MCR on a thresholded reward hurts counting: it shows the continuous evaluator is doing real work, not just decoration.

The soft spots are real, and the reviewer's concern about evaluator calibration is the right one. The entire box-level signal is a function of V(S,G) with hand-set weights alpha=2, beta=gamma=1, a 40-pixel cutoff, and point levels 1.0/0.3/0. No sensitivity analysis is reported for any of these choices. Since the credit is defined as a leave-one-out difference in V, if V rewards boxes that fail Acc@0.5 or inflate AP in ways that don't generalize, the model is being reinforced for the wrong behavior. Table 4 already shows the result is sensitive to the evaluator (continuous vs. thresholded changes counting by 8.6 points), so this is not a hypothetical worry. The paper also reports no error bars or multiple seeds, selects checkpoints around 400 steps because they give stronger performance (stated in Appendix A), and tunes lambda_mcr on the evaluation benchmarks. None of these are fatal — the method's logic survives — but together they make the SOTA claim conditional.

One note on circularity: I don't think it's a serious problem that credit is defined relative to the paper's own evaluator. The claim is that optimizing that credit improves external benchmarks, and that is testable. The burden is calibration, not circularity.

Who should read this: anyone doing RL fine-tuning of MLLMs for grounding, detection, or counting. It deserves a serious referee. The right referee report would ask for a sensitivity sweep over the evaluator hyperparameters, multiple seeds, and code release. I'd accept it for review and expect the revision to tighten the empirical claims.

Referee Report

3 major / 3 minor

Summary. The paper proposes MCR-GRPO, an RL fine-tuning method for structured visual perception (REC, DOD, segmentation, counting) that replaces response-level GRPO advantages with box-level marginal contribution credits. Given a sampled structured response parsed into object records, the method computes a continuous matched-set value V(S,G) via Hungarian matching (Eqs. 2-6), then estimates each box's contribution by leave-one-out difference ΔV_i (Eq. 7), z-normalizes within the response (Eqs. 8-9), and adds the resulting token-span credit to the GRPO advantage (Eq. 12). The authors report state-of-the-art results over GRPO-based baselines on RefCOCO, D3, ReasonSeg, PixMo-Count, and VQA benchmarks, with ablations showing MCR and the continuous evaluator are complementary. The supplementary material provides formal propositions about the sign and boundedness of MCR credits.

Significance. If the reported gains are reproducible, this is a useful contribution: it addresses a real granularity mismatch in GRPO for multi-object prediction without extra rollouts, and the LOO attribution is simple and theoretically grounded. The formal analysis in Appendix F is a clear strength: Propositions 1-4 are correct, including the regime-dependent raw signs and the bound |A_mcr| ≤ sqrt(K-1). The empirical scope is broad (REC, DOD, segmentation, counting, VQA, plus COCO in the appendix). The main weakness is that the entire box-level signal is a function of a hand-designed evaluator whose calibration is never tested, and the experimental reporting (checkpoint selection, no error bars) weakens the SOTA claim. With a sensitivity analysis and more rigorous evaluation, the result could be accepted.

major comments (3)

[Appendix A; Tables 1-2] The evaluation is performed at 'checkpoints around 400 training steps' selected because they 'empirically ... yield stronger performance' (Appendix A). With only 443 total steps, this means the reported numbers are a post hoc selection over the training trajectory, not a fixed protocol. This is load-bearing for the SOTA claim. Please report performance across the full trajectory (or at a prespecified final checkpoint) and justify the selection rule; otherwise the comparisons in Tables 1-2 are not reproducible.
[Eqs. (2)-(6), (7), (11); Table 4] The continuous evaluator V(S,G) is the common substrate for the response reward (Eq. 11) and for every raw MCR term ΔV_i (Eq. 7). The weights α=2, β=γ=1, the 40-pixel cutoff in Eq. (3), and the point-score levels 1.0/0.3/0 are fixed without any sensitivity study. Table 4 shows the outcome is sensitive to this design: replacing the continuous evaluator by the thresholded Accuracy Reward changes counting from 75.7 to 67.1 under MCR, i.e., the box-level signal can actively harm a cardinality-sensitive task if the evaluator is discretized. Because the paper never varies α/β/γ, the distance cutoff, or the point levels, the claimed gains are conditional on an untested proxy. Please add a sensitivity sweep and, ideally, a calibration analysis showing V(S,G) is monotonic in Acc@0.5/AP.
[Tables 1-2; Appendix D] All tables report single-run numbers. No standard errors, confidence intervals, or multiple seeds are provided for any benchmark. Given the small training set (7,099 examples), 8 rollouts per prompt, and 443 optimization steps, the reported improvements over strong baselines (e.g., REC Avg 86.8 vs 85.8 for GroupRevision in Table 1) are within plausible noise. Report mean±std over at least three seeds or a pairwise significance test for the main comparisons.

minor comments (3)

[Method, 'MCR-GRPO Optimization'] The text says 'exact cancellation holds at the record level rather than the token level, since spans differ in length'. Since the final objective (Eq. 16) is token-level, the zero-sum property is not exact at the optimization unit. Please state the implication for credit assignment more precisely.
[Eq. (4)] The evaluator is called 'Continuous Matched Set Value Evaluator', but the point term s_point is piecewise constant (1.0/0.3/0), not continuous. Consider renaming to 'graded' or clarifying the sense in which it is continuous (only the IoU and box-distance terms are continuous).
[Implementation Details] The paper does not mention whether code and trained checkpoints will be released. For a method whose main novelty is a training-time credit assignment procedure, code release is important for reproducibility; please state availability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; MCR is a reward-shaping construction evaluated against external benchmarks.

full rationale

The derivation chain is self-contained. MCR's box credit ΔV_i = V(S,G) − V(S\{d_i},G) is defined as a leave-one-out difference of the paper's own Continuous Matched Set Value Evaluator (Eqs. 2–7), and the response reward in Eq. (11) also uses V(S,G). This means 'helpful/harmful' is by construction relative to V, but that is a reward design choice, not an epistemic circle: V is an explicit, hand-set reward prior (α=2, β=γ=1, 40-px cutoff, point levels 1.0/0.3/0) rather than a parameter fitted to the reported metrics, and the paper's actual predictions—REC Acc@0.5, DOD AP, segmentation gIoU/cIoU, counting accuracy—are measured on external benchmarks (RefCOCO, D3, ReasonSeg, PixMo, COCO) that do not use V. The core claim is that optimizing this box-level credit improves those external metrics, which is falsifiable and tested in Tables 1–4 and D3–D4. The main caveat, that V's hand-set weights/cutoffs are never swept for sensitivity and may be miscalibrated w.r.t. the reported metrics, is a correctness/robustness risk, not circularity. There is also no load-bearing self-citation: the VisionReasoner/GroupRevision baselines, prompts, and training data are from different author groups, and no uniqueness theorem or prior result by these authors is invoked to force the MCR construction.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The central claim depends on hand-set evaluator constants and a within-response normalization convention that defines negative credit relative to the response mean. No new physical or architectural entities are introduced.

free parameters (6)

evaluator weights alpha, beta, gamma = alpha=2, beta=1, gamma=1
Hand-set in Eq. (2) to balance IoU, box-distance, and point-validity terms; no sensitivity analysis is provided.
sbox distance cutoff = 40 pixels
Eq. (3): coordinate-distance score is zero beyond 40 pixels mean absolute distance; introduced by hand.
point score levels = 1.0, 0.3, 0
Eq. (4): ad hoc gradation for point inside mask, inside box but outside mask, and outside box.
MCR ratio lambda_mcr = 0.10
Selected via ablation on the same evaluation benchmarks (Table 5); highest-scoring setting across REC, DOD, segmentation, counting.
base reward constant = 4
Eq. (11): chosen so format-valid responses have a base reward equal to maximum V(S,G); hand-set scale alignment.
non-repetition reward coefficient = 1.5
Eq. (11): weight of the R_nr term in the response-level reward; adopted without analysis.

assumptions (6)

standard math Pair scores are non-negative and Hungarian matching defines the optimal one-to-one assignment.
Used in Eqs. (5)-(6) and in Appendix F Lemma 1; standard result for maximum-weight bipartite matching.
domain assumption Ground truth is an unordered object set with a box and mask per object; predictions parse into an unordered box set with points.
Problem formulation; makes the count-aware denominator max(K,N) meaningful.
domain assumption The generated token span for each box can be deterministically identified from the response.
Eq. (10) assigns MCR credit to tokens in tau_i; the paper does not specify the span-extraction algorithm.
domain assumption A point inside the GT box and mask is a good SAM2 prompt; inside box but outside mask is worth 0.3.
Eq. (4); proxy for downstream segmentation prompt quality.
ad hoc to paper Within-response z-score normalization of delta_V_i yields sensible relative credit even when all raw deltas are non-negative (K<=N).
Main text after Eq. (9) and Appendix F Prop. 2: below-mean boxes get negative credit by construction, not by external evidence.
domain assumption The base MLLM Qwen2.5-VL-7B-Instruct and SAM2 provide sufficient vision and segmentation priors.
Initialization and mask generation; not verified in this paper.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Credit the Right Box: Marginal Contribution Assignment for Structured Visual Perception." pith.science (2026). https://pith.science/paper/3MUCY7KZ

@misc{pith2026260801055,
  author       = {Pith},
  title        = {Pith review of: Credit the Right Box: Marginal Contribution Assignment for Structured Visual Perception},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3MUCY7KZ}},
  note         = {Machine review of arXiv:2608.01055}
}

read the original abstract

Multimodal Large Language Models (MLLMs) are increasingly expected to solve structured perception tasks that require visual recognition, language-to-object binding, object cardinality preservation, and precisely localized grounding and segmentation outputs. However, existing group-relative reinforcement learning methods provide only response-level supervision, creating a granularity mismatch for structured multi-object prediction: a single advantage is broadcast to all tokens in a response, without distinguishing individual box contributions. To address this mismatch, we propose MCR-GRPO, a marginal contribution assignment framework that derives box-level credit directly from each sampled response. Specifically, Marginal Contribution Reward (MCR) estimates each predicted box's contribution through a leave-one-out comparison, measuring how the matched set value changes when the box is removed from the response. After within-response normalization, records that improve the set value receive positive credit, while redundant or harmful ones are suppressed. To make marginal attribution stable and informative, we further introduce a Continuous Matched Set Value Evaluator that integrates permutation-invariant matching, count-aware normalization, and graded localization. MCR-GRPO maps normalized box-level marginal advantages to the token spans that generated each box, preserving GRPO's response-level comparison while enabling box-aware optimization of structured multi-object grounding. Experiments across REC, DOD, segmentation, and counting benchmarks show state-of-the-art performance over prior GRPO-based baselines.

Figures

Figures reproduced from arXiv: 2608.01055 by the authors.

**Figure 1.** Three Paradigms of Supervision for Structured Visual Perception. Response supervision scores the whole structured answer with a single advantage. Trajectory supervision revises entire generated paths, recovering feedback at the cost of extra rollouts, and still updates at sequence granularity. Object-record Supervision (MCR-GRPO) supervises at the object-record level, providing native response-internal credit assign… view at source ↗

**Figure 2.** Overview of MCR-GRPO. Given an image-query pair, the policy samples structured responses and parses each response into box-indexed object records with their generated token spans. MCR-GRPO matches the predicted records to ground-truth objects, computes the matched set value, and estimates each object-record’s leave-one-out contribution to boxlevel MCR advantages. The advantages are mapped back to the token spans an… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

85 extracted references · 66 canonical work pages

[1]

2025 , eprint=

Qwen2.5-VL Technical Report , author=. 2025 , eprint=

2025
[2]

The Fourteenth International Conference on Learning Representations , year=

VisionReasoner: Unified Reasoning-Integrated Visual Perception via Reinforcement Learning , author=. The Fourteenth International Conference on Learning Representations , year=
[3]

Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

From Failure to Feedback: Group Revision Unlocks Hard Cases in Object-Level Grounding , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
[4]

European conference on computer vision , pages=

Modeling Context in Referring Expressions , author=. European conference on computer vision , pages=. 2016 , organization=

2016
[5]

Proceedings of the IEEE/CVF international conference on computer vision , pages=

Teaching CLIP to Count to Ten , author=. Proceedings of the IEEE/CVF international conference on computer vision , pages=
[6]

Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

work page
[7]

Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

LISA: Reasoning Segmentation via Large Language Model , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

work page
[8]

Proceedings of the Twentieth European Conference on Computer Systems , pages=

Hybridflow: A Flexible and Efficient RLHF Framework , author=. Proceedings of the Twentieth European Conference on Computer Systems , pages=

work page

Show all 85 references

[9]

Proceedings of the 29th symposium on operating systems principles , pages=

Efficient Memory Management for Large Language Model Serving with Pagedattention , author=. Proceedings of the 29th symposium on operating systems principles , pages=
[10]

International Conference on Learning Representations , volume=

SAM 2: Segment Anything in Images and Videos , author=. International Conference on Learning Representations , volume=
[11]

2024 , eprint=

DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models , author=. 2024 , eprint=

2024
[12]

Advances in neural information processing systems , volume=

Visual Instruction Tuning , author=. Advances in neural information processing systems , volume=
[13]

Advances in Neural Information Processing Systems , volume=

Perception-R1: Pioneering Perception Policy with Reinforcement Learning , author=. Advances in Neural Information Processing Systems , volume=
[14]

Advances in Neural Information Processing Systems , volume=

SAM-R1: Leveraging SAM for Reward Feedback in Multimodal Segmentation via Reinforcement Learning , author=. Advances in Neural Information Processing Systems , volume=
[15]

2025 , eprint=

VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model , author=. 2025 , eprint=

2025
[16]

2025 , eprint=

UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement Learning , author=. 2025 , eprint=

2025
[17]

Advances in Neural Information Processing Systems , volume=

Are We on the Right Way for Evaluating Large Vision-Language Models? , author=. Advances in Neural Information Processing Systems , volume=
[18]

ICDAR 2023 Competition on Document UnderstanDing of Everything (DUDE)

Van Landeghem, Jordy and Tito, Rub \`e n and Borchmann, ukasz and Pietruszka, Micha and Jurkiewicz, Dawid and Powalski, Rafa and J \'o ziak, Pawe and Biswas, Sanket and Coustaty, Micka \"e l and Stanis awek, Tomasz. ICDAR 2023 Competition on Document UnderstanDing of Everythin...

2023
[19]

Findings of the association for computational linguistics: ACL 2022 , pages=

ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning , author=. Findings of the association for computational linguistics: ACL 2022 , pages=

2022
[20]

Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , month=

Cheng, Xianfu and Zhang, Wei and Zhang, Shiwei and Yang, Jian and Guan, Xiangyuan and Wu, Xianjie and Li, Xiang and Zhang, Ge and Liu, Jiaheng and Mai, Yuying and Zeng, Yutao and Wen, Zhoufutu and Jin, Ke and Wang, Baorui and Zhou, Weixiao and Lu, Yunhong and Ji, Hangyuan and ...

2025
[21]

OCRBench: on the hidden mystery of OCR in large multimodal models , volume=

Liu, Yuliang and Li, Zhang and Huang, Mingxin and Yang, Biao and Yu, Wenwen and Li, Chunyuan and Yin, Xu-Cheng and Liu, Cheng-Lin and Jin, Lianwen and Bai, Xiang , year=. OCRBench: on the hidden mystery of OCR in large multimodal models , volume=. Science China Information Sci...

doi
[22]

MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans? , url =

Zhang, YiFan and Zhang, Huanyu and Tian, Haochen and Fu, Chaoyou and Zhang, Shuangqing and Wu, Junfei and Li, Feng and Wang, Kun and Wen, Qingsong and Zhang, Zhang and Wang, Liang and Jin, Rong , booktitle =. MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution R...
[23]

Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =

Pi, Renjie and Yao, Lewei and Gao, Jiahui and Zhang, Jipeng and Zhang, Tong , title =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =. 2024 , pages =

2024
[24]

Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =

Pramanick, Shraman and Han, Guangxing and Hou, Rui and Nag, Sayan and Lim, Ser-Nam and Ballas, Nicolas and Wang, Qifan and Chellappa, Rama and Almahairi, Amjad , title =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =. 2024 , pages =

2024
[25]

Elysium: Exploring Object-Level Perception in Videos via MLLM

Wang, Han and Ye, Yongjie and Wang, Yanjie and Nie, Yuxiang and Huang, Can. Elysium: Exploring Object-Level Perception in Videos via MLLM. Computer Vision -- ECCV 2024. 2025

2024
[26]

European Conference on Computer Vision , pages=

Groma: Localized visual tokenization for grounding multimodal large language models , author=. European Conference on Computer Vision , pages=. 2024 , organization=

2024
[27]

2026 , eprint=

Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcement , author=. 2026 , eprint=

2026
[28]

2025 , eprint=

Seg-R1: Segmentation Can Be Surprisingly Simple with Reinforcement Learning , author=. 2025 , eprint=

2025
[29]

2024 , eprint=

LLaVA-OneVision: Easy Visual Task Transfer , author=. 2024 , eprint=

2024
[30]

Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =

Ren, Zhongwei and Huang, Zhicheng and Wei, Yunchao and Zhao, Yao and Fu, Dongmei and Feng, Jiashi and Jin, Xiaojie , title =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =. 2024 , pages =

2024
[31]

International Conference on Learning Representations , author=

SegLLM: Multi-round Reasoning Segmentation with Large Language Models , url =. International Conference on Learning Representations , author=
[32]

Reasoning to Attend: Try to Understand How <SEG> Token Works , year=

Qian, Rui and Yin, Xin and Dou, Dejing , booktitle=. Reasoning to Attend: Try to Understand How <SEG> Token Works , year=
[33]

Grounding Multimodal Large Language Models to the World , url=

Peng, Zhiliang and Wang, Wenhui and Dong, Li and Hao, Yaru and Huang, Shaohan and Ma, Shuming and Ye, Qixiang and Wei, Furu , booktitle=. Grounding Multimodal Large Language Models to the World , url=
[34]

and Lo, Wan-Yen and Dollar, Piotr and Girshick, Ross , title=

Kirillov, Alexander and Mintun, Eric and Ravi, Nikhila and Mao, Hanzi and Rolland, Chloe and Gustafson, Laura and Xiao, Tete and Whitehead, Spencer and Berg, Alexander C. and Lo, Wan-Yen and Dollar, Piotr and Girshick, Ross , title=. Proceedings of the IEEE/CVF International C...

2023
[35]

Grounding DINO: Marrying DINO with Grounded Pre-training for Open-Set Object Detection

Liu, Shilong and Zeng, Zhaoyang and Ren, Tianhe and Li, Feng and Zhang, Hao and Yang, Jie and Jiang, Qing and Li, Chunyuan and Yang, Jianwei and Su, Hang and Zhu, Jun and Zhang, Lei. Grounding DINO: Marrying DINO with Grounded Pre-training for Open-Set Object Detection. Comput...

2024
[36]

Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =

Li, Liunian Harold and Zhang, Pengchuan and Zhang, Haotian and Yang, Jianwei and Li, Chunyuan and Zhong, Yiwu and Wang, Lijuan and Yuan, Lu and Zhang, Lei and Hwang, Jenq-Neng and Chang, Kai-Wei and Gao, Jianfeng , title =. Proceedings of the IEEE/CVF Conference on Computer Vi...

2022
[37]

Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =

Xiao, Bin and Wu, Haiping and Xu, Weijian and Dai, Xiyang and Hu, Houdong and Lu, Yumao and Zeng, Michael and Liu, Ce and Yuan, Lu , title =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =. 2024 , pages =

2024
[38]

Described Object Detection: Liberating Object Detection with Flexible Expressions , url =

Xie, Chi and Zhang, Zhao and Wu, Yixuan and Zhu, Feng and Zhao, Rui and Liang, Shuang , booktitle =. Described Object Detection: Liberating Object Detection with Flexible Expressions , url =. doi:10.52202/075280-3460 , editor =

doi
[39]

Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =

Liu, Chang and Ding, Henghui and Jiang, Xudong , title =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =. 2023 , pages =

2023
[40]

Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , month =

Liu, Ziyu and Sun, Zeyi and Zang, Yuhang and Dong, Xiaoyi and Cao, Yuhang and Duan, Haodong and Lin, Dahua and Wang, Jiaqi , title =. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , month =. 2025 , pages =

2025
[41]

Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=
[42]

Let s Verify Step by Step , url =

Lightman, Hunter and Kosaraju, Vineet and Burda, Yuri and Edwards, Harrison and Baker, Bowen and Lee, Teddy and Leike, Jan and Schulman, John and Sutskever, Ilya and Cobbe, Karl , booktitle =. Let s Verify Step by Step , url =
[44]

Bai, S.; Li, M.; Liu, Y.; Tang, J.; Zhang, H.; Sun, L.; Chu, X.; and Tang, Y. 2025 b . UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement Learning. arXiv:2505.14231

2025 arXiv
[45]

Chen, L.; Li, J.; Dong, X.; Zhang, P.; Zang, Y.; Chen, Z.; Duan, H.; Wang, J.; Qiao, Y.; Lin, D.; et al. 2024. Are We on the Right Way for Evaluating Large Vision-Language Models? Advances in Neural Information Processing Systems, 37: 27056--27087

2024
[46]

Cheng, X.; Zhang, W.; Zhang, S.; Yang, J.; Guan, X.; Wu, X.; Li, X.; Zhang, G.; Liu, J.; Mai, Y.; Zeng, Y.; Wen, Z.; Jin, K.; Wang, B.; Zhou, W.; Lu, Y.; Ji, H.; Li, T.; Huang, W.; and Li, Z. 2025. SimpleVQA: Multimodal Factuality Evaluation for Multimodal Large Language Model...

2025
[47]

S.; Salehi, M.; Muennighoff, N.; Lo, K.; Soldaini, L.; et al

Deitke, M.; Clark, C.; Lee, S.; Tripathi, R.; Yang, Y.; Park, J. S.; Salehi, M.; Muennighoff, N.; Lo, K.; Soldaini, L.; et al. 2025. Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models. In Proceedings of the Computer Vision and Pattern Recog...

2025
[48]

Huang, J.; Xu, Z.; Zhou, J.; Liu, T.; Xiao, Y.; Ou, M.; Ji, B.; Li, X.; and Yuan, K. 2026. SAM-R1: Leveraging SAM for Reward Feedback in Multimodal Segmentation via Reinforcement Learning. Advances in Neural Information Processing Systems, 38: 138362--138383

2026
[49]

C.; Lo, W.-Y.; Dollar, P.; and Girshick, R

Kirillov, A.; Mintun, E.; Ravi, N.; Mao, H.; Rolland, C.; Gustafson, L.; Xiao, T.; Whitehead, S.; Berg, A. C.; Lo, W.-Y.; Dollar, P.; and Girshick, R. 2023. Segment Anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 4015--4026

2023
[50]

H.; Gonzalez, J.; Zhang, H.; and Stoica, I

Kwon, W.; Li, Z.; Zhuang, S.; Sheng, Y.; Zheng, L.; Yu, C. H.; Gonzalez, J.; Zhang, H.; and Stoica, I. 2023. Efficient Memory Management for Large Language Model Serving with Pagedattention. In Proceedings of the 29th symposium on operating systems principles, 611--626

2023
[51]

Lai, X.; Tian, Z.; Chen, Y.; Li, Y.; Yuan, Y.; Liu, S.; and Jia, J. 2024. LISA: Reasoning Segmentation via Large Language Model. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 9579--9589

2024
[52]

Li, B.; Zhang, Y.; Guo, D.; Zhang, R.; Li, F.; Zhang, H.; Zhang, K.; Zhang, P.; Li, Y.; Liu, Z.; and Li, C. 2024. LLaVA-OneVision: Easy Visual Task Transfer. arXiv:2408.03326

2024 arXiv
[53]

H.; Zhang, P.; Zhang, H.; Yang, J.; Li, C.; Zhong, Y.; Wang, L.; Yuan, L.; Zhang, L.; Hwang, J.-N.; Chang, K.-W.; and Gao, J

Li, L. H.; Zhang, P.; Zhang, H.; Yang, J.; Li, C.; Zhong, Y.; Wang, L.; Yuan, L.; Zhang, L.; Hwang, J.-N.; Chang, K.-W.; and Gao, J. 2022. Grounded Language-Image Pre-Training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 10965--10975

2022
[54]

Lightman, H.; Kosaraju, V.; Burda, Y.; Edwards, H.; Baker, B.; Lee, T.; Leike, J.; Schulman, J.; Sutskever, I.; and Cobbe, K. 2024. Let s Verify Step by Step. In Kim, B.; Yue, Y.; Chaudhuri, S.; Fragkiadaki, K.; Khan, M.; and Sun, Y., eds., International Conference on Learning...

2024
[55]

Liu, C.; Ding, H.; and Jiang, X. 2023. GRES: Generalized Referring Expression Segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 23592--23601

2023
[56]

Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023. Visual Instruction Tuning. Advances in neural information processing systems, 36: 34892--34916

2023
[57]

Liu, S.; Zeng, Z.; Ren, T.; Li, F.; Zhang, H.; Yang, J.; Jiang, Q.; Li, C.; Yang, J.; Su, H.; Zhu, J.; and Zhang, L. 2025 a . Grounding DINO: Marrying DINO with Grounded Pre-training for Open-Set Object Detection. In Computer Vision -- ECCV 2024, 38--55. Cham: Springer Nature ...

2025
[58]

Liu, Y.; Ji, Y.; Le, A.; Zhu, J.; Pan, J.; Peng, C.; Deng, J.; Liu, F.; and Wu, J. 2026 a . From Failure to Feedback: Group Revision Unlocks Hard Cases in Object-Level Grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4816--4828

2026
[59]

Liu, Y.; Li, Z.; Huang, M.; Yang, B.; Yu, W.; Li, C.; Yin, X.-C.; Liu, C.-L.; Jin, L.; and Bai, X. 2024. OCRBench: on the hidden mystery of OCR in large multimodal models. Science China Information Sciences, 67(12)

2024
[60]

Liu, Y.; Peng, B.; Zhong, Z.; Yue, Z.; Lu, F.; Yu, B.; and Jia, J. 2026 b . Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcement. arXiv:2503.06520

2026 arXiv
[61]

Liu, Y.; Qu, T.; Zhong, Z.; Peng, B.; Liu, S.; Yu, B.; and Jia, J. 2026 c . VisionReasoner: Unified Reasoning-Integrated Visual Perception via Reinforcement Learning. In The Fourteenth International Conference on Learning Representations

2026
[62]

Liu, Z.; Sun, Z.; Zang, Y.; Dong, X.; Cao, Y.; Duan, H.; Lin, D.; and Wang, J. 2025 b . Visual-RFT: Visual Reinforcement Fine-Tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2034--2044

2025
[63]

Ma, C.; Jiang, Y.; Wu, J.; Yuan, Z.; and Qi, X. 2024. Groma: Localized visual tokenization for grounding multimodal large language models. In European Conference on Computer Vision, 417--435. Springer

2024
[64]

Q.; Joty, S.; Hoque, E.; et al

Masry, A.; Tan, J. Q.; Joty, S.; Hoque, E.; et al. 2022. ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning. In Findings of the association for computational linguistics: ACL 2022, 2263--2279

2022
[65]

Paiss, R.; Ephrat, A.; Tov, O.; Zada, S.; Mosseri, I.; Irani, M.; and Dekel, T. 2023. Teaching CLIP to Count to Ten. In Proceedings of the IEEE/CVF international conference on computer vision, 3170--3180

2023
[66]

Peng, Z.; Wang, W.; Dong, L.; Hao, Y.; Huang, S.; Ma, S.; Ye, Q.; and Wei, F. 2024. Grounding Multimodal Large Language Models to the World. In International Conference on Learning Representations, volume 2024, 51575--51598

2024
[67]

Pi, R.; Yao, L.; Gao, J.; Zhang, J.; and Zhang, T. 2024. PerceptionGPT: Effectively Fusing Visual Perception into LLM. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 27124--27133

2024
[68]

Pramanick, S.; Han, G.; Hou, R.; Nag, S.; Lim, S.-N.; Ballas, N.; Wang, Q.; Chellappa, R.; and Almahairi, A. 2024. Jack of All Tasks Master of Many: Designing General-Purpose Coarse-to-Fine Vision-Language Model. In Proceedings of the IEEE/CVF Conference on Computer Vision and...

2024
[69]

Qian, R.; Yin, X.; and Dou, D. 2025. Reasoning to Attend: Try to Understand How <SEG> Token Works. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 24722--24731

2025
[70]

Ravi, N.; Gabeur, V.; Hu, Y.-T.; Hu, R.; Ryali, C.; Ma, T.; Khedr, H.; R \"a dle, R.; Rolland, C.; Gustafson, L.; et al. 2025. SAM 2: Segment Anything in Images and Videos. In International Conference on Learning Representations, volume 2025, 28085--28128

2025
[71]

Ren, Z.; Huang, Z.; Wei, Y.; Zhao, Y.; Fu, D.; Feng, J.; and Jin, X. 2024. PixelLM: Pixel Reasoning with Large Multimodal Model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 26374--26383

2024
[72]

K.; Wu, Y.; and Guo, D

Shao, Z.; Wang, P.; Zhu, Q.; Xu, R.; Song, J.; Bi, X.; Zhang, H.; Zhang, M.; Li, Y. K.; Wu, Y.; and Guo, D. 2024. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300

2024 arXiv
[73]

Shen, H.; Liu, P.; Li, J.; Fang, C.; Ma, Y.; Liao, J.; Shen, Q.; Zhang, Z.; Zhao, K.; Zhang, Q.; Xu, R.; and Zhao, T. 2025. VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model. arXiv:2504.07615

2025 arXiv
[74]

Sheng, G.; Zhang, C.; Ye, Z.; Wu, X.; Zhang, W.; Zhang, R.; Peng, Y.; Lin, H.; and Wu, C. 2025. Hybridflow: A Flexible and Efficient RLHF Framework. In Proceedings of the Twentieth European Conference on Computer Systems, 1279--1297

2025
[75]

Van Landeghem, J.; Tito, R.; Borchmann, .; Pietruszka, M.; Jurkiewicz, D.; Powalski, R.; J \'o ziak, P.; Biswas, S.; Coustaty, M.; and Stanis awek, T. 2023. ICDAR 2023 Competition on Document UnderstanDing of Everything (DUDE). In Document Analysis and Recognition - ICDAR 2023...

2023
[76]

Wang, H.; Ye, Y.; Wang, Y.; Nie, Y.; and Huang, C. 2025 a . Elysium: Exploring Object-Level Perception in Videos via MLLM. In Computer Vision -- ECCV 2024, 166--185. Cham: Springer Nature Switzerland. ISBN 978-3-031-72670-5

2025
[77]

Wang, P.; Li, L.; Shao, Z.; Xu, R.; Dai, D.; Li, Y.; Chen, D.; Wu, Y.; and Sui, Z. 2024. Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long ...

2024
[78]

Wang, X.; Zhang, S.; Li, S.; Li, K.; Kallidromitis, K.; Kato, Y.; Kozuka, K.; et al. 2025 b . SegLLM: Multi-round Reasoning Segmentation with Large Language Models. In International Conference on Learning Representations, volume 2025, 56526--56547

2025
[79]

Xiao, B.; Wu, H.; Xu, W.; Dai, X.; Hu, H.; Lu, Y.; Zeng, M.; Liu, C.; and Yuan, L. 2024. Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 4818--4829

2024
[80]

Xie, C.; Zhang, Z.; Wu, Y.; Zhu, F.; Zhao, R.; and Liang, S. 2023. Described Object Detection: Liberating Object Detection with Flexible Expressions. In Oh, A.; Naumann, T.; Globerson, A.; Saenko, K.; Hardt, M.; and Levine, S., eds., Advances in Neural Information Processing S...

2023
[81]

You, Z.; and Wu, Z. 2025. Seg-R1: Segmentation Can Be Surprisingly Simple with Reinforcement Learning. arXiv:2506.22624

2025 arXiv
[82]

Yu, E.; Lin, K.; Zhao, L.; Wei, Y.; Peng, Y.; Wei, H.; Sun, J.; Han, C.; Ge, Z.; Zhang, X.; et al. 2026. Perception-R1: Pioneering Perception Policy with Reinforcement Learning. Advances in Neural Information Processing Systems, 38: 94827--94853

2026
[83]

C.; and Berg, T

Yu, L.; Poirson, P.; Yang, S.; Berg, A. C.; and Berg, T. L. 2016. Modeling Context in Referring Expressions. In European conference on computer vision, 69--85. Springer

2016
[84]

Zhang, Y.; Zhang, H.; Tian, H.; Fu, C.; Zhang, S.; Wu, J.; Li, F.; Wang, K.; Wen, Q.; Zhang, Z.; Wang, L.; and Jin, R. 2025. MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans? In Yue, Y.; Garg, A.; Peng, N.; S...

2025
[85]

Bai, S.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; Song, S.; Dang, K.; Wang, P.; Wang, S.; Tang, J.; Zhong, H.; Zhu, Y.; Yang, M.; Li, Z.; Wan, J.; Wang, P.; Ding, W.; Fu, Z.; Xu, Y.; Ye, J.; Zhang, X.; Xie, T.; Cheng, Z.; Zhang, H.; Yang, Z.; Xu, H.; and Lin, J. 2025. Qwen2.5-VL T...

2025 arXiv
[86]

Liu, Y.; Qu, T.; Zhong, Z.; Peng, B.; Liu, S.; Yu, B.; and Jia, J. 2026 b . VisionReasoner: Unified Reasoning-Integrated Visual Perception via Reinforcement Learning. In The Fourteenth International Conference on Learning Representations

2026

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

2025 , eprint=

Qwen2.5-VL Technical Report , author=. 2025 , eprint=

2025

[2] [2]

The Fourteenth International Conference on Learning Representations , year=

VisionReasoner: Unified Reasoning-Integrated Visual Perception via Reinforcement Learning , author=. The Fourteenth International Conference on Learning Representations , year=

[3] [3]

Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

From Failure to Feedback: Group Revision Unlocks Hard Cases in Object-Level Grounding , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

[4] [4]

European conference on computer vision , pages=

Modeling Context in Referring Expressions , author=. European conference on computer vision , pages=. 2016 , organization=

2016

[5] [5]

Proceedings of the IEEE/CVF international conference on computer vision , pages=

Teaching CLIP to Count to Ten , author=. Proceedings of the IEEE/CVF international conference on computer vision , pages=

[6] [6]

Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

work page

[7] [7]

Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

LISA: Reasoning Segmentation via Large Language Model , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

work page

[8] [8]

Proceedings of the Twentieth European Conference on Computer Systems , pages=

Hybridflow: A Flexible and Efficient RLHF Framework , author=. Proceedings of the Twentieth European Conference on Computer Systems , pages=

work page