Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

HART proposes a closed-loop framework where a vision-language model must answer from its own crops, making correct answers a direct signal for correct localization, and uses AP-GRPO to train grounding without annotations.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 20:13 UTC pith:TLZQ2T4O

load-bearing objection HART's empirical recipe — crops-only RL + SFT — looks promising and the reported gains are large, but Proposition 2 does not prove what it claims: AP-GRPO never observes the localization variable, so the 'direct grounding optimization' guarantee is unsupported; what remains is a solid empirical paper that needs reframing, error bars, and an isolated ablation. the 3 major comments →

arxiv 2602.23615 v3 pith:TLZQ2T4O submitted 2026-02-27 cs.CV

HART: High-Resolution Annotation-Free Reasoning Technique through a Closed-loop Framework

Jiacheng Yang , Anqi Chen , Yunkai Dang , Qi Fan , Cong Wang , Wenbin Li , Feng Miao , Yang Gao This is my paper
classification cs.CV
keywords visual groundinghigh-resolution LMMreinforcement learningGRPOannotation-freeclosed-loop reasoningregion-of-interest croppingvisual reasoning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to show that a vision-language model can be taught to find and zoom into the key region of a high-resolution image without human grounding labels. The idea is a feedback loop: have the model propose a crop, hide the original image, and ask it to answer from the crop alone; then a correct answer becomes evidence that the crop was right, so the usual final-answer reward now also measures localization. On top of this, AP-GRPO reweights training samples so that correct, high-advantage responses dominate the gradient update. The authors report gains of +20.1 points on MME-RealWorld, +6.7 on TreeBench, +2.1 on V* Bench, and +10.9 on HR-Bench-8K over strong baselines, and substantially better grounding accuracy on datasets with ground-truth boxes. If true, this would offer a low-cost way to improve high-resolution perception in multimodal models without expensive annotation.

Core claim

Reward misspecification—a positive reward for a correct answer with wrong localization—can be largely removed, the paper argues, by a closed-loop procedure (HART): the model predicts regions of interest, the original image is withheld, and the model must answer from the crops alone, making answer correctness a faithful proxy for grounding quality (pilot data: correct-answer-wrong-localization drops from 36.5% to 21.5% for Qwen2.5-VL-7B). The proposed AP-GRPO up-weights correct, advantageous responses and lowers their KL penalty, directly optimizing localization without annotations. Post-trained on Qwen2.5-VL-7B, HART beats baselines on high-resolution benchmarks, with grounding accuracy risi

What carries the argument

Closed-loop visual feedback: the model first outputs bounding-box coordinates for the region(s) of interest from the downsampled image; those regions are cropped from the original high-resolution image, the original is deliberately hidden, and the model must answer from the crops alone. This converts localization quality into a question-answer signal. AP-GRPO then applies group-relative advantages with two dynamic scalar weights (µ1 scales the policy-ratio term, µ2 scales the KL penalty) so that samples with correct, high-advantage answers receive larger updates and a smaller KL penalty, while incorrect samples are down-weighted. The scaling factor k is the only hyperparameter; k=0.6 in the

Load-bearing premise

The entire method rests on the proxy that a correct answer from the crops alone means the crop contained the necessary visual information; if the model can answer correctly from an incorrect crop, the reward is still misspecified and the proposed gradient correction (Proposition 2) is not directly observable because the training algorithm never sees the true localization label L.

What would settle it

On a held-out set with ground-truth boxes, measure the jointly correct-answer/incorrect-localization rate (L=0, r=1) after AP-GRPO training; if it does not drop materially below the 21.5% pilot figure, the reward remains misspecified and Proposition 2's claimed removal of these samples is not occurring. A complementary check: construct a dataset where the correct answer is obtainable from a deliberately wrong crop (e.g., a text question legible in the wrong region); if HART still receives maximum reward on those items, the method cannot distinguish faithful from unfaithful grounding.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If HART works as described, any LMM with a bounding-box or region-prediction output head can be post-trained to localize key regions using only final-answer supervision, removing the need for human grounding labels.
  • Because the reward signal now tracks localization, reinforcement fine-tuning on answer correctness should not merely improve the answer but also the model's pointing/cropping behavior, which the grounding measurements on TreeBench and Visual CoT support.
  • The two-stage pipeline (AP-GRPO for localization, then SFT with the full image visible) suggests a general recipe: train the model to find, then train it to reason from the found region; performance on MME-RealWorld-Lite and TreeBench depends on both stages.
  • Improved localization transfers out of distribution, at least from MME-RealWorld training to TreeBench, and the method also transfers to a second base model (InternVL3-8B), indicating the framework is not tied to one architecture.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The residual 21.5% of correct-answer-wrong-localization cases in the pilot data suggests the reward is still partially misspecified; a natural extension is to penalize cases where the crop-only answer is correct but a separate faithfulness check (e.g., answer confidence or answer change when the crop is perturbed) flags the localization as unreliable.
  • The closed-loop trick is a form of self-supervised grounding: the image is its own label. This could extend beyond cropping to other 'test-time information hiding' schemes, such as masking text regions or occluding objects, to train perception without annotations.
  • Because AP-GRPO's weighting is a simple function of the group-relative advantage, an ablation is available to readers: replacing µ1 and µ2 with constants should recover vanilla GRPO; the reported difference quantifies the contribution of the dynamic weighting alone versus the closed-loop itself.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes HART, a post-training framework for high-resolution LMMs that operates in a closed loop: the model first predicts region-of-interest (ROI) boxes from a downsampled image, then answers the question using only the corresponding high-resolution crops (the original image is withheld), and finally undergoes SFT on full-image data. To train the localization behavior without bounding-box annotations, the authors introduce AP-GRPO, a modified GRPO objective that reweights samples by their answer-correctness advantage (Eqs. 4–6). The paper claims that this directly optimizes visual grounding, supported by Proposition 2 (Eq. 7), and reports large improvements on MME-RealWorld-Lite, TreeBench, V* Bench, HR-Bench-4K/8K, and MMStar, along with grounding-accuracy evaluations on TreeBench and Visual CoT.

Significance. If the empirical results hold, HART is a practically useful annotation-free training recipe for high-resolution visual reasoning, with the valuable feature of evaluating grounding accuracy directly on two labeled benchmarks and reporting IoU curves. The paper also includes ablations over its single hyperparameter k and demonstrates transfer to a second base model (InternVL3-8B). However, the central novelty claimed in the title and abstract is that AP-GRPO 'directly optimizes visual grounding'; this rests on Proposition 2, which is not derivable from the defined objective. Without that theoretical support, the contribution reduces to an empirically motivated curriculum/self-verification effect, and the absence of error bars or significance tests makes the magnitude of the reported gains uncertain. The empirical evaluation is broad and the paper is clearly written, but the central claim needs either a valid derivation or a serious reframing.

major comments (3)
  1. [§4.3, Eq. (7)] Proposition 2 is not a consequence of Eqs. (4)–(6). The AP-GRPO objective depends only on r_i, the generated text o_i, and the group statistics; the localization-correctness variable L is never observed. Taking the gradient at θ=θ_old gives (1/G)Σ_i[(1+kδ_i)A_i∇logπθ(o_i|q) − β(1−kδ_i)∇D_KL(πθ||πref)], with δ_i = r_i − mean({r_i}) and A_i = δ_i/std. The term (1+kδ_i)A_i equals δ_i/std + kδ_i²/std; there is no mechanism that subtracts a contribution proportional to P(L=0,r=1), nor any sign selection that removes those samples. Thus Eq. (7) does not follow from the algorithm as stated. The authors should either derive a valid gradient identity that justifies 'direct grounding optimization' or explicitly reframe the contribution as an empirical curriculum/self-verification effect rather than a provable direct optimization.
  2. [Table 1 and §4.2] The paper's own pilot data show that the crops-only reward remains misspecified. For Qwen2.5-VL-7B, even under the HART prompt change, 502/2332 = 21.5% of correct answers still have incorrect grounding; for InternVL3-8B the proportion is 55.9%. This is a large residual fraction, so the assertion in §4.3 that 'rewarding correct answers also encourages faithful grounding' is an empirical assumption, not a guarantee. Proposition 1's mutual-information inequality, even if true, is a correlational statement and does not establish that AP-GRPO directly optimizes L. The paper should acknowledge this limitation explicitly and temper the 'directly optimizes grounding' language throughout, including the abstract.
  3. [§5.1, Tables 2–6] All experimental results are single-run point estimates without standard deviations, confidence intervals, or significance tests. This is particularly concerning for TreeBench (Table 3), where several subcategories have very small sample sizes (Material n=13, Phy. State n=23, Comparison n=44) and the overall 43.7% versus baselines could be sensitive to a few questions. Given the large claims (+20.1% on MME-RealWorld, +6.7% on TreeBench), the authors should provide multiple seeds (at least 3) or bootstrap confidence intervals for the main comparisons, and ideally for grounding accuracy in Table 6.
minor comments (5)
  1. [§4.3] The statement that 'k is the only hyperparameter in HART' is an overstatement. Equations (4)–(6) also contain β, group size G, and the training pipeline involves the D_RL/D_SFT split, the number of ROIs, and the IoU/cropping thresholds. Please clarify which hyperparameters were fixed identically across baselines and which were tuned.
  2. [Table 3] The small per-category sample sizes make some comparisons fragile. Consider reporting aggregate results with confidence intervals or pooling related categories where possible.
  3. [Figure 3] The caption does not specify the metric definition used on the y-axis (presumably Intersection over Ground-Truth coverage rate) or the evaluation protocol. Please state it explicitly and clarify whether the threshold is applied to the area coverage of the ground-truth box.
  4. [§4.2] The proof of Proposition 1 is deferred to Appendix A but the appendix is not included in the submitted text. Please ensure the proof is self-contained and also state the exact joint distribution used to compute mutual information.
  5. [General] The paper would benefit from a dedicated limitations section discussing the residual reward misspecification, the reliance on answer correctness as a proxy, and the lack of evaluation on diverse base models beyond Qwen2.5-VL-7B and InternVL3-8B.

Circularity Check

0 steps flagged

No significant circularity; the invalid Proposition 2 is a correctness/derivation flaw, not a circular reduction.

full rationale

Walking the derivation chain: HART's training objective (Eqs. 4-6) is a function only of r_i, the crops-only answer correctness; the paper does not fit any parameter to the grounding benchmarks, and the grounding improvements in Tables 3 and 6 are measured on external datasets (TreeBench, Visual CoT) with ground-truth boxes not used for training. The only formal bridge from the objective to 'directly optimizes grounding' is Proposition 2 (Eq. 7), which asserts a gradient identity involving P(L=0,r=1). That identity is not derivable from Eqs. (4)-(6) because L is never observed; the actual gradient difference is (1/G)Σ_i[k δ_i A_i ∇_θ logπ_θ + β k δ_i ∇_θ KL], which does not contain a term of the form -αP(L=0,r=1)E[∇logπ]. This is an invalid/unsupported proof, not a circular reduction of the output to the input. The sole self-citation ([6]) is a background survey and is not load-bearing. Hence no equation is equivalent to its inputs by construction, and no prediction is fitted from the evaluation targets.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The main free parameters are hyperparameters of the RL setup. The paper introduces no new physical or ontological entities. The critical assumption is that crops-only answer correctness is a faithful proxy for grounding quality.

free parameters (5)
  • k = 0.6
    Scaling factor in AP-GRPO (Eqs. 5–6); chosen by hand with sensitivity analysis in Table 7.
  • β (KL coefficient) = not reported
    Inherited from GRPO; not specified in main text.
  • Group size G = not reported
    Number of candidate responses per question in AP-GRPO; not specified.
  • D_RL/D_SFT split = 10K / rest
    Arbitrary split of MME-RealWorld training set (Sec. 5).
  • Number of ROIs = 1 (implicit)
    The framework crops 'the key ROIs' but details are not given.
axioms (4)
  • domain assumption The visual information needed to answer the question is localizable to a bounding-box crop of the original image.
    Entire HART pipeline (Sec. 4.2) crops ROIs and withholds original image; if the required info is distributed beyond a box, the reward is misspecified.
  • domain assumption The model's bounding-box output can be parsed reliably into coordinates and rendered as crops.
    The training loop assumes the textual bounding box is usable for cropping without failure; not discussed.
  • domain assumption Answer correctness from crops-only is a valid reward signal for grounding.
    The core feedback loop in Sec. 4.2; pilot data show it is still misspecified 21.5% of the time for Qwen.
  • standard math Standard GRPO convergence and policy-gradient assumptions.
    Used in Sec. 3.2 and extended in Sec. 4.3.

pith-pipeline@v1.3.0-alltime-deepseek · 17479 in / 16892 out tokens · 135283 ms · 2026-08-02T20:13:39.723565+00:00 · methodology

0 comments
read the original abstract

Current Large Multimodal Models (LMMs) struggle with high-resolution visual inputs during the reasoning process, as the number of image tokens increases quadratically with resolution, introducing substantial redundancy and irrelevant information. A common practice is to identify key image regions and refer to their high-resolution counterparts during reasoning, typically trained with external visual supervision. However, such visual supervision cues require costly grounding labels from human annotators. Meanwhile, it remains an open question how to enhance a model's grounding abilities to support reasoning without relying on additional annotations. In this paper, we propose High-resolution Annotation-free Reasoning Technique (HART), a closed-loop framework that enables LMMs to focus on and self-verify key regions of high-resolution visual inputs. HART incorporates a post-training paradigm in which we design Advantage Preference Group Relative Policy Optimization (AP-GRPO) to encourage accurate localization of key regions without external visual annotations. Notably, HART provides explainable reasoning pathways and enables efficient optimization of localization. Extensive experiments on MME-RealWorld-Lite, TreeBench, V* Bench, HR-Bench-4K/8K, and MMStar demonstrate that HART improves performance across a wide range of high-resolution visual tasks, consistently outperforming strong baselines.

Figures

Figures reproduced from arXiv: 2602.23615 by Anqi Chen, Cong Wang, Feng Miao, Jiacheng Yang, Qi Fan, Wenbin Li, Yang Gao, Yunkai Dang.

Figure 1
Figure 1. Figure 1: Optimization procedures of (a) general grounding-based methods without bounding-box annotations and (b) our proposed model. General models indirectly op￾timize grounding performance, while HART performs direct optimization by answering based solely on the ROIs. Abbreviations: Q—Question; A—Answer. and visual question answering [41, 48, 53]. They demonstrate strong capabilities in visual understanding and h… view at source ↗
Figure 2
Figure 2. Figure 2: Left: An example of Qwen-2.5-VL-7B where the final answer is correct but the grounding is incorrect. Right: HART Framework. The post-training strategy consists of Reinforcement Learning (RL) and Supervised Fine-Tuning (SFT). In stage 1, after identifying the ROIs, the model answers based solely on the sub-regions and the original question. AP-GRPO is introduced to improve the model’s grounding capabilities… view at source ↗
Figure 3
Figure 3. Figure 3: Grounding performance of AP-GRPO on TreeBench [46] [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Visualization of model outputs from InternVL3-8B [66], Qwen2.5-VL-7B [3], and our method HART-7B on TreeBench [46]. 5.4 Visualization Results We present a visualized comparison of our method against prior methods in Fig￾ure 4. In this example, the task requires attending to the right-most man before executing the reasoning step. Compared with InternVL3-7B [66] and Qwen2.5- VL-7B [3], our method identifies … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. UHR-BAT: Budget-Aware Token Compression Vision-Language model for Ultra-High-Resolution Remote Sensing

    cs.CV 2026-04 unverdicted novelty 6.0

    UHR-BAT is a budget-aware framework that uses text-guided multi-scale importance estimation plus region-wise preserve and merge strategies to compress visual tokens in ultra-high-resolution remote sensing vision-langu...

Reference graph

Works this paper leans on

65 extracted references · 35 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Anthropic: Claude 3.5 sonnet.https://www.anthropic.com/news/claude-3-5- sonnet(2024) 10

    2024

  2. [2]

    In: Proceedings of the AAAI Conference on Artificial In- telligence

    Arif, K.H.I., Yoon, J., Nikolopoulos, D.S., Vandierendonck, H., John, D., Ji, B.: Hired: Attention-guided token dropping for efficient inference of high-resolution vision-language models. In: Proceedings of the AAAI Conference on Artificial In- telligence. vol. 39, pp. 1773–1781 (2025) 2

    2025

  3. [3]

    5-vl technical report

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al.: Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923 (2025) 1, 2, 3, 6, 7, 10, 11, 12, 13, 14

    Pith/arXiv arXiv 2025

  4. [4]

    arXiv preprint arXiv:2509.06948 (2025) 9

    Chen, L., Han, X., Shen, L., Bai, J., Wong, K.F.: Beyond two-stage training: Co- operative sft and rl for llm reasoning. arXiv preprint arXiv:2509.06948 (2025) 9

    Pith/arXiv arXiv 2025

  5. [5]

    Chen, L., Li, J., Dong, X., Zhang, P., Zang, Y., Chen, Z., Duan, H., Wang, J., Qiao, Y., Lin, D., et al.: Are we on the right way for evaluating large vision- language models? Advances in Neural Information Processing Systems37, 27056– 27087 (2024) 11

    2024

  6. [6]

    arXiv preprint arXiv:2412.02104 (2024) 1

    Dang, Y., Huang, K., Huo, J., Yan, Y., Huang, S., Liu, D., Gao, M., Zhang, J., Qian,C.,Wang,K.,etal.:Explainableandinterpretablemultimodallargelanguage models: A comprehensive survey. arXiv preprint arXiv:2412.02104 (2024) 1

    Pith/arXiv arXiv 2024

  7. [7]

    Advances in Neural Information Processing Systems37, 42566–42592 (2024) 2, 4

    Dong, X., Zhang, P., Zang, Y., Cao, Y., Wang, B., Ouyang, L., Zhang, S., Duan, H., Zhang, W., Li, Y., et al.: Internlm-xcomposer2-4khd: A pioneering large vision- language model handling resolutions from 336 pixels to 4k hd. Advances in Neural Information Processing Systems37, 42566–42592 (2024) 2, 4

    2024

  8. [8]

    Google DeepMind: Gemini-2.5-flash.https://deepmind.google/models/gemini/ flash/(2025) 10, 11

    2025

  9. [9]

    Google DeepMind: Gemini-2.5-pro.https://deepmind.google/models/gemini/ pro/(2025) 1, 11

    2025

  10. [10]

    Nature645(8081), 633–638 (2025) 1, 4

    Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., Bi, X., et al.: Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nature645(8081), 633–638 (2025) 1, 4

    2025

  11. [11]

    arXiv preprint arXiv:2505.12432 (2025) 4 16 Authors Suppressed Due to Excessive Length

    Guo, Z., Hong, M., Jin, T.: Observe-r1: Unlocking reasoning abilities of mllms with dynamic progressive reinforcement learning. arXiv preprint arXiv:2505.12432 (2025) 4 16 Authors Suppressed Due to Excessive Length

    Pith/arXiv arXiv 2025

  12. [12]

    In: European Conference on Computer Vision

    Guo, Z., Xu, R., Yao, Y., Cui, J., Ni, Z., Ge, C., Chua, T.S., Liu, Z., Huang, G.: Llava-uhd: an lmm perceiving any aspect ratio and high-resolution images. In: European Conference on Computer Vision. pp. 390–406. Springer (2024) 4

    2024

  13. [13]

    CoRRabs/2403.12895(2024) 4

    Hu, A., Xu, H., Ye, J., Yan, M., Zhang, L., Zhang, B., Li, C., Zhang, J., Jin, Q., Huang, F., Zhou, J.: mplug-docowl 1.5: Unified structure learning for ocr-free document understanding. CoRRabs/2403.12895(2024) 4

    Pith/arXiv arXiv 2024

  14. [14]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Huang, R., Ding, X., Wang, C., Han, J., Liu, Y., Zhao, H., Xu, H., Hou, L., Zhang, W., Liang, X.: Hires-llava: Restoring fragmentation input in high-resolution large vision-language models. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 29814–29824 (2025) 2

    2025

  15. [15]

    arXiv preprint arXiv:2507.05920 (2025) 2, 3, 4, 6, 7, 10, 11, 12, 13, 14

    Huang, X., Dong, Y., Tian, W., Li, B., Feng, R., Liu, Z.: High-resolution visual reasoning via multi-turn grounding-based reinforcement learning. arXiv preprint arXiv:2507.05920 (2025) 2, 3, 4, 6, 7, 10, 11, 12, 13, 14

    Pith/arXiv arXiv 2025

  16. [16]

    arXiv preprint arXiv:2410.21276 (2024) 1, 10, 11

    Hurst, A., Lerer, A., Goucher, A.P., Perelman, A., Ramesh, A., Clark, A., Os- trow, A., Welihinda, A., Hayes, A., Radford, A., et al.: Gpt-4o system card. arXiv preprint arXiv:2410.21276 (2024) 1, 10, 11

    Pith/arXiv arXiv 2024

  17. [17]

    arXiv preprint arXiv:2412.16720 (2024) 1

    Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Carney, A., et al.: Openai o1 system card. arXiv preprint arXiv:2412.16720 (2024) 1

    Pith/arXiv arXiv 2024

  18. [19]

    arXiv preprint arXiv:2408.03326 (2024) 10, 11, 12

    Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., et al.: Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326 (2024) 10, 11, 12

    Pith/arXiv arXiv 2024

  19. [20]

    In: proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Li, Z., Yang, B., Liu, Q., Ma, Z., Zhang, S., Yang, J., Sun, Y., Liu, Y., Bai, X.: Monkey: Image resolution and text label are important things for large multi- modal models. In: proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 26763–26773 (2024) 2

    2024

  20. [21]

    arXiv preprint arXiv:2508.19652 (2025) 4, 7

    Li, Z., Yu, W., Huang, C., Liu, R., Liang, Z., Liu, F., Che, J., Yu, D., Boyd-Graber, J., Mi, H., et al.: Self-rewarding vision-language model via reasoning decomposi- tion. arXiv preprint arXiv:2508.19652 (2025) 4, 7

    Pith/arXiv arXiv 2025

  21. [22]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Liu, C., Yin, K., Cao, H., Jiang, X., Li, X., Liu, Y., Jiang, D., Sun, X., Xu, L.: Hrvda: High-resolution visual document assistant. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 15534– 15545 (2024) 4

    2024

  22. [23]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tun- ing. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 26296–26306 (2024) 4

    2024

  23. [24]

    In: European conference on computer vision

    Liu, S., Cheng, H., Liu, H., Zhang, H., Li, F., Ren, T., Zou, X., Yang, J., Su, H., Zhu, J., et al.: Llava-plus: Learning to use tools for creating multimodal agents. In: European conference on computer vision. pp. 126–142. Springer (2024) 1

    2024

  24. [25]

    CoRRabs/2403.04473 (2024) 4

    Liu, Y., Yang, B., Liu, Q., Li, Z., Ma, Z., Zhang, S., Bai, X.: Textmonkey: An ocr- free large multimodal model for understanding document. CoRRabs/2403.04473 (2024) 4

    Pith/arXiv arXiv 2024

  25. [26]

    arXiv preprint arXiv:2503.01785 (2025) 1

    Liu, Z., Sun, Z., Zang, Y., Dong, X., Cao, Y., Duan, H., Lin, D., Wang, J.: Visual- rft: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.01785 (2025) 1

    Pith/arXiv arXiv 2025

  26. [27]

    In: The Thirteenth Inter- national Conference on Learning Representations (2025) 2, 4 Title Suppressed Due to Excessive Length 17

    Liu, Z., Dong, Y., Liu, Z., Hu, W., Lu, J., Rao, Y.: Oryx MLLM: On-demand spatial-temporal understanding at arbitrary resolution. In: The Thirteenth Inter- national Conference on Learning Representations (2025) 2, 4 Title Suppressed Due to Excessive Length 17

    2025

  27. [28]

    CoRRabs/2403.12966(2024) 2, 4, 5

    Liu, Z., Dong, Y., Rao, Y., Zhou, J., Lu, J.: Chain-of-spot: Interactive reasoning improves large vision-language models. CoRRabs/2403.12966(2024) 2, 4, 5

    Pith/arXiv arXiv 2024

  28. [29]

    CoRRabs/2404.09797 (2024) 2, 4

    Luan, B., Feng, H., Chen, H., Wang, Y., Zhou, W., Li, H.: Textcot: Zoom in for enhanced multimodal text-rich image understanding. CoRRabs/2404.09797 (2024) 2, 4

    Pith/arXiv arXiv 2024

  29. [30]

    arXiv preprint arXiv:2503.07365 (2025) 4

    Meng, F., Du, L., Liu, Z., Zhou, Z., Lu, Q., Fu, D., Han, T., Shi, B., Wang, W., He, J., et al.: Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2503.07365 (2025) 4

    Pith/arXiv arXiv 2025

  30. [31]

    IEEE Transactions on Intelligent Transporta- tion Systems26(3), 2851–2879 (2025) 6

    Miri Rekavandi, A., Xu, L., Boussaid, F., Seghouane, A.K., Hoefs, S., Bennamoun, M.: A guide to image- and video-based small object detection using deep learning: Case study of maritime surveillance. IEEE Transactions on Intelligent Transporta- tion Systems26(3), 2851–2879 (2025) 6

    2025

  31. [32]

    arXiv preprint arXiv:2505.19702 (2025) 2, 4

    Ni, M., Yang, Z., Li, L., Lin, C.C., Lin, K., Zuo, W., Wang, L.: Point-rft: Improv- ing multimodal reasoning with visually grounded reinforcement finetuning. arXiv preprint arXiv:2505.19702 (2025) 2, 4

    Pith/arXiv arXiv 2025

  32. [33]

    OpenAI: Gpt-4o mini: advancing cost-efficient intelligence.https://openai.com/ index/gpt-4o-mini-advancing-cost-efficient-intelligence/(2024) 10

    2024

  33. [34]

    com / index / introducing - o3 - and - o4 - mini/(2025) 11

    OpenAI: Openai-o3.https : / / openai . com / index / introducing - o3 - and - o4 - mini/(2025) 11

    2025

  34. [35]

    arXiv preprint arXiv:2503.07536 (2025) 4

    Peng, Y., Zhang, G., Zhang, M., You, Z., Liu, J., Zhu, Q., Yang, K., Xu, X., Geng, X., Yang, X.: Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl. arXiv preprint arXiv:2503.07536 (2025) 4

    Pith/arXiv arXiv 2025

  35. [36]

    Ptito, M., Bleau, M., Bouskila, J.: The retina: a window into the brain (2021) 2

    2021

  36. [37]

    Advances in Neural Information Processing Systems37, 8612–8642 (2024) 2, 4, 6, 12, 13

    Shao, H., Qian, S., Xiao, H., Song, G., Zong, Z., Wang, L., Liu, Y., Li, H.: Vi- sual cot: Advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning. Advances in Neural Information Processing Systems37, 8612–8642 (2024) 2, 4, 6, 12, 13

    2024

  37. [38]

    arXiv preprint arXiv:2402.03300 (2024) 5, 8, 10, 11, 12, 13, 14

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al.: Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 (2024) 5, 8, 10, 11, 12, 13, 14

    Pith/arXiv arXiv 2024

  38. [39]

    arXiv preprint arXiv:2504.07615 (2025) 2, 4

    Shen, H., Liu, P., Li, J., Fang, C., Ma, Y., Liao, J., Shen, Q., Zhang, Z., Zhao, K., Zhang, Q., et al.: Vlm-r1: A stable and generalizable r1-style large vision-language model. arXiv preprint arXiv:2504.07615 (2025) 2, 4

    Pith/arXiv arXiv 2025

  39. [40]

    arXiv preprint arXiv:2505.15966 (2025) 2, 4, 10, 11

    Su, A., Wang, H., Ren, W., Lin, F., Chen, W.: Pixel reasoner: Incentivizing pixel-space reasoning with curiosity-driven reinforcement learning. arXiv preprint arXiv:2505.15966 (2025) 2, 4, 10, 11

    Pith/arXiv arXiv 2025

  40. [41]

    arXiv preprint arXiv:2408.08862 (2024) 2

    Sun, G., Jin, M., Wang, Z., Wang, C.L., Ma, S., Wang, Q., Geng, T., Wu, Y.N., Zhang, Y., Liu, D.: Visual agents as fast and slow thinkers. arXiv preprint arXiv:2408.08862 (2024) 2

    Pith/arXiv arXiv 2024

  41. [42]

    Sutton, R.S., Barto, A.G., et al.: Reinforcement learning: An introduction, vol. 1. MIT press Cambridge (1998) 2

    1998

  42. [43]

    arXiv preprint arXiv:2503.20752 (2025) 4

    Tan, H., Ji, Y., Hao, X., Lin, M., Wang, P., Wang, Z., Zhang, S.: Reason-rft: Reinforcement fine-tuning for visual reasoning. arXiv preprint arXiv:2503.20752 (2025) 4

    arXiv 2025

  43. [44]

    arXiv preprint arXiv:2403.05530 (2024) 10

    Team, G., Georgiev, P., Lei, V.I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vin- cent,D.,Pan,Z.,Wang,S.,etal.:Gemini1.5:Unlockingmultimodalunderstanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 (2024) 10

    Pith/arXiv arXiv 2024

  44. [45]

    CoRR (2025) 4 18 Authors Suppressed Due to Excessive Length

    Thawakar, O., Dissanayake, D., More, K., Thawkar, R., Heakl, A., Ahsan, N., Li, Y., Zumri, M., Lahoud, J., Anwer, R.M., et al.: Llamav-o1: Rethinking step-by-step visual reasoning in llms. CoRR (2025) 4 18 Authors Suppressed Due to Excessive Length

    2025

  45. [46]

    arXiv preprint arXiv:2507.07999 (2025) 2, 3, 4, 6, 10, 11, 12, 13, 14

    Wang, H., Li, X., Huang, Z., Wang, A., Wang, J., Zhang, T., Zheng, J., Bai, S., Kang, Z., Feng, J., et al.: Traceable evidence enhanced visual grounded reasoning: Evaluation and methodology. arXiv preprint arXiv:2507.07999 (2025) 2, 3, 4, 6, 10, 11, 12, 13, 14

    arXiv 2025

  46. [47]

    arXiv preprint arXiv:2504.08837 (2025) 4

    Wang, H., Qu, C., Huang, Z., Chu, W., Lin, F., Chen, W.: Vl-rethinker: Incen- tivizing self-reflection of vision-language models with reinforcement learning. arXiv preprint arXiv:2504.08837 (2025) 4

    Pith/arXiv arXiv 2025

  47. [48]

    arXiv preprint arXiv:2409.12191 (2024) 2

    Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al.: Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191 (2024) 2

    Pith/arXiv arXiv 2024

  48. [49]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Wang, W., Ding, L., Zeng, M., Zhou, X., Shen, L., Luo, Y., Yu, W., Tao, D.: Divide, conquer and combine: A training-free framework for high-resolution image perception in multimodal large language models. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 39, pp. 7907–7915 (2025) 3, 11

    2025

  49. [50]

    CoRRabs/2503.12605(March

    Wang, Y., Wu, S., Zhang, Y., Yan, S., Liu, Z., Luo, J., Fei, H.: Multimodal chain- of-thought reasoning: A comprehensive survey. CoRRabs/2503.12605(March

    Pith/arXiv arXiv

  50. [51]

    https://github.com/huggingface/trl(2020) 10

    von Werra, L., Belkada, Y., Tunstall, L., Beeching, E., Thrush, T., Lambert, N., Huang, S., Rasul, K., Gallouédec, Q.: Trl: Transformer reinforcement learning. https://github.com/huggingface/trl(2020) 10

    2020

  51. [52]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Wu, P., Xie, S.: V?: Guided visual search as a core mechanism in multimodal llms. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13084–13094 (2024) 3, 11

    2024

  52. [53]

    arXiv preprint arXiv:2412.10302 (2024) 2

    Wu,Z.,Chen,X.,Pan,Z.,Liu,X.,Liu,W.,Dai,D.,Gao,H.,Ma,Y.,Wu,C.,Wang, B., et al.: Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302 (2024) 2

    Pith/arXiv arXiv 2024

  53. [54]

    arXiv preprint arXiv:2503.10615 (2025) 4

    Yang, Y., He, X., Pan, H., Jiang, X., Deng, Y., Yang, X., Lu, H., Yin, D., Rao, F., Zhu, M., et al.: R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization. arXiv preprint arXiv:2503.10615 (2025) 4

    Pith/arXiv arXiv 2025

  54. [55]

    CoRRabs/2408.01800(2024) 4

    Yao, Y., Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., Cai, T., Li, H., Zhao, W., He, Z., Chen, Q., Zhou, H., Zou, Z., Zhang, H., Hu, S., Zheng, Z., Zhou, J., Cai, J., Han, X., Zeng, G., Li, D., Liu, Z., Sun, M.: Minicpm-v: A gpt-4v level mllm on your phone. CoRRabs/2408.01800(2024) 4

    Pith/arXiv arXiv 2024

  55. [56]

    arXiv preprint arXiv:2408.04840 (2024) 4

    Ye, J., Xu, H., Liu, H., Hu, A., Yan, M., Qian, Q., Zhang, J., Huang, F., Zhou, J.: mplug-owl3: Towards long image-sequence understanding in multi-modal large language models. arXiv preprint arXiv:2408.04840 (2024) 4

    Pith/arXiv arXiv 2024

  56. [57]

    Pattern Recognition155, 110714 (2024) 6

    Yu, Z., Huang, H., Chen, W., Su, Y., Liu, Y., Wang, X.: Yolo-facev2: A scale and occlusion aware face detector. Pattern Recognition155, 110714 (2024) 6

    2024

  57. [58]

    In: Proceedings of the IEEE/CVF International Conference on Com- puter Vision

    Zhan, Y., Zheng, S., Zhu, Y., Zhao, H., Yang, F., Tang, M., Wang, J.: Griffon v2: Advancing multimodal perception with high-resolution scaling and visual-language co-referring. In: Proceedings of the IEEE/CVF International Conference on Com- puter Vision. pp. 22947–22957 (2025) 2, 4

    2025

  58. [59]

    CoRR (2025) 4

    Zhan, Y., Zhu, Y., Zheng, S., Zhao, H., Yang, F., Tang, M., Wang, J.: Vision-r1: Evolving human-free alignment in large vision-language models via vision-guided reinforcement learning. CoRR (2025) 4

    2025

  59. [60]

    CoRRabs/2503.12937(March 2025) 4

    Zhang, J., Huang, J., Yao, H., Liu, S., Zhang, X., Lu, S., Tao, D.: R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization. CoRRabs/2503.12937(March 2025) 4

    Pith/arXiv arXiv 2025

  60. [61]

    In: Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision

    Zhang, R., Shao, R., Chen, G., Zhang, M., Zhou, K., Guan, W., Nie, L.: Falcon: Re- solving visual redundancy and fragmentation in high-resolution multimodal large Title Suppressed Due to Excessive Length 19 language models via visual registers. In: Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision. pp. 23530–23540 (2025)

    2025

  61. [62]

    CoRRabs/2406.08487 (2024) 2, 4

    Zhang,Y.F.,Wen,Q.,Fu,C.,Wang,X.,Zhang,Z.,Wang,L.,Jin,R.:Beyondllava- hd: Diving into high-resolution large multimodal models. CoRRabs/2406.08487 (2024) 2, 4

    Pith/arXiv arXiv 2024

  62. [63]

    Zhang, Y.F., Zhang, H., Tian, H., Fu, C., Zhang, S., Wu, J., Li, F., Wang, K., Wen, Q., Zhang, Z., et al.: Mme-realworld: Could your multimodal llm challenge high-resolution real-world scenarios that are difficult for humans? arXiv preprint arXiv:2408.13257 (2024) 3, 9, 10, 12, 13

    Pith/arXiv arXiv 2024

  63. [64]

    CoRRabs/2504.18397(April 2025) 2, 4, 5

    Zhao, K., Zhu, B., Sun, Q., Zhang, H.: Unsupervised visual chain-of-thought rea- soning via preference optimization. CoRRabs/2504.18397(April 2025) 2, 4, 5

    Pith/arXiv arXiv 2025

  64. [65]

    thinking with images

    Zheng, Z., Yang, M., Hong, J., Zhao, C., Xu, G., Yang, L., Shen, C., Yu, X.: Deepeyes: Incentivizing" thinking with images" via reinforcement learning. arXiv preprint arXiv:2505.14362 (2025) 2, 3, 4, 6, 7, 10, 11

    Pith/arXiv arXiv 2025

  65. [66]

    arXiv preprint arXiv:2504.10479 (2025) 2, 3, 6, 7, 10, 11, 12, 14

    Zhu, J., Wang, W., Chen, Z., Liu, Z., Ye, S., Gu, L., Tian, H., Duan, Y., Su, W., Shao, J., et al.: Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479 (2025) 2, 3, 6, 7, 10, 11, 12, 14

    Pith/arXiv arXiv 2025