Pith. sign in

REVIEW 1 major objections 1 minor 1 cited by

SSL-R1: Self-Supervised Visual Reinforcement Post-Training for Multimodal Large Language Models

T0 review · 1 major / 1 minor · reviewed 2026-05-10 · grok-4.3

Challenge this review Download PDF Read on arXiv ↗

Pith's one-line read Reformulating visual self-supervised tasks as verifiable puzzles supplies automatic rewards for reinforcement learning post-training of multimodal language models.

desk verdict SSL-R1 maps visual SSL tasks to RL puzzles for MLLM post-training without labels, but the abstract alone leaves the generalization claims untested. read the letter →

arxiv 2604.20705 v1 submitted 2026-04-22 cs.CV

Jiahao Xie , Alessio Tonioni , Nathalie Rauschmayr , Federico Tombari , Bernt Schiele This is my paper
classification cs.CV
keywords self-supervisedlearningreinforcementmultimodallargelanguagemodelsvisualpuzzlesverifiablerewardspost-trainingimageunderstanding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper presents SSL-R1, a framework that converts standard visual self-supervised learning tasks into puzzles whose solutions can be checked automatically from the image itself. These puzzles then supply the reward signal for reinforcement learning that refines multimodal large language models after their initial training. The goal is to strengthen the models' native visual understanding and reasoning without depending on human labels or separate language-based supervision. If successful, the method would allow post-training to scale using only abundant unlabeled images while reducing the dominance of language-centric priors in current pipelines.

What carries the argument

Reformulation of visual SSL tasks into verifiable puzzles that yield image-derived rewards for reinforcement learning.

What would settle it

Train an MLLM with SSL-R1 and measure its accuracy on held-out multimodal benchmarks such as visual question answering or reasoning tasks; no improvement or a drop relative to a standard baseline would falsify the central claim.

Watch

Extended reading notes

Core claim

SSL-R1 reformulates widely used visual self-supervised tasks into a collection of verifiable visual puzzles. These puzzles generate rewards directly from image data for RL post-training of MLLMs, requiring neither human annotations nor external model supervision. Models trained under this regime show substantial gains on multimodal understanding and reasoning benchmarks.

Load-bearing premise

Rewards obtained by solving these visual puzzles will strengthen the model's general visual understanding and reasoning instead of merely teaching it to solve the specific puzzles.

Editorial extensions

If this is right

  • MLLMs exhibit measurable gains on multimodal understanding and reasoning benchmarks after training on the visual puzzles.
  • RL post-training becomes feasible at larger scales because rewards no longer require human or external model supervision.
  • Vision-centric self-supervised signals can be used to counteract language-centric biases in MLLM training.
  • The framework supplies concrete experience for designing additional self-supervised verifiable rewards.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The method could be extended to video or 3D data by turning temporal or geometric SSL tasks into similarly verifiable puzzles.
  • Combining these visual rewards with existing language-based RLVR signals might produce hybrid training regimes that balance modalities more evenly.
  • Models refined this way may display improved transfer to downstream visual tasks that were never used as training puzzles.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 1 minor

Summary. The manuscript introduces SSL-R1, a self-supervised RL post-training framework for MLLMs. It reformulates standard visual SSL tasks (e.g., rotation prediction, jigsaw) into verifiable image-based puzzles that generate rewards directly from intrinsic image properties, without human annotations or external models. The central claim is that training MLLMs via RL on these tasks substantially improves performance on multimodal understanding and reasoning benchmarks.

Significance. If the gains prove generalizable rather than puzzle-specific, the work would meaningfully advance scalable RLVR for MLLMs by shifting reward design to vision-centric self-supervision. The approach correctly identifies the annotation bottleneck in prior RLVR methods and proposes a concrete alternative using existing SSL primitives.

major comments (1)
  1. [Experiments] Experiments section: The central claim that RL post-training on the reformulated puzzles produces transferable visual reasoning (rather than puzzle-format optimization) is load-bearing. The manuscript must include an ablation comparing full SSL-R1 RL against supervised fine-tuning on identical puzzle data; without it, benchmark gains could be explained by task exposure alone. The skeptic's concern is therefore a correctness risk that requires a concrete control experiment.
minor comments (1)
  1. [Abstract] Abstract: The phrase 'widely-used SSL tasks' should explicitly name the tasks (rotation, jigsaw, inpainting, etc.) and the exact reformulation into question-answer format for immediate clarity.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the positive assessment of the work and for the constructive major comment. We agree that the requested ablation is necessary to strengthen the central claim and will incorporate it in the revised manuscript.

read point-by-point responses

  1. Referee: [Experiments] Experiments section: The central claim that RL post-training on the reformulated puzzles produces transferable visual reasoning (rather than puzzle-format optimization) is load-bearing. The manuscript must include an ablation comparing full SSL-R1 RL against supervised fine-tuning on identical puzzle data; without it, benchmark gains could be explained by task exposure alone. The skeptic's concern is therefore a correctness risk that requires a concrete control experiment.

    Authors: We agree that this control experiment is essential to rule out the possibility that gains arise merely from exposure to the puzzle formats rather than from the RL optimization itself. In the revised manuscript we will add a direct comparison of SSL-R1 (RL post-training with verifiable rewards) against supervised fine-tuning on the identical set of puzzle instances, using the same verifiable ground-truth answers as supervision targets. This ablation will be reported alongside the existing results in the Experiments section, with details on training hyperparameters and evaluation to ensure fair comparison. We believe the new results will further substantiate that the RL stage yields transferable visual reasoning improvements beyond supervised task exposure. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity in the claimed derivation chain

full rationale

The paper introduces SSL-R1 by reformulating standard visual SSL tasks (e.g., rotation, jigsaw) into verifiable image-based puzzles whose rewards are computed directly from intrinsic image properties without human or external model labels. This reformulation and the subsequent RL post-training step constitute an independent methodological contribution; benchmark gains are reported as empirical outcomes rather than quantities derived by construction from fitted parameters or prior self-citations. No load-bearing uniqueness theorems, ansatzes, or self-referential definitions appear in the provided text, and the central premise does not reduce to renaming or tautological prediction of its own inputs.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

Only the abstract is available; no specific free parameters, axioms, or invented entities are identifiable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SSL-R1: Self-Supervised Visual Reinforcement Post-Training for Multimodal Large Language Models." pith.science (2026). https://pith.science/paper/2604.20705

@misc{pith2026260420705,
  author       = {Pith},
  title        = {Pith review of: SSL-R1: Self-Supervised Visual Reinforcement Post-Training for Multimodal Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2604.20705}},
  note         = {Machine review of arXiv:2604.20705}
}
read the original abstract

Reinforcement learning (RL) with verifiable rewards (RLVR) has demonstrated the great potential of enhancing the reasoning abilities in multimodal large language models (MLLMs). However, the reliance on language-centric priors and expensive manual annotations prevents MLLMs' intrinsic visual understanding and scalable reward designs. In this work, we introduce SSL-R1, a generic self-supervised RL framework that derives verifiable rewards directly from images. To this end, we revisit self-supervised learning (SSL) in visual domains and reformulate widely-used SSL tasks into a set of verifiable visual puzzles for RL post-training, requiring neither human nor external model supervision. Training MLLMs on these tasks substantially improves their performance on multimodal understanding and reasoning benchmarks, highlighting the potential of leveraging vision-centric self-supervised tasks for MLLM post-training. We think this work will provide useful experience in devising effective self-supervised verifiable rewards to enable RL at scale. Project page: https://github.com/Jiahao000/SSL-R1.

Figures

Figures reproduced from arXiv: 2604.20705 by the authors.

Figure 1
Figure 1. (a) Existing reinforcement learning with verifiable rewards (RLVR) for post-training MLLMs are supervised, requiring a large volume of high-quality language-centric multimodal data with human annotations, which is very expensive and unsustainable. (b) We propose SSL-R1, a generic self-supervised RLVR-based post-training framework that derives intrinsically verifiable rewards from input images, requiring neither huma… view at source ↗
Figure 2
Figure 2. Overview of our SSL-R1 tasks. We design five verifiable self-supervised tasks for RL post-training, ranging from the image level to the pixel level. Rotation Prediction: an image is rotated by a certain angle, and the model is tasked with predicting the angle. Visual Similarity: two augmented views are cropped from an image, with several additional views from other images, and the task is to select the most visually… view at source ↗
Figure 4
Figure 4. The training dynamics of SSL-R1. Left: Single-task rewards. Right: Multi-task rewards. All curves are exponentially smoothed for visualization. global spatial layout transfers to downstream tasks broadly. Rotation Prediction achieves the strongest performance im￾provement on MMVP (+9.34%), indicating that identify￾ing image orientations helps understand CLIP-blind visual patterns. Visual Similarity shines on general… view at source ↗
Figures from the paper (7 more)
Figure 5
Figure 5. Figure 5: Prompt templates for five self-supervised tasks. A format prompt (bottom) is appended at the end of each task prompt to enable the reasoning process. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Examples of the Rotation Prediction task. The ground-truth answer for each example is provided at the bottom. A B C D A B Answer: C Answer: D C D A B Answer: B C D Select the candidate image that is most visually similar to the reference image Visual Similarity [PITH_…
Figure 7
Figure 7. Figure 7: Examples of the Visual Similarity task. The ground-truth answer for each example is provided at the bottom. C D A B Answer: D A B C D Answer: A A C B D Answer: B Select the candidate image that best fits the missing part of the original image Region Inpainting [PITH_F…
Figure 8
Figure 8. Figure 8: Examples of the Region Inpainting task. The ground-truth answer for each example is provided at the bottom. 1 2 3 4 5 6 7 8 9 Answer: 3, 6, 7, 2, 1, 8, 9, 5, 4 1 2 3 4 5 6 7 8 9 Answer: 7, 6, 9, 2, 4, 8, 3, 5, 1 1 2 3 4 5 6 7 8 9 Answer: 6, 4, 7, 5, 9, 2, 8, 1, 3 Arran…
Figure 9
Figure 9. Figure 9: Examples of the Patch Ordering task. The ground-truth answer for each example is provided at the bottom. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Examples of the Geometric Correspondence task. The ground-truth answer for each example is provided at the bottom. Q: Does the caption describe the image correctly? Caption: A photograph of a mother goat sitting while a baby goat is standing in a field. Answer Yes or …
Figure 11
Figure 11. Figure 11: Qualitative examples on three types of vision-centric multimodal benchmarks. The wrong answers are marked in red while the correct answers are marked in green. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Failure-Informed Image Self-Augmentation for Multimodal Large Language Model Self-Improvement

    cs.AI 2026-08 conditional novelty 5.0 of 10

    FISA improves MLLM visual question answering by generating and filtering augmented images based on the model's own failure cases.

Reference graph

Works this paper leans on

88 extracted references · 88 canonical work pages · cited by 1 Pith paper

  1. [1]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. InNeurIPS,

    work page

  2. [2]

    Self-supervised learning from images with a joint-embedding predictive architecture

    Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bo- janowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. InCVPR, 2023. 2

    work page 2023

  3. [3]

    Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities.arXiv preprint arXiv:2308.12966, 2023. 1

    work page Pith review arXiv 2023

  4. [4]

    Qwen2.5-VL Technical Report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025. 1, 5, 7, 8

    work page Pith review arXiv 2025

  5. [5]

    Beit: Bert pre-training of image transformers

    Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. InICLR, 2022. 2

    work page 2022

  6. [6]

    Lan- guage models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners. InNeurIPS, 2020. 1, 2

    work page 2020

  7. [7]

    Deep clustering for unsupervised learning of visual features

    Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. InECCV, 2018. 2

    work page 2018

  8. [8]

    Unsupervised learn- ing of visual features by contrasting cluster assignments

    Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Pi- otr Bojanowski, and Armand Joulin. Unsupervised learn- ing of visual features by contrasting cluster assignments. In NeurIPS, 2020

    work page 2020

Show all 88 references
  1. [9]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In ICCV, 2021. 2

    2021

  2. [10]

    Are we on the right way for evaluating large vision-language models? InNeurIPS, 2024

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? InNeurIPS, 2024. 2, 5

    2024

  3. [11]

    Generative pre- training from pixels

    Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Hee- woo Jun, David Luan, and Ilya Sutskever. Generative pre- training from pixels. InICML, 2020. 2

    2020

  4. [12]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey Hinton. A simple framework for contrastive learning of visual representations. InICML, 2020. 2, 5

    2020

  5. [13]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. InCVPR,

  6. [14]

    Instructblip: Towards general-purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. InNeurIPS, 2023. 1

    2023

  7. [15]

    Bert: Pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding. InNAACL, 2019. 2

    2019

  8. [16]

    Unsuper- vised visual representation learning by context prediction

    Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsuper- vised visual representation learning by context prediction. In ICCV, 2015. 2

    2015

  9. [17]

    The 9 llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The 9 llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    arXiv

  10. [18]

    Sugarcrepe++ dataset: Vision-language model sensitivity to semantic and lexical alterations

    Sri Harsha Dumpala, Aman Jaiswal, Chandramouli Shama Sastry, Evangelos Milios, Sageev Oore, and Hassan Sajjad. Sugarcrepe++ dataset: Vision-language model sensitivity to semantic and lexical alterations. InNeurIPS,

  11. [19]

    Eva: Exploring the limits of masked visual representa- tion learning at scale

    Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representa- tion learning at scale. InCVPR, 2023. 2

    2023

  12. [20]

    Un- supervised representation learning by predicting image rota- tions

    Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Un- supervised representation learning by predicting image rota- tions. InICLR, 2018. 2

    2018

  13. [21]

    Bootstrap your own latent: A new approach to self-supervised learning

    Jean-Bastien Grill, Florian Strub, Florent Altch ´e, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Do- ersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Moham- mad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. InNe...

    2020

  14. [22]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025. 1, 2, 3

    2025 arXiv

  15. [23]

    Ssl4rl: Revisiting self-supervised learning as intrinsic reward for visual-language reasoning

    Xiaojun Guo, Runyu Zhou, Yifei Wang, Qi Zhang, Chen- heng Zhang, Stefanie Jegelka, Xiaohan Wang, Jiajun Chai, Guojun Yin, Wei Lin, et al. Ssl4rl: Revisiting self-supervised learning as intrinsic reward for visual-language reasoning. arXiv preprint arXiv:2510.16416, 2025. 2

    2025

  16. [24]

    Can mllms reason in multimodality? emma: An enhanced multimodal reasoning benchmark

    Yunzhuo Hao, Jiawei Gu, Huichen Will Wang, Linjie Li, Zhengyuan Yang, Lijuan Wang, and Yu Cheng. Can mllms reason in multimodality? emma: An enhanced multimodal reasoning benchmark. InICML, 2025. 9

    2025

  17. [25]

    Momentum contrast for unsupervised visual rep- resentation learning

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual rep- resentation learning. InCVPR, 2020. 2

    2020

  18. [26]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. InCVPR, 2022. 2

    2022

  19. [27]

    Vision-r1: Incentivizing reasoning capability in multimodal large language models.arXiv preprint arXiv:2503.06749,

    Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-r1: Incentivizing reasoning capability in multimodal large language models.arXiv preprint arXiv:2503.06749,

  20. [28]

    Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richard- son, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024. 2

    2024 arXiv

  21. [29]

    Omnispatial: Towards comprehensive spatial reasoning benchmark for vi- sion language models.arXiv preprint arXiv:2506.03135,

    Mengdi Jia, Zekun Qi, Shaochen Zhang, Wenyao Zhang, Xinqiang Yu, Jiawei He, He Wang, and Li Yi. Omnispatial: Towards comprehensive spatial reasoning benchmark for vi- sion language models.arXiv preprint arXiv:2506.03135,

  22. [30]

    Lisa: Reasoning segmenta- tion via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmenta- tion via large language model. InCVPR, 2024. 2, 5

    2024

  23. [31]

    Tulu 3: Pushing frontiers in open language model post- training.arXiv preprint arXiv:2411.15124, 2024

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post- training.arXiv preprint arXiv:2411.15124, 2024. 1

    2024

  24. [32]

    Otter: A multi-modal model with in-context instruction tuning.arXiv preprint arXiv:2305.03726, 2023

    Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning.arXiv preprint arXiv:2305.03726, 2023. 1

    2023

  25. [33]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. InICML,

  26. [34]

    Correlational image modeling for self-supervised visual pre-training

    Wei Li, Jiahao Xie, and Chen Change Loy. Correlational image modeling for self-supervised visual pre-training. In CVPR, 2023. 2

    2023

  27. [35]

    Improved visual-spatial reasoning via r1-zero-like training.arXiv preprint arXiv:2504.00883, 2025

    Zhenyi Liao, Qingsong Xie, Yanhao Zhang, Zijian Kong, Haonan Lu, Zhenyu Yang, and Zhijie Deng. Improved visual-spatial reasoning via r1-zero-like training.arXiv preprint arXiv:2504.00883, 2025. 2

    2025

  28. [36]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. 5

    2014

  29. [37]

    Visual spatial reasoning.TACL, 2023

    Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning.TACL, 2023. 2, 5

    2023

  30. [38]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InNeurIPS, 2023. 1

    2023

  31. [39]

    Mmbench: Is your multi-modal model an all-around player? InECCV, 2024

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? InECCV, 2024. 2, 5

    2024

  32. [40]

    Spatial-ssrl: Enhancing spatial understanding via self-supervised reinforcement learning.arXiv preprint arXiv:2510.27606, 2025

    Yuhong Liu, Beichen Zhang, Yuhang Zang, Yuhang Cao, Long Xing, Xiaoyi Dong, Haodong Duan, Dahua Lin, and Jiaqi Wang. Spatial-ssrl: Enhancing spatial understanding via self-supervised reinforcement learning.arXiv preprint arXiv:2510.27606, 2025. 2

    2025

  33. [41]

    Visual- rft: Visual reinforcement fine-tuning

    Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual- rft: Visual reinforcement fine-tuning. InICCV, 2025. 2

    2025

  34. [42]

    Deepseek-vl: towards real-world vision- language understanding.arXiv preprint arXiv:2403.05525,

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, et al. Deepseek-vl: towards real-world vision- language understanding.arXiv preprint arXiv:2403.05525,

  35. [43]

    Mathvista: Evaluating mathemat- ical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathemat- ical reasoning of foundation models in visual contexts. In ICLR, 2024. 9

    2024

  36. [44]

    Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforce- ment learning.arXiv preprint arXiv:2503.07365, 2025

    Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Tiancheng Han, Botian Shi, Wenhai Wang, Junjun He, et al. Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforce- ment learning.arXiv preprint arXiv:2503.07365, 2025. 1, 2

    2025

  37. [45]

    The llama 4 herd: The beginning of a new era of natively multimodal ai innovation.https://ai

    AI Meta. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation.https://ai. meta. 10 com/blog/llama-4-multimodal-intelligence/, checked on, 4 (7):2025, 2025. 1

    2025

  38. [46]

    Unsupervised learning of visual representations by solving jigsaw puzzles

    Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. InECCV,

  39. [47]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    OpenAI. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023. 1

    2023 arXiv

  40. [48]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 2

    2023 arXiv

  41. [49]

    Training lan- guage models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Car- roll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training lan- guage models to follow instructions with human feedback. InNeurIPS, 2022. 3

    2022

  42. [50]

    Context encoders: Feature learning by inpainting

    Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. InCVPR, 2016. 2

    2016

  43. [51]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. InICML, 2021. 2

    2021

  44. [52]

    Proximal policy optimization algo- rithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Rad- ford, and Oleg Klimov. Proximal policy optimization algo- rithms.arXiv preprint arXiv:1707.06347, 2017. 3

    2017 arXiv

  45. [53]

    Deepseekmath: Pushing the limits of math- ematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of math- ematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024. 2, 3

    2024 arXiv

  46. [54]

    Dinov3.arXiv preprint arXiv:2508.10104, 2025

    Oriane Sim ´eoni, Huy V V o, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Micha ¨el Ramamonjisoa, et al. Dinov3.arXiv preprint arXiv:2508.10104, 2025. 2

    2025 arXiv

  47. [55]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text.arXiv preprint arXiv:2403.05530, 2024

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text.arXiv preprint arXiv:2403.05530, 2024. 1

    2024

  48. [56]

    Gemma 3 technical report.arXiv preprint arXiv:2503.19786, 2025

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Ta- tiana Matejovicova, Alexandre Ram ´e, Morgane Rivi `ere, et al. Gemma 3 technical report.arXiv preprint arXiv:2503.19786, 2025. 1

    2025 arXiv

  49. [57]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms.arXiv preprint arXiv:2501.12599, 2025. 1

    2025

  50. [58]

    Winoground: Probing vision and language models for visio- linguistic compositionality

    Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio- linguistic compositionality. InCVPR, 2022. 2, 6

    2022

  51. [59]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. InCVPR, 2024. 2, 5

    2024

  52. [60]

    Siglip 2: Multilingual vision-language en- coders with improved semantic understanding, localization, and dense features.arXiv preprint arXiv:2502.14786, 2025

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muham- mad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language en- coders with improved semantic understanding, localization, ...

    2025

  53. [61]

    Pixel reasoner: Incentivizing pixel-space reasoning with curiosity-driven reinforcement learning

    Haozhe Wang, Alex Su, Weiming Ren, Fangzhen Lin, and Wenhu Chen. Pixel reasoner: Incentivizing pixel-space reasoning with curiosity-driven reinforcement learning. In NeurIPS, 2025. 2

    2025

  54. [62]

    Mea- suring multimodal mathematical reasoning with math-vision dataset

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Mea- suring multimodal mathematical reasoning with math-vision dataset. InNeurIPS, 2024. 9

    2024

  55. [63]

    Divide, conquer and combine: A training-free framework for high-resolution image perception in multimodal large language models

    Wenbin Wang, Liang Ding, Minyan Zeng, Xiabin Zhou, Li Shen, Yong Luo, Wei Yu, and Dacheng Tao. Divide, conquer and combine: A training-free framework for high-resolution image perception in multimodal large language models. In AAAI, 2025. 2, 5

    2025

  56. [64]

    Llava-critic-r1: Your critic model is secretly a strong policy model.arXiv preprint arXiv:2509.00676, 2025

    Xiyao Wang, Chunyuan Li, Jianwei Yang, Kai Zhang, Bo Liu, Tianyi Xiong, and Furong Huang. Llava-critic-r1: Your critic model is secretly a strong policy model.arXiv preprint arXiv:2509.00676, 2025. 2, 5, 7

    2025

  57. [65]

    Vicrit: A verifiable rein- forcement learning proxy task for visual perception in vlms

    Xiyao Wang, Zhengyuan Yang, Chao Feng, Yongyuan Liang, Yuhang Zhou, Xiaoyu Liu, Ziyi Zang, Ming Li, Chung-Ching Lin, Kevin Lin, et al. Vicrit: A verifiable rein- forcement learning proxy task for visual perception in vlms. InNeurIPS, 2025. 2

    2025

  58. [66]

    Sota with less: Mcts-guided sample selection for data-efficient visual reasoning self-improvement.arXiv preprint arXiv:2504.07934, 2025

    Xiyao Wang, Zhengyuan Yang, Chao Feng, Hongjin Lu, Lin- jie Li, Chung-Ching Lin, Kevin Lin, Furong Huang, and Li- juan Wang. Sota with less: Mcts-guided sample selection for data-efficient visual reasoning self-improvement.arXiv preprint arXiv:2504.07934, 2025. 1, 2, 5, 7, 8

    2025

  59. [67]

    Jigsaw-r1: A study of rule-based visual reinforcement learning with jigsaw puz- zles.arXiv preprint arXiv:2505.23590, 2025

    Zifu Wang, Junyi Zhu, Bo Tang, Zhiyu Li, Feiyu Xiong, Jiaqian Yu, and Matthew B Blaschko. Jigsaw-r1: A study of rule-based visual reinforcement learning with jigsaw puz- zles.arXiv preprint arXiv:2505.23590, 2025. 2

    2025

  60. [68]

    Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base llms.arXiv preprint arXiv:2506.14245, 2025

    Xumeng Wen, Zihan Liu, Shun Zheng, Shengyu Ye, Zhirong Wu, Yang Wang, Zhijian Xu, Xiao Liang, Junjie Li, Ziming Miao, et al. Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base llms.arXiv preprint arXiv:2506.14245, 2025. 2

    2025

  61. [69]

    V?: Guided visual search as a core mechanism in multimodal llms

    Penghao Wu and Saining Xie. V?: Guided visual search as a core mechanism in multimodal llms. InCVPR, 2024. 2, 5

    2024

  62. [70]

    Visual jigsaw post-training improves mllms

    Penghao Wu, Yushan Zhang, Haiwen Diao, Bo Li, Lewei Lu, and Ziwei Liu. Visual jigsaw post-training improves mllms. arXiv preprint arXiv:2509.25190, 2025. 2, 5, 7, 8, 9

    2025

  63. [71]

    Mimo-vl technical report.arXiv preprint arXiv:2506.03569, 2025

    LLM-Core-Team Xiaomi. Mimo-vl technical report.arXiv preprint arXiv:2506.03569, 2025. 9

    2025

  64. [72]

    Unsupervised object-level representation learning from scene images

    Jiahao Xie, Xiaohang Zhan, Ziwei Liu, Yew Soon Ong, and Chen Change Loy. Unsupervised object-level representation learning from scene images. InNeurIPS, 2021. 2

    2021

  65. [73]

    Delving into inter-image invariance for unsupervised visual representations.IJCV, 2022

    Jiahao Xie, Xiaohang Zhan, Ziwei Liu, Yew-Soon Ong, and Chen Change Loy. Delving into inter-image invariance for unsupervised visual representations.IJCV, 2022. 11

    2022

  66. [74]

    Masked frequency modeling for self-supervised visual pre-training

    Jiahao Xie, Wei Li, Xiaohang Zhan, Ziwei Liu, Yew Soon Ong, and Chen Change Loy. Masked frequency modeling for self-supervised visual pre-training. InICLR, 2023. 2

    2023

  67. [75]

    Depth any- thing v2

    Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiao- gang Xu, Jiashi Feng, and Hengshuang Zhao. Depth any- thing v2. InNeurIPS, 2024. 2, 5

    2024

  68. [76]

    How to evaluate the generalization of detection? a bench- mark for comprehensive open-vocabulary detection

    Yiyang Yao, Peng Liu, Tiancheng Zhao, Qianqian Zhang, Jiajia Liao, Chunxin Fang, Kyusong Lee, and Qing Wang. How to evaluate the generalization of detection? a bench- mark for comprehensive open-vocabulary detection. In AAAI, 2024. 2, 5

    2024

  69. [77]

    mplug-owl: Modularization empowers large language models with multimodality.arXiv preprint arXiv:2304.14178, 2023

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality.arXiv preprint arXiv:2304.14178, 2023. 1

    2023

  70. [78]

    Perception-r1: Pioneering perception policy with reinforcement learning.arXiv preprint arXiv:2504.07954,

    En Yu, Kangheng Lin, Liang Zhao, Jisheng Yin, Yana Wei, Yuang Peng, Haoran Wei, Jianjian Sun, Chunrui Han, Zheng Ge, et al. Perception-r1: Pioneering perception policy with reinforcement learning.arXiv preprint arXiv:2504.07954,

  71. [79]

    Vl-cogito: Progressive curriculum reinforcement learning for advanced multimodal reasoning

    Ruifeng Yuan, Chenghao Xiao, Sicong Leng, Jianyu Wang, Long Li, Weiwen Xu, Hou Pong Chan, Deli Zhao, Tingyang Xu, Zhongyu Wei, et al. Vl-cogito: Progressive curriculum reinforcement learning for advanced multimodal reasoning. arXiv preprint arXiv:2507.22607, 2025. 1, 2, 5, 7

    2025

  72. [80]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi. InCVPR, 2024. 9

    2024

  73. [81]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. InICCV, 2023. 2

    2023

  74. [82]

    Online deep clustering for unsupervised representation learning

    Xiaohang Zhan, Jiahao Xie, Ziwei Liu, Yew-Soon Ong, and Chen Change Loy. Online deep clustering for unsupervised representation learning. InCVPR, 2020. 2

    2020

  75. [83]

    Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In ECCV, 2024

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In ECCV, 2024. 9

    2024

  76. [84]

    Mme-realworld: Could your multimodal llm challenge high-resolution real-world scenar- ios that are difficult for humans? InICLR, 2025

    Yi-Fan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang, Junfei Wu, Feng Li, Kun Wang, Qing- song Wen, Zhang Zhang, et al. Mme-realworld: Could your multimodal llm challenge high-resolution real-world scenar- ios that are difficult for humans? InICLR, 2025. 2, 5

    2025

  77. [85]

    Deep- eyes: Incentivizing” thinking with images” via reinforce- ment learning.arXiv preprint arXiv:2505.14362, 2025

    Ziwei Zheng, Michael Yang, Jack Hong, Chenxiao Zhao, Guohai Xu, Le Yang, Chao Shen, and Xing Yu. Deep- eyes: Incentivizing” thinking with images” via reinforce- ment learning.arXiv preprint arXiv:2505.14362, 2025. 2

    2025

  78. [86]

    ibot: Image bert pre-training with online tokenizer

    Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. InICLR, 2022. 2

    2022

  79. [87]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023. 1

    2023

  80. [88]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shen- glong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025. 1 Appendix In thi...

    2025

Pith tools

Reviewed May 10, 2026 · model on record in the stance chip above.