Pith. sign in

REVIEW 4 major objections 5 minor 59 references

A procedurally generated visual-reasoning environment transfers to external benchmarks, raising average scores by 3.5–4.1 points.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 11:42 UTC pith:YAP6ZBJF

load-bearing objection TRACE is a solid environment plus a real two-scale RLVR transfer result, but the 'beyond generated distributions' claim outruns the evidence: the 24-benchmark suite sits close to the training domains and there's no distributional-distance check. the 4 major comments →

arxiv 2607.19790 v1 pith:YAP6ZBJF submitted 2026-07-22 cs.CV

Trace: A Taxonomy-Guided Environment for Multidomain Visual Reasoning

Md Tanvirul Alam This is my paper
classification cs.CV
keywords visual reasoningreinforcement learningverifiable rewardsprocedural data generationtaxonomyvision-language modelstransfer learningscene grammars
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper introduces TRACE, a taxonomy-guided environment that generates visual-reasoning tasks by separating a scene grammar from an executable task program. A shared semantic state determines the rendered image, the prompt, the typed answer, the verifier state, and a replayable trace, so supervision is exact and variation is controlled. The paper's central claim is that reinforcement learning with verifiable rewards on 64,000 TRACE instances transfers beyond the generated task distribution: the macro-average across 24 external benchmarks improves by 3.51 percentage points for a 3B vision-language model and 4.06 points for a 7B model. A sympathetic reader would care because it suggests that broad, structured procedural data can serve as a scalable source of exact-reward training for visual reasoning, rather than only improving scores on regenerated instances of the same tasks.

Core claim

The central discovery is that broad procedural training with exact rewards can transfer to external visual-reasoning benchmarks. TRACE factorizes task construction into scene grammars and executable task programs; each instance is generated from a semantic state on which the task program runs, and the same state determines the image, prompt, typed answer, verifier state, and replayable trace. Training two vision-language models (3B and 7B parameters) on the same 64,000 TRACE instances raises the macro-average of 24 external benchmarks by 3.51 and 4.06 percentage points respectively, with positive mean changes on 21 of 24 and all 24 benchmarks. The paper reads this as evidence that procedural

What carries the argument

The central object is TRACE's program-centered taxonomy, domain → scene grammar → task, where a task is the tuple (scene grammar, task program, answer schema, reward contract). The scene grammar generates a semantic state; the task program executes over that state to return a typed answer and verifier state; and the same state drives the renderer and prompt. This decoupling is what makes supervision exact, instances replayable, and task identity stable under semantic and visual variation. RLVR then uses a reward contract that combines exact answer correctness with a small JSON-format bonus, optimized with group relative policy optimization.

Load-bearing premise

The load-bearing premise is that the chosen 24-benchmark macro-average, measured under a single evaluation protocol, reflects general visual-reasoning ability rather than overlap between TRACE's task families and the benchmark suite; the paper itself concedes that gains are not attributed to domains or families and that transfer to every natural-image distribution is not established.

What would settle it

Run the same 64,000-instance training recipe on a set of external benchmarks whose visual formats and task structures are deliberately far from TRACE's domains—for example, natural-photograph question answering that requires fine-grained object recognition—and check whether the macro-average gain persists. If the gain vanishes or turns negative on such a suite, the claim that broad procedural training transfers beyond generated distributions would be falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Training on TRACE raises the 24-benchmark macro-average by 3.51 points at 3B and 4.06 points at 7B, with positive mean changes on 21 of 24 and all 24 benchmarks respectively.
  • All six benchmark categories improve at both scales; visual mathematics shows the largest category-level gain.
  • Because task identity is defined independently of prompt and rendering choices, the same environment can increase visual complexity without changing the reasoning objective or the verifier.
  • The explicit task program and reward contract make each instance exactly verifiable and deterministically replayable, supporting failure analysis and controlled variation studies.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • One testable extension of the paper's claim is to ablate the mixture by operation family or domain; if removing a family leaves external gains unchanged, the breadth of the taxonomy may matter less than the specific computational operations it instantiates.
  • The uniform task sampling used in the paper is a baseline; the explicit task identities and generation parameters could support difficulty-aware curricula or adaptive sampling, and one could test whether such reweighting further improves external transfer.
  • The paper's limitation section notes that transfer to every natural-image distribution is not established; a natural next step is to evaluate on natural-photograph benchmark groups whose formats are least similar to TRACE's renderings, where one might expect the gains to shrink if the mechanism is mainly format familiarity.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces TRACE, a procedural environment for visual-reasoning RLVR. It factorizes each instance into a scene grammar and an executable task program sharing a semantic state, so that the rendered image, prompt, typed answer, verifier state, and replayable trace are all derived from one source of truth. The environment contains 1,000 authored tasks over 277 scene grammars in 11 visual domains. The empirical study trains Qwen2.5-VL-3B and Qwen2.5-VL-7B on 64,000 TRACE instances with a GRPO-style verifiable reward, then evaluates on 24 external benchmarks. The headline results are gains of 3.51 points (3B) and 4.06 points (7B) in the unweighted 24-benchmark macro-average, with positive mean changes on 21/24 (3B) and 24/24 (7B) benchmarks. The paper also reports held-out TRACE gains, training dynamics, and comparisons with several released synthetic-data RLVR checkpoints.

Significance. If the empirical result is stable, the paper makes a useful contribution to multimodal RL: it demonstrates that a moderately sized, fully procedural, exactly verifiable visual dataset can transfer to a broad set of independently authored benchmarks, and it provides a concrete architecture for separating scene construction from task computation. The per-benchmark numbers in Table 2 are internally consistent with the reported category and overall macro-averages, and Table 8 gives paired item-bootstrap intervals, which is a methodological strength. I also found no evidence of circularity: rewards are computed from the semantic state before rendering, not from the external benchmarks. However, the central claims are supported by only one training run per scale, there is no ablation isolating the taxonomy or sampling design, and no code/data availability is stated. These limitations directly affect how strongly the abstract's 'provides evidence' claim can be read, though they do not invalidate the observed checkpoint-level results.

major comments (4)
  1. [§7; Table 2; §5.2] The headline gains of 3.51/4.06 points and the 21/24 and 24/24 counts are based on one training run per model scale, as acknowledged in §7. The paired item-bootstrap intervals in Table 8 quantify evaluation-item noise, not optimization stochasticity. Since GRPO is stochastic and the runs use only 500 updates, a second or third independent training run (or at least an explicit run-to-run variance estimate) is needed before 'RLVR on TRACE improves the macro-average' can be interpreted as a stable property of the data. If additional runs are infeasible, the abstract and conclusion should be explicitly scoped to the specific trained checkpoints rather than phrased as evidence about the method generally.
  2. [§7; contributions; §6.2] The paper does not provide any ablation that isolates the effect of the taxonomy-guided factorization, task-boundary rule, or uniform task sampling. The comparisons with Game-RL, Sphinx, and PC-GRPO are, as stated in §5.3, descriptive and unmatched in data, optimization, and compute. Therefore the results support the claim that this particular 64K TRACE mixture yields a strong checkpoint, but they do not support the contribution-level suggestion that taxonomy guidance, stable task units, or program–grammar separation are responsible for the gains. I would ask for a matched baseline (e.g., an un-factored template generator, a grammar-only variant, or a mixture without task-balanced sampling) or for removal/softening of the attribution language in the title and contribution list.
  3. [§5.3; §6.2; §7] The conclusion 'transfer beyond the generated task distributions' is stronger than the evidence because each of the six external benchmark groups has a close counterpart among TRACE's 11 visual domains (charts, graphs, physics, puzzles, counting, spatial/3D). The macro-average could be driven mainly by near-neighbor benchmarks. The §7 caveat about natural-image distributions is helpful, but it does not address the more immediate question of how far the external suite is from TRACE. I recommend either adding a quantitative distributional/overlap analysis (for example, nearest-TRACE-domain distances per benchmark, or per-group gains analyzed against domain similarity) or reformulating the conclusion as generalization to independently authored benchmarks within overlapping task families.
  4. [Abstract; §4; §7] The paper advertises a 'reproducible' and 'replayable' environment, but I could not find any code, data, or explicit availability statement in the manuscript beyond a project page URL. For an environment paper whose central value is exact supervision, deterministic replay, and controlled generation, the absence of a code/data release plan is a load-bearing reproducibility issue. Please add a Data/Code Availability section that states what will be released (generator, task definitions, renderer, verifier, training data, evaluation harness) and under what license, or clearly state any restrictions.
minor comments (5)
  1. [Eq. (8)] The sentence 'Answer correctness therefore determines the reward' is slightly overstated: if the answer is correct but the response is not a valid JSON object, the reward is 0.95, while a wrong answer with a valid JSON wrapper receives 0.05. Please rephrase to say that answer correctness is the dominant term.
  2. [Abstract] Typo: 'TRACEfactorizes' should be 'TRACE factorizes'.
  3. [Figure 14 caption] The caption contains 'Y ou are shown' — the spacing should be 'You are shown'.
  4. [§6.1 / Table 6] The numeric-answer interface shows a very large gain (+70.59 at 3B) but contains only 51 tasks, so this slice has high variance. Consider adding a caveat or confidence interval for that row.
  5. [Figure 7 caption] The figure reports 95% task-cluster bootstrap intervals, but the resampling procedure is not defined in the caption or the body. Please specify how task clusters are formed and how many replicates are used.

Circularity Check

0 steps flagged

No significant circularity: the central claim is an external-benchmark experiment, not a fit or self-citation reduction.

full rationale

The paper's main derivation is an empirical RLVR transfer study. TRACE instances are generated by Eqs. (5)-(7): a scene generator G_s constructs a semantic state x, the task program P_t computes the typed answer y and verifier state v from that state before rendering, and the reward contract is bound to the computed answer. Training and reward computation therefore do not use any of the 24 external benchmarks as inputs. The reported macro-average is a post-hoc summary of independent benchmark evaluations under a fixed protocol with three decoding seeds; it is not a training objective, and no parameter is fitted to the benchmark scores. The only self-citation is the author's prior SPHINX system, appearing as related work and as a descriptive comparison checkpoint; the TRACE taxonomy, environment, and transfer claim do not rely on SPHINX as evidence. Section 7 honestly limits the transfer claim by noting that gains are not attributed to domains and that transfer to every natural-image distribution is not established. These are validity caveats, not circular steps. No equation, definition, or cited result reduces the claimed prediction to its own input.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The environment and training recipe are engineering artifacts; no numerical constants are fitted to the external benchmarks. The listed items are the hand-chosen experimental settings and domain assumptions on which the transfer result depends.

free parameters (4)
  • Reward weight for exact answer = 0.95
    Eq. (8) sets R = 0.95 Ra + 0.05 Rf. This hand-chosen weighting defines the training signal; the claim depends on it.
  • Reward weight for JSON format = 0.05
    Eq. (8): small format bonus to encourage parseable output. Hand-chosen and not swept.
  • Learning rate and training length = 1e-6; 500 updates
    Table 5: LR 1e-6, one pass over 64,000 prompts. Picked without reported tuning; central to the measured improvements.
  • Uniform task sampling with 64 instances per task = 64 instances per task; uniform
    Section 5.1: fixed 64,000 instances = 64 per task. This design choice affects which tasks get trained on, and no mixture ablations are provided.
axioms (4)
  • domain assumption The type-aware canonicalization and reward contract correctly decide whether a submitted answer matches the target for every instance.
    Eqs. (5)-(8) and Section 4.4 rely on the verifier being exact. The paper describes validation checks but provides no independent audit or released verifier code.
  • domain assumption The rendered image and prompt faithfully realize the semantic state so that the visual question the model sees corresponds to the executed task program.
    Eqs. (5)-(6) assume renderer R_s and prompt function H_{s,t} preserve answer-relevant relations. Section 4.3 asserts invariants, but no code is shipped.
  • domain assumption Macro-averaging the chosen 24 external benchmarks is a meaningful measure of transfer.
    Section 5.3 defines the suite and unweighted averaging. Section 7 acknowledges the suite cannot establish transfer to every natural-image distribution.
  • domain assumption Base and TRACE-trained checkpoints are comparable under the same evaluation protocol.
    Section 5.3 states all checkpoints use the same prompts, parsers, scorers, and VLMEvalKit version; this is necessary for the reported deltas.

pith-pipeline@v1.3.0-alltime-deepseek · 27304 in / 10192 out tokens · 109222 ms · 2026-08-01T11:42:00.578586+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning with verifiable rewards (RLVR) has substantially improved language-model reasoning, yet its extension to vision-language models remains constrained by the lack of training data that are simultaneously broad, exactly verifiable, and reproducible. We introduce Trace, a taxonomy-guided environment for multidomain visual reasoning. Trace factorizes task construction into a scene grammar and an executable task program, separating visual realization from answer computation. A shared semantic state determines the rendered image, prompt, typed answer, verifier state, and replayable instance trace. The resulting environment comprises 1,000 tasks over 277 scene grammars and 11 visual domains, with controlled semantic and visual variation. RLVR on 64,000 Trace instances improves the macro-average across 24 external benchmarks by 3.51 percentage points for Qwen2.5-VL-3B and 4.06 points for Qwen2.5-VL-7B, providing evidence that broad procedural training can transfer beyond the generated task distributions. Project page: https://maveryn.github.io/trace/.

Figures

Figures reproduced from arXiv: 2607.19790 by Md Tanvirul Alam.

Figure 1
Figure 1. Figure 1: Representative instances from the 11 visual domains in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Examples of the task boundary in TRACE. Left: Changing extremum direction modifies only a bounded query argument and therefore remains within one boxplot task. Center: Adding a conjunctive attribute changes the candidate predicate and selected set, producing a new icon-filtering task. Right: Changes to board dimensions, layout, and visual treatment remain generation parameters because the reachability prog… view at source ↗
Figure 3
Figure 3. Figure 3: End-to-end construction of a TRACE instance. Four-neighbor reachability is computed over the semantic cell board, and the resulting state determines the typed answer, rendered image, prompt, scorer, and replayable instance trace. tasks applied to their semantic states, and additional reasoning objectives can reuse an existing scene grammar without duplicating its visual construction. 4 The TRACE Environmen… view at source ↗
Figure 4
Figure 4. Figure 4: Composition of TRACE across 11 visual domains. The panels report the number of task programs and scene grammars in each domain, together with the overall distribution of answer interfaces. 4.2 Environment composition TRACE comprises 1,000 tasks defined over 277 scene grammars spanning 11 visual domains. The do￾mains organize related visual structures, rendering conventions, and object vocabularies for repo… view at source ↗
Figure 5
Figure 5. Figure 5: Distribution of task-program operations across visual domains. Cell labels report the [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Answer-preserving realizations of a fixed semantic graph. Theme, typeface, layout, non [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Held-out TRACE accuracy gains by visual domain (left) and operation family (right). Whiskers show 95% task-cluster bootstrap intervals; parentheses give task counts, and operation￾family assignments overlap. −2 0 2 4 6 8 10 12 3B improvement over base (points) −2 0 2 4 6 8 10 12 7B improvement over base (points) Spearman ρs = 0.60 EvoChart ChartQAPro TreeBench Benchmark group Charts & Tables Visual Math Sc… view at source ↗
Figure 8
Figure 8. Figure 8: Benchmark gains at 3B versus 7B. Colors denote benchmark groups; dashed axes mark [PITH_FULL_IMAGE:figures/full_fig_p010_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Optimization trajectories over 500 RLVR updates. Faint lines show per-update values, and [PITH_FULL_IMAGE:figures/full_fig_p012_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Control flow for visual realization. Structural and surface controls are treated as render [PITH_FULL_IMAGE:figures/full_fig_p020_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Representative chart tasks. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Representative game tasks. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Representative geometry tasks. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Representative graph tasks. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Representative icon tasks. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Representative illustration tasks. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Representative page tasks. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Representative physics tasks. 31 [PITH_FULL_IMAGE:figures/full_fig_p031_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Representative puzzle tasks. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Representative symbolic tasks. 33 [PITH_FULL_IMAGE:figures/full_fig_p033_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Representative 3D tasks. 34 [PITH_FULL_IMAGE:figures/full_fig_p034_21.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

59 extracted references · 1 canonical work pages

  1. [1]

    SPHINX: A synthetic environment for visual perception and reasoning

    Md Tanvirul Alam, Saksham Aggarwal, Justin Yang Chae, and Nidhi Rastogi. SPHINX: A synthetic environment for visual perception and reasoning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9489–9499, 2026

    2026

  2. [2]

    Qwen2.5-VL technical report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-VL technical report. a...

    Pith/arXiv arXiv 2025

  3. [3]

    Enigmata: Scaling logical reasoning in large language models with synthetic verifiable puzzles.arXiv preprint arXiv:2505.19914, 2025

    Jiangjie Chen, Qianyu He, Siyu Yuan, Aili Chen, Zhicheng Cai, Weinan Dai, et al. Enigmata: Scaling logical reasoning in large language models with synthetic verifiable puzzles.arXiv preprint arXiv:2505.19914, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    Are we on the right way for evaluating large vision-language models?arXiv preprint arXiv:2403.20330, 2024

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for evaluating large vision-language models?arXiv preprint arXiv:2403.20330, 2024

    Pith/arXiv arXiv 2024

  5. [5]

    PuzzleVQA: Diagnosing multimodal reasoning challenges of language models with abstract visual patterns.arXiv preprint arXiv:2403.13315, 2024

    Yew Ken Chia, Vernon Toh Yan Han, Deepanway Ghosal, Lidong Bing, and Soujanya Poria. PuzzleVQA: Diagnosing multimodal reasoning challenges of language models with abstract visual patterns.arXiv preprint arXiv:2403.13315, 2024

    Pith/arXiv arXiv 2024

  6. [6]

    EmbSpatial-Bench: Benchmarking spatial understanding for embodied tasks with large vision-language models

    Mengfei Du, Binhao Wu, Zejun Li, Xuanjing Huang, and Zhongyu Wei. EmbSpatial-Bench: Benchmarking spatial understanding for embodied tasks with large vision-language models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024. 13

    2024

  7. [7]

    VLMEvalKit: An open-source toolkit for evaluating large multi-modality models

    Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. VLMEvalKit: An open-source toolkit for evaluating large multi-modality models. InProceedings of the 32nd ACM International Conference on Multimedia, pages 11198–11201, 2024

    2024

  8. [8]

    VisualSphinx: Large-scale synthetic vision logic puzzles for RL.arXiv preprint arXiv:2505.23977, 2025

    Yichen Feng, Zhangchen Xu, Fengqing Jiang, Yuetai Li, Bhaskar Ramasubramanian, Luyao Niu, Bill Yuchen Lin, and Radha Poovendran. VisualSphinx: Large-scale synthetic vision logic puzzles for RL.arXiv preprint arXiv:2505.23977, 2025

    Pith/arXiv arXiv 2025

  9. [9]

    Smith, Wei-Chiu Ma, and Ranjay Krishna

    Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, and Ranjay Krishna. BLINK: Multimodal large language models can see but not perceive.arXiv preprint arXiv:2404.12390, 2024

    Pith/arXiv arXiv 2024

  10. [10]

    Embodied reasoning question answer (ERQA) benchmark

    Google DeepMind. Embodied reasoning question answer (ERQA) benchmark. GitHub reposi- tory, 2025. URLhttps://github.com/embodiedreasoning/ERQA

    2025

  11. [11]

    Composition-grounded data synthesis for visual reasoning.arXiv preprint arXiv:2510.15040, 2025

    Xinyi Gu, Jiayuan Mao, Zhang-Wei Hong, Zhuoran Yu, Pengyuan Li, Dhiraj Joshi, Rogerio Feris, and Zexue He. Composition-grounded data synthesis for visual reasoning.arXiv preprint arXiv:2510.15040, 2025

    arXiv 2025

  12. [12]

    OpenThoughts: Data recipes for reasoning models.arXiv preprint arXiv:2506.04178, 2025

    Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, et al. OpenThoughts: Data recipes for reasoning models.arXiv preprint arXiv:2506.04178, 2025

    Pith/arXiv arXiv 2025

  13. [13]

    DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    EvoChart: A benchmark and a self-training approach towards real-world chart understanding

    Muye Huang, Han Lai, Xinyu Zhang, Wenjun Wu, Jie Ma, Lingling Zhang, and Jun Liu. EvoChart: A benchmark and a self-training approach towards real-world chart understanding. arXiv preprint arXiv:2409.01577, 2025

    Pith/arXiv arXiv 2025

  15. [15]

    Vision- R1: Incentivizing reasoning capability in multimodal large language models.arXiv preprint arXiv:2503.06749, 2025

    Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, et al. Vision- R1: Incentivizing reasoning capability in multimodal large language models.arXiv preprint arXiv:2503.06749, 2025

    Pith/arXiv arXiv 2025

  16. [16]

    Hudson and Christopher D

    Drew A. Hudson and Christopher D. Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019

    2019

  17. [17]

    Derpanis, Babak Taati, and Radek Grzeszczuk

    Ahmadreza Jeddi, Hakki Can Karaimer, Hue Nguyen, Zhongling Wang, Ke Zhao, Javad Rajabi, Ran Zhang, Raghav Goyal, Konstantinos G. Derpanis, Babak Taati, and Radek Grzeszczuk. PuzzleCraft: Exploration-aware curriculum learning for puzzle-based RLVR in VLMs.arXiv preprint arXiv:2512.14944, 2025. URLhttps://arxiv.org/abs/2512.14944

    arXiv 2025

  18. [18]

    Lawrence Zitnick, and Ross Girshick

    Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017

    2017

  19. [19]

    TableVQA-Bench: A visual question answering benchmark on multiple table domains.arXiv preprint arXiv:2404.19205, 2024

    Yoonsik Kim, Moonbin Yim, and Ka Yeon Song. TableVQA-Bench: A visual question answering benchmark on multiple table domains.arXiv preprint arXiv:2404.19205, 2024

    Pith/arXiv arXiv 2024

  20. [20]

    Truth in the few: High-value data selection for efficient multi-modal reasoning.arXiv preprint arXiv:2506.04755, 2025

    Shenshen Li, Xing Xu, Kaiyuan Deng, Lei Wang, Heng Tao Shen, and Fumin Shen. Truth in the few: High-value data selection for efficient multi-modal reasoning.arXiv preprint arXiv:2506.04755, 2025

    arXiv 2025

  21. [21]

    MoDoMoDo: Multi-domain data mixtures for multimodal LLM reinforcement learning.arXiv preprint arXiv:2505.24871, 2025

    Yiqing Liang, Jielin Qiu, Wenhao Ding, Zuxin Liu, James Tompkin, Mengdi Xu, Mengzhou Xia, Zhengzhong Tu, Laixi Shi, and Jiacheng Zhu. MoDoMoDo: Multi-domain data mixtures for multimodal LLM reinforcement learning.arXiv preprint arXiv:2505.24871, 2025

    Pith/arXiv arXiv 2025

  22. [22]

    Visual-RFT: Visual reinforcement fine-tuning.arXiv preprint arXiv:2503.01785, 2025

    Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-RFT: Visual reinforcement fine-tuning.arXiv preprint arXiv:2503.01785, 2025. 14

    Pith/arXiv arXiv 2025

  23. [23]

    MathVista: Evaluating mathematical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2024

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. MathVista: Evaluating mathematical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2024

    Pith/arXiv arXiv 2024

  24. [24]

    ChartQAPro: A more diverse and challenging benchmark for chart question answering.arXiv preprint arXiv:2504.05506, 2025

    Ahmed Masry, Mohammed Saidul Islam, Mahir Ahmed, Aayush Bajaj, Firoz Kabir, Aaryaman Kartha, Md Tahmid Rahman Laskar, Mizanur Rahman, Shadikur Rahman, Mehrad Shahmo- hammadi, Megh Thakkar, Md Rizwan Parvez, Enamul Hoque, and Shafiq Joty. ChartQAPro: A more diverse and challenging benchmark for chart question answering.arXiv preprint arXiv:2504.05506, 2025

    Pith/arXiv arXiv 2025

  25. [25]

    MM-Eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning.arXiv preprint arXiv:2503.07365, 2025

    Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, et al. MM-Eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning.arXiv preprint arXiv:2503.07365, 2025

    Pith/arXiv arXiv 2025

  26. [26]

    Teaching CLIP to count to ten.arXiv preprint arXiv:2302.12066, 2023

    Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Irani, and Tali Dekel. Teaching CLIP to count to ten.arXiv preprint arXiv:2302.12066, 2023

    Pith/arXiv arXiv 2023

  27. [27]

    LMM-R1: Empowering 3b LMMs with strong reasoning abilities through two-stage rule-based RL.arXiv preprint arXiv:2503.07536, 2025

    Yingzhe Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, et al. LMM-R1: Empowering 3b LMMs with strong reasoning abilities through two-stage rule-based RL.arXiv preprint arXiv:2503.07536, 2025

    Pith/arXiv arXiv 2025

  28. [28]

    We-Math: Does your large multimodal model achieve human-like mathematical reasoning? InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, 2025

    Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Jiapeng Wang, Zhuoma Gong Que, Shanglin Lei, Yifan Zhang, Zhe Wei, Miaoxuan Zhang, Runfeng Qiao, Xiao Zong, Yida Xu, Peiqing Yang, Zhimin Bao, Muxi Diao, Chen Li, and Honggang Zhang. We-Math: Does your large multimodal model achieve human-like mathematical reasoning? InProceedings...

    2025

  29. [29]

    Vero: An open RL recipe for general visual reasoning.arXiv preprint arXiv:2604.04917, 2026

    Gabriel Sarch, Linrong Cai, Qunzhong Wang, Haoyang Wu, Danqi Chen, and Zhuang Liu. Vero: An open RL recipe for general visual reasoning.arXiv preprint arXiv:2604.04917, 2026

    Pith/arXiv arXiv 2026

  30. [30]

    DeepSeek- Math: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, et al. DeepSeek- Math: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  31. [31]

    VLM-R1: A stable and generalizable R1-style large vision-language model.arXiv preprint arXiv:2504.07615, 2025

    Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, et al. VLM-R1: A stable and generalizable R1-style large vision-language model.arXiv preprint arXiv:2504.07615, 2025

    Pith/arXiv arXiv 2025

  32. [32]

    PhyX: Does your model have the wits for physical reasoning?arXiv preprint arXiv:2505.15929, 2025

    Hui Shen, Taiqiang Wu, Qi Han, Yunta Hsieh, Jizhou Wang, Yuyue Zhang, Yuxin Cheng, Zijian Hao, Yuansheng Ni, Xin Wang, Zhongwei Wan, Kai Zhang, Wendong Xu, Jing Xiong, Ping Luo, Wenhu Chen, Chaofan Tao, Zhuoqing Mao, and Ngai Wong. PhyX: Does your model have the wits for physical reasoning?arXiv preprint arXiv:2505.15929, 2025

    Pith/arXiv arXiv 2025

  33. [33]

    VisualPuz- zles: Decoupling multimodal reasoning evaluation from domain knowledge.arXiv preprint arXiv:2504.10342, 2025

    Yueqi Song, Tianyue Ou, Yibo Kong, Zecheng Li, Graham Neubig, and Xiang Yue. VisualPuz- zles: Decoupling multimodal reasoning evaluation from domain knowledge.arXiv preprint arXiv:2504.10342, 2025

    Pith/arXiv arXiv 2025

  34. [34]

    Reasoning Gym: Reasoning environments for reinforcement learning with verifiable rewards.arXiv preprint arXiv:2505.24760, 2025

    Zafir Stojanovski, Oliver Stanley, Joe Sharratt, Richard Jones, Abdulhakeem Adefioye, Jean Kaddour, and Andreas Köpf. Reasoning Gym: Reasoning environments for reinforcement learning with verifiable rewards.arXiv preprint arXiv:2505.24760, 2025

    arXiv 2025

  35. [35]

    DeepVision-103K: A visually diverse, broad-coverage, and verifiable mathematical dataset for multimodal reasoning.arXiv preprint arXiv:2602.16742, 2026

    Haoxiang Sun, Lizhen Xu, Bing Zhao, Wotao Yin, Wei Wang, Boyu Yang, Rui Wang, and Hu Wei. DeepVision-103K: A visually diverse, broad-coverage, and verifiable mathematical dataset for multimodal reasoning.arXiv preprint arXiv:2602.16742, 2026

    arXiv 2026

  36. [36]

    CountQA: How well do MLLMs count in the wild?arXiv preprint arXiv:2508.06585, 2025

    Jayant Sravan Tamarapalli, Rynaa Grover, Nilay Pande, and Sahiti Yerramilli. CountQA: How well do MLLMs count in the wild?arXiv preprint arXiv:2508.06585, 2025

    Pith/arXiv arXiv 2025

  37. [37]

    Reason-RFT: Reinforcement fine-tuning for visual reasoning of vision language models.arXiv preprint arXiv:2503.20752, 2025

    Huajie Tan, Yuheng Ji, Xiaoshuai Hao, Xiansheng Chen, Pengwei Wang, Zhongyuan Wang, and Shanghang Zhang. Reason-RFT: Reinforcement fine-tuning for visual reasoning of vision language models.arXiv preprint arXiv:2503.20752, 2025. 15

    arXiv 2025

  38. [38]

    Game-RL: Synthesizing multimodal verifiable game data to boost VLMs’ general reasoning

    Jingqi Tong, Jixin Tang, Hangcheng Li, Yurong Mou, Ming Zhang, Jun Zhao, Yanbo Wen, Fan Song, Jiahao Zhan, Yuyang Lu, et al. Game-RL: Synthesizing multimodal verifiable game data to boost VLMs’ general reasoning. InInternational Conference on Learning Representations,

  39. [39]

    Cambrian-1: A fully open, vision-centric exploration of multimodal LLMs.arXiv preprint arXiv:2406.16860, 2024

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Ziteng Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: A fully open, vision-centric exploration of multimodal LLMs.arXiv preprint arXiv:2406.16860, 2024

    Pith/arXiv arXiv 2024

  40. [40]

    Traceable evidence enhanced visual grounded reasoning: Evaluation and methodology.arXiv preprint arXiv:2507.07999, 2026

    Haochen Wang, Xiangtai Li, Zilong Huang, Anran Wang, Jiacong Wang, Tao Zhang, Jiani Zheng, Sule Bai, Zijian Kang, Jiashi Feng, Zhuochen Wang, and Zhaoxiang Zhang. Traceable evidence enhanced visual grounded reasoning: Evaluation and methodology.arXiv preprint arXiv:2507.07999, 2026

    arXiv 2026

  41. [41]

    Mea- suring multimodal mathematical reasoning with MATH-Vision dataset.arXiv preprint arXiv:2402.14804, 2024

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. Mea- suring multimodal mathematical reasoning with MATH-Vision dataset.arXiv preprint arXiv:2402.14804, 2024

    Pith/arXiv arXiv 2024

  42. [42]

    SpatialViz-Bench: A cognitively-grounded benchmark for diagnosing spatial visualization in MLLMs.arXiv preprint arXiv:2507.07610, 2026

    Siting Wang, Minnan Pei, Luoyang Sun, Cheng Deng, Yuchen Li, Kun Shao, Zheng Tian, Haifeng Zhang, and Jun Wang. SpatialViz-Bench: A cognitively-grounded benchmark for diagnosing spatial visualization in MLLMs.arXiv preprint arXiv:2507.07610, 2026

    arXiv 2026

  43. [43]

    ViCrit: A verifiable reinforcement learning proxy task for visual perception in VLMs.arXiv preprint arXiv:2506.10128, 2025

    Xiyao Wang, Zhengyuan Yang, Chao Feng, Yongyuan Liang, Yuhang Zhou, Xiaoyu Liu, et al. ViCrit: A verifiable reinforcement learning proxy task for visual perception in VLMs.arXiv preprint arXiv:2506.10128, 2025

    Pith/arXiv arXiv 2025

  44. [44]

    SoTA with less: MCTS-guided sample selection for data-efficient visual reasoning self-improvement.arXiv preprint arXiv:2504.07934, 2025

    Xiyao Wang, Zhengyuan Yang, Chao Feng, Hongjin Lu, Linjie Li, Chung-Ching Lin, Kevin Lin, Furong Huang, and Lijuan Wang. SoTA with less: MCTS-guided sample selection for data-efficient visual reasoning self-improvement.arXiv preprint arXiv:2504.07934, 2025

    Pith/arXiv arXiv 2025

  45. [45]

    Blaschko

    Zifu Wang, Junyi Zhu, Bo Tang, Zhiyu Li, Feiyu Xiong, Jiaqian Yu, and Matthew B. Blaschko. Jigsaw-R1: A study of rule-based visual reinforcement learning with jigsaw puzzles.arXiv preprint arXiv:2505.23590, 2025

    arXiv 2025

  46. [46]

    CharXiv: Charting gaps in realistic chart understanding in multimodal LLMs.arXiv preprint arXiv:2406.18521, 2024

    Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, Alexis Chevalier, Sanjeev Arora, and Danqi Chen. CharXiv: Charting gaps in realistic chart understanding in multimodal LLMs.arXiv preprint arXiv:2406.18521, 2024

    Pith/arXiv arXiv 2024

  47. [47]

    SynthRL: Scaling visual reasoning with verifiable data synthesis.arXiv preprint arXiv:2506.02096, 2025

    Zijian Wu, Jinjie Ni, Xiangyan Liu, Zichen Liu, Hang Yan, and Michael Qizhe Shieh. SynthRL: Scaling visual reasoning with verifiable data synthesis.arXiv preprint arXiv:2506.02096, 2025

    Pith/arXiv arXiv 2025

  48. [48]

    RealWorldQA

    xAI. RealWorldQA. Hugging Face dataset, 2024. URL https://huggingface.co/ datasets/xai-org/RealworldQA

    2024

  49. [49]

    LogicVista: Multimodal LLM logical reasoning benchmark in visual contexts.arXiv preprint arXiv:2407.04973, 2024

    Yijia Xiao, Edward Sun, Tianyu Liu, and Wei Wang. LogicVista: Multimodal LLM logical reasoning benchmark in visual contexts.arXiv preprint arXiv:2407.04973, 2024

    Pith/arXiv arXiv 2024

  50. [50]

    WeThink: Toward general-purpose vision-language reasoning via reinforcement learn- ing.arXiv preprint arXiv:2506.07905, 2025

    Jie Yang, Feipeng Ma, Zitian Wang, Dacheng Yin, Kang Rong, Fengyun Rao, and Ruimao Zhang. WeThink: Toward general-purpose vision-language reasoning via reinforcement learn- ing.arXiv preprint arXiv:2506.07905, 2025

    Pith/arXiv arXiv 2025

  51. [51]

    TRON: Targeted rule-verifiable online environments for visual reasoning RL.arXiv preprint arXiv:2606.01599, 2026

    Tianze Yang, Yucheng Shi, Ruitong Sun, Jingyuan Huang, Ninghao Liu, and Jin Sun. TRON: Targeted rule-verifiable online environments for visual reasoning RL.arXiv preprint arXiv:2606.01599, 2026. URLhttps://arxiv.org/abs/2606.01599

    Pith/arXiv arXiv 2026

  52. [52]

    R1- Onevision: Advancing generalized multimodal reasoning through cross-modal formalization

    Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, et al. R1- Onevision: Advancing generalized multimodal reasoning through cross-modal formalization. arXiv preprint arXiv:2503.10615, 2025

    Pith/arXiv arXiv 2025

  53. [53]

    Worldbench: A challenging and visually diverse multimodal reasoning benchmark.arXiv preprint arXiv:2606.06538, 2026

    Yida Yin, Harish Krishnakumar, Chung Peng Lee, Boya Zeng, Wenhao Chai, Shengbang Tong, Wenhu Chen, Hu Xu, Xingyu Fu, Gabriel Sarch, Aleksandra Korolova, and Zhuang Liu. Worldbench: A challenging and visually diverse multimodal reasoning benchmark.arXiv preprint arXiv:2606.06538, 2026. 16

    Pith/arXiv arXiv 2026

  54. [54]

    MME-Reasoning: A comprehensive benchmark for logical reasoning in MLLMs.arXiv preprint arXiv:2505.21327, 2025

    Jiakang Yuan, Tianshuo Peng, Yilei Jiang, Yiting Lu, Renrui Zhang, Kaituo Feng, Chaoyou Fu, Tao Chen, Lei Bai, Bo Zhang, and Xiangyu Yue. MME-Reasoning: A comprehensive benchmark for logical reasoning in MLLMs.arXiv preprint arXiv:2505.21327, 2025

    Pith/arXiv arXiv 2025

  55. [55]

    MMMU- Pro: A more robust multi-discipline multimodal understanding benchmark.arXiv preprint arXiv:2409.02813, 2025

    Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and Graham Neubig. MMMU- Pro: A more robust multi-discipline multimodal understanding benchmark.arXiv preprint arXiv:2409.02813, 2025

    Pith/arXiv arXiv 2025

  56. [56]

    Xing, and Zhiting Hu

    Yuheng Zha, Kun Zhou, Yujia Wu, Yushu Wang, Jie Feng, Zhi Xu, Shibo Hao, Zhengzhong Liu, Eric P. Xing, and Zhiting Hu. Vision-G1: Towards general reasoning vision-language models via reinforcement learning.Proceedings of the AAAI Conference on Artificial Intelligence, 40 (33):28131–28139, 2026. doi: 10.1609/aaai.v40i33.40039

    work page doi:10.1609/aaai.v40i33.40039 2026

  57. [57]

    Task me anything.arXiv preprint arXiv:2406.11775, 2024

    Jieyu Zhang, Weikai Huang, Zixian Ma, Oscar Michel, Dong He, Tanmay Gupta, Wei-Chiu Ma, Ali Farhadi, Aniruddha Kembhavi, and Ranjay Krishna. Task me anything.arXiv preprint arXiv:2406.11775, 2024

    Pith/arXiv arXiv 2024

  58. [58]

    S9W6" through

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, and Hongsheng Li. MathVerse: Does your multi-modal LLM truly see the diagrams in visual math problems?arXiv preprint arXiv:2403.14624, 2024. 17 A Environment Construction and Validation A.1 Domain construction profiles Prompts are ass...

    Pith/arXiv arXiv 2024

  59. [2026]

    URLhttps://openreview.net/forum?id=e4FqU4SyHL