Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

A VLM's own correct and incorrect answers can be spliced together to teach it self-correction.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 03:14 UTC pith:PBHJ67ND

load-bearing objection A genuinely novel rollout-augmentation idea for teaching VLMs self-correction, with solid ablations, but the headline 1-point gain rests on a single run and the central paired-o1/o2 assumption deserves a direct test. the 4 major comments →

arxiv 2602.08503 v2 pith:PBHJ67ND submitted 2026-02-09 cs.CV cs.CLcs.LG

Learning Self-Correction in Vision-Language Models via Rollout Augmentation

Yi Ding , Ziliang Qiu , Bolian Li , Ruqi Zhang This is my paper
classification cs.CV cs.CLcs.LG
keywords self-correctionvision-language modelsreinforcement learningrollout augmentationRLVRresponse maskingtest-time scalingreasoning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that effective self-correction—a model catching and fixing its own mistakes mid-reasoning—is too rare to learn under standard reinforcement learning, but that the missing training signal already exists inside ordinary RL rollouts. It introduces Octopus, a rollout augmentation method that pairs the pre-correction segment of one response with the post-correction segment of another response from the same input, turning a sparse handful of genuine wrong-to-correct transitions into a dense, balanced training set at no extra generation cost. On top of this, a response-masking strategy separates the learning of direct reasoning from the learning of self-correction, preventing the model from gaming the reward by deliberately making an easy mistake and then 'correcting' it. The resulting Octopus-8B model beats the strongest RL baseline by 1.0 average accuracy point across seven vision-language benchmarks while using roughly 0.72x the training time per step, and its self-correction continues to improve with appended tokens at inference time.

Core claim

The paper's central claim is that self-correction can be learned as a controllable, reliable behavior by recombining existing rollouts. For a given visual question, a policy generates n responses in an explicit self-correction format: o1 ⊕ <sc> ⊕ o2. Because correct and incorrect trajectories naturally coexist in the rollout group, pairing any o1 with any o2 yields n^2 synthetic trajectories, of which the wrong→correct pairs directly demonstrate how an error should be repaired. The authors show that this augmentation, rather than merely increasing data volume, is what drives gains: random concatenation improves the average by only 1.2 points, while Octopus augmentation gains 4.3 points over

What carries the argument

The central mechanism is the paired-rollout recombination: given rollouts {(o1_i ⊕ <sc> ⊕ o2_i)}, the algorithm constructs o1_i ⊕ <sc> ⊕ o2_j for all i,j, producing four labeled categories (wrong→correct, correct→correct, correct→wrong, wrong→wrong). This creates dense, explicit self-correction demonstrations and balances positive and negative examples within each training group. A second key mechanism is the response-masking strategy: in Stage I the o1 tokens are masked and held fixed so only the correction segment o2 is trained, with a KL penalty keeping o1 on-distribution; in Stage II o1 is unmasked only for samples whose correctness is unchanged before and after <sc>, preventing the mode

Load-bearing premise

Pairing an o1 from one rollout with an o2 from a different rollout produces a valid self-correction example, in the sense that the model learns to identify and fix the specific error in o1 instead of merely uncritically regenerating a fresh answer after the <sc> marker.

What would settle it

Measure whether Octopus-8B's post-<sc> accuracy changes when the pre-correction response o1 is replaced by a different o1 for the same question; if accuracy is roughly unchanged, the model is treating <sc> as a restart rather than correcting the specific error.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim holds, RL for reasoning models no longer has to wait for self-correction to emerge spontaneously; the signal can be synthesized from rollouts the model already generates, at no extra inference cost.
  • The same recombination idea should transfer to text-only reasoning LLMs and to other structured behaviors beyond correction, such as verification or backtracking, wherever correct and incorrect trajectories coexist.
  • The gains should scale with rollout diversity rather than rollout count: because n rollouts yield n^2 pairs, any policy that maintains diverse successes and failures provides an increasingly rich correction curriculum.
  • The controllable <sc> token makes self-correction a test-time dial: accuracy and token efficiency both improve when extra correction tokens are appended, suggesting a cheap inference-time scaling axis for VLMs.
  • The decoupled two-stage objective offers a template for training multiple entangled capabilities in one pass without reward hacking, which prior shaped-reward approaches failed to achieve.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A reader might push further: if splicing correct and incorrect segments is what teaches correction, the framework could be inverted to deliberately teach models to avoid over-correction by balancing how many correct→wrong negatives are included; the paper already samples negatives but does not ablate their ratio.
  • One testable extension is to apply Octopus to domains without verifiable rewards, using the model's own confidence or self-consistency to label o1 and o2 as correct or wrong—this would extend the method beyond math and chart QA.
  • The synthetic-pair validity assumption implies an experiment the paper does not run: measure whether the model actually conditions on the specific error in o1 when generating o2, rather than treating <sc> as a restart signal; attention or counterfactual editing of o1 would settle this.
  • Because the augmented pairs are off-policy, the paper's reliance on GSPO's sequence-level importance weights is load-bearing; a natural follow-up is to test whether a token-level method with the same augmentation collapses, which would clarify whether the gains come from augmentation or from the off-policy stabilizer.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Octopus, an RL rollout augmentation method for vision-language models that recombines pre-correction and post-correction segments (o1 and o2) from existing rollouts to synthesize dense self-correction training examples. It introduces a two-stage training recipe: Stage I decouples self-correction learning by masking o1 and applying KL regularization, and Stage II selectively unmasks o1 only for samples with consistent correctness. Experiments on seven benchmarks with Qwen3-VL-8B-Instruct report an average accuracy of 71.7, which is 1.0 point above the best RLVR baseline (GSPO with 16 rollouts) and 1.2 points above Qwen3-VL-8B-Thinking, while using 0.72x the training time per step. The paper also reports test-time-scaling gains from appending additional <sc> tokens and a pass@k improvement over baselines.

Significance. If the central claim holds, the paper makes a useful and timely contribution: it identifies a real sparsity problem in RL-based self-correction learning, proposes a cheap augmentation that reuses existing rollouts, and provides a masking scheme to mitigate reward hacking. The method is clearly specified, the ablations are informative, and the random-augmentation control is a valuable attempt to separate augmentation quality from simple data quantity. The efficiency result is also interesting. However, the interpretation of the gains as learned, error-specific self-correction rather than a controllable 'restart' behavior is not yet established by the presented experiments. In addition, the evaluation rests on a single run with no error bars and uses a geometry validation set with potential overlap with test benchmarks. These issues are addressable within the manuscript's scope, but they are load-bearing for the main scientific claim.

major comments (4)
  1. [§3.2, Algorithm 1, Eq. (3), Eq. (6)] The central claim that splicing o1 from one rollout with o2 from another produces valid self-correction training signal is not directly tested. The rewards in Eq. (3) and Eq. (6) depend only on the final correctness of o1 and o2, not on whether o2 actually addresses the specific error in o1. Since the same o2 can be paired with many different o1's, a policy can maximize the reward by regenerating a fresh answer after <sc>, which would improve accuracy through a second attempt rather than through error-specific correction. The random-augmentation row in Table 3 does not isolate this shortcut, because random concatenation differs in distribution and coherence. The case studies in Appendix D are selected examples, not systematic evidence. I request a direct test of o2's dependence on o1, for example: (i) compare o2 accuracy when conditioned on the true o1 versus on a different wrong o1 from
  2. [§5.1, §5.2, Appendix A.1] The headline 1.0-point average gain over GSPO (Table 2) is reported from a single run with no error bars or significance testing. For a 7-benchmark average, run-to-run variance can easily exceed 1 point for 8B-scale RL training. Additionally, Appendix A.1 states that Geometry-3k is used as a validation set to select the best training checkpoint. Geometry-3k is itself a geometry reasoning benchmark, and the test set includes MathVista, MathVerse, and WeMath, which contain substantial geometry and diagram-reasoning content. This creates a risk that the reported numbers are partly driven by checkpoint selection on a distribution overlapping the eval benchmarks. Please report variance over at least three seeds, and clarify how the Geometry-3k validation selection is isolated from the reported test benchmarks, ideally by selecting checkpoints on a held-out non-geometry set or by reporting the
  3. [§5.1, §5.2, Table 2, Table 3] The comparison against RLVR baselines is confounded by the SFT cold-start and by the inference prompt. Octopus starts from a 10k-sample SFT cold-start built with outputs from Qwen3-VL-30B-A3B-Instruct, while GRPO, DAPO, GSPO, and SRPO are trained directly from Qwen3-VL-8B-Instruct with standard prompts (Appendix A.3). The 'w/o RL (SFT only)' row in Table 3 shows a +1.2 gain from SFT alone, and the 'w/o Octopus Augmentation' row (67.4) is only +0.8 above GSPO with n=8 (66.6). The main comparison to GSPO with n=16 (70.7) differs not only in augmentation but also in initialization, prompt, and effective sample count. Please add a control where GSPO (or the strongest baseline) is trained from the same cold-start checkpoint, or at least from the same SFT data, with the standard prompt and no augmentation. This would isolate the contribution of the augmentation from the contribution of cold-st
  4. [Appendix A.3, Fig. 7] The test-time-scaling experiment in Fig. 7(a) is described as 'appending additional <sc> tokens to trigger further correction,' but the implementation is not fully specified: how many tokens are appended, at what position, and does the model produce a full new reasoning chain each time? Since the inference prompt already instructs the model to 'regenerate a new reasoning process and a new answer from scratch' after <sc>, the blue curve may reflect a restart mechanism rather than iterative refinement. Please provide the exact inference procedure and, if possible, a control where the same number of extra tokens is forced without the <sc> marker. This is relevant to interpreting the TTS claim as evidence of self-correction.
minor comments (5)
  1. [Algorithm 1] 'originical' typo. More importantly, the pseudocode uses N for both the training set size and the loop bound in the augmentation buffer construction, while the text defines n as the number of original rollouts. The loops should be over n, and N should be used only for the final training set size.
  2. [Table 1] The dataset and evaluation protocol for the sparsity quantification in Table 1 are not stated. Please specify the benchmark, number of samples, decoding parameters, and whether the 'Wait'/'Alternatively' prompts are used during RL training or only during evaluation. Without this, the numbers are hard to interpret.
  3. [Table 2] The order of the DAPO and GRPO rows with n=16 and n=8 appears reversed: the n=16 entries are listed first but the Gen./Total columns are populated for the n=8 rows. This makes the timing comparison easy to misread. Please reorder consistently.
  4. [Appendix C] Appendix C describes a failed attempt that 'randomly mixes elements from o1 and o2', while Table 3 reports 'Randomly Augmentation' as random concatenation. These are different operations; please clarify the distinction to avoid confusion about what the Table 3 control actually controls for.
  5. [Appendix A.3] The Octopus prompt says 'generate <self-correction> </self-correction> tags enclosed with no content, and regenerate...' The phrasing is awkward. Please clarify whether the model should emit an empty tag pair and then continue with a new reasoning chain, as the case studies show.

Circularity Check

1 steps flagged

Internal diagnostic curves partly restate the reward/selection definition, but the headline 7-benchmark results are held out and independent.

specific steps
  1. fitted input called prediction [§3.2 (augmentation selection rules, Algorithm 1) + §4.3, Eq. (6), Fig. 6]
    "Among them, wrong → correct is the most informative, as it directly encodes effective self-correction behavior. ... The rule-based reward is defined as: ... rsc(x, o1, o2) = 0.9·r′(x, o1, o2) + 0.1·rf(x, o1, o2), where rf is the format reward, and r′ is the shaped reward defined in Eq. (3) to strengthen self-correction behavior."

    Effective self-correction is operationalized as the wrong→correct transition, which receives reward 1.0 in Eq. (3). The augmentation algorithm in §3.2/Algorithm 1 preferentially selects exactly these wrong→correct pairs into the training batch, and Eq. (6) then rewards that same transition. The in-training evidence that self-correction is learned — Fig. 6(a)'s growing r(x,o2)−r(x,o1) gap and Fig. 6(b)'s stable rsc — is therefore a restatement of the selection/reward design rather than an independent measurement: the model is trained to increase a quantity the augmentation already engineered to be present and balanced. This does not make the held-out 7-benchmark comparisons circular, but the internal diagnostic is partly self-fulfilling.

full rationale

No equation in the paper defines a held-out result in terms of its own fitted values. The headline claims are measured on seven independent benchmarks with external baselines trained on the same data, so the main empirical result is not forced by Eq. (3)/(6). The TTS and pass@k analyses on MMStar also provide held-out behavioral evidence beyond the training reward. The only self-citation (Ding & Zhang 2025) is used for motivation and is backed by the paper's own sparsity measurements (Table 1, Fig. 2), so it is not load-bearing. The mild circularity is confined to the internal proof-of-concept diagnostics, which use the same wrong→correct categorization that defines both the augmentation selection and the shaped reward; those particular curves are partly by construction. Thus the overall circularity is low.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 1 invented entities

The central method introduces a new training-data construction (rollout recombination) and a two-stage objective. The free parameters are mostly hand-chosen design constants (N/n, reward coefficients, dataset sizes). The key load-bearing assumption is that spliced o1/o2 pairs teach genuine self-correction; this remains empirically plausible but not directly verified. Checkpoint selection on Geometry-3k introduces a mild selection-bias assumption that touches the benchmark claims.

free parameters (4)
  • Augmentation budget n=8 generated rollouts, N=16 effective training samples = n=8, N=16
    Hand-chosen in §3.2 to balance generation cost and augmented sample count; no sensitivity analysis is given.
  • Shaped-reward coefficients (0.9 and 0.1) in Eq. (6) = 0.9, 0.1
    Chosen by hand to balance self-correction reward and format reward; no ablation on the coefficient values is reported.
  • Cold-start dataset splits = 10k inputs; 4k/6k instances
    Approximately chosen in §4.1 to construct o1⊕<sc>⊕o2 examples; no analysis of the effect of these sizes.
  • KL coefficient in Stage I objective = unspecified
    Eq. (4) includes a KL term to keep o1 near the reference, but no coefficient value is given; this affects reproducibility.
axioms (4)
  • domain assumption Synthetic pairs (o1 from one rollout, o2 from another) are valid self-correction training examples
    §3.2 and Algorithm 1 build training samples by splicing unrelated pre- and post-correction segments; the paper assumes this teaches targeted error recovery rather than a blind restart behavior.
  • ad hoc to paper The shaped reward r' (Eq. 3) correctly orders correction quality
    The chosen ordering wrong→correct (1.0) > correct→correct (0.75) > wrong→wrong (0) > correct→wrong (-0.25) is a design choice used to incentivize self-correction; no evidence that this ordering is optimal.
  • domain assumption Checkpoint selection on Geometry-3k transfers to the 7 evaluation benchmarks
    Appendix A.1 uses Geometry-3k as the validation set for choosing the best checkpoint, while MathVista, MathVerse, and WeMath contain geometry-heavy questions; if the validation distribution is too close to the test distribution, reported gains can be inflated.
  • ad hoc to paper Masking o1 in Stage I and selectively unmasking in Stage II prevents reward hacking without degrading direct reasoning
    The two-stage masking is presented as the solution to the reward-hacking failure in §4.2; this is a method design assumption rather than a proven property.
invented entities (1)
  • None introduced no independent evidence
    purpose: No new physical or architectural entities are postulated.
    The <sc> token is a format marker, not a new model component or external entity.

pith-pipeline@v1.3.0-alltime-deepseek · 18382 in / 15087 out tokens · 157276 ms · 2026-08-03T03:14:53.639047+00:00 · methodology

0 comments
read the original abstract

Self-correction is essential for solving complex reasoning problems in vision-language models (VLMs). However, existing reinforcement learning (RL) methods struggle to learn it, as effective self-correction behaviors emerge only rarely, making learning signals extremely sparse. To address this challenge, we propose correction-specific rollouts (Octopus), an RL rollout augmentation framework that synthesizes dense self-correction examples by recombining existing rollouts. This augmentation simultaneously improves sample efficiency due to rollout reuse and stabilizes RL optimization through balanced supervision. Furthermore, we introduce a response-masking strategy that decouples self-correction from direct reasoning, avoiding signal conflicts and enabling both behaviors to be learned effectively. Building on this, we introduce Octopus-8B, a reasoning VLM with controllable self-correction capability. Across 7 benchmarks, it achieves SoTA performance among open-source VLMs, outperforming the best RLVR baseline by 1.0 score while requiring only $0.72\times$ training time per step.

Figures

Figures reproduced from arXiv: 2602.08503 by Bolian Li, Ruqi Zhang, Yi Ding, Ziliang Qiu.

Figure 1
Figure 1. Figure 1: Comparison of accuracy and training efficiency across different RL methods initialized on Qwen3-8B-VL-Instruct. Octo￾pus achieves the best average accuracy across seven benchmarks while requiring substantially less rollout time. et al., 2025). These behaviors resemble how humans tackle challenging problems, suggesting that self-correction is an important capability for strong and robust reasoning. However,… view at source ↗
Figure 2
Figure 2. Figure 2: The percentage of different correction behaviors during RL training with a self-correction–encouraging prompt. Assuming that N rollouts are required for each policy up￾date, we keep the n originally generated rollouts to avoid relying entirely on offline data. We then select an additional N − n samples from the augmented pool while balancing positive and negative examples. For positive examples, we priorit… view at source ↗
Figure 3
Figure 3. Figure 3: Left: Octopus augmentation pairs responses before and after the <sc> token to explicitly construct effective self-correction examples (wrong → correct), increasing their count from 0 to 4. It also produces an equal number of positive and negative samples (4 each), balancing the advantage distribution within each training group. Right: Our two-stage RL pipeline. In Stage I, we decouple self-correction learn… view at source ↗
Figure 4
Figure 4. Figure 4: Training dynamics of different methods. GSPO is ini￾tialized from the base πθ and trained with standard RL. In-dis and Mixed Sampling are initialized from their corresponding SFT models and trained with Octopus RL strategy introduced in § 4.3. Takeaway for Cold-start Data Construction SFT-based cold-start is necessary for learning a self￾correction format. Mixed sampling avoids entropy col￾lapse and leads … view at source ↗
Figure 5
Figure 5. Figure 5: Teaching self-correction with binary and shaped rewards. (a) Reward curves before and after self-correction under a binary reward setting, showing limited self-correction learning. (b) Re￾ward curves with the shaped reward defined in Eq. (3), highlighting the emergence of reward hacking. Setup. We compare two reward designs: (i) a standard binary reward (0/1) based on the correctness of o2, and (ii) a shap… view at source ↗
Figure 7
Figure 7. Figure 7: Test-time scaling (TTS) performance on MMStar. Left: Sequential TTS achieved by appending <sc> tokens to trigger self-correction. Green points denote the original performance without TTS, and blue points indicate responses with TTS triggers. The x-axis shows the average cumulative number of tokens during inference. Right: Comparison of pass@k performance. racy and better token efficiency than those generat… view at source ↗
Figure 8
Figure 8. Figure 8: Failure attempts. Training collapse during RL training. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. BUS: Brain-Inspired Unsupervised Self-Reflection via Backward Prediction for Multimodal Reasoning

    cs.CV 2026-07 conditional novelty 6.0

    BUS enables unsupervised self-reflection in VLMs by training models to perform backward prediction—identifying which reasoning paths precede a sampled answer—yielding improvements on 8 visual reasoning benchmarks with...

  2. BUS: Brain-Inspired Unsupervised Self-Reflection via Backward Prediction for Multimodal Reasoning

    cs.CV 2026-07 conditional novelty 6.0

    BUS trains VLMs for self-reflection without labels by verifying which sampled reasonings precede their own answers via backward prediction, improving multimodal reasoning.

Reference graph

Works this paper leans on

44 extracted references · 23 linked inside Pith · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Claude 3.5 sonnet model card addendum, 2024

    Anthropic . Claude 3.5 sonnet model card addendum, 2024. URL https://www.anthropic.com/claude-3-5-sonnet-model-card-addendum

    2024

  3. [3]

    V., R \'e , C., and Mirhoseini, A

    Brown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V., R \'e , C., and Mirhoseini, A. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787, 2024

    Pith/arXiv arXiv 2024

  4. [4]

    Are we on the right way for evaluating large vision-language models? Advances in Neural Information Processing Systems, 37: 0 27056--27087, 2024

    Chen, L., Li, J., Dong, X., Zhang, P., Zang, Y., Chen, Z., Duan, H., Wang, J., Qiao, Y., Lin, D., et al. Are we on the right way for evaluating large vision-language models? Advances in Neural Information Processing Systems, 37: 0 27056--27087, 2024

    2024

  5. [5]

    Towards reasoning era: A survey of long chain-of-thought for reasoning large language models

    Chen, Q., Qin, L., Liu, J., Peng, D., Guan, J., Wang, P., Hu, M., Zhou, Y., Gao, T., and Che, W. Towards reasoning era: A survey of long chain-of-thought for reasoning large language models. arXiv preprint arXiv:2503.09567, 2025

    Pith/arXiv arXiv 2025

  6. [6]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities

    Comanici, G., Bieber, E., Schaekermann, M., Pasupat, I., Sachdeva, N., Dhillon, I., Blistein, M., Ram, O., Zhang, D., Rosen, E., et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025

    Pith/arXiv arXiv 2025

  7. [7]

    and Zhang, R

    Ding, Y. and Zhang, R. Sherlock: Self-correcting reasoning in vision-language models. arXiv preprint arXiv:2505.22651, 2025

    arXiv 2025

  8. [8]

    Vlmevalkit: An open-source toolkit for evaluating large multi-modality models

    Duan, H., Yang, J., Qiao, Y., Fang, X., Chen, L., Liu, Y., Dong, X., Zang, Y., Zhang, P., Wang, J., et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM international conference on multimedia, pp.\ 11198--11201, 2024

    2024

  9. [9]

    Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models

    Guan, T., Liu, F., Wu, X., Xian, R., Li, Z., Liu, X., Wang, X., Chen, L., Huang, F., Yacoob, Y., et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 14375--14385, 2024

    2024

  10. [10]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  11. [11]

    P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al

    Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    Pith/arXiv arXiv 2024

  12. [12]

    Openai o1 system card

    Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Carney, A., et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    Look again, think slowly: Enhancing visual reflection in vision-language models

    Jian, P., Wu, J., Sun, W., Wang, C., Ren, S., and Zhang, J. Look again, think slowly: Enhancing visual reflection in vision-language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp.\ 9262--9281, 2025

    2025

  14. [14]

    D., Singh, A., Baumli, K., Iqbal, S., Bishop, C., Roelofs, R., et al

    Kumar, A., Zhuang, V., Agarwal, R., Su, Y., Co-Reyes, J. D., Singh, A., Baumli, K., Iqbal, S., Bishop, C., Roelofs, R., et al. Training language models to self-correct via reinforcement learning. arXiv preprint arXiv:2409.12917, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    H., Gonzalez, J

    Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023

    2023

  16. [16]

    Lambert, N., Morrison, J., Pyatkin, V., Huang, S., Ivison, H., Brahman, F., Miranda, L. J. V., Liu, A., Dziri, N., Lyu, S., et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Liu, X., Ni, J., Wu, Z., Du, C., Dou, L., Wang, H., Pang, T., and Shieh, M. Q. Noisyrollout: Reinforcing visual reasoning with data augmentation. arXiv preprint arXiv:2504.13055, 2025

    arXiv 2025

  18. [18]

    Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning

    Lu, P., Gong, R., Jiang, S., Qiu, L., Huang, S., Liang, X., and Zhu, S.-C. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. arXiv preprint arXiv:2105.04165, 2021

    Pith/arXiv arXiv 2021

  19. [19]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., and Gao, J. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023

    Pith/arXiv arXiv 2023

  20. [20]

    Self-refine: Iterative refinement with self-feedback

    Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36: 0 46534--46594, 2023

    2023

  21. [21]

    L., Tan, J

    Masry, A., Do, X. L., Tan, J. Q., Joty, S., and Hoque, E. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the association for computational linguistics: ACL 2022, pp.\ 2263--2279, 2022

    2022

  22. [22]

    Skywork r1v: Pioneering multimodal reasoning with chain-of-thought

    Peng, Y., Wang, P., Wang, X., Wei, Y., Pei, J., Qiu, W., Jian, A., Hao, Y., Pan, J., Xie, T., et al. Skywork r1v: Pioneering multimodal reasoning with chain-of-thought. arXiv preprint arXiv:2504.05599, 2025

    Pith/arXiv arXiv 2025

  23. [23]

    Qiao, R., Tan, Q., Dong, G., MinhuiWu, M., Sun, C., Song, X., Wang, J., Gongque, Z., Lei, S., Zhang, Y., et al. We-math: Does your large multimodal model achieve human-like mathematical reasoning? In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 20023--20070, 2025

    2025

  24. [24]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  25. [25]

    Srpo: Enhancing multimodal llm reasoning via reflection-aware reinforcement learning

    Wan, Z., Dou, Z., Liu, C., Zhang, Y., Cui, D., Zhao, Q., Shen, H., Xiong, J., Xin, Y., Jiang, Y., et al. Srpo: Enhancing multimodal llm reasoning via reflection-aware reinforcement learning. arXiv preprint arXiv:2506.01713, 2025

    arXiv 2025

  26. [26]

    Vl-rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning

    Wang, H., Qu, C., Huang, Z., Chu, W., Lin, F., and Chen, W. Vl-rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning. arXiv preprint arXiv:2504.08837, 2025 a

    Pith/arXiv arXiv 2025

  27. [27]

    Internvl3

    Wang, W., Gao, Z., Gu, L., Pu, H., Cui, L., Wei, X., Liu, Z., Jing, L., Ye, S., Shao, J., et al. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025 b

    Pith/arXiv arXiv 2025

  28. [28]

    Sota with less: Mcts-guided sample selection for data-efficient visual reasoning self-improvement

    Wang, X., Yang, Z., Feng, C., Lu, H., Li, L., Lin, C.-C., Lin, K., Huang, F., and Wang, L. Sota with less: Mcts-guided sample selection for data-efficient visual reasoning self-improvement. arXiv preprint arXiv:2504.07934, 2025 c

    Pith/arXiv arXiv 2025

  29. [29]

    Charxiv: Charting gaps in realistic chart understanding in multimodal llms

    Wang, Z., Xia, M., He, L., Chen, H., Liu, Y., Zhu, R., Liang, K., Wu, X., Liu, H., Malladi, S., et al. Charxiv: Charting gaps in realistic chart understanding in multimodal llms. Advances in Neural Information Processing Systems, 37: 0 113569--113697, 2024

    2024

  30. [30]

    V., Zhou, D., et al

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 0 24824--24837, 2022

    2022

  31. [31]

    Mimo-vl technical report, 2025

    Xiaomi, L.-C.-T. Mimo-vl technical report, 2025. URL https://arxiv.org/abs/2506.03569

    Pith/arXiv arXiv 2025

  32. [32]

    Llava-cot: Let vision language models reason step-by-step

    Xu, G., Jin, P., Wu, Z., Li, H., Song, Y., Sun, L., and Yuan, L. Llava-cot: Let vision language models reason step-by-step. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 2087--2098, 2025

    2087

  33. [33]

    Qwen3 technical report

    Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  34. [34]

    Dapo: An open-source llm reinforcement learning system at scale

    Yu, Q., Zhang, Z., Zhu, R., Yuan, Y., Zuo, X., Yue, Y., Dai, W., Fan, T., Liu, G., Liu, L., et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025

    Pith/arXiv arXiv 2025

  35. [35]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 9556--9567, 2024

    2024

  36. [36]

    Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837, 2025

    Yue, Y., Chen, Z., Lu, R., Zhao, A., Wang, Z., Song, S., and Huang, G. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837, 2025

    Pith/arXiv arXiv 2025

  37. [37]

    Evolving llms' self-refinement capability via iterative preference optimization

    Zeng, Y., Cui, X., Jin, X., Liu, G., Sun, Z., Li, D., Yang, N., Hao, J., Zhang, H., and Wang, J. Evolving llms' self-refinement capability via iterative preference optimization. arXiv preprint arXiv:2502.05605, 2025

    arXiv 2025

  38. [38]

    R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization

    Zhang, J., Huang, J., Yao, H., Liu, S., Zhang, X., Lu, S., and Tao, D. R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint arXiv:2503.12937, 2025

    Pith/arXiv arXiv 2025

  39. [39]

    Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pp.\ 169--186

    Zhang, R., Jiang, D., Zhang, Y., Lin, H., Guo, Z., Qiu, P., Zhou, A., Lu, P., Chang, K.-W., Qiao, Y., et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pp.\ 169--186. Springer, 2024 a

    2024

  40. [40]

    Small language models need strong verifiers to self-correct reasoning

    Zhang, Y., Khalifa, M., Logeswaran, L., Kim, J., Lee, M., Lee, H., and Wang, L. Small language models need strong verifiers to self-correct reasoning. In ACL (Findings), 2024 b

    2024

  41. [41]

    Mme-realworld: Could your multimodal llm challenge high-resolution real-world scenarios that are difficult for humans? arXiv preprint arXiv:2408.13257, 2024 c

    Zhang, Y.-F., Zhang, H., Tian, H., Fu, C., Zhang, S., Wu, J., Li, F., Wang, K., Wen, Q., Zhang, Z., et al. Mme-realworld: Could your multimodal llm challenge high-resolution real-world scenarios that are difficult for humans? arXiv preprint arXiv:2408.13257, 2024 c

    Pith/arXiv arXiv 2024

  42. [42]

    Group sequence policy optimization

    Zheng, C., Liu, S., Li, M., Chen, X.-H., Yu, B., Gao, C., Dang, K., Liu, Y., Men, R., Yang, A., et al. Group sequence policy optimization. arXiv preprint arXiv:2507.18071, 2025 a

    Pith/arXiv arXiv 2025

  43. [43]

    Llamafactory: Unified efficient fine-tuning of 100+ language models

    Zheng, Y., Zhang, R., Zhang, J., Ye, Y., Luo, Z., Feng, Z., and Ma, Y. Llamafactory: Unified efficient fine-tuning of 100+ language models. arXiv preprint arXiv:2403.13372, 2024

    Pith/arXiv arXiv 2024

  44. [44]

    Easyr1: An efficient, scalable, multi-modality rl training framework

    Zheng, Y., Lu, J., Wang, S., Feng, Z., Kuang, D., and Xiong, Y. Easyr1: An efficient, scalable, multi-modality rl training framework. https://github.com/hiyouga/EasyR1, 2025 b

    2025