Pith. sign in

REVIEW 4 major objections 5 minor 33 references

MMRefine: Unveiling the Obstacles to Robust Refinement in Multimodal Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MMRefine measures whether multimodal large language models can find and fix their own math errors by splitting every refinement attempt into six scenarios instead of comparing final accuracy alone.

desk verdict Useful benchmark with an honest limitations section, but the headline error-correction gaps inherit uncertainty from a 72%-agreement judge and a closed-model-filtered gold standard. read the letter →

arxiv 2506.04688 v1 pith:ORVL3UB4 submitted 2025-06-05 cs.CL cs.AIcs.CV

Gio Paik , Geewook Kim , Jinbae Im This is my paper
classification cs.CLcs.AIcs.CV
keywords multimodallargelanguagemodelsself-refinementerrordetectioncorrectionmathematicalreasoningLLM-as-a-judgebenchmarkRefScore
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces MMRefine, a benchmark that measures whether multimodal large language models can find and fix errors in their own mathematical reasoning. Instead of comparing only the final answer before and after refinement, MMRefine classifies each refinement attempt into one of six scenarios: a model can falsely 'correct' a right solution, confirm it, fail to spot an error, spot it, correct it, or produce a fully repaired solution. Evaluated on 17 models over 200 text and visual math problems, the benchmark reports that closed-source models repair more than 32% of flawed initial solutions while most open-source models repair less than 20%. The paper also breaks down failures by six error types and finds that model size is not the whole story: some small models outperform large ones on image-related errors. If these measurements are right, MMRefine offers a reusable instrument for deciding when self-correction helps and where it hurts.

What carries the argument

The central object is the six-scenario taxonomy of refinement outcomes. For initially correct solutions, the model either verifies the solution (VS) or wrongly flags an error (FD); for incorrect solutions, it either misses the error (RF), detects it (ED), corrects it (EC), or produces a flawless revised solution (RS). The evaluation protocol asks the model to review a solution step-by-step, stop at the first error, correct it, and regenerate from that point, and uses GPT-4O as a judge to classify the result against reference feedback generated by OpenAI O1. The metrics RefScore $= RS - FD$ and mRecall $= (ED + VS)/2$ compress the taxonomy into two numbers, while the raw scenario proportions reveal where in the chain each model loses the most.

What would settle it

Re-annotate the 200 MMRefine items with expert human judges, replace GPT-4O's scenario labels with the human labels, and recompute RefScore for all 17 models; if the closed-source advantage shrinks or reverses (for instance, if GPT-4O's RefScore falls below the best open-source model), the benchmark's central gap would not be robust.

Watch

Extended reading notes

Core claim

The central claim is that refinement is not a single skill but a chain of distinct stages—detecting an error, correcting it, and then continuing to a flawless solution—and that each stage can be measured separately. The paper defines two scalar metrics, RefScore $= RS - FD$ and mRecall $= (ED + VS)/2$, where RS and FD are the shares of fully repaired and falsely 'corrected' solutions and ED and VS are the shares of successful error detection and correct-solution verification. On its 200-problem set, the benchmark reports that closed-source models repair more than 32% of flawed solutions while most open-source models repair less than 20%, and that RefScore correlates at 0.82 with the actual score change from self-reflection on MATH-500 and MathVista. The paper further claims that error-type analysis reveals a qualitative split: larger models handle text-related errors better, while smaller models often handle image-related errors better, and spatial-reasoning errors behave almost independently of other error types.

Load-bearing premise

The load-bearing premise is that GPT-4O's automatic classification of open-ended math reasoning into the six scenarios is accurate enough to trust the reported RefScores, even though it agrees with human experts only 72% of the time.

Editorial extensions

If this is right

  • If RefScore is predictive, model developers can use MMRefine to forecast whether a self-correction step will improve or degrade a model's math performance on unseen problems.
  • The 0.82 correlation with MATH-500 and MathVista self-reflection deltas suggests that benchmark scores can stand in for expensive full self-correction evaluations in some settings.
  • Because open-source models fail mostly at error detection (high RF), the paper points to detection, not correction, as the priority bottleneck for most small models.
  • The low correlation of spatial-reasoning RefScore with other error types implies that scaling alone may not fix spatial reasoning; specialised training or prompting may be needed.
  • The PRM experiments suggest that best-of-N selection with a process reward model raises verification but lowers error detection, so reward models must be tuned with refinement, not just answer accuracy, in mind.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 72% judge agreement means roughly a quarter of scenario labels may be wrong; a natural check is to recompute RefScore with full human annotation, which could either confirm or erode the reported closed-versus-open gap.
  • The benchmark's single-reference-solution design may penalise valid alternative solution paths; an extension allowing multiple correct references would test whether the six-scenario taxonomy changes when diverse proofs are accepted.
  • The scenario taxonomy should transfer to non-mathematical multimodal reasoning such as diagram comprehension or chart reading, where 'first error' is equally well defined; that would reveal which findings are math-specific.
  • One testable extension the paper does not run: fine-tune a small open-source model on MMRefine-style detection feedback and measure whether its RefScore and downstream self-reflection gain both improve.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces MMRefine, a benchmark that evaluates how well multimodal large language models (MLLMs) detect and correct errors in mathematical solutions, including both text-only and visual problems. The benchmark assigns each refinement attempt to one of six scenarios (e.g., Refinement Failure, Error Detection Success, Refinement Success) and categorizes the first error in the initial solution into one of six error types. The authors evaluate 17 open- and closed-source MLLMs, propose two aggregated metrics (RefScore and mRecall), analyze bottlenecks by scenario and error type, and provide a correlation study with self-reflection performance on MATH-500 and MathVista. The dataset and code are released publicly.

Significance. If the benchmark's measurements are reliable, MMRefine would be a useful fine-grained instrument for studying refinement capabilities, going beyond before/after accuracy comparisons and providing a reusable taxonomy of failure modes. The paper has notable strengths: it ships public code and data, includes human verification of the annotation process, and explicitly discusses limitations such as the single-reference-solution bias. However, the central claims of a closed-source versus open-source gap and of specific bottleneck scenarios rest on (i) an aggregated metric whose definition mixes proportions over different subpopulations, and (ii) an automated judge with 72% human agreement and no reported per-category reliability breakdown. These issues are load-bearing for the paper's main conclusions, so the benchmark requires additional validation before its rankings and headline numbers can be taken as definitive.

major comments (4)
  1. [§2.2, Eq. (1)] The definition RefScore = RS − FD subtracts two proportions that are computed over different populations: RS is the proportion of incorrect initial solutions that are refined successfully, while FD is the proportion of correct initial solutions that are falsely flagged as erroneous. Consequently, RefScore is not an aggregated accuracy over a fixed population, and its scale and ranking behavior depend on the ratio of correct to incorrect initial solutions in the benchmark. A model with high RS but also high FD can be ranked below a model with low RS and low FD, purely because of the composition of the dataset. Since RefScore is used as the primary ranking metric in Tables 1, 4, 6, and 8, and for the efficiency analysis in Table 2, the metric's definition should be justified or replaced by a per-sample joint score (e.g., a weighted combination defined over all instances) with RS and FD reported separately.
  2. [§3.4] The GPT-4O judge is reported to agree with human judgments 72% of the time and with OpenAI O1 73% of the time, but the paper does not report the sample size, a confusion matrix, or a per-scenario/per-error-type breakdown. All six scenario rates, RefScore, mRecall, and the Section 3.1 claim that closed-source models outperform open-source models are computed from these labels. If the 28% of disagreements are systematic rather than random, per-model rankings and the closed-versus-open gap can change materially. The paper should provide agreement statistics per scenario, per error type, and per model, along with a chance-corrected agreement measure such as Cohen's kappa.
  3. [§2.3 and Figure 7] The evaluation protocol defines Refinement Success (RS) by whether the model's feedback 'reaches the same answer and logic as the reference feedback' (Figure 7 prompt). The reference feedback is generated by OpenAI O1 and retained only when GPT-4O, Gemini-1.5-Pro, and Claude-3.5-Sonnet all succeed in refining the initial solution using it. As a result, RS measures fidelity to a single reference repair path, not necessarily mathematical correctness. A model that corrects the error through a different but equally valid method can be scored as a failure, and the paper's own Limitations section concedes this single-reference bias. Because RS is one of the two components of RefScore and the basis for the headline 'correct errors in over 32%' claim, this conflation of correctness with reference-path agreement is a load-bearing issue that should be addressed by evaluating final-answer correctness independently of the reference feedback.
  4. [§3.2 and Table 3] The paper claims a correlation coefficient of 0.82 between MMRefine scores and self-reflection improvements on MATH-500 and MathVista, but this is computed on only four models (and on three after excluding Gemini as an exception). With n = 4, a single outlier can determine the correlation, and the text does not state whether a Pearson or Spearman coefficient is used, nor does it report the confidence interval. The claim that 'MMRefine scores are valuable indicators of refinement capability' would be stronger with more models in the correlation study or a clearly reported full-set coefficient with the outlier's influence quantified.
minor comments (5)
  1. [§3.1] The sentence 'closed-source models correct errors in over 32% of initial solutions' is ambiguous: if 'correct errors' refers to RS, Claude-3.5-Sonnet's RS is 18.95 in Table 1, contradicting the statement; if it refers to EC, the metric should be named explicitly to avoid confusion.
  2. [Table 2] The header 'RefinementCoT Refinement Efficiency' is malformed; it should likely read 'Refinement CoT' or 'Refinement Chain-of-Thought'.
  3. [§3.2] The correlation coefficient is reported without stating whether it is Pearson or Spearman; please specify the measure and the number of data points.
  4. [§2.3] The notation 'OPENAI O1' is inconsistent with the rest of the paper; use 'OpenAI O1' for consistency.
  5. [Table 4] The caption says 'Maximum and minimum values are highlighted', but no highlighting is visible in the text; please ensure the final version includes the color or boldface cues.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MMRefine measures models against an external reference and validates against independent benchmarks; its headline gaps are computed from those measurements, not from fitted inputs.

full rationale

MMRefine's claims are benchmark measurements rather than derived predictions. The six scenario labels are assigned by an external GPT-4O judge with human and O1 verification (Section 3.4), and RefScore and mRecall are arithmetic aggregates (RS−FD and (ED+VS)/2) of those labels. The reference feedback is generated by O1 and filtered through three closed-source MLLMs, but this is dataset construction, not a fitted parameter that forces the measured outcomes: open-source models are evaluated on the same items and can, and do, fail to reproduce the reference repair. Section 3.2 validates the benchmark against independent MATH-500 and MathVista self-reflection deltas (correlation 0.82), so the central claim has external content. The reported 72% human/GPT-4O agreement and the single-reference bias, which the Limitations section explicitly acknowledges, are validity limitations rather than circular derivation: they affect measurement accuracy but do not make the benchmark's conclusions equivalent to its inputs by construction.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The benchmark depends on several external judgments: the automatic judge, the reference feedback generator, the choice of initial-solution generators, and the problem sampling. None of these are fitted to the central claim, but each shapes the measurements. The only hand-tuned numeric parameter in the analyses is the PRM threshold in Appendix G.

free parameters (2)
  • Dataset sample size per source = 100 problems from MathOdyssey, 100 from MathVision
    The authors hand-picked 200 problems to construct MMRefine; all conclusions generalize from this fixed sample.
  • VisualPRM error detection threshold = 0.6
    Appendix G scans thresholds and reports optimal mRecall at 0.6; used only in the PRM analysis, not the main benchmark.
assumptions (4)
  • domain assumption GPT-4O judge categorizations are accurate enough (72% human agreement) to label refinement scenarios.
    Section 3.4 reports the agreement rates and then uses GPT-4O as the judge for all of MMRefine.
  • domain assumption O1-generated reference feedback, after validation by three MLLMs, is a valid ground truth for successful correction.
    Section 2.3 describes the reference-feedback generation and validation protocol.
  • domain assumption Initial solutions from four MLLMs are representative of realistic refinement inputs.
    Section 2.3 uses unrestricted solutions from GPT-4O, GEMINI-1.5-PRO, CLAUDE-3.5-SONNET, and LLAMA-3.2-VISION-11B.
  • domain assumption The sampled 200 problems from MathOdyssey and MathVision are diverse enough to support general conclusions about MLLM refinement.
    Appendix B describes the subject and difficulty distributions; the paper generalizes from these 200 problems to refinement capability broadly.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MMRefine: Unveiling the Obstacles to Robust Refinement in Multimodal Large Language Models." pith.science (2026). https://pith.science/paper/ORVL3UB4

@misc{pith2026250604688,
  author       = {Pith},
  title        = {Pith review of: MMRefine: Unveiling the Obstacles to Robust Refinement in Multimodal Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ORVL3UB4}},
  note         = {Machine review of arXiv:2506.04688}
}
read the original abstract

This paper introduces MMRefine, a MultiModal Refinement benchmark designed to evaluate the error refinement capabilities of Multimodal Large Language Models (MLLMs). As the emphasis shifts toward enhancing reasoning during inference, MMRefine provides a framework that evaluates MLLMs' abilities to detect and correct errors across six distinct scenarios beyond just comparing final accuracy before and after refinement. Furthermore, the benchmark analyzes the refinement performance by categorizing errors into six error types. Experiments with various open and closed MLLMs reveal bottlenecks and factors impeding refinement performance, highlighting areas for improvement in effective reasoning enhancement. Our code and dataset are publicly available at https://github.com/naver-ai/MMRefine.

Figures

Figures reproduced from arXiv: 2506.04688 by the authors.

Figure 1
Figure 1. Motivation. Error refinement fails for a variety of scenarios and error types. Systematic evaluation is vital for providing accurate feedback and enhancing performance. causing incorrect corrections and unnecessarily pro￾longed response times (Huang et al., 2024a). There￾fore, it is essential to investigate whether MLLMs can accurately identify and correct errors in their reasoning processes. However, previous studi… view at source ↗
Figure 2
Figure 2. Evaluation Protocol. We define six scenarios to evaluate MLLM refinement capabilities. cess for correct solutions, and Refinement Failure, Error Detection Success, Error Correction Success, and Refinement Success for incorrect solutions, as shown in [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 4
Figure 4. Correlation Between Refscore by Error Types. We calculate the correlation coefficients of Ref￾Score for each error type across 17 models. Because the human evaluation of such complex rea￾soning is not only highly demanding but can even be inaccurate, automated methods such as LLM-as￾a-Judge (Zheng et al., 2023) have been proposed to evaluate MLLMs’ reasoning processes. In MM￾Refine, we use GPT-4O as a judge. To ensu… view at source ↗
Figures from the paper (9 more)
Figure 5
Figure 5. Figure 5: Prompt to Perform Refinement [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Prompt to Evaluate Samples with Correct Initial Solutions You are an expert evaluator assessing the quality of feedback provided on an initial solution to a problem. Your task is to determine if the feedback is effective in guiding the initial solution towards a correc…
Figure 7
Figure 7. Figure 7: Prompt to Evaluate Samples with Incorrect Initial Solutions [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Error Type Distribution of Initial Solutions by Model [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Problem Source, Subject, and Error Type Distribution in MMRefine Review your previous reasoning about the question, then finally answer the question. Question: {question} Your Previous Solution: {previous_solution} [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Prompt to Perform Self-Reflection [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: Difficulty Distribution of Problems for Which Initial Solutions Are Incorrect by Model. Problem difficulty is determined by the number of MLLMs that correctly solve it. Specifically, if three out of the four models (GPT-4O, GEMINI-1.5-PRO, CLAUDE-3.5-SONNET, and LLAMA…
Figure 12
Figure 12. Figure 12: Refinement Failure Example. Model (CLAUDE 3.5 SONNET) fails to detect an error in step 4 of initial solution [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]
Figure 13
Figure 13. Figure 13: Error Detection Success Example. Model (GEMINI-1.5-PRO) manages to detect the initial error but fails to correct it due to a visual perception error in the refinement process [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

33 extracted references · 7 canonical work pages

  1. [1]

    Anthropic . 2024. https://www-cdn.anthropic.com/fed9cc193a14b84131812372d8d5857f8f304c52/Model_Card_Claude_3_Addendum.pdf Claude 3.5 sonnet model card addendum

    work page 2024

  2. [2]

    Justin Chih-Yao Chen, Archiki Prasad, Swarnadeep Saha, Elias Stengel-Eskin, and Mohit Bansal. 2024. https://arxiv.org/abs/2409.12147 Magicore: Multi-agent, iterative, coarse-to-fine refinement for reasoning . Preprint, arXiv:2409.12147

    arXiv 2024

  3. [3]

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li, Yimin Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, Conghui He, Botian Shi, Xingcheng Zhang, Han Lv, Yi Wang, Wenqi Shao, Pei Chu, Zhongying Tu, Tong He, Zhiyong Wu, Huipeng Deng, Jia...

    arXiv 2025

  4. [4]

    Meng Fang, Xiangpeng Wan, Fei Lu, Fei Xing, and Kai Zou. 2024. https://arxiv.org/abs/2406.18321 Mathodyssey: Benchmarking mathematical problem-solving skills in large language models using odyssey math data . Preprint, arXiv:2406.18321

    arXiv 2024

  5. [5]

    Google . 2024. https://deepmind.google/technologies/gemini/pro Gemini 1.5 pro

    work page 2024

  6. [6]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava S...

    arXiv 2024

  7. [7]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/be83ab3ecd0db773eb2dc1b0a17836a1-Abstract-round2.html Measuring mathematical problem solving with the MATH dataset . In The Thirty-fifth Conference on Neural Informati...

    work page 2021

  8. [8]

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2024 a . https://openreview.net/forum?id=IkmD3fKBPQ Large language models cannot self-correct reasoning yet . In The Twelfth International Conference on Learning Representations

    work page 2024

Show all 33 references
  1. [9]

    Xiang Huang, Sitao Cheng, Shanshan Huang, Jiayu Shen, Yong Xu, Chaoyun Zhang, and Yuzhong Qu. 2024 b . https://doi.org/10.18653/v1/2024.acl-long.274 Q uery A gent: A reliable and efficient reasoning framework with environmental feedback based self-correction . In Proceedings o...

    2024 doi

  2. [10]

    Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, Lei M Zhang, Kay McKinney, Disha Shrivastava, Cosmin Paduraru, George Tucker, Doina Precup, Feryal Behbahani, and Aleksandra Faust. 2025...

    2025

  3. [11]

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2025. https://openreview.net/forum?id=zKv8qULV6n Llava-onevision: Easy visual task transfer . Transactions on Machine Learning Research

    2025

  4. [12]

    Loka Li, Zhenhao Chen, Guangyi Chen, Yixuan Zhang, Yusheng Su, Eric Xing, and Kun Zhang. 2024 a . https://arxiv.org/abs/2402.12563 Confidence matters: Revisiting intrinsic self-correction capabilities of large language models . Preprint, arXiv:2402.12563

    2024 arXiv

  5. [13]

    Xiaoyuan Li, Wenjie Wang, Moxin Li, Junrong Guo, Yang Zhang, and Fuli Feng. 2024 b . https://doi.org/10.18653/v1/2024.findings-acl.673 Evaluating mathematical reasoning of large language models: A focus on error identification and correction . In Findings of the Association fo...

    2024 doi

  6. [14]

    Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.992 Encouraging divergent thinking in large language models through multi-agent debate . In Proceedings of the 2024 ...

    2024 doi

  7. [15]

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. https://openreview.net/forum?id=v8L0pN6EOi Let's verify step by step . In The Twelfth International Conference on Learning Re...

    2024

  8. [16]

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024. https://llava-vl.github.io/blog/2024-01-30-llava-next/ Llava-next: Improved reasoning, ocr, and world knowledge

    2024

  9. [17]

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2024. https://openreview.net/forum?id=KUNzEQMWU7 Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts . I...

    2024

  10. [18]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. https://proceed...

    2023

  11. [19]

    Oikantik Nath, Hanani Bathina, Mohammed Safi Ur Rahman Khan, and Mitesh M. Khapra. 2025. https://arxiv.org/abs/2501.07244 Can vision-language models evaluate handwritten math? Preprint, arXiv:2501.07244

    2025 arXiv

  12. [20]

    OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Mądry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Pai...

    2024 arXiv

  13. [21]

    OpenAI, :, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennet...

    2024 arXiv

  14. [22]

    Yashar Talebirad and Amirhossein Nadiri. 2023. https://arxiv.org/abs/2306.03314 Multi-agent collaboration: Harnessing the power of intelligent llm agents . Preprint, arXiv:2306.03314

    2023 arXiv

  15. [23]

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, Soroosh Mariooryad, Yifan Ding, Xinyang Geng, Fred Alcober, Roy Frostig, Mark Omernick, Lexi Walker, Cosmin Paduraru, Christina Sorokin, A...

    2024 arXiv

  16. [24]

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. 2024 a . https://openreview.net/forum?id=QWTCcxMpPA Measuring multimodal mathematical reasoning with MATH -vision dataset . In The Thirty-eight Conference on Neural Information...

    2024

  17. [25]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024 b . https://arxiv.org/abs/2409.12191 Qwen...

    2024 arXiv

  18. [26]

    Weiyun Wang, Zhangwei Gao, Lianjie Chen, Zhe Chen, Jinguo Zhu, Xiangyu Zhao, Yangzhou Liu, Yue Cao, Shenglong Ye, Xizhou Zhu, et al. 2025. https://arxiv.org/abs/2503.10291 Visualprm: An effective process reward model for multimodal reasoning . Preprint, arXiv:2503.10291

    2025 arXiv

  19. [27]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf Chain-of-thought prompting elicits reasoning...

    2022

  20. [28]

    Xueqing Wu, Yuheng Ding, Bingxuan Li, Pan Lu, Da Yin, Kai-Wei Chang, and Nanyun Peng. 2024. https://arxiv.org/abs/2412.02172 Visco: Benchmarking fine-grained critique and correction towards self-improvement in visual reasoning . Preprint, arXiv:2412.02172

    2024 arXiv

  21. [29]

    Yu, Xuming Hu, and Qingsong Wen

    Yibo Yan, Shen Wang, Jiahao Huo, Hang Li, Boyan Li, Jiamin Su, Xiong Gao, Yi-Fan Zhang, Tianlong Xu, Zhendong Chu, Aoxiao Zhong, Kun Wang, Hui Xiong, Philip S. Yu, Xuming Hu, and Qingsong Wen. 2024. https://arxiv.org/abs/2410.04509 Errorradar: Benchmarking complex mathematical...

    2024 arXiv

  22. [30]

    Yunxiang Zhang, Muhammad Khalifa, Lajanugen Logeswaran, Jaekyeom Kim, Moontae Lee, Honglak Lee, and Lu Wang. 2024. https://doi.org/10.18653/v1/2024.findings-acl.924 Small language models need strong verifiers to self-correct reasoning . In Findings of the Association for Compu...

    2024 doi

  23. [31]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595--46623

    2023

  24. [32]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  25. [33]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.