Pith. sign in

REVIEW 4 major objections 5 minor 5 cited by

BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Dataset

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read BMMR is a 109,449-question bilingual, multimodal, multi-discipline reasoning benchmark on which the strongest tested models score only about 50 percent.

desk verdict BMMR is a substantial new bilingual multimodal reasoning resource, but its headline scores rest on unindependently verified labels and inconsistent evaluation subsets; worth reviewing, not yet fully trustworthy. read the letter →

arxiv 2507.03483 v2 pith:R6E7ZAHU submitted 2025-07-04 cs.CL cs.AI

Zhiheng Xi , Guanyu Li , Yutao Fan , Honglin Guo , Yufang Liu , Xiaoran Fan , Jiaqi Liu , Jingchao Ding
show 8 more authors
Wangmeng Zuo Zhenfei Yin Lei Bai Tao Ji Tao Gui Qi Zhang Philip Torr Xuanjing Huang
This is my paper
classification cs.CLcs.AI
keywords bilingualmultimodaldatasetmultidisciplinaryreasoningcollege-levelquestionansweringpathsprocessrewardmodellargemodelsbenchmarkevaluationopen-sourcefine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

BMMR is a newly built dataset of 109,449 college-level questions in English and Chinese, drawn from books, exams, and quizzes and organized into a 20,458-item evaluation set and an 88,991-item training set. The paper tries to establish that this is both a hard evaluation instrument and a useful training resource for multimodal AI models: the strongest tested models score only about 38–50 percent, while fine-tuning open-source models on the training split lifts their scores, with a 78-billion-parameter model improving by 19.07 percent overall. Every instance is paired with a reasoning path, and the authors add a process-based verifier that grades each step of a model's reasoning. If the dataset is sound, the field gains a common instrument for measuring and improving broad, cross-disciplinary multimodal reasoning rather than math alone.

What carries the argument

The central object is the BMMR dataset itself, split into BMMR-Eval and BMMR-Train. The companion mechanism is BMMR-Verifier, a process reward model that takes a question, a reference solution, and a model response, labels each reasoning step as positive or negative, and averages the step scores into a response-level score. The machinery that produces the dataset is a six-stage curation pipeline: taxonomy-guided collection from print and digital sources, preprocessing, hierarchical discipline classification, safety and objectivity filtering with self-consistency validation, format transformation and augmentation, and cascaded quality control with human verification. This pipeline is what carries the paper's quality and balance claims.

What would settle it

Select 500 random BMMR-Eval instances, have subject-matter experts who did not help build the dataset solve and grade them from scratch, and compare their verdicts with the published reference answers; if expert agreement falls well below the 95 percent agreement the paper reports between human annotators and its verifier, the benchmark scores and fine-tuning conclusions are not reliable.

Watch

Extended reading notes

Core claim

The paper's central claim is that BMMR fills a gap left by existing multimodal benchmarks: it is larger (109,449 instances), broader (300 subject areas under a standard education taxonomy), bilingual (English and Chinese), and college-level, with every question paired with a reference reasoning path. The authors argue that BMMR-Eval is genuinely hard, reporting that the highest-scoring tested model, Gemini-2.5-Pro, reaches 50.15 and that o3 reaches 38.06 on a matched subset, while open-source models trail by a wide margin. They also argue that BMMR-Train is useful, because fine-tuning five open-source models on it improves their performance, with the 78B InternVL2.5 model gaining 19.07 percent overall. A companion finding is that reasoning-specialized models are not uniformly stronger across disciplines and exhibit clear subject bias, which the authors take as evidence that broad multi-discipline evaluation is necessary.

Load-bearing premise

The reference answers and reasoning paths in BMMR are assumed to be correct; they are checked by automated self-consistency sampling and by a pool of 40 human annotators, but not by independent expert review of every item.

Editorial extensions

If this is right

  • BMMR-Eval leaves substantial headroom for every model tested, so claims of broad expert-level multimodal reasoning should be checked against a more diverse college-level instrument than current benchmarks offer.
  • Fine-tuning open-source models on BMMR-Train improves all five tested sizes, with the 78B model gaining 19.07 percent overall, so the dataset can serve as training data rather than only as an exam.
  • Reasoning-specialized models show discipline bias, outperforming general multimodal models mainly in technical subjects, so aggregate scores should be reported per discipline to avoid hiding the bias.
  • BMMR-Verifier's step-level scores agree with human judgments at 93.71 percent and locate flawed steps, making failure modes such as overthinking and hallucination visible in analysis.
  • The eval set spans 300 subject areas in two languages, giving a single instrument for comparing knowledge breadth and reasoning depth across disciplines and languages.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next experiment the paper does not run is to use BMMR-Train's reasoning paths and BMMR-Verifier step labels as reward signals for reinforcement learning in non-mathematical disciplines; if it works, the verifier becomes a training tool, not just an analysis tool.
  • Because BMMR is bilingual, a matched translation study could separate language ability from domain knowledge by comparing model scores on the same questions in both languages; the paper reports balanced averages but does not construct those matched pairs.
  • The source pool overlaps with the book and exam styles models have likely seen, so a fresh exam set drawn from the same 300 subject areas but from new sources would test whether the headroom on BMMR-Eval reflects reasoning rather than unfamiliar formatting.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. BMMR is a proposed bilingual (English/Chinese) multimodal reasoning dataset of 109,449 college-level instances spanning 8 top-level UNESCO disciplines and 300 subfields, split into BMMR-Eval (20,458) and BMMR-Train (88,991), with each instance paired with a reference answer and a reasoning path. The paper also presents BMMR-Verifier, a process-based verifier trained on 140k sampled responses with rule-based step labels, and reports experiments on 24 open and proprietary models, finding that SOTA models score 38–50%, that reasoning models show subject bias, that open models trail proprietary ones, and that fine-tuning on BMMR-Train gives gains of up to about 19%. The central claims are the dataset's scale, diversity, and usefulness as both an evaluation and a training resource.

Significance. If the reference answers and reasoning paths are correct, BMMR would fill a real gap in the benchmarking landscape: it is larger than MMMU, bilingual, organized by a principled UNESCO taxonomy, and it provides a training split in addition to an eval split. The fine-tuning results across five open-source models and the proposed process verifier are potentially useful contributions. However, the benchmark's value rests on the correctness of its ground truth, and the paper's own curation appendix shows that most items never receive independent human verification of the reference answer or reasoning path. With even a few percent label errors, the reported model rankings, open-versus-proprietary gaps, and fine-tuning gains could shift. The contribution is therefore promising but not yet established at the claimed level.

major comments (4)
  1. [Appendix A (Quality control and distribution balancing)] The curation cascade retains open-ended items with agreement rates between 0.2 and 0.6 and multiple-choice items between 0.3 and 0.6, computed against 'our annotated ground truth'; only items below these bands after GPT-4o sampling are sent to the 40 annotators for manual verification. Items in the retained medium-agreement bands therefore never receive independent verification of the reference answer or reasoning path, even though they form a large part of BMMR-Eval. GPT-4o self-consistency cannot detect a reference answer that is wrong but consistently reproducible, and the medium-agreement band by construction includes cases where weaker models disagree with the reference. Since every headline number in the paper (o3 38.06, Gemini-2.5-Pro 50.15, the 19.07% fine-tuning gain, and the BMMR-Verifier agreement rates) is computed against these reference labels, an unmeasured label-error rate can reorder model rankings. Please audit a random sample from all agreement bands with expert annotators, report per-band label-error rates, and either release per-item provenance or filter the eval set accordingly.
  2. [§5.1 (Implementation details) and Table 2] Proprietary models (Gemini-2.5-Pro, o3, o4-mini) are evaluated on TestMini, a 5.4k subset, while open-source models are evaluated on the full BMMR-Eval; Table 2 reports these scores in the same columns without marking the subset difference. The claims that 'SOTA models leave substantial headroom' and that 'open-source models still trail proprietary counterparts' are based on cross-subset comparisons. Distribution matching on categories does not establish equal item difficulty. Please evaluate all models on the full BMMR-Eval, or at least report matched comparisons on the same TestMini for the open models, with confidence intervals; otherwise the open-versus-proprietary gap can be an artifact of subset difficulty.
  3. [§5.4 (Effectiveness of BMMR-Verifier) and Table 3] The verifier's agreement is measured against GPT-4o judgments and annotations from 'college students from diverse academic backgrounds', not against domain experts, and the reference reasoning paths used as the gold standard are the same unverified paths from Appendix A. High agreement with a non-expert human standard does not establish that the process scores are correct; it can reflect shared mistakes. Please report inter-annotator agreement (e.g., Cohen's kappa), expert spot-checks for a sample of disciplines, and agreement broken down by discipline, language, and question type.
  4. [Appendix A versus §5.2 (GPT-4o's dual role)] GPT-4o is used at multiple curation stages (discipline tagging, safety filtering, self-consistency validation, and final agreement filtering) and also appears as an evaluated model in Table 2. This is not a formal circularity, but it creates a risk that the eval set is calibrated to GPT-4o's own error patterns; the cascade uses GPT-4o agreement rates to decide retention, and GPT-4o is then evaluated on the resulting set. To assess the impact, please compare model rankings on a held-out set curated without GPT-4o involvement, or at minimum report how scores change when items in the highest and lowest agreement bands are excluded. A sentence in the limitations section stating this risk would be a minimum requirement.
minor comments (5)
  1. [§5.2 and Table 2] The text says 'Gemini Pro attains a performance of 51.15', but Table 2 and the abstract report 50.15; please correct the inconsistency.
  2. [§5.1 and Figure 5] The text says 24 models are evaluated, while the Figure 5 caption says '23 models from 8 distinct series'; please make the count consistent.
  3. [§5.3 and Figure 1] Fine-tuning gains are expressed inconsistently: Section 5.3 reports a '72.28% improvement on ICTs' and '43.34% improvement on Health', while Figure 1 labels gains as '+69.38%', '+19.29%', and similar; please specify whether these are relative or absolute improvements and state the per-discipline basis.
  4. [Throughout] Several typos should be fixed: 'splited' (§3.1), 'sampld' and 'varients' (§5.1), 'realing' (§5.2), 'frow' (Appendix A), and 'Projectet Site' (page 1).
  5. [Table 2] Please define how the 'Avg.' column is computed (macro-average over disciplines versus micro-average over items); the two can differ when discipline sizes are unbalanced, as the current table suggests.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: BMMR's headline results are empirical measurements against reference answers, not derivations that reduce to their inputs.

full rationale

This is a benchmark construction and empirical evaluation paper, so the classical circular-derivation failure mode does not apply. The central claims (model scores on BMMR-Eval, fine-tuning gains from BMMR-Train, and BMMR-Verifier agreement rates) are measured outcomes, not quantities derived from the same fitted parameters or from the paper's own definitions. The reference answers were produced by a curation pipeline involving Qwen2.5-7B, Qwen2.5-72B, GPT-4o self-consistency sampling, and a pool of 40 human annotators; even though GPT-4o participates in curation and also appears in the evaluated model set, the benchmark labels are not defined as GPT-4o's outputs, and the reported scores are independently measured against those labels. The BMMR-Verifier is trained with rule-based labels and then validated against GPT-4o scores and human annotations (Table 3), which are external checks rather than the verifier's own training labels. The self-citations in the paper, such as references [44] and [56], appear only in related-work or methodological-context passages and are not load-bearing for the paper's main claims. The difficulty filter that retains questions with agreement rates between 0.2 and 0.6 is an explicit selection criterion for benchmark construction, not a hidden fit that is later repackaged as a prediction. The acknowledged limitations, such as coverage of only English and Chinese and incomplete UNESCO subject coverage, are data-coverage limitations and do not indicate circular reasoning. Overall, no circular step could be exhibited with a specific reduction, so the appropriate finding is no significant circularity.

Assumptions & free parameters 1 free parameters · 2 assumptions · 0 invented entities

The dataset construction relies on manual agreement thresholds and on the correctness of the curation pipeline. There is no fitted physical or mathematical parameter, and no new theoretical entity is introduced; the only invented artifact is the BMMR-Verifier model, which is a trained model rather than a postulated entity.

free parameters (1)
  • Agreement retention thresholds in quality control = open-ended 0.2-0.6; multiple-choice 0.3-0.6
    Chosen by hand in Appendix A 'Quality control and distribution balancing' to keep mid-difficulty questions; they determine which items enter BMMR-Eval and BMMR-Train and thus affect every reported accuracy and fine-tuning gain.
assumptions (2)
  • domain assumption Reference answers and reasoning paths in BMMR are correct after human-in-the-loop validation.
    Appendix A describes validation by GPT-4o self-consistency and 40 annotators, not independent per-item expert review; all benchmark scores, fine-tuning gains, and verifier labels depend on this ground truth.
  • domain assumption GPT-4o self-consistency and quality thresholds select questions that are genuinely multimodal and at college difficulty.
    Appendix A 'Safety and objectivity check, and self-consistency validation' uses GPT-4o to flag domain-knowledge requirements and image-text alignment; this assumes the filter does not encode GPT-4o's blind spots into the benchmark.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Dataset." pith.science (2026). https://pith.science/paper/R6E7ZAHU

@misc{pith2026250703483,
  author       = {Pith},
  title        = {Pith review of: BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Dataset},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R6E7ZAHU}},
  note         = {Machine review of arXiv:2507.03483}
}
read the original abstract

In this paper, we introduce BMMR, a large-scale bilingual, multimodal, multi-disciplinary reasoning dataset for the community to develop and evaluate large multimodal models (LMMs). BMMR comprises 110k college-level questions spanning 300 UNESCO-defined subjects, spanning diverse formats-multiple-choice, fill-in-the-blank, and open-ended QA-and sourced from both print and digital media such as books, exams, and quizzes. All data are curated and filtered via a human-in-the-loop and scalable framework, and each instance is paired with a high-quality reasoning path. The dataset is organized into two parts: BMMR-Eval that comprises 20,458 high-quality instances to comprehensively assess LMMs' knowledge and reasoning across multiple disciplines in both Chinese and English; and BMMR-Train that contains 88,991 instances to support further research and development, extending the current focus on mathematical reasoning to diverse disciplines and domains. In addition, we propose the process-based multi-discipline verifier (i.e., BMMR-Verifier) for accurate and fine-grained evaluation of reasoning paths. Extensive experiments on 24 models reveal that (i) even SOTA models (e.g., o3 and Gemini-2.5-Pro) leave substantial headroom on BMMR-Eval; (ii) reasoning models exhibit discipline bias and outperform LMMs only on specific subjects; (iii) open-source models still trail their proprietary counterparts; and (iv) fine-tuning on BMMR-Train narrows this gap. Additionally, we conduct reasoning-chain analyses using BMMR-Verifier and other in-depth studies, uncovering the challenges LMMs currently face in multidisciplinary reasoning. We will release the data, and we hope our work can offer insights and contributions to the community.

Figures

Figures reproduced from arXiv: 2507.03483 by the authors.

Figure 1
Figure 1. Overview of the BMMR dataset. It encompasses [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Performance of fine-tuned open-source models with BMMR-Train. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Score distribution in different models Overall Health Bus. ICTs Arts Agri. Soc. Sci. Nat. Sci. Eng. 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Average Process Score InternVL3-14B LLaVA-OneVision-72B InternVL2.5-8B-MPO QVQ-72B-Preview InternVL2.5-78B-MPO InternVL3-78B Qwen2.5-VL-72B-Instruct Gemini-2.5-flash-thinking [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Average reasoning path scores across top-level disciplines predicted by BMMR-Verifier. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Overall performance on BMMR-Eval of 23 models from 8 distinct series with respect to [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Error distribution on BMMR-Eval. In this section, we conduct a fine-grained error analysis on 19k re￾sponses sampled from different models. We provide the incorrect reasoning responses to GPT-4o for error classification, and the results are presented in [PITH_FULL_IMA…
Figure 7
Figure 7. Figure 7: Error case of overthinking. Inputs. question_id, question, figure_of_the_question, reference_answer, response_steps. Allowed tools. Same as above. Procedure. 1. Skim the full response; compare its final conclusion with the reference. 2. For every step, mark it Correct,…
Figure 8
Figure 8. Figure 8: Error case of hallucination. as possible, while the dataset still does not cover all the subjects in the International Standard Classification of Education released by the UNESCO [20]. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Zone of Proximal Policy Optimization: Teacher in Prompts, Not Gradients

    cs.CL 2026-06 unverdicted novelty 7.0 of 10

    ZPPO improves distillation to small vision-language models by using binary and negative candidate prompts plus a replay buffer for hard questions, outperforming standard distillation and GRPO on a 31-benchmark suite w...

  2. Towards Characterizing Scientific Image Utility and Upgradability

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    The SIU²A framework evaluates scientific images for error detection, repair feasibility, and correction quality, showing current multimodal systems have major limitations in preserving scientific validity.

  3. When Prompts Become Pixels: Prompt-Region Grounding for Multimodal Reasoning

    cs.AI 2026-08 conditional novelty 6.0 of 10

    Moving a question from typed prompt to pixels drops MLLM accuracy by 17.8 points on average, and prompt-region grounding recovers 8.3 points of the four-benchmark VTS loss at matched training cost.

  4. SciAgentGym: Benchmarking Multi-Step Scientific Tool-use in LLM Agents

    cs.CL 2026-02 conditional novelty 6.0 of 10

    SciAgent-8B, fine-tuned on trajectories synthesized from a tool dependency graph, outperforms Qwen3-VL-235B-Instruct on SciAgentBench, a new 259-task benchmark for multi-step scientific tool-use.

  5. InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency

    cs.CV 2025-08 unverdicted novelty 6.0 of 10

    InternVL3.5 advances open-source multimodal models with Cascade RL for +16% reasoning gains and ViR for 4x inference speedup, with the 241B model reaching SOTA among open-source MLLMs on multimodal, reasoning, and age...

Reference graph

Works this paper leans on

93 extracted references · 17 canonical work pages · cited by 5 Pith papers

  1. [1]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  2. [2]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024

    arXiv 2024

  3. [3]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479, 2025

    arXiv 2025

  4. [4]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

  5. [5]

    MMMU: A Massive Multi-discipline Multimodal Understanding and Reason- ing Benchmark for Expert AGI, November 2023

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. MMMU: A Massive Multi-discipline Multimodal Understanding and Reason- ing Benchmark for E...

    2023

  6. [6]

    Sci- enceqa: A novel resource for question answering on scholarly articles

    Tanik Saikh, Tirthankar Ghosal, Amish Mittal, Asif Ekbal, and Pushpak Bhattacharyya. Sci- enceqa: A novel resource for question answering on scholarly articles. International Journal on Digital Libraries, 23(3):289–301, 2022

    2022

  7. [7]

    What can large language models do in chemistry? a comprehensive benchmark on eight tasks

    Taicheng Guo, Bozhao Nan, Zhenwen Liang, Zhichun Guo, Nitesh Chawla, Olaf Wiest, Xi- angliang Zhang, et al. What can large language models do in chemistry? a comprehensive benchmark on eight tasks. Advances in Neural Information Processing Systems, 36:59662– 59688, 2023

    2023

  8. [8]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    arXiv 2024

Show all 93 references
  1. [9]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  2. [10]

    Introducing openai o3 and o4-mini

    OpenAI. Introducing openai o3 and o4-mini. https://openai.com/index/ introducing-o3-and-o4-mini , 2025

    2025

  3. [11]

    Claude 3.7 sonnet

    Anthropic. Claude 3.7 sonnet. https://www.anthropic.com/news/claude-3-7-sonnet , 2025

    2025

  4. [12]

    Supergpqa: Scaling llm evaluation across 285 graduate disciplines

    Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, King Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, et al. Supergpqa: Scaling llm evaluation across 285 graduate disciplines. arXiv preprint arXiv:2502.14739, 2025. 10

    2025 arXiv

  5. [13]

    P-mmeval: A parallel multilingual multitask benchmark for consistent evaluation of llms, 2024

    Yidan Zhang, Yu Wan, Boyi Deng, Baosong Yang, Haoran Wei, Fei Huang, Bowen Yu, Junyang Lin, Fei Huang, and Jingren Zhou. P-mmeval: A parallel multilingual multitask benchmark for consistent evaluation of llms, 2024. URL https://arxiv.org/abs/2411.09116

    2024

  6. [14]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Information Pr...

    2024

  7. [15]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024

    2024

  8. [16]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  9. [17]

    Cumulative reasoning with large language models

    Yifan Zhang, Jingqin Yang, Yang Yuan, and Andrew Chi-Chih Yao. Cumulative reasoning with large language models. arXiv preprint arXiv:2308.04371, 2023

    2023 arXiv

  10. [19]

    Multimodal ArXiv: A dataset for improving scientific comprehension of large vision-language models

    Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. Multimodal ArXiv: A dataset for improving scientific comprehension of large vision-language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Mee...

    2024 doi

  11. [20]

    International standard classification of education

    UNESCO. International standard classification of education. fields of education and training 2013 (isced f 2013). detailed field descriptions. UNESCO Institute for Statistics, 2015

    2013

  12. [21]

    Mind with eyes: from language reasoning to multimodal reasoning

    Zhiyu Lin, Yifei Gao, Xian Zhao, Yunfan Yang, and Jitao Sang. Mind with eyes: from language reasoning to multimodal reasoning. arXiv preprint arXiv:2503.18071, 2025

    2025 arXiv

  13. [22]

    Multimodal chain-of-thought reasoning: A comprehensive survey

    Yaoting Wang, Shengqiong Wu, Yuecheng Zhang, Shuicheng Yan, Ziwei Liu, Jiebo Luo, and Hao Fei. Multimodal chain-of-thought reasoning: A comprehensive survey. arXiv preprint arXiv:2503.12605, 2025

    2025 arXiv

  14. [23]

    Perception, reason, think, and plan: A survey on large multimodal reasoning models

    Yunxin Li, Zhenyu Liu, Zitao Li, Xuanyu Zhang, Zhenran Xu, Xinyu Chen, Haoyuan Shi, Shenyuan Jiang, Xintong Wang, Jifang Wang, et al. Perception, reason, think, and plan: A survey on large multimodal reasoning models. arXiv preprint arXiv:2505.04921, 2025

    2025 arXiv

  15. [24]

    Generative verifiers: Reward modeling as next-token prediction

    Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint arXiv:2408.15240, 2024

    2024 arXiv

  16. [25]

    Math-shepherd: Verify and reinforce llms step-by-step without human annotations

    Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. arXiv preprint arXiv:2312.08935, 2023

    2023 arXiv

  17. [26]

    Visualprm: An effective process reward model for multimodal reasoning

    Weiyun Wang, Zhangwei Gao, Lianjie Chen, Zhe Chen, Jinguo Zhu, Xiangyu Zhao, Yangzhou Liu, Yue Cao, Shenglong Ye, Xizhou Zhu, et al. Visualprm: An effective process reward model for multimodal reasoning. arXiv preprint arXiv:2503.10291, 2025

    2025 arXiv

  18. [27]

    Do not think that much for 2+ 3=? on the overthinking of o1-like llms

    Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al. Do not think that much for 2+ 3=? on the overthinking of o1-like llms. arXiv preprint arXiv:2412.21187, 2024. 11

    2024 arXiv

  19. [28]

    Missing premise exacerbates overthinking: Are reasoning models losing critical thinking skill? arXiv preprint arXiv:2504.06514, 2025

    Chenrui Fan, Ming Li, Lichao Sun, and Tianyi Zhou. Missing premise exacerbates overthinking: Are reasoning models losing critical thinking skill? arXiv preprint arXiv:2504.06514, 2025

    2025 arXiv

  20. [29]

    Halle-control: controlling object hallucination in large multimodal models

    Bohan Zhai, Shijia Yang, Chenfeng Xu, Sheng Shen, Kurt Keutzer, Chunyuan Li, and Manling Li. Halle-control: controlling object hallucination in large multimodal models. arXiv preprint arXiv:2310.01779, 2023

    2023 arXiv

  21. [30]

    Hal-eval: A universal and fine-grained hallucination evaluation framework for large vision language models

    Chaoya Jiang, Hongrui Jia, Mengfan Dong, Wei Ye, Haiyang Xu, Ming Yan, Ji Zhang, and Shikun Zhang. Hal-eval: A universal and fine-grained hallucination evaluation framework for large vision language models. In Proceedings of the 32nd ACM International Conference on Multimedia,...

    2024

  22. [31]

    The second half

    Shunyu Yao. The second half. https://ysymyth.github.io/The-Second-Half , 2025

    2025

  23. [32]

    Elevater: A benchmark and toolkit for evaluating language-augmented visual models

    Chunyuan Li, Haotian Liu, Liunian Li, Pengchuan Zhang, Jyoti Aneja, Jianwei Yang, Ping Jin, Houdong Hu, Zicheng Liu, Yong Jae Lee, et al. Elevater: A benchmark and toolkit for evaluating language-augmented visual models. Advances in Neural Information Processing Systems, 35:92...

    2022

  24. [33]

    Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models

    Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision...

    2015

  25. [34]

    Shikra: Unleashing multimodal llm’s referential dialogue magic

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023

    2023 arXiv

  26. [35]

    Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark

    Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, et al. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. arXiv preprint arXiv:2409.02813, 2024

    2024 arXiv

  27. [36]

    Gemini 2.5

    Google DeepMind. Gemini 2.5. https://blog.google/technology/google-deepmind/ gemini-model-thinking-updates-march-2025 , 2025

    2025

  28. [37]

    MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts, January 2024

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts, January 2024. URL http://arxiv. org/abs/2310.02255....

    2024 arXiv

  29. [38]

    MathVerse: Does Your Multi- modal LLM Truly See the Diagrams in Visual Math Problems?, August 2024

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, and Hongsheng Li. MathVerse: Does Your Multi- modal LLM Truly See the Diagrams in Visual Math Problems?, August 2024. URL http: //arxiv.org/abs/2403.14624...

    2024 arXiv

  30. [39]

    Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402....

    2024 arXiv

  31. [40]

    Dynamath: A dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models

    Chengke Zou, Xingang Guo, Rui Yang, Junyu Zhang, Bin Hu, and Huan Zhang. Dynamath: A dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models. arXiv preprint arXiv:2411.00836, 2024

    2024 arXiv

  32. [41]

    Cmmu: A benchmark for chinese multi-modal multi-type question understanding and reasoning

    Zheqi He, Xinya Wu, Pengfei Zhou, Richeng Xuan, Guang Liu, Xi Yang, Qiannan Zhu, and Hua Huang. Cmmu: A benchmark for chinese multi-modal multi-type question understanding and reasoning. arXiv preprint arXiv:2401.14011, 2024

    2024 arXiv

  33. [42]

    R-bench: Graduate-level multi-disciplinary benchmarks for llm & mllm complex reasoning evaluation

    Meng-Hao Guo, Jiajun Xu, Yi Zhang, Jiaxi Song, Haoyang Peng, Yi-Xuan Deng, Xinzhi Dong, Kiyohiro Nakayama, Zhengyang Geng, Chen Wang, et al. R-bench: Graduate-level multi-disciplinary benchmarks for llm & mllm complex reasoning evaluation. arXiv preprint arXiv:2505.02018, 2025. 12

    2025 arXiv

  34. [43]

    Mmsci: A multimodal multi-discipline dataset for phd-level scientific comprehension

    Zekun Li, Xianjun Yang, Kyuri Choi, Wanrong Zhu, Ryan Hsieh, HyeonJung Kim, Jin Hyuk Lim, Sungyoung Ji, Byungju Lee, Xifeng Yan, et al. Mmsci: A multimodal multi-discipline dataset for phd-level scientific comprehension. In AI for Accelerated Materials Design-Vienna 2024, 2024

    2024

  35. [44]

    Worldsimbench: Towards video generation models as world simulators

    Yiran Qin, Zhelun Shi, Jiwen Yu, Xijun Wang, Enshen Zhou, Lijun Li, Zhenfei Yin, Xihui Liu, Lu Sheng, Jing Shao, et al. Worldsimbench: Towards video generation models as world simulators. arXiv preprint arXiv:2410.18072, 2024

    arXiv

  36. [45]

    Spatialvlm: Endowing vision-language models with spatial reasoning capabilities

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14455–14465, 2024

    2024

  37. [46]

    Mm-spatial: Exploring 3d spatial understanding in multimodal llms

    Erik Daxberger, Nina Wenzel, David Griffiths, Haiming Gang, Justin Lazarow, Gefen Kohavi, Kai Kang, Marcin Eichner, Yinfei Yang, Afshin Dehghan, et al. Mm-spatial: Exploring 3d spatial understanding in multimodal llms. arXiv preprint arXiv:2503.13111, 2025

    2025 arXiv

  38. [47]

    Spatialbot: Precise spatial understanding with vision language models

    Wenxiao Cai, Iaroslav Ponomarenko, Jianhao Yuan, Xiaoqi Li, Wankou Yang, Hao Dong, and Bo Zhao. Spatialbot: Precise spatial understanding with vision language models. arXiv preprint arXiv:2406.13642, 2024

    2024 arXiv

  39. [48]

    Llava-cot: Let vision language models reason step-by-step, 2024

    Guowei Xu, Peng Jin, Hao Li, Yibing Song, Lichao Sun, and Li Yuan. Llava-cot: Let vision language models reason step-by-step, 2024. URL https://arxiv.org/abs/2411.10440

    2024 arXiv

  40. [49]

    Mammoth-vl: Eliciting multimodal reasoning with instruction tuning at scale, 2024

    Jarvis Guo, Tuney Zheng, Yuelin Bai, Bo Li, Yubo Wang, King Zhu, Yizhi Li, Graham Neubig, Wenhu Chen, and Xiang Yue. Mammoth-vl: Eliciting multimodal reasoning with instruction tuning at scale, 2024. URL https://arxiv.org/abs/2412.05237

    2024 arXiv

  41. [50]

    Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning

    Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Tiancheng Han, Botian Shi, Wenhai Wang, Junjun He, et al. Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2503.07365, 2025

    2025 arXiv

  42. [51]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In The Twelfth International Conference on Learning Representations, 2023

    2023

  43. [52]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  44. [53]

    Interpretable preferences via multi-objective reward modeling and mixture-of-experts

    Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. arXiv preprint arXiv:2406.12845, 2024

    2024 arXiv

  45. [54]

    Llm critics help catch llm bugs

    Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishinskaya, Maja Trebacz, and Jan Leike. Llm critics help catch llm bugs. arXiv preprint arXiv:2407.00215, 2024

    2024 arXiv

  46. [55]

    Improve mathematical reasoning in language models by automated process supervision

    Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, et al. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint arXiv:2406.06592, 2024

    2024 arXiv

  47. [56]

    Better process supervision with bi-directional rewarding signals

    Wenxiang Chen, Wei He, Zhiheng Xi, Honglin Guo, Boyang Hong, Jiazheng Zhang, Rui Zheng, Nijun Li, Tao Gui, Yun Li, et al. Better process supervision with bi-directional rewarding signals. arXiv preprint arXiv:2503.04618, 2025

    2025 arXiv

  48. [57]

    Bandit based monte-carlo planning

    Levente Kocsis and Csaba Szepesvári. Bandit based monte-carlo planning. In European conference on machine learning, pages 282–293. Springer, 2006

    2006

  49. [58]

    Efficient selectivity and backup operators in monte-carlo tree search

    Rémi Coulom. Efficient selectivity and backup operators in monte-carlo tree search. In International conference on computers and games, pages 72–83. Springer, 2006. 13

    2006

  50. [59]

    Ovm, outcome-supervised value models for planning in mathematical reasoning

    Fei Yu, Anningzhe Gao, and Benyou Wang. Ovm, outcome-supervised value models for planning in mathematical reasoning. arXiv preprint arXiv:2311.09724, 2023

    2023 arXiv

  51. [60]

    Process reward model with q-value rankings

    Wendi Li and Yixuan Li. Process reward model with q-value rankings. arXiv preprint arXiv:2410.11287, 2024

    2024 arXiv

  52. [61]

    Rewarding progress: Scaling automated process verifiers for llm reasoning

    Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for llm reasoning. arXiv preprint arXiv:2410.08146, 2024

    arXiv

  53. [62]

    Salmon: Self-alignment with instructable reward models.arXiv preprint arXiv:2310.05910, 2023

    Zhiqing Sun, Yikang Shen, Hongxin Zhang, Qinhong Zhou, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Salmon: Self-alignment with instructable reward models.arXiv preprint arXiv:2310.05910, 2023

    2023 arXiv

  54. [63]

    Constitutional ai: Harmlessness from ai feedback

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022

    2022 arXiv

  55. [64]

    Improv- ing reinforcement learning from human feedback using contrastive rewards

    Wei Shen, Xiaoying Zhang, Yuanshun Yao, Rui Zheng, Hongyi Guo, and Yang Liu. Improv- ing reinforcement learning from human feedback using contrastive rewards. arXiv preprint arXiv:2403.07708, 2024

    2024 arXiv

  56. [65]

    Examining false positives under inference scaling for mathematical reasoning

    Yu Wang, Nan Yang, Liang Wang, and Furu Wei. Examining false positives under inference scaling for mathematical reasoning. CoRR, abs/2502.06217, 2025. doi: 10.48550/ARXIV .2502. 06217. URL https://doi.org/10.48550/arXiv.2502.06217

    2025 doi

  57. [66]

    Llm reasoners: New evaluation, library, and analysis of step-by-step reasoning with large language models, 2024

    Shibo Hao, Yi Gu, Haotian Luo, Tianyang Liu, Xiyan Shao, Xinyuan Wang, Shuhua Xie, Haodi Ma, Adithya Samavedhi, Qiyue Gao, Zhen Wang, and Zhiting Hu. Llm reasoners: New evaluation, library, and analysis of step-by-step reasoning with large language models, 2024. URL https://ar...

    2024 arXiv

  58. [67]

    When benchmarks are targets: Revealing the sen- sitivity of large language model leaderboards

    Norah Alzahrani, Hisham Alyahya, Yazeed Alnumay, Sultan AlRashed, Shaykhah Alsub- aie, Yousef Almushayqih, Faisal Mirza, Nouf Alotaibi, Nora Al-Twairesh, Areeb Alow- isheq, M Saiful Bari, and Haidar Khan. When benchmarks are targets: Revealing the sen- sitivity of large langua...

  59. [68]

    LLMs may perform MCQA by selecting the least incorrect option

    Haochun Wang, Sendong Zhao, Zewen Qiang, Nuwa Xi, Bing Qin, and Ting Liu. LLMs may perform MCQA by selecting the least incorrect option. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert, editors, Proceedings of the 31s...

    2025

  60. [69]

    Llm-evaluation tropes: Perspectives on the validity of llm-evaluations, 2025

    Laura Dietz, Oleg Zendel, Peter Bailey, Charles Clarke, Ellese Cotterill, Jeff Dalton, Faegheh Hasibi, Mark Sanderson, and Nick Craswell. Llm-evaluation tropes: Perspectives on the validity of llm-evaluations, 2025. URL https://arxiv.org/abs/2504.19076

    2025

  61. [70]

    Right answer, wrong score: Uncovering the inconsistencies of llm evaluation in multiple-choice question answering, 2025

    Francesco Maria Molfese, Luca Moroni, Luca Gioffrè, Alessandro Scirè, Simone Conia, and Roberto Navigli. Right answer, wrong score: Uncovering the inconsistencies of llm evaluation in multiple-choice question answering, 2025. URL https://arxiv.org/abs/2503.14996

    2025 arXiv

  62. [71]

    xfinder: Large language models as automated evaluators for reliable evaluation

    Qingchen Yu, Zifan Zheng, Shichao Song, Zhiyu li, Feiyu Xiong, Bo Tang, and Ding Chen. xfinder: Large language models as automated evaluators for reliable evaluation. In The Thirteenth International Conference on Learning Representations , 2025. URL https://openreview.net/foru...

    2025

  63. [72]

    Gemini 2.5 flash

    Google DeepMind. Gemini 2.5 flash. https://deepmind.google/technologies/ gemini/flash, 2025. 14

    2025

  64. [73]

    Enhancing the reasoning ability of multimodal large language models via mixed preference optimization

    Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, and Jifeng Dai. Enhancing the reasoning ability of multimodal large language models via mixed preference optimization. arXiv preprint arXiv:2411.10442, 2024

    2024 arXiv

  65. [74]

    Internvl3: Advancing open-source multimodal models with native mul- timodal pretraining

    OpenGVLab Team. Internvl3: Advancing open-source multimodal models with native mul- timodal pretraining. https://internvl.github.io/blog/2025-04-11-InternVL-3.0 , 2025

    2025

  66. [75]

    Qvq: To see the world with wisdom, December 2024

    Qwen Team. Qvq: To see the world with wisdom, December 2024. URL https://qwenlm. github.io/blog/qvq-72b-preview/

    2024

  67. [76]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024

    2024 arXiv

  68. [77]

    Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras

    Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, et al. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras. arXiv preprint a...

    2025 arXiv

  69. [78]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  70. [79]

    L1: Controlling how long a reasoning model thinks with reinforcement learning

    Pranjal Aggarwal and Sean Welleck. L1: Controlling how long a reasoning model thinks with reinforcement learning. arXiv preprint arXiv:2503.04697, 2025

    2025 arXiv

  71. [80]

    s1: Simple test-time scaling

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393, 2025

    2025 arXiv

  72. [81]

    Qwen3 technical report, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jia...

    2025 arXiv

  73. [82]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  74. [83]

    Reft: Reasoning with reinforced fine-tuning

    Trung Quoc Luong, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Reasoning with reinforced fine-tuning. arXiv preprint arXiv:2401.08967, 3, 2024

    2024 arXiv

  75. [84]

    Llamafactory: Unified efficient fine-tuning of 100+ language models

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System ...

    2024 arXiv

  76. [85]

    Swift:a scal- able lightweight infrastructure for fine-tuning, 2024

    Yuze Zhao, Jintao Huang, Jinghan Hu, Xingjun Wang, Yunlin Mao, Daoze Zhang, Zeyinzi Jiang, Zhikai Wu, Baole Ai, Ang Wang, Wenmeng Zhou, and Yingda Chen. Swift:a scal- able lightweight infrastructure for fine-tuning, 2024. URL https://arxiv.org/abs/2408. 05517

    2024

  77. [86]

    fact verification

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Syste...

    2023

  78. [88]

    Read both Question and Answer; verify facts as needed

  79. [89]

    Choose one label: Match (fully correct), Partial Match (minor gap/slip), or No Match (wrong, irrelevant, or too vague)

  80. [90]

    the southeastern part has rich forest resources

    Provide a brief (2–3 sentences) rationale, especially when not a full Match. E.2 Scenario 2 — Step-by-Step Verification Goal. Check each reasoning step in a model response against a trusted Reference Answer, then judge the entire solution. ‡https://github.com/OpenGVLab/InternV...

  81. [91]

    Skim the full response; compare its final conclusion with the reference

  82. [92]

    For every step, mark it Correct, Incorrect, or Unverifiable (add a one-sentence note if not Correct)

  83. [93]

    Overall label is Correct only when all steps are Correct and the final answer matches the reference

  84. [94]

    F Limitations and Broader Impact BMMR is a dataset that focus on multidisciplinary reasoning for multimodal models

    Summarise the decisive error chain in a short overall comment. F Limitations and Broader Impact BMMR is a dataset that focus on multidisciplinary reasoning for multimodal models. We acknowledge that BMMR is created for research purposes only and should not be applied for other...

  85. [2024]

    doi: 10.18653/v1/2024.acl-long.744

    Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.744. URL https://aclanthology.org/2024.acl-long.744/

    2024 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.