Pith. sign in

REVIEW 4 major objections 4 minor 12 cited by

R-Bench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation

T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper introduces R-Bench, a graduate-level English-Chinese benchmark spanning 108 text and 83 multimodal subjects, and claims that the strongest model tested reaches only 69.0% on text questions and 53.2% on multimodal ones.

desk verdict New bilingual graduate-level benchmark with real breadth, but the difficulty calibration rests on an unvalidated o1 token threshold and the data is still unreleased. read the letter →

arxiv 2505.02018 v1 pith:B4WFNZQM submitted 2025-05-04 cs.CV

Meng-Hao Guo , Jiajun Xu , Yi Zhang , Jiaxi Song , Haoyang Peng , Yi-Xuan Deng , Xinzhi Dong , Kiyohiro Nakayama
show 9 more authors
Zhengyang Geng Chen Wang Bolin Ni Guo-Wei Yang Yongming Rao Houwen Peng Han Hu Gordon Wetzstein Shi-min Hu
This is my paper · ORCID
classification cs.CV
keywords complexreasoningevaluationmultimodalbenchmarkmultidisciplinarybilingualdifficultycalibrationLLMMLLMchain-of-thought
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

R-Bench is a new benchmark for testing whether language and multimodal models can do slow, deliberate reasoning, not just recall facts. It spans 1,094 text questions across 108 subjects and 665 multimodal questions across 83 subjects, with every question available in both English and Chinese. The paper argues that established tests such as MMLU and MMMU have become too easy to discriminate among advanced models, and that R-Bench restores headroom: the strongest model tested reaches 69.0% on the text set and 53.2% on the multimodal set. If the difficulty calibration is sound, the field gains a measurement that can separate models on genuine multi-step reasoning.

What carries the argument

The load-bearing mechanism is the model-based difficulty filter: the o1 reasoning model returns a count of reasoning tokens per question, and any question below 2,000 tokens is discarded so that R-Bench reflects deliberate reasoning rather than quick recall. Around this filter, the pipeline combines expert screening of source questions, manual review for completeness and ambiguity, conversion of all questions to six-option single-choice format for automatic scoring, and bilingual English-Chinese translation with expert proofreading. The token threshold carries the paper's difficulty-calibration claim; it is what lets the authors assert that low model scores mean hard reasoning rather than badly written questions.

What would settle it

Give a random sample of discarded and kept questions to human experts who rank difficulty blindly. If experts judge many of the discarded sub-2,000-token questions to be as hard as the kept ones, or if o1's token count correlates mainly with question length, the difficulty calibration fails. A simpler check: run the full model panel on the discarded questions; if scores are no higher than on R-Bench, the filter did not select for difficulty.

Watch

Extended reading notes

Core claim

The central claim is that complex reasoning can be measured at graduate level in a way that is broad, bilingual, and multimodal, and that current models are far from mastering it. The construction pipeline starts with more than one hundred university courses, keeps only questions experts judge to be reasoning-based rather than knowledge-based, and then removes any question that the o1 reasoning model solves with fewer than 2,000 reasoning tokens, on the grounds that short thinking time marks an easy or memory-driven problem. Expert and model ratings plus o1 thinking time are used to argue that R-Bench questions are harder than MMLU and MMMU by a large margin. On the resulting benchmark, o1 scores 69.0% on text-only questions and 53.2% on multimodal questions, while chat-style models score lower, so the paper concludes that multimodal reasoning is a major remaining bottleneck.

Load-bearing premise

The difficulty filter assumes o1's reasoning-token count measures how much thinking a question requires, so removing every question that takes fewer than 2,000 tokens is supposed to leave only hard reasoning problems; if token count instead tracks question length, wording, or OCR ambiguity, the benchmark's difficulty claim and its model rankings collapse.

Editorial extensions

If this is right

  • If R-Bench is accepted as the right difficulty level, the roughly 69% text ceiling means there is clear room before reasoning benchmarks saturate again.
  • The 15-point drop from text to multimodal accuracy on the same benchmark family indicates that improving vision-language integration is now one of the fastest levers on overall reasoning performance.
  • The finding that explicit chain-of-thought prompting helps chat models but not reasoning models suggests future prompting work should target the models that do not already reason internally.
  • High English-Chinese consistency on equally difficult questions implies that cross-lingual reasoning is already a relative strength, and score gaps between languages should be read as language-specific overfitting rather than reasoning failure.
  • Since scores vary by up to 37.9 percentage points across departments, focused single-subject training will not move a model's overall reasoning score as much as balanced multi-discipline improvement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One testable extension the paper does not run: apply the same expert-vs-token comparison to the discarded questions. If models already score low on the sub-2,000-token questions, the threshold is not doing the difficulty work the paper assigns to it.
  • A contamination check would be a natural next step, since graduate coursework problems circulate publicly; the benchmark's future value depends on keeping those questions out of model training data.
  • The English-Chinese consistency measure could be reused as a diagnostic for whether a model reasons from an abstract problem representation or from language-specific pattern matching.
  • If reasoning-token counts are shown to track human effort across subjects, the same filtering recipe could be applied to build harder versions of benchmarks in law, medicine, or engineering without relying on olympiad mathematics.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. R-Bench is a new bilingual (English/Chinese) benchmark for evaluating complex reasoning in LLMs and MLLMs. It contains 1,094 text-only questions across 108 subjects and 665 multimodal questions across 83 subjects, collected from Tsinghua graduate/undergraduate courses, digitized with OCR, filtered in three rounds (expert screening, an o1 reasoning-token threshold, and manual review), converted to six-option single choice, translated by experts, and evaluated with 27 LLMs and 11 MLLMs. The paper reports that even o1 achieves 69.0% on R-Bench-T and 53.2% on R-Bench-M, and uses expert/o1 pairwise comparisons and thinking-time comparisons to argue that R-Bench requires more reasoning than MMLU and MMMU.

Significance. If the difficulty-calibration claims are substantiated, R-Bench would be a valuable community resource because it combines breadth of disciplines, bilingual equal-difficulty question pairs, multimodal and text-only splits, and an automatic single-choice answer format that avoids proof-verification problems. The paper also publicly releases data and code and evaluates a wide range of open and closed models, which supports reproducibility and direct use. However, the evidence that the benchmark actually measures 'graduate-level complex reasoning' rather than length- or ambiguity-related difficulty is currently thin; in particular, the o1 reasoning-token filter is the central load-bearing assumption and it is not validated independently of o1 itself. For these reasons the contribution is promising but not yet rigorously established.

major comments (4)
  1. [§2.3, Step 4 (model screening)] The paper removes every question for which OpenAI o1 generates fewer than 2,000 reasoning tokens and states that this 'ensures that R-Bench is a benchmark for reasoning evaluation.' This is the sole difficulty-calibration mechanism, yet no evidence is given that reasoning-token count is a valid or transferable measure of human reasoning difficulty. The count is a policy-dependent property of o1 and can be inflated by input length, OCR artifacts, ambiguity, unfamiliar notation, or cheap uncertainty, and depressed for questions o1 shortcuts. Because the abstract and Section 1 base the 'rigorous difficulty calibration' and 'Olympiad-level' claims on this step, the authors should provide the distribution of token counts, a correlation with independent human difficulty ratings on a larger sample, a sensitivity analysis around the 2,000 threshold, and a funnel report showing how many questions were removed at each stage. Without this, the difficulty claim and the headline accuracies are not supported.
  2. [§3.1, Tables 2–3] The reasoning-comparison study uses 30 randomly selected questions per benchmark, with no sample-size justification, no confidence intervals or inter-annotator agreement, and no description of how the 30 questions were sampled. Moreover, the o1 voting compares R-Bench against MMLU/MMMU, not against the questions removed by the §2.3 filter, so it does not validate the threshold. Using o1 as the filter, as the judge in Tables 2–3, and as an evaluated model in Table 5 means that o1's poor accuracy is not independent evidence of difficulty. The authors should report expert-level agreement with error bars on a substantially larger sample, and should directly compare retained versus filtered questions.
  3. [§3.2, Tables 5–7; Fig. 5] Accuracy differences are reported without confidence intervals or repeated evaluations. For example, in Table 5 the difference between o1-20241217 (69.0) and Gemini-2.0-flash-thinking (68.4) is 0.6 percentage points on 1,094 questions, well within binomial sampling noise; similarly, the CoT effect for GPT-4o in Table 7 is 2.1 points. The ordering claims and the 'no impact of CoT on o1-mini' conclusion need bootstrap intervals or paired statistical tests. This is important because the paper's stated purpose is to provide guidance for model improvement, and several cross-model conclusions are currently indistinguishable from noise.
  4. [§2.4 and Fig. 5] The paper asserts that the English and Chinese versions are of equal difficulty and interprets the high consistency in Fig. 5 as evidence of cross-lingual reasoning. The translation review is described, but no procedure or metric establishes that the two versions are equally difficult; a model could answer both versions of the same question correctly or incorrectly for reasons related to translation quality or format rather than reasoning. The authors should either measure equal difficulty (for example, by obtaining separate human difficulty ratings on each language version) or soften the cross-linguistic-alignment claim.
minor comments (4)
  1. [Fig. 1] The R-Bench labels appear truncated in the figure as '-Bench-T' and '-Bench-M', and the figure lacks confidence intervals, which is especially relevant because several accuracy differences in Tables 5 and 6 are within sampling noise.
  2. [Fig. 3] In the example for complex analysis, the polynomial is rendered in a garbled form ('53 2() 5 2pz z z z=++ +'), making it impossible for a reader to verify the question; if this reflects the actual typeset version, it undermines the claim that the digitization was carefully proofread.
  3. [Appendix A.3] The text says 'Table 8 and Table 8' when introducing the subject distributions; the second reference should be Table 9.
  4. [§2.1] The paper reports 10,270 collected questions and 1,759 retained questions, but does not report how many questions were removed at each of the three filtering stages; providing a full funnel would help readers assess the effect of the o1-token threshold and the manual review.

Circularity Check

1 steps flagged · score 6.0 of 10

R-Bench's validation that it is more reasoning-demanding than MMLU/MMMU is partly circular: the same o1 reasoning-token threshold used to select questions is reused as evidence of difficulty.

  1. self definitional [Section 2.3 'Data filtering' (Model-screening); Section 3.1 'Reasoning comparison with other benchmarks', Tables 2-4]
    "When we call its API, it returns the number of reasoning tokens, which, to some extent, reflects the difficulty of the question. ... We filter out the questions with less than 2,000 reasoning tokens to ensure that our R-Bench is a benchmark for reasoning evaluation. ... we used the o1 model to determine which question required more reasoning ability based on the number of reasoning tokens (reasoning time)"

    R-Bench items were admitted only if OpenAI o1's reasoning-token count was at least 2,000. Section 3.1 then uses that same token count as evidence that R-Bench requires more reasoning than MMLU/MMMU (o1 voting and thinking-time ratios in Tables 2-4). Because MMLU/MMMU were not thresholded by o1 tokens, the comparison is biased by construction: the two benchmark sets differ by exactly the variable used to validate difficulty, so the token-based leg of the validation restates the selection rule rather than independently confirming difficulty calibration. Expert scoring and manual review provide separate evidence, so the circularity is partial, but the paper's claim that o1-based scores demonstrate R-Bench's greater reasoning demand is not an independent test.

full rationale

The main benchmark numbers on R-Bench are not themselves derived from fitted parameters, and there is no load-bearing self-citation or imported uniqueness theorem, so the paper is not wholly circular. However, one load-bearing validation step is self-referential: Section 2.3 defines R-Bench's difficulty by OpenAI o1's reasoning-token count, filtering out every question below 2,000 tokens, and Section 3.1 uses the same token count and thinking time to conclude that R-Bench is harder than MMLU and MMMU. That argument reduces to the construction rule, and the abstract's headline framing of o1's low accuracy as evidence of challenge is weakened because o1 was also the difficulty oracle. The independent expert pairwise scoring (85.94% and 76.88% win rates in Tables 2-3) and the manual review for ambiguity, completeness, and repetition give the benchmark substantial independent grounding, so the circularity is partial rather than total. A fully non-circular calibration would require validating the 2,000-token threshold against an external criterion not supplied by o1 itself, or showing that filtering on reasoning-token count does not simply select for length, ambiguity, or OCR artifacts.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

R-Bench is an empirical resource, so its 'parameters' are design choices that shape the benchmark composition. The central claims rest on several unvalidated assumptions about the filtering and translation processes; none of these are machine-checked or externally benchmarked.

free parameters (3)
  • o1 reasoning-token threshold for difficulty filter = 2000 tokens
    Chosen by hand in Section 2.3 Step 4; all questions where o1 uses fewer than 2000 reasoning tokens are removed. This parameter directly defines the benchmark's difficulty and affects all downstream evaluations.
  • Maximum questions per subject = 50
    Set in Section 2.3 manual review to control subject balance; affects the distribution shown in Fig 4 and the claim of subject balance.
  • Number of answer options = 6 (5 generated plus 'All other answers are incorrect')
    Introduced in Section 2.4; determines the random baseline and the MCQ format for auto-verification.
assumptions (5)
  • domain assumption o1 reasoning-token count is a valid proxy for reasoning difficulty
    Invoked in Section 2.3 Step 4 to filter questions and in Section 3.1 Table 4 as evidence of complexity. No validation that this proxy is uncorrelated with confounding factors.
  • domain assumption Expert screening can reliably distinguish reasoning-based from knowledge-based questions
    Section 2.1 relies on 51 recruited experts to filter out 'knowledge-based' questions; no quality-control metric for inter-expert agreement is provided.
  • domain assumption Universally verifiable answers can be obtained without proof-based questions
    Section 2.1 excludes proof-based questions because automated methods cannot verify them, so the benchmark measures only a subset of reasoning.
  • domain assumption Manual review removes all ambiguity and errors
    Section 2.3 Step 5 asserts completeness, repetition, and ambiguity checks without providing audit statistics or inter-annotator agreement.
  • domain assumption English and Chinese translations preserve difficulty and meaning
    Section 2.4 describes manual translation with expert review, but no back-translation or difficulty-equivalence validation is reported, and the cross-lingual consistency analysis (Fig 5) depends on this.

how reviews work

0 comments
Cite this review

Pith. "Pith review of R-Bench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation." pith.science (2026). https://pith.science/paper/B4WFNZQM

@misc{pith2026250502018,
  author       = {Pith},
  title        = {Pith review of: R-Bench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B4WFNZQM}},
  note         = {Machine review of arXiv:2505.02018}
}
read the original abstract

Reasoning stands as a cornerstone of intelligence, enabling the synthesis of existing knowledge to solve complex problems. Despite remarkable progress, existing reasoning benchmarks often fail to rigorously evaluate the nuanced reasoning capabilities required for complex, real-world problemsolving, particularly in multi-disciplinary and multimodal contexts. In this paper, we introduce a graduate-level, multi-disciplinary, EnglishChinese benchmark, dubbed as Reasoning Bench (R-Bench), for assessing the reasoning capability of both language and multimodal models. RBench spans 1,094 questions across 108 subjects for language model evaluation and 665 questions across 83 subjects for multimodal model testing in both English and Chinese. These questions are meticulously curated to ensure rigorous difficulty calibration, subject balance, and crosslinguistic alignment, enabling the assessment to be an Olympiad-level multi-disciplinary benchmark. We evaluate widely used models, including OpenAI o1, GPT-4o, DeepSeek-R1, etc. Experimental results indicate that advanced models perform poorly on complex reasoning, especially multimodal reasoning. Even the top-performing model OpenAI o1 achieves only 53.2% accuracy on our multimodal evaluation. Data and code are made publicly available at here.

Figures

Figures reproduced from arXiv: 2505.02018 by the authors.

Figure 1
Figure 1. Top-1 accuracy comparison of different models on MMLU, MMMU, and R-Bench. R-Bench poses a greater chal￾lenge to current models. Reasoning, the systematic process of synthesizing knowl￾edge to solve novel problems, lies at the heart of intelligence. Yet, as foundation models grow increasingly sophisticated, existing benchmarks fail to comprehensively assess their complex reasoning capabilities. As shown in the above … view at source ↗
Figure 2
Figure 2. Pipeline of building R-Bench. The process is divided into six steps, which are detailed in Sec. 2. The funnel represents screening. We always filter out the blue ball and preserve the brown one. In Step 2, KQ and RQ denote knowledge-based questions and reasoning-based questions, respectively. In Step 4, < 2000 indicates that the reasoning tokens of o1 are less than 2000. Finally, in Step 5, AQ and CQ represent ambig… view at source ↗
Figure 3
Figure 3. Some examples in R-Bench. These examples show that R-Bench is multidisciplinary, multimodal, and multilingual. As shown in the figure, the problems in R-Bench are complex and cannot be solved by quick thinking, which shows that R-Bench focuses on deep reasoning problems rather than knowledge problems, such as conceptual problems. To do so, we recruit a data annotation team of about 20 people. They are responsible fo… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: According to statistics on R-Bench, the benchmark spans 19 departments, including mathematics, physics, biology, computer science, and chemistry, covering over 100 subjects such as Inorganic Chemistry, Chemical Reaction Kinetics, and Electromagnetism. It features 1,094…
Figure 5
Figure 5. Figure 5: The performance of different models on questions of the same difficulty in Chinese and English. Biology Physics E. Electronic E. Aerospace ChemistryMath Automation Physics Microelectronics Mechanical E. Statistics Chemical E. Computer S. Materials Civil E. Vehicle E. E…
Figure 6
Figure 6. Figure 6: GPT-4o on R-Bench-T across different departments, which shows large variation among different disciplines.. different areas of the R-Bench-T benchmark. From the figure, it can be observed that the performance varies sig￾nificantly across different domains, with a range…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning

    cs.CL 2026-08 conditional novelty 7.0 of 10

    Skill entropy, a reference-model-based measure of skill-switching difficulty, calibrates a new cross-skill benchmark and serves as an RL reward, more than doubling small models' scores.

  2. When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research

    cs.CL 2025-05 conditional novelty 7.0 of 10

    SPOT shows that state-of-the-art AI models detect fewer than one in five known errors in full scientific papers, with precision below 7%.

  3. Enjoy Your Talk: A Human-Centered Benchmark for Multi-Turn Dialogue with Decoupled User Simulation, Target Modeling, and Judging

    cs.CL 2026-07 conditional novelty 6.0 of 10

    A three-party-decoupled multi-turn chat benchmark finds closed and open models nearly tied on subjective empathy/persona scores but separated by up to 9× on objective intent tracking, with reasoning and persona format...

  4. IRIS: A Real-World Benchmark for Inverse Recovery and Identification of Physical Dynamic Systems from Monocular Video

    cs.CV 2026-03 accept novelty 6.0 of 10

    IRIS releases 220 real 4K videos of eight dynamical systems with ground-truth parameters plus a protocol that measures parameter recovery, equation selection, and multi-body failure modes of unsupervised video-to-phys...

  5. MapTab: A Diagnostic Benchmark for Long-Horizon Multi-Criteria Multimodal Reasoning on Heterogeneous Topological Graphs

    cs.LG 2026-02 unverdicted novelty 6.0 of 10

    MapTab is a new multimodal benchmark with 328 images and nearly 200k queries that shows current MLLMs have substantial difficulty with multi-criteria route planning when visual and tabular information must be combined.

  6. SciAgentGym: Benchmarking Multi-Step Scientific Tool-use in LLM Agents

    cs.CL 2026-02 conditional novelty 6.0 of 10

    SciAgent-8B, fine-tuned on trajectories synthesized from a tool dependency graph, outperforms Qwen3-VL-235B-Instruct on SciAgentBench, a new 259-task benchmark for multi-step scientific tool-use.

  7. MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models

    cs.CL 2025-08 conditional novelty 6.0 of 10

    A manually curated, multilingual, multimodal science benchmark shows that even top MLLMs struggle, with fine-grained knowledge-point labels revealing specific weaknesses.

  8. BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Dataset

    cs.CL 2025-07 conditional novelty 6.0 of 10

    BMMR provides a 110k-question bilingual, multimodal, college-level dataset across 300 subjects where state-of-the-art models score at most about 50%.

  9. OPT-BENCH: Evaluating LLM Agent on Large-Scale Search Spaces Optimization Problems

    cs.AI 2025-06 conditional novelty 6.0 of 10

    OPT-BENCH is a 30-task benchmark showing that LLM agents generally improve optimization results when they are given their own past solutions and feedback.

  10. RBench-V: A Primary Assessment for Visual Reasoning Models with Multi-modal Outputs

    cs.CV 2025-05 reject novelty 6.0 of 10

    The RBench-V benchmark finds that the best current AI models score 25.8%, far below 82.3% for humans, on visual reasoning problems claimed to require multi-modal outputs.

  11. Blind-Spots-Bench: Evaluating Blind Spots in Multimodal Models

    cs.AI 2026-07 conditional novelty 5.0 of 10

    A 235-item multimodal stress-test shows frontier closed models outpace open-weight peers by ~10% and leaves shared failures on counting, spatial, and character-level tasks.

  12. Boogu-Image-0.1: Boosting Open Agentic Multimodal Generation via Understanding under a Minimal Budget

    cs.CV 2026-07 conditional novelty 4.0 of 10

    An open-source image-generation family shows that agentic prompt rewriting and a stronger text encoder can lift quality to near closed-source levels with only 208.62M images and about $400K of training compute.

Reference graph

Works this paper leans on

34 extracted references · 33 linked inside Pith · cited by 12 Pith papers

  1. [1]

    Deepseek-r1: Incentivizing reasoning capabil- ity in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    AI, D. Deepseek-r1: Incentivizing reasoning capabil- ity in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    arXiv

  2. [3]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901,

    1901

  3. [4]

    Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y ., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374,

    arXiv

  4. [6]

    N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonza- lez, J

    Chiang, W.-L., Zheng, L., Sheng, Y ., Angelopoulos, A. N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonza- lez, J. E., et al. Chatbot arena: An open platform for evaluating llms by human preference.arXiv preprint arXiv:2403.04132,

    arXiv

  5. [7]

    A survey on in-context learning.arXiv preprint arXiv:2301.00234,

    Dong, Q., Li, L., Dai, D., Zheng, C., Ma, J., Li, R., Xia, H., Xu, J., Wu, Z., Liu, T., et al. A survey on in-context learning.arXiv preprint arXiv:2301.00234,

    arXiv

  6. [8]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    arXiv

  7. [9]

    Dubois, Y ., Galambosi, B., Liang, P., and Hashimoto, T. B. Length-controlled alpacaeval: A simple way to debias automatic evaluators.arXiv preprint arXiv:2404.04475,

    arXiv

  8. [10]

    Omni-math: A universal olympiad level mathematic benchmark for large language models.arXiv preprint arXiv:2410.07985,

    Gao, B., Song, F., Yang, Z., Cai, Z., Miao, Y ., Dong, Q., Li, L., Ma, C., Chen, L., Xu, R., et al. Omni-math: A universal olympiad level mathematic benchmark for large language models.arXiv preprint arXiv:2410.07985,

    arXiv

Show all 34 references
  1. [11]

    F., Denain, J.-S., Ho, A., Santos, E

    9 R-Bench Glazer, E., Erdil, E., Besiroglu, T., Chicharro, D., Chen, E., Gunning, A., Olsson, C. F., Denain, J.-S., Ho, A., Santos, E. d. O., et al. Frontiermath: A benchmark for evaluating advanced mathematical reasoning in ai.arXiv preprint arXiv:2411.04872,

    arXiv

  2. [12]

    Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., and Stoica, I

    URL https:// openreview.net/forum?id=d7KBjmI3GmQ. Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., and Stoica, I. Livecodebench: Holistic and contamination free eval- uation of large language models for code.arXiv preprint arXiv:24...

    arXiv

  3. [13]

    Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b.arXiv preprint arXiv:2310.06825,

    arXiv

  4. [15]

    Seed-bench: Benchmarking multimodal llms with gener- ative comprehension.arXiv preprint arXiv:2307.16125,

    Li, B., Wang, R., Wang, G., Ge, Y ., Ge, Y ., and Shan, Y . Seed-bench: Benchmarking multimodal llms with gener- ative comprehension.arXiv preprint arXiv:2307.16125,

    arXiv

  5. [16]

    Llava- onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326,

    Li, B., Zhang, Y ., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y ., Liu, Z., et al. Llava- onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326,

    arXiv

  6. [17]

    Let’s verify step by step.arXiv preprint arXiv:2305.20050,

    Lightman, H., Kosaraju, V ., Burda, Y ., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let’s verify step by step.arXiv preprint arXiv:2305.20050,

    arXiv

  7. [18]

    Y ., Deng, Y ., Chandu, K., Brahman, F., Ravichander, A., Pyatkin, V ., Dziri, N., Bras, R

    Lin, B. Y ., Deng, Y ., Chandu, K., Brahman, F., Ravichander, A., Pyatkin, V ., Dziri, N., Bras, R. L., and Choi, Y . Wild- bench: Benchmarking llms with challenging tasks from real users in the wild.arXiv preprint arXiv:2406.04770,

    arXiv

  8. [19]

    Truthfulqa: Measuring how models mimic human falsehoods.arXiv preprint arXiv:2109.07958,

    Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring how models mimic human falsehoods.arXiv preprint arXiv:2109.07958,

    arXiv

  9. [20]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434, 2024a

    Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Dengr, C., Ruan, C., Dai, D., Guo, D., et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434, 2024a. Liu, H., Li, C., Wu, Q., and Lee, Y . J. Visual in...

    arXiv

  10. [22]

    Smaug: Fixing failure modes of pref- erence optimisation with dpo-positive.arXiv preprint arXiv:2402.13228,

    Pal, A., Karkhanis, D., Dooley, S., Roberts, M., Naidu, S., and White, C. Smaug: Fixing failure modes of pref- erence optimisation with dpo-positive.arXiv preprint arXiv:2402.13228,

    arXiv

  11. [23]

    S., Wang, H., Killamsetty, K., Su- dalairaj, S., Zhao, W., Han, S., Bhandwaldar, A., Xu, G., Xu, K., et al

    Pareja, A., Nayak, N. S., Wang, H., Killamsetty, K., Su- dalairaj, S., Zhao, W., Han, S., Bhandwaldar, A., Xu, G., Xu, K., et al. Unveiling the secret recipe: A guide for supervised fine-tuning small llms.arXiv preprint arXiv:2412.13337,

    arXiv

  12. [25]

    Proximal policy optimization algorithms

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

    arXiv

  13. [26]

    M., Hauth, A., Millican, K., et al

    Team, G., Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Sori- cut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805,

    arXiv

  14. [27]

    I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al

    Team, G., Georgiev, P., Lei, V . I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context.arXiv preprint arXiv:2403.05530,

    arXiv

  15. [28]

    Llama: Open and efficient foundation lan- guage models.arXiv preprint arXiv:2302.13971,

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi`ere, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models.arXiv preprint arXiv:2302.13971,

    arXiv

  16. [30]

    Deepseek-vl2: Mixture- of-experts vision-language models for advanced multi- modal understanding.arXiv preprint arXiv:2412.10302,

    Wu, Z., Chen, X., Pan, Z., Liu, X., Liu, W., Dai, D., Gao, H., Ma, Y ., Wu, C., Wang, B., et al. Deepseek-vl2: Mixture- of-experts vision-language models for advanced multi- modal understanding.arXiv preprint arXiv:2412.10302,

    arXiv

  17. [31]

    Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115,

    arXiv

  18. [32]

    Mm-vet: Evaluating large multi- modal models for integrated capabilities.arXiv preprint arXiv:2308.02490,

    Yu, W., Yang, Z., Li, L., Wang, J., Lin, K., Liu, Z., Wang, X., and Wang, L. Mm-vet: Evaluating large multi- modal models for integrated capabilities.arXiv preprint arXiv:2308.02490,

    arXiv

  19. [33]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Yue, X., Ni, Y ., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y ., et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. InProceedings of the IEEE/CVF Conference on Computer Vision and Patte...

    arXiv

  20. [34]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592,

    Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592,

    arXiv

  21. [2017]

    cc/paper_files/paper/2017/file/ 3f5ee243547dee91fbd053c1c4a845aa-Paper

    URL https://proceedings.neurips. cc/paper_files/paper/2017/file/ 3f5ee243547dee91fbd053c1c4a845aa-Paper. pdf. Wang, K., Pan, J., Shi, W., Lu, Z., Zhan, M., and Li, H. Mea- suring multimodal mathematical reasoning with math- vision dataset.arXiv preprint arXiv:2402.14804, 2024a...

    2017 arXiv

  22. [2020]

    L., Stickland, A

    Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y ., Dirani, J., Michael, J., and Bowman, S. R. Gpqa: A graduate-level google-proof q&a benchmark.arXiv preprint arXiv:2311.12022,

    arXiv

  23. [2021]

    Expanding per- formance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271,

    Chen, Z., Wang, W., Cao, Y ., Liu, Y ., Gao, Z., Cui, E., Zhu, J., Ye, S., Tian, H., Liu, Z., et al. Expanding per- formance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271,

    arXiv

  24. [2022]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255,

    Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., and Gao, J. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255,

    arXiv

  25. [2023]

    E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K

    Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. Swe-bench: Can language mod- els resolve real-world github issues?arXiv preprint arXiv:2310.06770,

    arXiv

  26. [2024]

    Deepseek llm: Scaling open-source language models with longtermism

    Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., Du, Q., Fu, Z., et al. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954,

    arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.