Pith. sign in

REVIEW 3 major objections 3 minor 34 references

BRUCE: Benchmarking Robustness Under Corruption Escalation for Scientific Vision-Language Reasoning

T0 review · 3 major / 3 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read BRUCE is a benchmark that measures how quickly scientific vision-language models lose reasoning accuracy as input images are progressively corrupted, using two new metrics, RCI and T-RCI, to quantify the speed of that collapse.

desk verdict Abstract-only benchmark for VLM corruption robustness; the framing is good and the failure taxonomy is useful, but RCI/T-RCI are undefined in the abstract, so soundness can't yet be assessed. read the letter →

arxiv 2608.07742 v1 pith:KJZY7A4V submitted 2026-08-07 cs.CV

Saim Rehman , Muhammad Shafique This is my paper · ORCID
classification cs.CV
keywords visual-languagemodelsrobustnessbenchmarkcorruptionescalationmultimodalreasoningscientificmetricsfailureanalysisimageperturbations
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces BRUCE, a benchmark for measuring how robust scientific vision-language models are when input images are progressively degraded by blur, low contrast, and other corruptions. Most evaluation compares clean-task accuracy; BRUCE instead asks how quickly reasoning performance collapses as corruption severity increases. It defines two metrics, Robustness Corruption Index (RCI) and Traversal-RCI (T-RCI), to quantify the rate of deterioration, and classifies the resulting errors into OCR-dependent, spatial, symbolic, and semantic reasoning failures. The goal is to make robustness in multimodal scientific reasoning measurable, comparable, and interpretable rather than treating it as a single average accuracy number.

What carries the argument

The central object is BRUCE, a corruption-escalation benchmark that systematically applies increasing levels of image perturbations, such as blur and contrast degradation, to scientific reasoning tasks. The load-bearing metrics are RCI, which aggregates the overall accuracy drop caused by corruption, and T-RCI, which measures how rapidly that drop occurs as severity increases. The supporting taxonomy maps causal failures into four high-level reasoning domains -- OCR-dependent, spatial, symbolic, and semantic -- providing the interpretive layer that turns raw accuracy losses into a diagnosis of which reasoning abilities break first.

What would settle it

A concrete observation that would settle the claim: if applying the same corruption types in a different order (for example, blur before contrast versus contrast before blur) or with coarser or finer severity steps substantially reorders models according to RCI or T-RCI, then the measured rate of deterioration is partly an artifact of the escalation schedule. Alternatively, if the four reasoning-domain failure types can be predicted entirely from a generic low-level image-quality score, the metrics would not be measuring reasoning-specific fragility.

Watch

Extended reading notes

Core claim

The central claim is that reasoning robustness in scientific vision-language models can be characterized by how performance changes along a progression of corruption severity, not just by accuracy at a fixed noise level. BRUCE applies a wide range of perturbations to chemistry and mathematical reasoning tasks and measures the point and speed at which each model's reasoning degrades, using RCI to capture the overall corruption-induced drop and T-RCI to capture the traversal rate as severity escalates. The paper argues that these metrics reveal distinct fragility patterns: OCR-dependent reasoning fails under different corruptions and at different rates than spatial, symbolic, or semantic reasoning, and mapping failures to these four domains gives an interpretable account of why a model breaks. If correct, the benchmark provides a standard way to ask not only whether a vision-language model is robust, but how its robustness is distributed across reasoning types and perturbation scales.

Load-bearing premise

The benchmark's chosen corruption types, severity levels, and aggregation rules accurately capture a general property called reasoning robustness; if a different corruption schedule changes which models look robust, then the measured traversal rate is a property of the benchmark rather than of the models.

Editorial extensions

If this is right

  • Model rankings can change depending on the corruption-severity schedule, so reporting RCI and T-RCI alongside clean accuracy gives a fuller picture of practical reliability.
  • The failure taxonomy lets developers identify which reasoning skill a model loses first under visual stress, such as OCR dependence vs spatial reasoning.
  • Scientific vision-language model accuracy claims become more meaningful when accompanied by corruption-traversal metrics, since real images are rarely pristine.
  • The taxonomy may support targeted robustness training, such as augmenting specific corruption types that cause early semantic or OCR failures.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • RCI and T-RCI could be applied outside the scientific domains in the paper to general-purpose vision-language models, but that extrapolation is not established in the abstract.
  • The two scalar metrics compress a high-dimensional corruption space; a hidden assumption is that severity ordering is commensurable across corruption types, which the abstract does not defend.
  • A testable extension is to check whether the failure taxonomy predicts human error patterns or only model-specific artifacts.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes BRUCE, a benchmark for evaluating vision-language model robustness under progressively escalating image corruptions in scientific reasoning tasks. The abstract introduces two metrics, RCI (Robustness Corruption Index) and T-RCI (Traversal-RCI), intended to quantify how rapidly multimodal reasoning performance degrades as corruption severity increases. The benchmark is applied to chemistry and mathematics reasoning tasks, and corruption-induced failures are classified into four reasoning domains: OCR-dependent, spatial, symbolic, and semantic. The abstract provides no mathematical definitions, no experimental results, and no validation of the proposed metrics.

Significance. The motivation is timely: robustness evaluation for scientific vision-language models currently relies heavily on clean-task accuracy, and a benchmark that characterizes degradation trajectories could be a useful community resource. The proposed four-domain failure taxonomy is a sensible high-level structure for interpretable analysis. However, because the abstract does not define RCI and T-RCI or provide any empirical evidence, the scientific contribution is not yet assessable. If the full paper supplies rigorous definitions, invariance or sensitivity analyses, and comparative experiments, the benchmark could become a valuable tool, but those elements are essential and currently missing.

major comments (3)
  1. [Abstract, paras. 2–3] The central claim that RCI and T-RCI quantify "how rapidly" performance deteriorates is not assessable because the functional forms of these metrics are entirely unspecified. Without formulas showing how severity levels are parameterized and aggregated, there is no way to determine whether the measured "traversal rate" is a stable property of a model or an artifact of the chosen severity schedule. The paper must present the exact definitions and, ideally, a sensitivity analysis over severity spacing and number of levels.
  2. [Abstract, para. 2] RCI aggregates multiple corruption types (blur, low contrast, etc.) into a single index, but the abstract does not address commensurability across qualitatively different perturbations. If corruption types are simply averaged, the index depends on the arbitrary list of included corruptions and their severity ranges. The authors should either prove that RCI is invariant to such choices or report a sensitivity analysis showing that model rankings do not change when the corruption schedule is varied.
  3. [Abstract, para. 3] The claimed "interpretable failure analysis" rests on a taxonomy of four reasoning domains and "fine-grained corruption specific failure subtypes," but the abstract gives no operational definition of how failures are assigned to these categories. Without a rule-based or human-annotation protocol with reliability statistics, the failure analysis cannot be reproduced or compared across studies. The full paper must specify the labeling procedure and report agreement measures.
minor comments (3)
  1. [Abstract, para. 1] The parenthetical expansion of BRUCE is incomplete: the opening parenthesis after "BRUCE" is never closed, and the phrase "a multimodal reasoning fragility framework" reads as an appositive that is not grammatically integrated.
  2. [Abstract, para. 2] The term "Traversal-RCI" is introduced without any explanation of what is traversed; a brief description of the conceptual meaning (e.g., moving across severity levels or across corruption types) would improve clarity.
  3. [Abstract, para. 3] The abstract does not mention whether each evaluated dataset provides ground-truth labels for the failure subtypes, which would be needed to support the quantitative failure analysis.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity detectable from the abstract; metric definitions and derivations are absent, so no reduction to inputs can be exhibited.

full rationale

The abstract-only manuscript contains no derivation chain, no equations, no fitted parameters, and no load-bearing citations. It proposes a benchmark and two metrics, RCI and T-RCI, and states that they quantify how rapidly multimodal reasoning performance deteriorates under progressive corruption escalation, but it does not define the metrics mathematically or claim that any quantity is predicted from a fitted input. The concern that the metrics might depend on the arbitrary choice of corruption types or severity spacing is a validity and sensitivity question, not a circularity of the kind defined in the rubric. Without formulas or prior results, there is no specific reduction to quote, so the honest finding is no significant circularity.

Assumptions & free parameters 1 free parameters · 2 assumptions · 0 invented entities

The central claim rests on the chosen corruption protocol and the failure taxonomy. The abstract does not disclose free parameters beyond the severity schedule, and the metrics themselves are not free parameters in the fitting sense. No new physical entities are introduced.

free parameters (1)
  • Corruption severity levels and perturbation types = unspecified
    The choice of blur, low contrast, and the number or step of severity levels determines what RCI and T-RCI measure; the abstract does not say how these were chosen.
assumptions (2)
  • domain assumption Vision-language reasoning failures can be classified into OCR-dependent, spatial, symbolic, and semantic reasoning domains, and these categories are exhaustive for scientific tasks.
    The abstract introduces this taxonomy in the last sentence; the benchmark's interpretability claim depends on this classification being meaningful and complete.
  • domain assumption The corruption types evaluated (e.g., blur, low contrast) are representative of real-world image quality variations in scientific applications.
    The abstract says VLMs struggle due to low or varying quality images; the benchmark's validity as a robustness measure assumes these selected distortions capture the relevant failure modes.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BRUCE: Benchmarking Robustness Under Corruption Escalation for Scientific Vision-Language Reasoning." pith.science (2026). https://pith.science/paper/KJZY7A4V

@misc{pith2026260807742,
  author       = {Pith},
  title        = {Pith review of: BRUCE: Benchmarking Robustness Under Corruption Escalation for Scientific Vision-Language Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KJZY7A4V}},
  note         = {Machine review of arXiv:2608.07742}
}
read the original abstract

Visual-language models (VLMs) frequently struggle with robustness issues in real-world situations due to low- or varying-quality input images. In this paper, we aim at analyzing VLMs' robustness by applying perturbations and distortions to the input images, such as blur or low contrast. Toward this goal, we propose BRUCE (Benchmarking Robustness Under Corruption Escalation, a multimodal reasoning fragility framework for scientific vision-language reasoning. State-of-the-art evaluation frameworks/studies primarily focus on clean-task accuracy and rarely analyze how reasoning stability degrades across robustness dimensions. Besides varying over a wide-range of input perturbations, BRUCE employs two novel metrics -- Robustness Corruption Index (RCI) and Traversal-RCI (T-RCI) -- to quantify how rapidly multimodal reasoning performance deteriorates in VLMs as visual corruption severity increases under progressive perturbation scaling. We evaluate BRUCE across chemistry and mathematical reasoning tasks for multiple datasets, while analyzing corruption-induced prediction failures in terms of four high-level reasoning domains: OCR-dependent reasoning, spatial reasoning, symbolic reasoning, and semantic failures, with each containing fine-grained corruption specific failure subtypes, thereby enabling an interpretable failure analysis.

Figures

Figures reproduced from arXiv: 2608.07742 by the authors.

Figure 1
Figure 1. A Comprehensive Overview of BRUCE Framework illustrating Different Steps [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Examples of Single Perturbation Types: (a) Blurry Image; (b) Low-Contrast [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Left: Pareto Robustness Map showing the trade-off between mean corrupted accuracy [%] and Mean Robustness Corruption Index (RCIµ ) across evaluated models and datasets. Lower-right regions indicate stronger robustness to corruption/perturbations. Right: Pareto Traversal Robustness map showing the trade-off between mean traversal ac￾curacy [%] and Mean Traversal Robustness Corruption Index (Tµ ). Lower-right regions … view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Global traversal-induced failure taxonomy across evaluated vision-language mod [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Spatial traversal robustness curves under progressive directional cropping. Accu [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Fragility curves plotted for all models across datasets [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Example traversal-induced failure on a chemistry reasoning question. [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]
Figure 8
Figure 8. Figure 8: Global traversal-induced failure taxonomy across evaluated vision-language mod [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 15 canonical work pages

  1. [1]

    Mvtamperbench: Evaluating robustness of vision-language models.arXiv preprint arXiv:2412.19794, 2024

    Amit Agarwal, Srikant Panda, Angeline Charles, et al. Mvtamperbench: Evaluating robustness of vision-language models.arXiv preprint arXiv:2412.19794, 2024

    arXiv 2024

  2. [2]

    Introducing claude opus 4.7.https://www.anthropic.com/ news/claude-opus-4-7, 2025

    Anthropic. Introducing claude opus 4.7.https://www.anthropic.com/ news/claude-opus-4-7, 2025. Accessed 2026

    work page 2025

  3. [3]

    Qwen-vl: A frontier large vision-language model with versatile abili- ties.arXiv preprint arXiv:2308.12966, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Sheng Wang, Silu Tan, Ke Wang, Xinghao Huang, et al. Qwen-vl: A frontier large vision-language model with versatile abili- ties.arXiv preprint arXiv:2308.12966, 2023

    arXiv 2023

  4. [4]

    A Causally Grounded Taxonomy for Image Degradation Robustness Evaluation

    Stefan Becker, Simon Weiss, Wolfgang Hübner, and Michael Arens. A causally grounded taxonomy for image degradation robustness evaluation.arXiv preprint arXiv:2605.15906, 2026. URLhttps://arxiv.org/abs/2605.15906

    work page Pith review arXiv 2026

  5. [5]

    Mmstar: Are we on the right way for evalu- ating large vision-language models?arXiv preprint arXiv:2403.20330, 2024

    Lin Chen, Yilun Zhao, Yuxuan Liu, et al. Mmstar: Are we on the right way for evalu- ating large vision-language models?arXiv preprint arXiv:2403.20330, 2024

    arXiv 2024

  6. [6]

    Internvl 2.0: Scaling up vi- sion foundation models and aligning for generic visual-linguistic tasks.arXiv preprint arXiv:2403.20377, 2024

    Zhe Chen, Weiyun Wang, Yue Cao, Zheng Liu, et al. Internvl 2.0: Scaling up vi- sion foundation models and aligning for generic visual-linguistic tasks.arXiv preprint arXiv:2403.20377, 2024

    work page arXiv 2024

  7. [7]

    Evaluating large language models on multimodal chemistry olympiad exams.Communications Chemistry, 8(1):402, 2025

    Yiming Cui, Xin Yao, Yuxuan Qin, Xin Li, Shijin Wang, and Guoping Hu. Evaluating large language models on multimodal chemistry olympiad exams.Communications Chemistry, 8(1):402, 2025

    work page 2025

  8. [8]

    Instructblip: Towards general-purpose vision-language models with instruction tuning.arXiv preprint arXiv:2305.06500, 2023

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Nicu Sebe, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning.arXiv preprint arXiv:2305.06500, 2023

    arXiv 2023

Show all 34 references
  1. [9]

    Interpretable explanations of black boxes by meaning- ful perturbation

    Ruth Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaning- ful perturbation. InProceedings of the IEEE International Conference on Computer Vision (ICCV), pages 3429–3437, 2017

    2017

  2. [10]

    Can llms solve molecule puzzles? a multi- modal benchmark for molecular structure elucidation

    Kehan Guo, Yifei Zhang, Shuo Liu, et al. Can llms solve molecule puzzles? a multi- modal benchmark for molecular structure elucidation. InAdvances in Neural Informa- tion Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2024

    2024

  3. [11]

    Benchmarking neural network robustness to common corruptions and perturbations

    Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. InICLR, 2019

    2019

  4. [12]

    The many faces of robustness: A critical analysis of out-of-distribution generalization

    Dan Hendrycks, Steven Basart, Norman Mu, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 8340–8349, 2021

    2021

  5. [13]

    Azam Hossain

    Md Farhan Ishmam, Ishmam Tashdeed, Talukder Asir Saadat, Md Hamjajul Ashmafee, Abu Raihan Mostofa Kamal, and Md. Azam Hossain. Visual robustness benchmark for visual question answering (vqa). InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (W...

    2025

  6. [14]

    R-bench: Are your large multimodal models robust to real-world corruptions?arXiv preprint arXiv:2410.05474, 2024

    Chunyi Li, Jianbo Zhang, Zicheng Zhang, Haoning Wu, Yuan Tian, Wei Sun, Guo Lu, Xiaohong Liu, Xiongkuo Min, Weisi Lin, and Guangtao Zhai. R-bench: Are your large multimodal models robust to real-world corruptions?arXiv preprint arXiv:2410.05474, 2024

    arXiv

  7. [15]

    Chemvlm: Exploring the power of multi- modal large language models in chemistry area.Proceedings of the AAAI Conference on Artificial Intelligence, 39(1):415–423, 2025

    Junxian Li, Di Zhang, Xunzhi Wang, et al. Chemvlm: Exploring the power of multi- modal large language models in chemistry area.Proceedings of the AAAI Conference on Artificial Intelligence, 39(1):415–423, 2025. doi: 10.1609/aaai.v39i1.32020

    2025 doi

  8. [16]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023

    2023 arXiv

  9. [17]

    Mmbench: Is your multi-modal model an all-around player?arXiv preprint arXiv:2307.06281, 2023

    Yuan Liu, Haodong Duan, Wenqi Zhang, et al. Mmbench: Is your multi-modal model an all-around player?arXiv preprint arXiv:2307.06281, 2023

    2023 arXiv

  10. [18]

    Learn to explain: Multimodal rea- soning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal rea- soning via thought chains for science question answering. InAdvances in Neural In- formation Processing Systems (NeurIPS...

    2022 arXiv

  11. [19]

    Mathvista: Evaluat- ing mathematical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2023

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluat- ing mathematical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2023. URLhttps://...

    arXiv

  12. [20]

    mmjee-eval: A bilingual multimodal bench- mark for evaluating scientific reasoning in vision-language models

    Arka Mukherjee and Shreya Ghosh. mmjee-eval: A bilingual multimodal bench- mark for evaluating scientific reasoning in vision-language models. InFindings of the Association for Computational Linguistics: IJCNLP 2025, 2025. URLhttps: //arxiv.org/abs/2511.09339

    2025

  13. [21]

    Introducing gpt-4.1 in the api.https://openai.com/index/ gpt-4-1/, 2025

    OpenAI. Introducing gpt-4.1 in the api.https://openai.com/index/ gpt-4-1/, 2025. Accessed 2026

    2025

  14. [22]

    Do cifar- 10 classifiers generalize to cifar-10?arXiv preprint arXiv:1806.00451, 2018

    Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do cifar- 10 classifiers generalize to cifar-10?arXiv preprint arXiv:1806.00451, 2018

    2018 arXiv

  15. [23]

    Do im- agenet classifiers generalize to imagenet? InProceedings of the 36th International Conference on Machine Learning (ICML), pages 5389–5400, 2019

    Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do im- agenet classifiers generalize to imagenet? InProceedings of the 36th International Conference on Machine Learning (ICML), pages 5389–5400, 2019

    2019

  16. [24]

    Hughes, and Finale Doshi-Velez

    Andrew Slavin Ross, Michael C. Hughes, and Finale Doshi-Velez. Right for the right reasons: Training differentiable models by constraining their explanations. InPro- ceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI), pages 2662–2670...

    2017 doi

  17. [25]

    Runcie et al

    Nicholas T. Runcie et al. Assessing the chemical intelligence of large language models. arXiv preprint arXiv:2505.07735, 2025

    2025 arXiv

  18. [26]

    Benchmarking corruption robustness of lvlms: A discriminative benchmark and ro- bustness alignment metric.arXiv preprint arXiv:2511.19032, 2025

    Xiangjie Sui, Songyang Li, Hanwei Zhu, Baoliang Chen, Yuming Fang, and Xin Sun. Benchmarking corruption robustness of lvlms: A discriminative benchmark and ro- bustness alignment metric.arXiv preprint arXiv:2511.19032, 2025. 18REHMAN & SHAFIQUE: BRUCE

    2025 arXiv

  19. [27]

    Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    Qwen Team et al. Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    2025 arXiv

  20. [28]

    Analysing the robustness of vision-language-models to common corruptions.CoRR, abs/2504.13690, 2025

    Muhammad Usama, Syeda Aishah Asim, Syed Bilal Ali, Syed Talal Wasim, and Umair Bin Mansoor. Analysing the robustness of vision-language-models to common corruptions.CoRR, abs/2504.13690, 2025

    2025 arXiv

  21. [29]

    Cogvlm: Visual expert for pretrained language models.arXiv preprint arXiv:2311.03079, 2023

    Yizhong Wang, Swaroop Mishra, Jesujoba Alabi, et al. Cogvlm: Visual expert for pretrained language models.arXiv preprint arXiv:2311.03079, 2023

    2023 arXiv

  22. [30]

    Demystifying the visual quality para- dox in multimodal large language models.CoRR, abs/2506.15645, 2025

    Shuo Xing, Lanqing Guo, Hongyuan Hua, et al. Demystifying the visual quality para- dox in multimodal large language models.CoRR, abs/2506.15645, 2025

    2025 arXiv

  23. [31]

    Mmt-bench: A comprehensive multi- modal benchmark for evaluating large vision-language models towards multitask agi

    Kaining Ying, Fanqing Meng, Jin Wang, et al. Mmt-bench: A comprehensive multi- modal benchmark for evaluating large vision-language models towards multitask agi. InProceedings of the 41st International Conference on Machine Learning (ICML), pages 57116–57198, 2024

    2024

  24. [32]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.arXiv preprint arXiv:2311.16502, 2023

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A m...

    2023 arXiv

  25. [33]

    Zeiler and Rob Fergus

    Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional net- works. InComputer Vision – ECCV 2014, pages 818–833. Springer, 2014

    2014

  26. [34]

    Benchmarking multi- modal llms on recognition and understanding over chemical tables.arXiv preprint arXiv:2506.11375, 2025

    Yitong Zhou, Mingyue Cheng, Qingyang Mao, et al. Benchmarking multi- modal llms on recognition and understanding over chemical tables.arXiv preprint arXiv:2506.11375, 2025. URLhttps://arxiv.org/abs/2506.11375. REHMAN & SHAFIQUE: BRUCE19 A Detailed Discussion on Further Observa...

    2025

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.