Pith Number

pith:JATB3AB7

pith:2024:JATB3AB7PTUEINKWCUPAMJTI3S

not attested not anchored not stored refs resolved

Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset

Hongsheng Li, Junting Pan, Ke Wang, Mingjie Zhan, Weikang Shi, Zimu Lu

The MATH-Vision dataset of 3,040 competition-sourced visual math problems reveals a large performance gap between current large multimodal models and human solvers.

arxiv:2402.14804 v1 · 2024-02-22 · cs.CV · cs.AI · cs.CL · cs.LG · math.HO

Open paper page JSON What is a Pith Number?

Record completeness

1 Bitcoin timestamp

2 Internet Archive

3 Author claim open · sign in to claim

4 Citations open

5 Replications open

Claims

C1strongest claim

Through extensive experimentation, we unveil a notable performance gap between current LMMs and human performance on MATH-V, underscoring the imperative for further advancements in LMMs.

C2weakest assumption

The curation process from real competitions produces a representative and unbiased sample of visual mathematical reasoning challenges without introducing selection effects that favor certain problem types.

C3one line summary

MATH-Vision is a new benchmark of 3,040 visual mathematical competition problems that reveals substantial gaps between large multimodal models and human performance in mathematical reasoning.

References

27 extracted · 27 resolved · 2 Pith anchors

[1] GeoQA: A geometric question answering benchmark towards multimodal numerical reasoning 2023

[2] Fangyu Liu, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Yasemin Altun, Nigel Collier, and Julian Martin Eisenschlos

[3] MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts 2023 · arXiv:2310.02255

[4] MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI 2022 · arXiv:2311.16502

[5] ". If it is a multiple choice question, only one letter is allowed in the 2010

Formal links

2 machine-checked theorem links

Cited by

20 papers in Pith

Mixture-of-Visual-Thoughts: Exploring Context-Adaptive Reasoning Mode Selection for General Visual Reasoning

Mitigating Visual Context Degradation in Large Multimodal Models: A Training-Free Decoupled Agentic Framework

OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning

MetaMorph: Multimodal Understanding and Generation via Instruction Tuning

Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs

Receipt and verification

First computed	2026-05-17T23:38:13.127495Z
Builder	pith-number-builder-2026-05-17-v1
Signature	Pith Ed25519 (`pith-v1-2026-05`) · public key
Schema	pith-number/v1.0

Canonical hash

48261d803f7ce8443556151e062668dcb02f1c5399b48b445cea402c82a6e058

Aliases

arxiv: 2402.14804 · arxiv_version: 2402.14804v1 · doi: 10.48550/arxiv.2402.14804 · pith_short_12: JATB3AB7PTUE · pith_short_16: JATB3AB7PTUEINKW · pith_short_8: JATB3AB7

Agent API

Resolver JSON Graph JSON Events JSON Schema Signing key

Verify this Pith Number yourself

curl -sH 'Accept: application/ld+json' https://pith.science/pith/JATB3AB7PTUEINKWCUPAMJTI3S \
  | jq -c '.canonical_record' \
  | python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: 48261d803f7ce8443556151e062668dcb02f1c5399b48b445cea402c82a6e058

Canonical record JSON

{
  "metadata": {
    "abstract_canon_sha256": "ab8db3b1d9ffae8b1caabd3e948c239ccc005888bae1db02f29069b09bca720b",
    "cross_cats_sorted": [
      "cs.AI",
      "cs.CL",
      "cs.LG",
      "math.HO"
    ],
    "license": "http://creativecommons.org/licenses/by-nc-sa/4.0/",
    "primary_cat": "cs.CV",
    "submitted_at": "2024-02-22T18:56:38Z",
    "title_canon_sha256": "cfba9a1c2871cc9808744fdd45c3ddacdb2bc0720bce42dbe33b3e93b236e091"
  },
  "schema_version": "1.0",
  "source": {
    "id": "2402.14804",
    "kind": "arxiv",
    "version": 1
  }
}