Pith. sign in

REVIEW 3 cited by

ConceptMath: A Bilingual Concept-wise Benchmark for Measuring Mathematical Reasoning of Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.14660 v2 pith:VQ2A3KPY submitted 2024-02-22 cs.CL cs.AI

Yanan Wu , Jie Liu , Xingyuan Bu , Jiaheng Liu , Zhanhui Zhou , Yuanxing Zhang , Chenchen Zhang , Zhiqi Bai
show 5 more authors
Haibin Chen Tiezheng Ge Wanli Ouyang Wenbo Su Bo Zheng
This is my paper · ORCID
classification cs.CLcs.AI
keywords mathematicalconceptmathllmsmodelsreasoningconcept-wisemathaccuracies
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

This paper introduces ConceptMath, a bilingual (English and Chinese), fine-grained benchmark that evaluates concept-wise mathematical reasoning of Large Language Models (LLMs). Unlike traditional benchmarks that evaluate general mathematical reasoning with an average accuracy, ConceptMath systematically organizes math problems under a hierarchy of math concepts, so that mathematical reasoning can be evaluated at different granularity with concept-wise accuracies. Based on our ConcepthMath, we evaluate a broad range of LLMs, and we observe existing LLMs, though achieving high average accuracies on traditional benchmarks, exhibit significant performance variations across different math concepts and may even fail catastrophically on the most basic ones. Besides, we also introduce an efficient fine-tuning strategy to enhance the weaknesses of existing LLMs. Finally, we hope ConceptMath could guide the developers to understand the fine-grained mathematical abilities of their models and facilitate the growth of foundation models.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. One Example Shown, Many Concepts Known! Counterexample-Driven Conceptual Reasoning in Mathematical LLMs

    cs.LG 2025-02 conditional novelty 6.0 of 10

    LLMs score poorly on CounterMATH, a new counterexample-based university math benchmark, and a 1,025-sample counterexample fine-tune yields small and partly inconsistent gains.

  2. Mixed-Precision Graph Neural Quantization for Low Bit Large Language Models

    cs.CL 2025-01 conditional novelty 6.0 of 10

    A graph neural network trained on Hessian-derived graphs of LLM weights adaptively assigns per-column quantization bit-widths, improving low-bit perplexity over GPTQ on LLaMA models.

  3. End-to-End Bangla AI for Solving Math Olympiad Problem Benchmark: Leveraging Large Language Model Using Integrated Approach

    cs.CL 2025-01 conditional novelty 4.0 of 10

    Fine-tuning Qwen2.5-7B with translated math datasets plus retrieval and tool-integrated reasoning yields 71/100 on a Bangla math olympiad test set, versus 77/100 for a larger base model.

Pith tools