REVIEW 7 cited by
HARDMath: A Benchmark Dataset for Challenging Problems in Applied Mathematics
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Advanced applied mathematics problems are underrepresented in existing Large Language Model (LLM) benchmark datasets. To address this, we introduce HARDMath, a dataset inspired by a graduate course on asymptotic methods, featuring challenging applied mathematics problems that require analytical approximation techniques. These problems demand a combination of mathematical reasoning, computational tools, and subjective judgment, making them difficult for LLMs. Our framework auto-generates a large number of problems with solutions validated against numerical ground truths. We evaluate both open- and closed-source LLMs on HARDMath-mini, a sub-sampled test set of 366 problems, as well as on 40 word problems formulated in applied science contexts. Even leading closed-source models like GPT-4 achieve only 43.8% overall accuracy with few-shot Chain-of-Thought prompting, and all models demonstrate significantly lower performance compared to results on existing mathematics benchmark datasets. We additionally conduct a detailed error analysis to gain insights into the failure cases of LLMs. These results demonstrate limitations of current LLM performance on advanced graduate-level applied math problems and underscore the importance of datasets like HARDMath to advance mathematical abilities of LLMs.
Forward citations
Cited by 7 Pith papers
-
DPrivBench: Benchmarking LLMs' Reasoning for Differential Privacy
DPrivBench shows that top LLMs handle basic differential privacy mechanisms but fail on advanced algorithms, exposing gaps in automated DP reasoning.
-
DPrivBench: Benchmarking LLMs' Reasoning for Differential Privacy
DPrivBench is a new benchmark for evaluating LLMs on differential privacy reasoning, with results showing good performance on textbook mechanisms but substantial failures on advanced algorithms.
-
Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models
Evaluations of 53 LLMs on 14 basic math tasks show reasoning models use ~18x more tokens with sometimes lower accuracy, non-monotonic gains from extended budgets, and sharp performance drops under token constraints.
-
AdvancedMathBench: A Benchmark Suite for Advanced Mathematical Proof Generation and Verification
A 245-problem advanced proof benchmark plus 888 expert-labeled trajectories shows frontier LLMs remain far from reliable advanced proof generation and verification.
-
SPM-Bench: Benchmarking Large Language Models for Scanning Probe Microscopy
A new benchmark of 2,703 automatically generated multimodal questions for scanning probe microscopy, plus a modified F1 metric that penalizes over-selection and labels model 'personalities'.
-
EngTrace: A Symbolic Benchmark for Verifiable Process Supervision of Engineering Reasoning
EngTrace, a 1,350-instance symbolic engineering benchmark with gold reasoning traces, shows frontier LLMs outperform math-specialized small models and that trace verification reveals a complexity cliff.
-
ECHO-2: A Large-Scale Distributed Rollout Framework for Cost-Efficient Reinforcement Learning
Distributed rollout workers under a bounded-staleness budget can keep a centralized learner saturated and cut LLM post-training cost by roughly a third at matched reward.
Discussion (0). Sign in to comment.