REVIEW 11 cited by
TheoremQA: A Theorem-driven Question Answering dataset
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The recent LLMs like GPT-4 and PaLM-2 have made tremendous progress in solving fundamental math problems like GSM8K by achieving over 90% accuracy. However, their capabilities to solve more challenging math problems which require domain-specific knowledge (i.e. theorem) have yet to be investigated. In this paper, we introduce TheoremQA, the first theorem-driven question-answering dataset designed to evaluate AI models' capabilities to apply theorems to solve challenging science problems. TheoremQA is curated by domain experts containing 800 high-quality questions covering 350 theorems (e.g. Taylor's theorem, Lagrange's theorem, Huffman coding, Quantum Theorem, Elasticity Theorem, etc) from Math, Physics, EE&CS, and Finance. We evaluate a wide spectrum of 16 large language and code models with different prompting strategies like Chain-of-Thoughts and Program-of-Thoughts. We found that GPT-4's capabilities to solve these problems are unparalleled, achieving an accuracy of 51% with Program-of-Thoughts Prompting. All the existing open-sourced models are below 15%, barely surpassing the random-guess baseline. Given the diversity and broad coverage of TheoremQA, we believe it can be used as a better benchmark to evaluate LLMs' capabilities to solve challenging science problems. The data and code are released in https://github.com/wenhuchen/TheoremQA.
Forward citations
Cited by 11 Pith papers
-
FeynmanBench: Benchmarking Multimodal LLMs on Diagrammatic Physics Reasoning
FeynmanBench is the first benchmark for evaluating multimodal LLMs on diagrammatic reasoning with Feynman diagrams, revealing systematic failures in enforcing physical constraints and global topology.
-
Empirical Characterization of Inference-Time Elicited Probability Transformations in Large Language Models
Across 4,975 reasoning problems and multiple LLM families, post-evidence answer probabilities follow an approximate log-ratio relation log q̃ ≈ α(log q + log b) + c with mean R² ≈ 0.76, where α varies by prompting con...
-
StatEval: A Comprehensive Benchmark for Large Language Models in Statistics
StatEval is a new 16,000-question statistics benchmark showing that even strong LLMs score below 60% on research-level statistical proof tasks.
-
A Simple "Try Again" Can Elicit Multi-Turn LLM Reasoning
Multi-turn RL with only unary 'try again' feedback improves multi-turn reasoning accuracy by up to 14% while preserving single-turn performance.
-
VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videos
VRBench is a benchmark of 960 long narrative videos with 8,243 human-written multi-step questions, plus a two-level evaluation of answer accuracy and reasoning quality for 31 large models.
-
PhySense: Principle-Based Physics Reasoning Benchmarking for Large Language Models
A new benchmark of 380 principle-based physics problems shows that state-of-the-art LLMs struggle to apply symmetry, conservation, and dimensional-analysis shortcuts, achieving under 50 percent average accuracy with h...
-
Structured Thoughts For Improved Reasoning And Context Pruning
Structured try/outcome SFT improves math reasoning by up to 8% over standard SFT and enables pruning ~85% of context with ~9% accuracy drop.
-
Coupled Variational Reinforcement Learning for Language Model General Reasoning
CoVRL trains an LLM on a mixture of question-only and answer-guided reasoning traces, using the model's own answer probability as reward, and reports consistent gains on math and general-reasoning benchmarks.
-
CLPO: Curriculum Learning meets Policy Optimization for LLM Reasoning
CLPO: an online curriculum that rewrites each problem according to the model's own current accuracy and scales the KL penalty by difficulty, reporting strong but internally inconsistent gains over GRPO and DAPO.
-
Technical Report of TeleChat2, TeleChat2.5 and T1
The released T1-115B open-weight model outperforms OpenAI's o1-mini and GPT-4o on MATH500, AlignBench, and IFEval, despite using a standard dense transformer architecture.
-
AI4Research: A Survey of Artificial Intelligence for Scientific Research
A survey that organizes AI-for-research work into five tasks, comprehension, survey, discovery, writing, and peer review, and compiles associated tools and benchmarks.
Discussion (0). Sign in to comment.