Pith. sign in

REVIEW 18 cited by

Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gap

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.19450 v1 pith:O7B3SOF5 submitted 2024-02-29 cs.AI cs.CL

Saurabh Srivastava , Annarose M B , Anto P V , Shashank Menon , Ajay Sukumar , Adwaith Samod T , Alan Philipose , Stevin Prince
show 1 more author
Sooraj Thomas
This is my paper · ORCID
classification cs.AIcs.CL
keywords reasoningmodelsfunctionalbenchmarksevaluationmathgapsperformance
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

We propose a framework for robust evaluation of reasoning capabilities of language models, using functional variants of benchmarks. Models that solve a reasoning test should exhibit no difference in performance over the static version of a problem compared to a snapshot of the functional variant. We have rewritten the relevant fragment of the MATH benchmark into its functional variant MATH(), with functionalization of other benchmarks to follow. When evaluating current state-of-the-art models over snapshots of MATH(), we find a reasoning gap -- the percentage difference between the static and functional accuracies. We find reasoning gaps from 58.35% to 80.31% among the state-of-the-art closed and open weights models that perform well on static benchmarks, with the caveat that the gaps are likely to be smaller with more sophisticated prompting strategies. Here we show that models which anecdotally have good reasoning performance over real-world tasks, have quantifiable lower gaps, motivating the open problem of building "gap 0" models. Code for evaluation and new evaluation datasets, three MATH() snapshots, are publicly available at https://github.com/consequentai/fneval/.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 18 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. New Lower and Upper Bounds for the Grothendieck Constant

    cs.CC 2026-08 conditional novelty 8.0 of 10

    New rigorous bounds pin the Grothendieck constant to [6pi/11, pi/(2 log(1+sqrt 2)) - 10^-4], improving both known lower and upper bounds.

  2. MirrorCode: AI can rebuild entire programs from behavior alone

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    MirrorCode benchmark shows current AI models achieving up to 56% success reimplementing 25 diverse full programs from behavior alone, including a 16,000-line bioinformatics toolkit.

  3. Robust Reasoning Benchmark

    cs.LG 2026-03 unverdicted novelty 7.0 of 10

    The Robust Reasoning Benchmark shows frontier LLMs are mostly resilient to textual perturbations on AIME problems while open-weight models suffer up to 54% accuracy drops and exhibit accuracy decay on later problems d...

  4. Long-Horizon AI Research for Grothendieck Constant: A Case Study in Human-AI Mathematical Collaboration

    cs.AI 2026-08 conditional novelty 6.0 of 10

    A human-AI team reports new bounds on the Grothendieck constant, 6π/11 ≤ K_G ≤ π/(2 log(1+√2)) - 3.47e-4, crediting an AI model with the core idea for the lower bound.

  5. Probing for Arithmetic Errors in Language Models

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Lightweight probes trained on hidden states decode both predicted and correct arithmetic results, detect errors with over 90% accuracy, and transfer from isolated addition to chain-of-thought reasoning.

  6. RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Language models score substantially lower on automatically generated variations of standard math, logic, and code benchmarks, indicating partial reliance on memorization.

  7. ASyMOB: Algebraic Symbolic Mathematical Operations Benchmark

    cs.CL 2025-05 conditional novelty 6.0 of 10

    A new benchmark of perturbed symbolic math problems shows that large language models' performance drops sharply under minor numeric, symbolic, and equivalence transformations.

  8. UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models

    cs.CL 2025-02 conditional novelty 6.0 of 10

    UGPhysics is a new bilingual benchmark of 5,520 undergraduate physics problems; the strongest tested LLM, OpenAI o1-mini, reaches only 49.8% accuracy.

  9. UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models

    cs.CL 2025-01 conditional novelty 6.0 of 10

    UGMathBench provides a 5,062-problem dynamic benchmark for undergraduate math reasoning and shows that leading LLMs solve all versions of only about half the problems.

  10. Critical-Questions-of-Thought: Steering LLM reasoning with Argumentative Querying

    cs.AI 2024-12 conditional novelty 6.0 of 10

    CQoT, a pipeline that uses argumentation-theoretic critical questions to check LLM reasoning plans, improves MT-Bench reasoning and math scores by roughly 5% over baseline and CoT prompting.

  11. AntiLeakBench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledge

    cs.CL 2024-12 conditional novelty 6.0 of 10

    AntiLeakBench automatically constructs QA benchmarks from knowledge updated after each model's cutoff, and its experiments suggest that pre-cutoff evaluation overstates LLM ability.

  12. ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended Capabilities

    cs.LG 2024-12 conditional novelty 6.0 of 10

    ONEBench treats each benchmark sample as a voter in a Plackett-Luce aggregation, enabling open-ended, capability-specific, and incomplete-data model rankings.

  13. Evaluating the Robustness of Analogical Reasoning in Large Language Models

    cs.CL 2024-11 conditional novelty 6.0 of 10

    GPT models solve original analogy tasks but fail many simple variants that humans handle easily, showing their analogy performance is not robust.

  14. Probing the Capacity of Language Model Agents to Operationalize Disparate Experiential Context Despite Distraction

    cs.CL 2024-11 conditional novelty 6.0 of 10

    On the OEDD benchmark, LLM agents select the worse of two actions below chance when the correct choice requires combining two earlier facts and ignoring a recent distractor in contexts over 1,615 tokens.

  15. Artificial Intelligence for Mathematical Reasoning: An Integrated Survey of Language Models, Neuro-symbolic Systems, and Verified Discovery

    cs.AI 2026-06 unverdicted novelty 5.0 of 10

    An integrated survey organizing AI mathematical reasoning into informal, formal, discovery, and technique axes while cataloging benchmarks and assessing failure modes.

  16. Reasoning Language Models: A Blueprint

    cs.AI 2025-01 accept novelty 5.0 of 10

    A modular blueprint and open-source framework (x1) that presents existing reasoning language model designs as special cases of one unified toolbox.

  17. Ask-Before-Detection: Identifying and Mitigating Conformity Bias in LLM-Powered Error Detector for Math Word Problem Solutions

    cs.CL 2024-12 conditional novelty 5.0 of 10

    LLM error detectors favor conventional solution formats; generating an adaptive reference solution before grading mitigates this conformity bias on a 200-example GSM8K subset.

  18. Causality can systematically address the monsters under the bench(marks)

    cs.LG 2025-02 conditional novelty 4.0 of 10

    A position paper making the case that causal inference should be the organizing framework for evaluating large language models, with template causal graphs (CATs) for benchmark problems.

Pith tools