Pith. sign in

REVIEW 4 cited by

ThinkBench: Dynamic Out-of-Distribution Evaluation for Robust LLM Reasoning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2502.16268 v1 pith:OHRCXXQJ submitted 2025-02-22 cs.CL

ThinkBench: Dynamic Out-of-Distribution Evaluation for Robust LLM Reasoning

Shulin Huang , Linyi Yang , Yan Song , Shuang Chen , Leyang Cui , Ziyu Wan , Qingcheng Zeng , Ying Wen
show 4 more authors
Kun Shao Weinan Zhang Jun Wang Yue Zhang
This is my paper
classification cs.CL
keywords llmsthinkbenchdataevaluationreasoningmodelschallengescontamination
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

Evaluating large language models (LLMs) poses significant challenges, particularly due to issues of data contamination and the leakage of correct answers. To address these challenges, we introduce ThinkBench, a novel evaluation framework designed to evaluate LLMs' reasoning capability robustly. ThinkBench proposes a dynamic data generation method for constructing out-of-distribution (OOD) datasets and offers an OOD dataset that contains 2,912 samples drawn from reasoning tasks. ThinkBench unifies the evaluation of reasoning models and non-reasoning models. We evaluate 16 LLMs and 4 PRMs under identical experimental conditions and show that most of the LLMs' performance are far from robust and they face a certain level of data leakage. By dynamically generating OOD datasets, ThinkBench effectively provides a reliable evaluation of LLMs and reduces the impact of data contamination.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. The Generalization Spectrum: A Chromatographic Approach to Evaluating Learning Algorithms

    cs.LG 2026-06 unverdicted novelty 7.0

    Introduces the Generalization Spectrum evaluation framework to track per-example generalization across transfer distances in competitive programming tasks.

  2. The Generalization Spectrum: A Chromatographic Approach to Evaluating Learning Algorithms

    cs.LG 2026-06 unverdicted novelty 6.0

    The Generalization Spectrum framework shows RL turns memorization into near-transfer more efficiently than SFT, ICL transfer depends on correspondence, and local gains often fail to expand far-transfer radius.

  3. ReasonBENCH: Benchmarking the (In)Stability of LLM Reasoning

    cs.AI 2025-12 reject novelty 5.0

    LLM reasoning benchmark scores vary substantially across repeated runs under the same model, strategy, and task, so single-run evaluation can misrank systems.

  4. Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models

    cs.AI 2025-03 unverdicted novelty 5.0

    The paper unifies perspectives on Long CoT in reasoning LLMs by introducing a taxonomy, detailing characteristics of deep reasoning and reflection, and discussing emergence phenomena and future directions.