REVIEW 10 cited by
Can Large Language Models Detect Errors in Long Chain-of-Thought Reasoning?
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Recently, o1-like models have drawn significant attention, where these models produce the long Chain-of-Thought (CoT) reasoning steps to improve the reasoning abilities of existing Large Language Models (LLMs). In this paper, to understand the qualities of these long CoTs and measure the critique abilities of existing LLMs on these long CoTs, we introduce the DeltaBench, including the generated long CoTs from different o1-like models (e.g., QwQ, DeepSeek-R1) for different reasoning tasks (e.g., Math, Code, General Reasoning), to measure the ability to detect errors in long CoT reasoning. Based on DeltaBench, we first perform fine-grained analysis of the generated long CoTs to discover the effectiveness and efficiency of different o1-like models. Then, we conduct extensive evaluations of existing process reward models (PRMs) and critic models to detect the errors of each annotated process, which aims to investigate the boundaries and limitations of existing PRMs and critic models. Finally, we hope that DeltaBench could guide developers to better understand the long CoT reasoning abilities of their models.
Forward citations
Cited by 10 Pith papers
-
Stepwise Reasoning Checkpoint Analysis: A Test Time Scaling Method to Enhance LLMs' Reasoning
A checkpoint-based search and candidate augmentation method improves small LLM mathematical reasoning accuracy over existing test-time scaling baselines.
-
Can Large Language Models Reason About Complex Execution Paths? An Empirical Study on Python
State-of-the-art LLMs solve over 60% of complex Python path constraints in test generation, but their path classification is unreliable, especially for infeasible paths.
-
Can Large Multimodal Models Actively Recognize Faulty Inputs? A Systematic Evaluation Framework of Their Input Scrutiny Ability
Large multimodal models mostly fail to proactively detect flawed textual premises, and their performance depends on error type and on how they weight text versus images.
-
RefCritic: Training Long Chain-of-Thought Critic Models with Refinement Feedback
RefCritic uses reinforcement learning with two rule-based rewards, judgment accuracy and refinement accuracy, to train long-chain-of-thought critic models that improve LLM math reasoning and detect erroneous steps.
-
CriticLean: Critic-Guided Reinforcement Learning for Mathematical Formalization
A critic model trained with reinforcement learning judges semantic correctness of Lean 4 formalizations, and using it as a filter sharply improves autoformalization accuracy.
-
What Makes a Good Reasoning Chain? Uncovering Structural Patterns in Long Chain-of-Thought Reasoning
Reasoning trees parsed from long chain-of-thought text predict answer correctness better than length and expose failure patterns such as over-branching.
-
Are Large Language Models Reliable AI Scientists? Assessing Reverse-Engineering of Black-Box Systems
LLMs struggle to use passive observations for reverse engineering, but active intervention improves performance, largely through the process of generating queries rather than the data obtained.
-
R1-ACT: Efficient Reasoning Model Safety Alignment by Activating Safety Knowledge
Adding an explicit 'is this harmful?' step to the reasoning chain, trained on just 1,000 examples, substantially reduces harmful responses from reasoning models while roughly preserving benchmark reasoning performance.
-
CoRT: Code-integrated Reasoning within Thinking
Inserting targeted hints into a few training examples teaches reasoning models to compute with Python instead of text, improving accuracy and cutting token use by 30 to 50 percent.
-
Can A Gamer Train A Mathematical Reasoning Model?
Fine-tuning Qwen2.5-Math-1.5B with LoRA and GRPO on one RTX 3080 Ti improves GSM8K accuracy from 71.65 to 73.69, matching or beating several larger base models.
Discussion (0). Sign in to comment.