REVIEW 10 cited by
LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Generating accurate step-by-step reasoning is essential for Large Language Models (LLMs) to address complex problems and enhance robustness and interpretability. Despite the flux of research on developing advanced reasoning approaches, systematically analyzing the diverse LLMs and reasoning strategies in generating reasoning chains remains a significant challenge. The difficulties stem from the lack of two key elements: (1) an automatic method for evaluating the generated reasoning chains on different tasks, and (2) a unified formalism and implementation of the diverse reasoning approaches for systematic comparison. This paper aims to close the gap: (1) We introduce AutoRace for fully automated reasoning chain evaluation. Existing metrics rely on expensive human annotations or pre-defined LLM prompts not adaptable to different tasks. In contrast, AutoRace automatically creates detailed evaluation criteria tailored for each task, and uses GPT-4 for accurate evaluation following the criteria. (2) We develop LLM Reasoners, a library for standardized modular implementation of existing and new reasoning algorithms, under a unified formulation of the search, reward, and world model components. With the new evaluation and library, (3) we conduct extensive study of different reasoning approaches (e.g., CoT, ToT, RAP). The analysis reveals interesting findings about different factors contributing to reasoning, including the reward-guidance, breadth-vs-depth in search, world model, and prompt formats, etc.
Forward citations
Cited by 10 Pith papers
-
SidConArena: An Environment Evaluating Agents in Open-Ended,Positive-Sum Bargaining Game
SidConArena is a new multi-phase benchmark framework formalizing a partially observable stochastic game for evaluating LLM agents in open-ended positive-sum bargaining with negotiation, converter production, and seale...
-
Training Large Language Models to Reason in a Continuous Latent Space
Coconut lets LLMs perform reasoning directly in continuous latent space by recycling hidden states as inputs, outperforming standard chain-of-thought on search-intensive logical tasks with better accuracy-efficiency t...
-
Supervised Fine-tuning with Synthetic Rationale Data Hurts Real-World Disease Prediction
Across 504 configurations on five-year ADRD prediction, rationale-based supervised fine-tuning consistently degrades performance relative to label-only fine-tuning, despite high-quality rationales validated by experts.
-
MoG: Mixture of Experts for Graph-based Retrieval-Augmented Generation
MoG uses hub graphs for shared context and sparsely activates expert graphs with a topology-aware router, reporting over 20% relative gains on MuSiQue.
-
Vision-aligned Latent Reasoning for Multi-modal Large Language Model
VaLR generates vision-aligned latent tokens before each reasoning step to preserve perceptual cues, improving VSI-Bench accuracy from 33.0% to 52.9%.
-
General Agentic Planning Through Simulative Reasoning with World Models
SiRA uses LLM world models for simulative reasoning to achieve up to 124% higher task completion and 32.2% navigation success versus reactive baselines in web environments.
-
TreeThink: A Modular Tree Search Library for Mathematical Reasoning with LLMs
TreeThink provides a modular, asynchronous tree-search library for neural theorem proving with unified REPL clients for Lean, Rocq, and Isabelle and up to 6.3× wall-clock speedup.
-
Improving Factuality in LLMs via Inference-Time Knowledge Graph Construction
A framework for inference-time knowledge graph construction and expansion improves factual accuracy in LLMs on three QA benchmarks by combining internal LLM knowledge with selective external retrieval.
-
Agentic Reasoning for Large Language Models
The survey structures agentic reasoning for LLMs into foundational, self-evolving, and collective multi-agent layers while distinguishing in-context orchestration from post-training optimization and reviewing applicat...
-
CodeGrad: Integrating Multi-Step Verification with Gradient-Based LLM Refinement
CodeGrad improves code generation by routing structured critic feedback into up to two refinement rounds, but its formal-verification claim rests on AI-written proofs judged by the AI.
Discussion (0). Sign in to comment.