REVIEW 5 cited by
Revealing the Barriers of Language Agents in Planning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Autonomous planning has been an ongoing pursuit since the inception of artificial intelligence. Based on curated problem solvers, early planning agents could deliver precise solutions for specific tasks but lacked generalization. The emergence of large language models (LLMs) and their powerful reasoning capabilities has reignited interest in autonomous planning by automatically generating reasonable solutions for given tasks. However, prior research and our experiments show that current language agents still lack human-level planning abilities. Even the state-of-the-art reasoning model, OpenAI o1, achieves only 15.6% on one of the complex real-world planning benchmarks. This highlights a critical question: What hinders language agents from achieving human-level planning? Although existing studies have highlighted weak performance in agent planning, the deeper underlying issues and the mechanisms and limitations of the strategies proposed to address them remain insufficiently understood. In this work, we apply the feature attribution study and identify two key factors that hinder agent planning: the limited role of constraints and the diminishing influence of questions. We also find that although current strategies help mitigate these challenges, they do not fully resolve them, indicating that agents still have a long way to go before reaching human-level intelligence.
Forward citations
Cited by 5 Pith papers
-
Beyond Theorem Proving: Formulation, Framework and Benchmark for Formal Problem-Solving
Formulates problem-solving as a sound Markov decision process, implements it in Lean as FPS and D-FPS, and introduces three formal problem-solving benchmarks plus the RPE answer-equivalence checker.
-
Leveraging Trajectory Graphs for Pre-Execution Error Diagnosis in Agentic LLM Systems
Modeling agent trajectories as action-centric probabilistic graphs lets a GNN warn LLM agents of likely step-level errors before execution, improving pass ratio ~14.7% across four benchmarks.
-
Agentomics-ML: Autonomous Machine Learning Experimentation Agent for Genomic and Transcriptomic Data
Agentomics-ML, an LLM-based agent with reflection, produced working classification code for genomic benchmarks in 93% of runs and beat all compared AI methods on six datasets.
-
Agent-R: Training Language Model Agents to Reflect via Iterative Self-Training
Agent-R iteratively fine-tunes language agents on trajectories that splice the agent's own failed prefix at a model-identified error step with a successful continuation, improving scores on WebShop, ScienceWorld, and ...
-
Large Language Models for Planning: A Comprehensive and Systematic Survey
A structured survey of LLM planning methods, benchmarks, and interpretability work, organized around a three-way taxonomy.
Discussion (0). Continue with ORCID to comment.