REVIEW 31 cited by
LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
There is considerable confusion about the role of Large Language Models (LLMs) in planning and reasoning tasks. On one side are over-optimistic claims that LLMs can indeed do these tasks with just the right prompting or self-verification strategies. On the other side are perhaps over-pessimistic claims that all that LLMs are good for in planning/reasoning tasks are as mere translators of the problem specification from one syntactic format to another, and ship the problem off to external symbolic solvers. In this position paper, we take the view that both these extremes are misguided. We argue that auto-regressive LLMs cannot, by themselves, do planning or self-verification (which is after all a form of reasoning), and shed some light on the reasons for misunderstandings in the literature. We will also argue that LLMs should be viewed as universal approximate knowledge sources that have much more meaningful roles to play in planning/reasoning tasks beyond simple front-end/back-end format translators. We present a vision of {\bf LLM-Modulo Frameworks} that combine the strengths of LLMs with external model-based verifiers in a tighter bi-directional interaction regime. We will show how the models driving the external verifiers themselves can be acquired with the help of LLMs. We will also argue that rather than simply pipelining LLMs and symbolic components, this LLM-Modulo Framework provides a better neuro-symbolic approach that offers tighter integration between LLMs and symbolic components, and allows extending the scope of model-based planning/reasoning regimes towards more flexible knowledge, problem and preference specifications.
Forward citations
Cited by 31 Pith papers
-
Agent vs. Parametric World Models: Hybrid Planning for Reliable Language Agents
GILP trains a parameterized backbone for valid actions and state predictions, then uses a consistency gate with LLM drafts to reduce hallucinated-state rate from 0.176 to 0.035 on GPT-4o-mini while raising success fro...
-
Monte Carlo Tree Search for Comprehensive Exploration in LLM-Based Automatic Heuristic Design
MCTS-AHD organizes LLM-generated heuristics in a Monte Carlo tree and reports better or comparable heuristics than population-based baselines on several NP-hard problems and a Bayesian optimization task.
-
TREK: A Travel Reasoning and Evaluation Kit for LLM Agents in Complex Trip Planning
On 800 jointly constrained trip tasks with a deterministic scorer and achievable gold, the best of 15 LLM agents fully solves only 46.2% of feasible plans, with unstated persona needs as the universal bottleneck.
-
SymStep: Symbolic Step Verification for Logical Reasoning
SymStep couples atomic LLM deductions to a deterministic constraint propagator with MRV hints, reaching ~97–100% on constraint-dense logic puzzles where CoT scores 0%.
-
LLM-WikiRace Benchmark: How Far Can LLMs Plan over Real-World Knowledge Graphs?
Frontier LLMs exceed human performance on easy Wikipedia navigation tasks but finish fewer than 25% of hard games, with failures driven by looping and an inability to replan after mistakes.
-
A Systematic Study of Large Language Models for Task and Motion Planning With PDDLStream
LLM-based planners inside an LLM-Modulo TAMP loop have lower success rates and higher planning times than engineered planners, and fast direct variants outperform slow thinking variants in most settings.
-
Can LLMs Generate Good Stories? Insights and Challenges from a Narrative Planning Perspective
A new automatically verified benchmark shows GPT-4 tier LLMs can do small-scale causal story planning, but character intentionality and dramatic conflict remain hard except for reasoning models like o1.
-
Token Signature: Predicting Chain-of-Thought Gains with Token Decoding Feature in Large Language Models
The monotonicity of token probabilities during initial decoding predicts chain-of-thought gains, enabling dynamic selection between CoT and direct answers.
-
Integrating Neural and Symbolic Components in a Model of Pragmatic Question-Answering
A neuro-symbolic Rational Speech Act model with LLM proposers and evaluators predicts human question-answer patterns about as well as the fully hand-specified probabilistic model.
-
Learning Compositional Behaviors from Demonstration and Language
BLADE learns structured, planable action representations from language-annotated demonstrations and composes them with a symbolic planner, outperforming latent and LLM/VLM baselines on new manipulation tasks.
-
xInv: Explainable Optimization of Inverse Problems
An explainability method that instruments differentiable optimizers to emit natural language events and uses a language model to synthesize human-readable explanations of inverse problem optimization.
-
Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistant
User involvement in plan-then-execute LLM agents fails to calibrate trust and only inconsistently improves performance, while plausible but wrong plans can mislead users into over-trust.
-
Disentangling Exploration of Large Language Models by Optimal Exploitation
Exploration by LLM agents can be measured separately from exploitation using an optimal exploitation oracle, and most models explore poorly, with exploration performance correlated to reasoning ability.
-
On the Limit of Language Models as Planning Formalizers
On simple planning domains, state-of-the-art LLMs can generate complete, solvable PDDL that often outperforms direct plan generation, but natural-language descriptions and complex domains sharply reduce accuracy.
-
BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games
A benchmark of six long-horizon game environments shows current LLMs and VLMs struggle on hard tasks and often do worse when given images.
-
Confidently Wrong: Exception Chain Collapse in Frontier LLM Rule Evaluation
Nested exception-chain eligibility breaks frontier LLMs in unstable ways; an SMT execution layer makes outcomes deterministic given authored rules.
-
Search, Fail, Recover: A Training Framework for Correction-Aware Reasoning
Pyligent trains LLMs to search, detect failures via task validators, and backtrack to recoverable prefixes, improving solve rates by 13–73 points over gold-only SFT on hidden graphs, Sudoku, and Blocksworld.
-
Initial Steps in Integrating Large Reasoning and Action Models for Service Composition
A conceptual framework integrating LRMs and LAMs for end-to-end automated service composition, with no empirical validation.
-
Large Reasoning Models are not thinking straight: on the unreliability of thinking trajectories
Large reasoning models frequently ignore explicitly injected ground-truth solutions mid-trajectory and continue to overthink, often ending on incorrect answers.
-
Spectral Journey: How Transformers Predict the Shortest Path
Two-layer transformers learn shortest paths on small graphs by building embeddings that correlate with spectral decomposition of the line graph, yielding an approximate spectral path-finding algorithm.
-
Policy Guided Tree Search for Enhanced LLM Reasoning
PGTS trains a graph-transformer policy via PPO to guide tree search over LLM reasoning steps, reporting improved accuracy and lower token use than chain-of-thought and MCTS baselines.
-
LLMER: Crafting Interactive Extended Reality Worlds with JSON Data Generated by Large Language Models
LLMER uses LLM-generated JSON data instead of code to create interactive XR worlds, cutting token use and task completion time in a small user study.
-
Towards Specification-Driven LLM-Based Generation of Embedded Automotive Software
A feasibility study in which GPT-4 and GPT-3.5 generated C code for three Scania automotive modules, and some of that code passed Frama-C verification against hand-derived ACSL specifications without iterative feedback.
-
LOOP: A Plug-and-Play Neuro-Symbolic Framework for Enhancing Planning in Autonomous Systems
Iterative feedback between an LLM and a PDDL validator is claimed to reach 85.8% success on six IPC planning domains, beating LLM+P at 55.0%, with no error bars or protocol details given in the abstract.
-
One For All: LLM-based Heterogeneous Mission Planning in Precision Agriculture
An LLM-based planner with XML-schema validation generates behavior-tree missions in plain language for both a Husky rover and a Kinova arm in agricultural scenarios.
-
Whispers of Many Shores: Cultural Alignment through Collaborative Cultural Expertise
A multi-agent router that selects culturally specialized LLM personas reports a jump in self-scored cultural alignment from 0.208 to 0.820, but the metric and the claimed method are not independently validated.
-
One to rule them all: natural language to bind communication, perception and action
The authors combine an LLM-based ReAct planner, a semantic scene graph, and a hand-built failure memory to let a humanoid robot follow natural-language instructions and recover from errors.
-
One STEP at a time: Language Agents are Stepwise Planners
A stepwise planner with memory-guided execution and evaluation raises ScienceWorld scores to 67.4, outperforming CLIN and published SOTA baselines.
-
GenPlanX. Generation of Plans and Execution
GenPlanX couples an LLM-based natural-language to PDDL translation with a classical planner and an execution monitor to solve office automation tasks with plan guarantees.
-
A Survey on Human-Centric LLMs
A review that sorts existing evidence on how well large language models imitate individual human skills and collective social dynamics into one taxonomy.
- Transformers Struggle to Learn to Search
Discussion (0). Continue with ORCID to comment.