REVIEW 4 cited by
On the Planning Abilities of Large Language Models : A Critical Investigation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Intrigued by the claims of emergent reasoning capabilities in LLMs trained on general web corpora, in this paper, we set out to investigate their planning capabilities. We aim to evaluate (1) the effectiveness of LLMs in generating plans autonomously in commonsense planning tasks and (2) the potential of LLMs in LLM-Modulo settings where they act as a source of heuristic guidance for external planners and verifiers. We conduct a systematic study by generating a suite of instances on domains similar to the ones employed in the International Planning Competition and evaluate LLMs in two distinct modes: autonomous and heuristic. Our findings reveal that LLMs' ability to generate executable plans autonomously is rather limited, with the best model (GPT-4) having an average success rate of ~12% across the domains. However, the results in the LLM-Modulo setting show more promise. In the LLM-Modulo setting, we demonstrate that LLM-generated plans can improve the search process for underlying sound planners and additionally show that external verifiers can help provide feedback on the generated plans and back-prompt the LLM for better plan generation.
Forward citations
Cited by 4 Pith papers
-
Training Small LLMs as Spatial Multi-Agent Policies
Small frozen LLMs trained over state-filtered symbolic option menus with per-agent LoRA adapters reach competent play in three cooperative spatial games, while behavioral audits show reward and cooperation decouple.
-
TREK: A Travel Reasoning and Evaluation Kit for LLM Agents in Complex Trip Planning
On 800 jointly constrained trip tasks with a deterministic scorer and achievable gold, the best of 15 LLM agents fully solves only 46.2% of feasible plans, with unstated persona needs as the universal bottleneck.
-
Information-seeking failures of large language models in agentic clinical reasoning
LLMs systematically under-request critical molecular and cytogenetic data in multi-round oncology, capping accuracy at 68% despite high knowledge scores and coherent reasoning traces.
-
AnnoBench: A Benchmark for Visualization Annotation Generation
A benchmark for chart annotation generation with a five-dimensional rubric shows current LLMs annotate code-based charts well but distort raster charts and over-rely on explicit instructions.
Discussion (0). Sign in to comment.