REVIEW 6 cited by
Can We Rely on LLM Agents to Draft Long-Horizon Plans? Let's Take TravelPlanner as an Example
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Large language models (LLMs) have brought autonomous agents closer to artificial general intelligence (AGI) due to their promising generalization and emergent capabilities. There is, however, a lack of studies on how LLM-based agents behave, why they could potentially fail, and how to improve them, particularly in demanding real-world planning tasks. In this paper, as an effort to fill the gap, we present our study using a realistic benchmark, TravelPlanner, where an agent must meet multiple constraints to generate accurate plans. We leverage this benchmark to address four key research questions: (1) are LLM agents robust enough to lengthy and noisy contexts when it comes to reasoning and planning? (2) can few-shot prompting adversely impact the performance of LLM agents in scenarios with long context? (3) can we rely on refinement to improve plans, and (4) can fine-tuning LLMs with both positive and negative feedback lead to further improvement? Our comprehensive experiments indicate that, firstly, LLMs often fail to attend to crucial parts of a long context, despite their ability to handle extensive reference information and few-shot examples; secondly, they still struggle with analyzing the long plans and cannot provide accurate feedback for refinement; thirdly, we propose Feedback-Aware Fine-Tuning (FAFT), which leverages both positive and negative feedback, resulting in substantial gains over Supervised Fine-Tuning (SFT). Our findings offer in-depth insights to the community on various aspects related to real-world planning applications.
Forward citations
Cited by 6 Pith papers
-
Evolution and The Knightian Blindspot of Machine Learning
ML's formalisms, particularly RL's, exclude Knightian uncertainty, and evolution's diversify-and-filter mechanisms point toward a direct remedy.
-
LAD-VF: LLM-Automatic Differentiation Enables Fine-Tuning-Free Robot Planning from Formal Methods Feedback
LAD-VF iteratively rewrites LLM prompts using formal verification failure rates as a loss, lifting safety-spec compliance from about 60% to over 90% without model fine-tuning.
-
LLMER: Crafting Interactive Extended Reality Worlds with JSON Data Generated by Large Language Models
LLMER uses LLM-generated JSON data instead of code to create interactive XR worlds, cutting token use and task completion time in a small user study.
-
Efficient Prompt Compression with Evaluator Heads for Long-Context Transformer Inference
EHPC uses a handful of attention heads, selected via a synthetic needle-in-a-haystack test, to compress long prompts and speed up LLM inference while retaining task performance.
-
Can LLM-Reasoning Models Replace Classical Planning? A Benchmark Study
Reasoning-optimized LLMs reach at most 63.4% success across the five PDDL benchmarks, versus 97.9% for Fast Downward, and frequently produce plans that fail during execution.
-
A Survey on Multi-Turn Interaction Capabilities of Large Language Models
A comprehensive review of how large language models are evaluated, trained, and improved for multi-turn interaction, organized into evaluation practices, core capabilities, and general algorithms.
Discussion (0). Continue with ORCID to comment.