REVIEW 10 cited by
Chain of Thoughtlessness? An Analysis of CoT in Planning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language model (LLM) performance on reasoning problems typically does not generalize out of distribution. Previous work has claimed that this can be mitigated with chain of thought prompting-a method of demonstrating solution procedures-with the intuition that it is possible to in-context teach an LLM an algorithm for solving the problem. This paper presents a case study of chain of thought on problems from Blocksworld, a classical planning domain, and examines the performance of two state-of-the-art LLMs across two axes: generality of examples given in prompt, and complexity of problems queried with each prompt. While our problems are very simple, we only find meaningful performance improvements from chain of thought prompts when those prompts are exceedingly specific to their problem class, and that those improvements quickly deteriorate as the size n of the query-specified stack grows past the size of stacks shown in the examples. We also create scalable variants of three domains commonly studied in previous CoT papers and demonstrate the existence of similar failure modes. Our results hint that, contrary to previous claims in the literature, CoT's performance improvements do not stem from the model learning general algorithmic procedures via demonstrations but depend on carefully engineering highly problem specific prompts. This spotlights drawbacks of chain of thought, especially the sharp tradeoff between possible performance gains and the amount of human labor necessary to generate examples with correct reasoning traces.
Forward citations
Cited by 10 Pith papers
-
From Custom-Fit to Portable: Bridging the Gap Between Synthesized and Engineered GPU Query Execution
LLM-synthesized GPU query kernels outperform engineered engines by 7.4x, but a portable SYCL engine with lifted optimizations closes the gap to 1.27x, suggesting engineering remains preferable on GPUs.
-
From Reasoning to Super-Intelligence: A Search-Theoretic Perspective
The Diligent Learner, a reverse-curriculum algorithm with explicit backtracking and a validator, is proven to learn chain-of-thought reasoning efficiently under two learnability assumptions, while standard methods fai...
-
LLM+AL: Bridging Large Language Models and Action Languages for Complex Reasoning about Actions
LLM+AL, an LLM-to-BC+ pipeline with solver-feedback self-revision, outperforms direct LLMs and LLM-generated Python on complex action-reasoning puzzles, though it still needs a few manual corrections on most hard cases.
-
Self-guided Knowledgeable Network of Thoughts: Amplifying Reasoning with Large Language Models
A large language model can plan and write an executable network of elementary reasoning steps for itself, and this self-guided workflow beats prior prompting schemes on sorting, arithmetic, and language counting tasks.
-
On the Limit of Language Models as Planning Formalizers
On simple planning domains, state-of-the-art LLMs can generate complete, solvable PDDL that often outperforms direct plan generation, but natural-language descriptions and complex domains sharply reduce accuracy.
-
Agentic Workflows for Conversational Human-AI Interaction Design
A four-cycle design study found that agentic workflows with contextualization, goal formulation, and prompt articulation help users and designers manage ambiguity and transience in conversational AI.
-
Chasing Progress, Not Perfection: Revisiting Strategies for End-to-End LLM Plan Generation
Fine-tuned LLMs fail out-of-distribution planning tests; chain-of-thought and related strategies improve executability but not validity, while a similarity-based RL reward gives modest gains.
-
Recursive Decomposition of Logical Thoughts: Framework for Superior Reasoning and Knowledge Propagation in Large Language Models
A prompting framework that recursively decomposes reasoning tasks and self-scores candidate thoughts is reported to improve LLM accuracy on math and letter-concatenation benchmarks, though the headline improvement is ...
-
CoreThink: A Symbolic Reasoning Layer to reason over Long Horizon Tasks with LLMs
The paper claims a symbolic orchestration layer, CoreThink, achieves state-of-the-art results on seven coding and reasoning benchmarks with no training, but provides no verifiable implementation or method details.
-
Large Language Models for Planning: A Comprehensive and Systematic Survey
A structured survey of LLM planning methods, benchmarks, and interpretability work, organized around a three-way taxonomy.
Discussion (0). Continue with ORCID to comment.