REVIEW 8 cited by
On the Brittle Foundations of ReAct Prompting for Agentic Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
On the Brittle Foundations of ReAct Prompting for Agentic Large Language Models
read the original abstract
The reasoning abilities of Large Language Models (LLMs) remain a topic of debate. Some methods such as ReAct-based prompting, have gained popularity for claiming to enhance sequential decision-making abilities of agentic LLMs. However, it is unclear what is the source of improvement in LLM reasoning with ReAct based prompting. In this paper we examine these claims of ReAct based prompting in improving agentic LLMs for sequential decision-making. By introducing systematic variations to the input prompt we perform a sensitivity analysis along the claims of ReAct and find that the performance is minimally influenced by the "interleaving reasoning trace with action execution" or the content of the generated reasoning traces in ReAct, contrary to original claims and common usage. Instead, the performance of LLMs is driven by the similarity between input example tasks and queries, implicitly forcing the prompt designer to provide instance-specific examples which significantly increases the cognitive burden on the human. Our investigation shows that the perceived reasoning abilities of LLMs stem from the exemplar-query similarity and approximate retrieval rather than any inherent reasoning abilities.
Forward citations
Cited by 8 Pith papers
-
NeSyFS: A Neuro-symbolic Fast-Slow Thinking Framework for LLM Agent under Partial Observability
NeSyFS combines a knowledge-graph belief state, a reflection step, and TSMC-style particle planning to improve LLM agent success rates on ALFWorld, WebShop, and ScienceWorld.
-
Consistent but Miscalibrated: Evaluating LLM Limitations for Risk Communication in Natural Language
Current LLMs produce consistent but miscalibrated natural-language descriptors of likelihood and uncertainty from probabilistic predictions and are not yet reliable zero-shot risk communicators.
-
Consistent but Miscalibrated: Evaluating LLM Limitations for Risk Communication in Natural Language
Current LLMs are consistent but miscalibrated when selecting verbal descriptors for likelihood and uncertainty of probabilistic predictions, with the bottleneck in verbalization itself.
-
Where Do CoT Training Gains Land in LLM based Agents?
CoT training in LLM agents improves prompt-action quality more than the advantage of generated reasoning, and selectively masking action supervision improves out-of-domain generalization.
-
RIMRULE: Improving Tool-Using Language Agents via MDL-Guided Rule Learning
RIMRULE distills LLM tool-use failures into MDL-compressed symbolic rules that, when injected at inference, improve tool-calling accuracy and transfer across models.
-
NeSyFS: A Neuro-symbolic Fast-Slow Thinking Framework for LLM Agent under Partial Observability
Combining a knowledge-graph belief state, KG-augmented reflection, and TSMC-style particle planning lifts LLM agent success rates by roughly 30-95% over ReAct/Reflexion baselines on three benchmarks.
-
Novelty-based Tree-of-Thought Search for LLM Reasoning and Planning
Novelty estimation via LLM prompts enables pruning in Tree-of-Thought search, reducing overall token usage on language planning benchmarks.
-
CogniConsole: Externalizing Inference-Time Control as a Formal Abstraction for Reliable LLM Interactions
Under a fixed model, increasing structural scaffolding of inference-time control systematically reduces LLM output variance and failure rates on controllability probes.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.