REVIEW 7 cited by
Put Your Money Where Your Mouth Is: Evaluating Strategic Planning and Execution of LLM Agents in an Auction Arena
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Recent advancements in Large Language Models (LLMs) showcase advanced reasoning, yet NLP evaluations often depend on static benchmarks. Evaluating this necessitates environments that test strategic reasoning in dynamic, competitive scenarios requiring long-term planning. We introduce AucArena, a novel evaluation suite that simulates auctions, a setting chosen for being highly unpredictable and involving many skills related to resource and risk management, while also being easy to evaluate. We conduct controlled experiments using state-of-the-art LLMs to power bidding agents to benchmark their planning and execution skills. Our research demonstrates that LLMs, such as GPT-4, possess key skills for auction participation, such as budget management and goal adherence, which improve with adaptive strategies. This highlights LLMs' potential in modeling complex social interactions in competitive contexts. However, variability in LLM performance and occasional outperformance by simpler methods indicate opportunities for further advancements in LLM design and the value of our simulation environment for ongoing testing and refinement.
Forward citations
Cited by 7 Pith papers
-
Can LLM Agents Price Competitively? A Dynamic Multi-Attribute Auction Benchmark for Agentic Commerce
LLM merchant agents in a new dynamic auction benchmark capture at most 32% of hindsight-optimal profit; profit tracks margin per win more than win rate, and fast pre-shock learners adapt poorly to preference shocks.
-
Chain-of-Models: Cross-Model Auditing for Bias-Robust LLM Judges
Per-bias selection of a cross-family LLM auditor lifts biased-judgment accuracy from 0.805/0.824 baselines to 0.884.
-
Simulating Human-Like Learning Dynamics with LLM-Empowered Agents
An LLM multi-agent simulation of a year-long classroom shows that a persona-free base LLM defaults to a 'diligent but brittle surface learner' profile, industrious but lacking deep understanding.
-
Position Auctions in AI-Generated Content
New mechanism-design results for position auctions with context-dependent click-through rates under multinomial logit and cascade user models, with exact optimality in the MNL case and an O(log m) approximation in the...
-
AI Agent Behavioral Science
AI agents should be studied as behavioral entities shaped by context and interaction, not only as trained models.
-
Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
ACBench tests compressed LLMs on agentic tasks and finds 4-bit quantization keeps tool use and workflow generation strong while hurting real-world application performance.
-
Large Language Models for Planning: A Comprehensive and Systematic Survey
A structured survey of LLM planning methods, benchmarks, and interpretability work, organized around a three-way taxonomy.
Discussion (0). Continue with ORCID to comment.