REVIEW 18 cited by
SmartPlay: A Benchmark for LLMs as Intelligent Agents
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Recent large language models (LLMs) have demonstrated great potential toward intelligent agents and next-gen automation, but there currently lacks a systematic benchmark for evaluating LLMs' abilities as agents. We introduce SmartPlay: both a challenging benchmark and a methodology for evaluating LLMs as agents. SmartPlay consists of 6 different games, including Rock-Paper-Scissors, Tower of Hanoi, Minecraft. Each game features a unique setting, providing up to 20 evaluation settings and infinite environment variations. Each game in SmartPlay uniquely challenges a subset of 9 important capabilities of an intelligent LLM agent, including reasoning with object dependencies, planning ahead, spatial reasoning, learning from history, and understanding randomness. The distinction between the set of capabilities each game test allows us to analyze each capability separately. SmartPlay serves not only as a rigorous testing ground for evaluating the overall performance of LLM agents but also as a road-map for identifying gaps in current methodologies. We release our benchmark at github.com/Microsoft/SmartPlay
Forward citations
Cited by 18 Pith papers
-
DungeonBench: A Benchmark for Rules-Rich Tactical Reasoning in Dungeons & Dragons Combat
DungeonBench scores LLM tactical play on D&D combat, finding frontier policies clear ~80% of single encounters but only 40% of linked multi-encounter days.
-
Mastering PokeGym: Graph-Guided Multimodal Evolution at Test Time
The manuscript body introduces PokeGym, a vision-only automated 3D-game benchmark, while the abstract claims a G-EvoMAC method and 60.18% success rate absent from the body.
-
LLM-WikiRace Benchmark: How Far Can LLMs Plan over Real-World Knowledge Graphs?
Frontier LLMs exceed human performance on easy Wikipedia navigation tasks but finish fewer than 25% of hard games, with failures driven by looping and an inability to replan after mistakes.
-
WGSR-Bench: Wargame-based Game-theoretic Strategic Reasoning Benchmark for Large Language Models
A new wargame-based benchmark finds large language models score far below human experts on strategic reasoning across situation awareness, opponent modeling, and policy generation.
-
lmgame-Bench: How Good are LLMs at Playing Games?
lmgame-Bench turns six classic games into a scaffolded LLM evaluation suite, ranks 13 models, detects contamination, and reports RL transfer from Sokoban or Tetris to unseen games and planning tasks.
-
G1: Bootstrapping Perception and Reasoning Abilities of Vision-Language Model via Reinforcement Learning
A 7B vision-language model trained with reinforcement learning in a new four-game environment, VLM-Gym, outperforms larger proprietary models and improves its perception and reasoning abilities together.
-
Reasoning Capabilities of Large Language Models on Dynamic Tasks
Prompting strategies on dynamic game benchmarks do not produce emergent reasoning in open-source LLMs; they add variance and help smaller models only sometimes.
-
Disentangling Exploration of Large Language Models by Optimal Exploitation
Exploration by LLM agents can be measured separately from exploitation using an optimal exploitation oracle, and most models explore poorly, with exploration performance correlated to reasoning ability.
-
SPAgent: Adaptive Task Decomposition and Model Selection for General Video Generation and Editing
SPAgent is an MLLM-based coordinator that decomposes user instructions, plans execution routes, and selects among open-source video generation and editing models, outperforming single models in MOS.
-
BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games
A benchmark of six long-horizon game environments shows current LLMs and VLMs struggle on hard tasks and often do worse when given images.
-
Probing the Capacity of Language Model Agents to Operationalize Disparate Experiential Context Despite Distraction
On the OEDD benchmark, LLM agents select the worse of two actions below chance when the correct choice requires combining two earlier facts and ignoring a recent distractor in contexts over 1,615 tokens.
-
Spatial Reasoning in LLM Game Agents: Impact of Causal Context and Multi-Step Planning
On a new three-game spatial benchmark, larger Qwen3 models with thinking mode and multi-step planning achieve higher win rates, while small models struggle to localize and causal prompt hints give only marginal, model...
-
Large Language Model-Enhanced Multi-Armed Bandits
Using an LLM as a reward predictor inside Thompson sampling and regression-oracle bandits outperforms LLM direct arm selection in the tested tasks.
-
VideoCogQA: A Controllable Benchmark for Evaluating Cognitive Abilities in Video-Language Models
A new controllable synthetic-video benchmark shows that even state-of-the-art video-language models struggle with abstract and symbolic video cognition, with accuracy falling as task difficulty rises.
-
Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey
A survey that classifies AI agent evaluation benchmarks along environment and capability axes, and proposes five traits that distinguish agents from chatbots.
-
Humanizing LLMs: A Survey of Psychological Measurements with Tools, Datasets, and Human-Agent Applications
A survey of six dimensions of LLM psychological assessment concludes that results are strongly affected by test design and remain inconsistent across models and settings.
-
Generative to Agentic AI: Survey, Conceptualization, and Challenges
Agentic AI is characterized over Generative AI by iterative reasoning, environment interaction, memory, and tool use, with autonomy as the defining difference.
-
PLANET: A Collection of Benchmarks for Evaluating LLMs' Planning Capabilities
A survey that catalogs and categorizes existing planning benchmarks for LLM agents and lists open gaps, without introducing new benchmarks, data, or experimental results.
Discussion (0). Continue with ORCID to comment.