{"id":"c8a3c03e-8d3a-44c4-ad09-e6f7221d0440","arxiv_id":"2506.23689","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A text-based LLM battle agent reaches an 80.8% win rate on Pokémon Red wild battles, close to a single human run of 86%, with different models showing distinct playstyles.","lead":"PokéAI is a text-only AI system that plays the battle part of Pokémon Red, winning 80.8% of wild battles in one test level versus an experienced human's 86%. The paper also reports a link between a model's language scores and battle performance, but the full multi-agent game-playing system is not yet built.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's 'win rate' appears to count fleeing as a win, which would undermine the headline comparison to the human's defeat-only 86%.","rationale":"The reader's weakest_assumption focused on lack of human replication. While real, that is a statistical uncertainty; the metric inconsistency is a construct validity problem that could invalidate the comparison even with perfect replication. I therefore partially agree with the reader. The paper has strengths: ten AI runs, a clear ablation, and candid discussion of component contributions. However, the central claim's meaning depends on a definition of 'win' that is never stated. Checking the outcome logs is cheap and decisive. The verdict should remain CONDITIONAL because the fix is to define the metric transparently and rerun the comparison; the underlying system and ablations may still be valid. If the metric check shows escape-inflation, the specific 'only 6% lower' claim should be removed or corrected. The reader's conditional verdict already requires revision, so I leave it unchanged.","tokens_in":136,"tokens_out":4303,"duration_ms":56870,"concrete_test":"Instrument the battle module or re-analyze the existing gameplay logs to classify each of the 50 encounters per run as 'opponent fainted', 'player fled', or 'player lost'. Recompute the win rate with only 'opponent fainted' counted as a win, then compare that defeat-only rate to the human's 43/50. If the defeat-only rate is materially below 80.8%, the headline comparison is inflated by escapes and must be revised; if all 40.4 wins were by knockout, the concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"In Section III-A, the battle agent's 80.8% win rate is compared to an experienced human player who 'managed to defeat 43 wild Pokémon' (86%). The ablation in Section III-B treats escape as a component of the full agent, and the authors report that the full model sometimes 'chose to flee unnecessarily'; such escapes would presumably be counted as wins under the paper's definition, whereas the human's 43 are explicitly defeats. If the AI's 40.4 average includes successful escapes, the two numbers measure different things, and the 'only 6% lower' claim is unsupported. The paper never defines 'win' and does not report whether escape outcomes were logged separately. This is the most load-bearing issue because the central headline result rests entirely on this comparison; the ablation and LLM rankings also depend on the same ambiguous metric.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PokéAI, a three-agent (Planning, Execution, Critique) LLM system for playing Pokémon Red, implemented as a text-only agent loop with vector memory and function calling. The present manuscript focuses on a preliminary battle module inside the Execution Agent. The module reads memory addresses (e.g., 0xD057) to detect battles, sends state to an LLM, parses a JSON action, and loops until battle end. Using a Mt. Moon checkpoint (Level 15 Charmander and Pidgey, five potions), the authors report an 80.8% average win rate across 10 repetitions of 50 wild encounters with DeepSeek-V3, compare this with an experienced human player who defeated 43 of 50 encounters (86%), run ablations over switching, item use, and escape, compare several LLM backends, analyze action distributions by model, and pilot a long-term-memory mechanism with Letta.","tokens_in":5434,"tokens_out":5529,"duration_ms":60074,"significance":"The paper's value is as an existence proof: a purely text-based LLM loop can control a nontrivial game subtask, and the public code release supports reproducibility. The battle-module design is simple and the ablation idea is sensible. The play-style observations (e.g., Claude 3.5's single early switch, GPT-4o's aggressive play) are concrete and falsifiable in principle. The quantitative headline, however, is not yet established: 'win' is undefined, escape may be counted as a win, the human baseline is one run, and the claimed LLM Arena correlation has no reported statistic. These issues are fixable with additional logging, re-analysis, and replication, so the work is better treated as a promising preliminary report than a settled result.","major_comments":[{"comment":"The central metric is ambiguous: 'win' is never defined, and the human comparison is explicitly 'managed to defeat 43 wild Pokémon' (86%), while the agent's 'wins an average of 40.4 out of 50 battles' may include successful escapes. Section III-B strengthens this concern by listing escape as a component and noting that the full model 'chose to flee unnecessarily'; if those escapes are counted as wins, the AI's 80.8% and the human's 86% are not measuring the same outcome. Please report the per-run counts of opponent KO, escape, own faint, and early run termination, and recompute the win rate under a defeat-only definition, or justify a definition of 'win' that treats fleeing as success.","section":"Section III-A and III-B"},{"comment":"The human baseline consists of one run by one player (43 wins out of 50). The abstract's claim that the battle AI is 'only 6% lower' than human performance rests on a 5.2 percentage point difference between a 10-run mean and a single observation, with no human error bar and no statement of the AI run-to-run spread in the text (error bars appear in figures but no numeric CI). Please report the AI distribution, conduct additional human runs or at least provide a confidence interval, and clarify whether the human played exactly 50 encounters and how the denominator of 50 was obtained.","section":"Section III-A"},{"comment":"The abstract states that battle performance 'correlates strongly with its LLM Arena score', but Section III-C only asserts proportionality 'with the exception of Claude 3.5 Sonnet' and reports no correlation coefficient, significance test, or point count. Given the small number of models and acknowledged outlier, the strong correlation claim is unsupported. Please report the statistic (e.g., Spearman rho with confidence interval) or downgrade the claim to a qualitative observation.","section":"Section III-C / Abstract"}],"minor_comments":[{"comment":"The text says 'As shown in Fig.4' for the ablation study, but the ablation figure is captioned as Fig. 3; figure cross-references should be corrected.","section":"Section III-B"},{"comment":"The 'LLM Arena Score' is never defined or cited; please state which leaderboard and snapshot is used.","section":"Section III-C"},{"comment":"The long-term-memory pilot consists of a single injected-memory scenario (Level 6 Squirtle vs Level 9 Pikachu) in which the agent flees; the sentence 'improve strategic decision-making and overall battle performance' overstates what one anecdote can show and should be labeled as preliminary.","section":"Section III-E"},{"comment":"The citation [13] for GPT-4o's 'creative rule-bending' is to an instruction-following paper; reference [14] on in-context scheming is more directly relevant and should be cited here.","section":"Section III-D"},{"comment":"The claim of being 'the first text-based ... open-source agent framework' for Pokémon Red is not supported by a literature comparison beyond the multimodal examples given; please either substantiate the 'first' claim or weaken it, and clarify the repository's license.","section":"Abstract / Section I"}],"recommendation":"major_revision","confidential_remarks":"The paper is a concise preliminary report that fits the cs.AI scope. The headline result is the only load-bearing quantitative claim, and it is currently under-specified; the revision needs outcome-level logging and a statistical treatment of the human comparison. I saw no signs of data fabrication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nBottom line up front: the paper has an honest, measurable battle module with real experiments, but the headline '80.8% win rate vs human 86%' is on shaky ground because 'win' is never defined. If successful escapes count as wins, the two numbers aren't measuring the same thing—the human played to defeat, not flee.\n\nWhat's actually new: a text-only LLM battle module for Pokémon Red that reads memory addresses and outputs JSON actions. Several prior works use vision-language models on the same game; this is the first text-only one I've seen with results. The experimental setup is concrete: a fixed checkpoint, 10 runs of 50 encounters, encounter rates, and standard error bars on the ablation and LLM comparisons. The playstyle analysis (GPT-4o aggressive, DeepSeek-V3 resource-preserving, Claude switching once) is qualitative but plausible and could be useful to others building such agents. Code is promised on GitHub.\n\nThe soft spots, in order of severity. First, the win metric. The abstract says 'wins' but the human comparison is 'defeated 43 wild Pokémon'. The ablation includes escape as a component, and the text says the full model sometimes 'chose to flee unnecessarily'. That means a successful flee likely counts as a 'win' in the AI's tally. Without a clear definition and separate logs for escapes, the 'only 6% lower than an experienced human' claim is unsubstantiated. This is the load-bearing result, so it needs fixing before I'd trust the number. Second, the human baseline is a single run—no replication, no error bar. Third, the correlation with LLM Arena scores is read off a bar chart without coefficient or significance. Fourth, the abstract and introduction claim 'the first fully text-based, multi-agent LLM framework', but only the battle module is implemented; the Planning and Critique agents are future work. The body is honest about this, but the framing overreaches.\n\nNone of these are fatal—they're fixable in a revision. The central observation, that LLM battle performance roughly tracks language-task ability, is interesting but not yet a strong claim.\n\nThis is a workshop-to-short-paper level contribution. I'd bring it to a reading group for the metric blunder and the LLM playstyle discussion, but I wouldn't cite it yet. It deserves a serious referee—there's enough experimental substance—but only after the authors define 'win', log escape outcomes, and scale back the 'first' claim.","headline":"Useful empirical battle module, but the undefined win metric makes the human comparison unreliable, and the 'first multi-agent framework' framing outruns what is actually built.","tokens_in":5990,"tokens_out":3600,"would_cite":false,"duration_ms":37545,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fully text-based multi-agent LLM system plays Pokemon Red battles at an 80.8% win rate, within 5.2 points of an experienced human.","keywords":["game AI","large language models","multi-agent systems","Pokemon Red","battle strategy","text-based gameplay","LLM playstyles","long-term memory"],"falsifier":"Run the same 50-battle Mt. Moon checkpoint with at least ten human players and compute the distribution of their win rates; if the human average falls at or below the agent's 80.8%, the near-human claim collapses.","tokens_in":5111,"feed_emoji":"⚔️","tokens_out":7998,"duration_ms":80340,"temperature":0.7,"pith_summary":"The paper introduces PokeAI, a three-agent system—Planning, Execution, and Critique—that plays Pokemon Red entirely through text. As a first self-contained component, it built and tested a battle module: reading game memory, prompting a large language model, and executing its JSON moves. In 50 wild encounters at Mt. Moon, the module won 80.8% of battles, compared with 86% for one experienced human player. The paper also reports that battle win rates track general language-benchmark scores and that different LLMs show distinct battle styles. A pilot test suggests injected long-term memories can steer later decisions, such as fleeing a losing matchup.","feed_headline":"Text-only LLM wins 80.8% of Pokemon Red battles","feed_subtitle":"A three-agent loop plans, acts, and verifies, landing within 5.2 points of a human player.","key_machinery":"The central object is the closed-loop battle module inside the Execution Agent: it monitors memory address 0xD057, reads battle information, sends it to the LLM, receives a JSON action, and performs that action until the battle ends. This sits inside a three-agent system—Planning decomposes goals into tasks, Execution invokes a toolkit through function calling, and Critique verifies task completion—with each agent owning a vector memory bank. The ablation method (disabling switching, escape, or items) is what isolates which capabilities carry the win rate.","core_discovery":"The paper claims that a fully text-based multi-agent LLM architecture can autonomously play Pokemon Red battles at near-human level. The battle loop—detect battle via memory address 0xD057, read state, ask the LLM, act on its response—achieved an average 80.8% win rate over 50 Mt. Moon wild encounters with a Level 15 Charmander and Pidgey and five potions. The paper claims this is only 5.2 percentage points below an experienced human player's 86% in the same scenario. It further claims that performance correlates with a model's score on a general public language ranking, and that action logs show each model develops a recognizable playstyle.","pith_inferences":["Beyond the paper, the single-human baseline means the 5.2-point gap is not a stable measurement; a multi-player study would be needed to know whether the agent truly trails humans.","Beyond the paper, the memory pilot tests only one injected episode, so the promise of long-term memory for full-game planning remains untested; a natural next experiment is the whole 50-battle suite with memory on and off.","Beyond the paper, the overthinking observation suggests reasoning-heavy backends may hurt low-level game control; this could be tested by running deep-reasoning and fast-response models on the same battles."],"forward_implications":["The battle module is a self-contained component that can be integrated into the full Planning–Execution–Critique loop without further work.","Removing item use drops the win rate to 32.6%, so healing items carry most of the agent's survival capability.","Removing strategic switching drops the win rate to 58.8%, showing that choosing the right Pokemon is a major part of the agent's edge.","Battle win rates rise with a model's general language ranking, so language ability and strategic play move together in this setting.","Different LLMs produce distinct action distributions, which the paper points toward as a basis for varied, human-like NPC behavior."],"supporting_citations":[{"why":"It is a prior vision-language playthrough that the paper contrasts with its text-only design.","marker":"[4]"},{"why":"It is another vision-language playthrough used as a baseline for the text-only novelty claim.","marker":"[5]"},{"why":"It is a third prior multimodal Pokemon player used to situate the framework as the first text-based one.","marker":"[6]"},{"why":"It supplies the function-calling mechanism the Execution Agent uses to invoke navigation and battle tools.","marker":"[7]"},{"why":"It provides the Mt. Moon wild-encounter rates that define the 50-battle evaluation scenario.","marker":"[8]"},{"why":"It underlies the pilot study in which an injected past memory changes a later battle decision to flee.","marker":"[15]"}],"fun_headline_variants":["Text-only AI wins 80.8% of Pokemon Red battles","Pokemon Red AI: 80.8% win rate, 5 points from human","Multi-agent LLM nears human skill in Pokemon Red","Battle IQ in Pokemon AI mirrors language ability","Text-only agents win 80.8% of Pokemon Red fights"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that one experienced player's 86% win rate fairly represents human performance in this scenario; without replication, the 'only 5.2 points lower' conclusion has no stable yardstick.","fun_headline_variants_meta":{"raw":{"variants":["Text-only AI wins 80.8% of Pokemon Red battles","Pokemon Red AI: 80.8% win rate, 5 points from human","Multi-agent LLM nears human skill in Pokemon Red","Battle IQ in Pokemon AI mirrors language ability","Text-only agents win 80.8% of Pokemon Red fights"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000959,"raw_usage":{"total_tokens":4073,"prompt_tokens":922,"completion_tokens":3151,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":3062}},"tokens_in":538,"tokens_out":3151,"duration_ms":26478,"temperature":1.0,"reasoning_tokens":3062,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:34:00.723738+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 50-battle Mt. Moon checkpoint with at least ten human players and compute the distribution of their win rates; if the human average falls at or below the agent's 80.8%, the near-human claim collapses.","supporting_citations":[{"cited_title":"Gemini plays pokemon","cited_arxiv_id":null,"evidence_quote":"It is a prior vision-language playthrough that the paper contrasts with its text-only design."},{"cited_title":"Claudeplayspokemon","cited_arxiv_id":null,"evidence_quote":"It is another vision-language playthrough used as a baseline for the text-only novelty claim."},{"cited_title":"Beating the world record in pok ´emon emerald","cited_arxiv_id":null,"evidence_quote":"It is a third prior multimodal Pokemon player used to situate the framework as the first text-based one."},{"cited_title":"Function calling in large language models: Industrial practices, challenges, and future directions","cited_arxiv_id":null,"evidence_quote":"It supplies the function-calling mechanism the Execution Agent uses to invoke navigation and battle tools."},{"cited_title":"Altissimo’s pok ´emon site","cited_arxiv_id":null,"evidence_quote":"It provides the Mt. Moon wild-encounter rates that define the 50-battle evaluation scenario."}],"review_version":1}