{"id":"96e85338-92c0-4883-89a4-fb1db6224a9d","arxiv_id":"2508.10428","paper_version":1,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The authors propose SC2Arena, a full-coverage StarCraft II benchmark for LLMs, and StarEvolve, a planner-executor-verifier self-improvement framework, claiming superior strategic planning.","lead":"SC2Arena is a new evaluation environment for testing large language models on the full StarCraft II game, and StarEvolve is a companion training framework that breaks play into planning and execution steps and improves itself by fine-tuning on its own best games. The paper claims StarEvolve plans better than previous approaches, but the full manuscript text supplied with this report belongs to a different paper, so the claim could not be checked.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SC2Arena's core claim rests on text observations preserving full game state; without evidence of information retention for spatial layout, action legality, and simultaneity, both benchmark completeness and StarEvolve's gains are unsubstantiated.","rationale":"The reader's verdict is CONDITIONAL, based on the missing manuscript body and the abstract's unsupported superiority claim. My stress-test identifies the most load-bearing technical assumption underlying those claims: the sufficiency of the text-based observation representation. If that representation is lossy, the benchmark cannot deliver 'full complexity' and StarEvolve's measured improvements do not reflect strategic planning in the actual game. This aligns exactly with the reader's weakest_assumption, so I agree. The concrete test—checking uniqueness and action legality across real replays—would empirically settle whether the representation preserves the information needed for low-level actions and spatial reasoning. The document mismatch (supplied full text belonging to MM-Food-100K) prevents in-scope verification of the encoding, but that is a data-availability issue rather than an internal inconsistency. I do not see a need to move the verdict: CONDITIONAL remains appropriate until the encoding is demonstrated sufficient or the actual paper provides the missing details.","tokens_in":4968,"tokens_out":4095,"duration_ms":44517,"concrete_test":"Using a sample of 1,000 diverse mid-game states from human StarCraft II replays, compute the SC2Arena text observation for each and apply a state-reconstruction oracle (e.g., a human expert or a scripted parser) to determine whether the text uniquely identifies the game state. Count the number of distinct states that map to the same text (collision rate) and the fraction of text observations from which a legal low-level action cannot be unambiguously inferred. If the collision rate exceeds 0% or any action ambiguity exists, the representation is lossy and the benchmark does not 'fully support' the game; if all states are uniquely reconstructed with complete legal action sets, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—SC2Arena 'fully supports all playable races, low-level action spaces' while using 'optimized text-based observations'—is only as strong as the assumption that the text serialization faithfully preserves the information needed for real-time strategic decisions. StarCraft II has a continuous spatial map, simultaneous unit actions, fog of war, and timing constraints. If the text encoding compresses or discretizes these, it introduces an artificial bottleneck: (1) legal low-level actions may be under-specified (e.g., selecting a screen coordinate for a move/ability); (2) spatial relations (e.g., distance, line-of-sight, chokepoints) may be approximated in ways that change strategic options; (3) simultaneity/real-time pressure may be serialized into turn-based text, removing the real-time adaptation aspect that defines SC2. The abstract explicitly acknowledges 'spatial reasoning challenges' from text observations, making this the load-bearing point. Because the supplied body text is an unrelated paper (MM-Food-100K), there is no in-scope description of the encoding scheme, no comparison of state equivalence, and no evidence that the measured 'superior performance' of StarEvolve generalizes beyond the text-limited environments. Without a demonstration that the text representation is informationally equivalent (or at least provably sufficient) for the full action space, both the benchmark's completeness claim and the meaning of StarEvolve's improvements are uncertain.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The submitted manuscript, as received, consists of an abstract for arXiv:2508.10428 titled \"SC2Arena and StarEvolve: Benchmark and Self-Improvement Framework for LLMs in Complex Decision-Making Tasks,\" followed by a full-text file whose title, content, and contributions are entirely different: \"MM-Food-100K: A 100,000-Sample Multimodal Food Intelligence Dataset with Verifiable Provenance.\" The abstract claims that SC2Arena provides a StarCraft II benchmark with full support for all playable races, low-level action spaces, and optimized text-based observations for spatial reasoning, and that StarEvolve achieves superior strategic-planning performance through a Planner-Executor-Verifier loop and fine-tuning on internally selected high-quality gameplay data. No methods, environment descriptions, experiments, tables, baselines, or code-level details for SC2Arena or StarEvolve are present in the supplied full text.","tokens_in":5160,"tokens_out":2121,"duration_ms":26150,"significance":"If the abstract's claims were fully realized, the work would be significant: SC2Arena would be the first StarCraft II LLM benchmark to cover the entire game context across all races and low-level action spaces, and StarEvolve would demonstrate a practical self-improvement loop for LLM strategic play. The stated release of code, environment, and algorithms would also be valuable for reproducibility. However, because the manuscript body does not contain the claimed work, these potential contributions cannot be evaluated. The significance assessment is therefore conditional on the existence of a correct manuscript, which was not provided for review.","major_comments":[{"comment":"The full-text file is not the SC2Arena/StarEvolve paper. It is a completely unrelated manuscript on a food dataset (MM-Food-100K), with no mention of StarCraft II, SC2Arena, StarEvolve, the Planner-Executor-Verifier architecture, or any of the claimed experiments. Consequently, every technical claim in the abstract—benchmark design, state representation, training procedure, and experimental results—is unverifiable. This is not a presentation issue; it is the absence of the object under review.","section":"Full text"},{"comment":"The claim that SC2Arena \"fully supports all playable races, low-level action spaces, and optimizes text-based observations\" is load-bearing but entirely unsupported. In particular, the abstract gives no evidence that the text-based serialization preserves the game's spatial layout, action legality, fog of war, or simultaneity of low-level actions. If the encoding discretizes or approximates the continuous SC2 state, the benchmark's completeness claims and any measured performance gains could reflect encoding artifacts rather than decision-making ability. Because the manuscript body is missing, there is no description of the encoding scheme, no comparison of state equivalence, and no analysis of information loss.","section":"Abstract"},{"comment":"StarEvolve's claimed \"continuous improvement via fine-tuning on high-quality gameplay data\" relies on an internal scoring system to select its own rollouts. This is structurally similar to expert iteration, which is not inherently circular, but without details on the scoring function, threshold selection, data diversity, and safeguards against self-generated-data collapse, the claim of continuous improvement is not supported. The abstract reports no comparison to baselines such as random rollouts, behavior cloning from human data, or a non-improving control, so superior strategic planning is asserted rather than demonstrated.","section":"Abstract"},{"comment":"The abstract states \"Comprehensive analysis using SC2Arena provides valuable insights\" and \"experimental results also demonstrate that our proposed StarEvolve achieves superior performance,\" but no quantitative evidence is given: no numbers, no baselines, no metrics, no confidence intervals, and no ablation. A formal referee report cannot assess the strength of a claim without the associated results, which are absent from the submitted full text.","section":"Abstract"}],"minor_comments":[{"comment":"The rendered full text contains a broken symbol in Table 4 (\"R�\" instead of R²), but this is minor compared with the substantive mismatch.","section":"Full text"},{"comment":"The paper's title and abstract do not match its full text. Even setting aside the content mismatch, the manuscript needs to be reassembled and resubmitted.","section":"Title/Abstract"}],"recommendation":"reject","confidential_remarks":"This submission appears to contain the wrong full-text file: the abstract and the body are different papers. I recommend desk rejection and a resubmission with the correct SC2Arena/StarEvolve manuscript. Given the state of the submission, a technical review on the merits is not possible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the honest state of play on arXiv:2508.10428. The abstract describes exactly the kind of benchmark the LLM-agent community lacks: SC2Arena covering all playable races, low-level action spaces, and text observations designed for spatial reasoning, plus StarEvolve, a Planner-Executor-Verifier loop that fine-tunes on self-scored gameplay data. If the body delivers on even half of that, it's a real contribution.\n\nBut I have to flag a serious data problem before anything else: the full text I was sent is not this paper. It's MM-Food-100K, a food dataset paper with a different arXiv ID. So my read is based on the abstract alone. That means every load-bearing claim—no numbers, no baselines, no descriptions of the encoding scheme—is an assertion.\n\nWhat earns credit: the differentiators are concrete and checkable. 'All playable races,' 'low-level action spaces,' and 'optimized text-based observations' are specific enough that a reviewer could verify them against the code and environment, which the authors say they're releasing. That's a good sign. StarEvolve's structure is also named concretely: Planner-Executor-Verifier, internal scoring, fine-tuning on selected samples. This isn't a hand-wave.\n\nThe soft spots are exactly where you'd expect them. First, the superiority claim is unsupported even at the abstract level. 'Achieves superior performance in strategic planning' appears without a single metric, baseline, or comparison. That could be fine if the body backs it up, but we can't see that.\n\nSecond, the stress-test concern is legitimate: SC2Arena's completeness claim depends on text serialization preserving the full game state—spatial layout, action legality, simultaneity, fog of war. If the encoding discretizes or approximates these, 'full support' means something weaker than advertised, and StarEvolve's gains might be artifacts of the bottleneck. I can't tell from the abstract whether the paper addresses this, and the supplied text doesn't help.\n\nThird, StarEvolve curates its own training data via its own scoring system and evaluates in its own benchmark. That's not circular by construction—expert iteration does the same thing—but whether the quality proxy is sound and whether the benchmark gives fair comparisons are things only the body can settle.\n\nBottom line: this deserves a serious referee if the actual manuscript exists and is complete. The claims are significant, specific, and potentially reproducible. But it should not be accepted on the abstract, and I'd want the reviewers to push hard on the information-equivalence question and on the baselines. If I were you, I'd skip citing it until the body is out and checkable.","headline":"The abstract promises a real missing piece—full-race, low-level StarCraft II for LLM agents—but we only have the abstract; the supplied full text is a different paper, so nothing is auditable yet.","tokens_in":5789,"tokens_out":2691,"would_cite":false,"duration_ms":26547,"reading_group":"maybe","serious_thinker":"unclear","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SC2Arena presents a StarCraft II benchmark covering every playable race and low-level action, and StarEvolve improves LLM strategic planning through a Planner-Executor-Verifier loop.","keywords":["StarCraft II","LLM benchmark","strategic planning","low-level action space","self-improvement","Planner-Executor-Verifier","text-based observations","real-time decision-making"],"falsifier":"Run StarEvolve on SC2Arena using a deliberately lossy text serialization (e.g., dropping unit coordinates or omitting one of the three races) and compare strategy-level win rates; if performance degrades sharply or saturates regardless of serialization detail, the benchmark's completeness claim and the framework's measured gains would be artifacts of the encoding rather than evidence about decision-making capability.","tokens_in":4754,"feed_emoji":"🎮","tokens_out":2032,"duration_ms":24057,"temperature":0.7,"pith_summary":"This paper claims that existing LLM benchmarks for StarCraft II miss the game's full complexity—all three races, low-level action spaces, and a complete game context—so they cannot measure whether LLMs genuinely plan strategically or just react to simplified scenarios. To fix that, it introduces SC2Arena, a benchmark that supports every playable race, uses low-level actions, and converts spatial game states into optimized text-based observations to make spatial reasoning testable. Alongside the benchmark, the paper proposes StarEvolve, a hierarchical framework that combines strategic planning, tactical execution, iterative self-correction, and fine-tuning on internally selected high-quality gameplay data. The paper's key claim is that StarEvolve achieves superior strategic planning performance in SC2Arena, and that the benchmark reveals insights about generalist agents that earlier benchmarks could not.","feed_headline":"Benchmark spans all StarCraft II races and actions","feed_subtitle":"SC2Arena tests LLM strategy with full game context; StarEvolve self-improves via Planner-Executor-Verifier.","key_machinery":"The load-bearing identity is the Planner-Executor-Verifier (PEV) loop inside StarEvolve, coupled with a scoring mechanism that filters gameplay data for fine-tuning. The Planner converts high-level goals into tactical subgoals; the Executor maps those subgoals to low-level actions in SC2Arena; the Verifier evaluates the executed state and triggers iterative self-correction. The scoring system ranks self-play or sampled gameplay by quality, and only high-scoring traces are used to fine-tune the LLM, closing the self-improvement loop. SC2Arena itself is the measurement instrument: its text-based observation optimization is what makes spatial reasoning representable to an LLM while preserving t","core_discovery":"The central discovery is the pair of a benchmark and a training method. SC2Arena is claimed to be the first StarCraft II benchmark that fully supports all playable races, low-level action spaces, and optimized text-based observations that preserve enough of the spatial layout to pose spatial reasoning challenges. StarEvolve is a hierarchical framework with a Planner-Executor-Verifier structure: the Planner decomposes the game into strategic steps, the Executor carries out low-level actions, and the Verifier checks results and feeds back corrections; a scoring system then selects high-quality gameplay traces for fine-tuning, enabling continuous self-improvement. The paper asserts that, evalua","pith_inferences":["The paper implicitly assumes that optimizing text-based observations does not lose the spatial and temporal information essential to strategic decision-making; a separate analysis could test how sensitive StarEvolve's measured gains are to the serialization order or granularity of the map text.","If the claim holds, a natural extension is to port the PEV framework to other real-time strategy environments or to partially observable decision tasks, suggesting that the mechanism is not StarCraft-specific but a general architecture for LLM agents.","The published abstract does not report absolute win rates or baselines against strong non-LLM agents, so the 'superior performance' claim invites a stricter comparison to scripted StarCraft II bots or RL agents in the same low-level action space.","The mismatch between the paper's abstract and its body (which describes a food dataset, not StarCraft II) suggests the reader should verify the full text before treating the abstract's experimental results as supported."],"forward_implications":["If SC2Arena indeed preserves full game context, LLM strategic planning can be measured on the complete StarCraft II game rather than on simplified subsets, changing how generalist agents are evaluated for real-time decision-making.","StarEvolve's self-improvement loop, if effective, would show that LLMs can improve their strategic play in a complex environment without human expert demonstrations, relying only on self-generated and self-scored gameplay.","The benchmark's support for all playable races and low-level action spaces could make StarCraft II a standard testbed for comparing LLM-based agents with traditional reinforcement learning agents on tasks requiring spatial reasoning and long-horizon planning.","The text-based observation design, if successful, suggests that spatial reasoning in LLMs can be evaluated without image encoders, potentially simplifying other real-time strategy benchmarks.","The scoring-system-directed fine-tuning, if sound, would provide a template for continuous improvement in other interactive decision-making domains where high-quality data is scarce."],"supporting_citations":[],"fun_headline_variants":["SC2Arena covers every StarCraft II race and action","StarEvolve: self-improving LLM for StarCraft II strategy","Planner-Executor-Verifier framework enables LLM learning in StarCraft","SC2Arena + StarEvolve: full-game benchmark and self-training","Self-improving LLM strategy in StarCraft via SC2Arena and StarEvolve"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The central premise is that converting the full StarCraft II game state into text-based observations preserves the complete game context—spatial layout, simultaneous low-level actions, and timing—so that what the benchmark measures is genuine strategic decision-making and not artifacts of lossy serialization.","fun_headline_variants_meta":{"raw":{"variants":["SC2Arena covers every StarCraft II race and action","StarEvolve: self-improving LLM for StarCraft II strategy","Planner-Executor-Verifier framework enables LLM learning in StarCraft","SC2Arena + StarEvolve: full-game benchmark and self-training","Self-improving LLM strategy in StarCraft via SC2Arena and StarEvolve"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001807,"raw_usage":{"total_tokens":6945,"prompt_tokens":730,"completion_tokens":6215,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":474,"completion_tokens_details":{"reasoning_tokens":6109}},"tokens_in":474,"tokens_out":6215,"duration_ms":43760,"temperature":1.0,"reasoning_tokens":6109,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T20:27:13.511174+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run StarEvolve on SC2Arena using a deliberately lossy text serialization (e.g., dropping unit coordinates or omitting one of the three races) and compare strategy-level win rates; if performance degrades sharply or saturates regardless of serialization detail, the benchmark's completeness claim and the framework's measured gains would be artifacts of the encoding rather than evidence about decision-making capability.","supporting_citations":[],"review_version":1}