{"id":"e311c16d-99cd-49a7-a105-c50720b0656b","arxiv_id":"2605.29512","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Mindgames introduces a four-game evaluation platform for multi-agent LLM reasoning, runs a 944-agent competition, surfaces rule-adherence and error-survival limitations, and releases a 29k-game dataset with an offline scoring protocol.","lead":"Mindgames creates a live multi-game arena with four environments to test LLM agents on sustained social and strategic reasoning like belief attribution and deception. A smart generalist should read it to understand current bottlenecks in multi-agent AI and access the released dataset of nearly 30,000 games for testing new agents.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Whether the four games plus TrueSkill + error-attribution lens supply a non-confounded measure of the claimed reasoning demands remains the load-bearing assumption.","rationale":"The reader's weakest_assumption directly matches the point where the argument is least secured; the abstract already surfaces the confound but provides no quantitative check that the lens neutralizes it. No other internal inconsistency appears in the provided abstract, and the paper's own acknowledgment of brittle rule adherence and environment-specific validity differences supports rather than undermines the need to verify the measurement claim.","tokens_in":1804,"tokens_out":358,"duration_ms":25853,"concrete_test":"Locate the exact definition and implementation of the error-attribution lens and TrueSkill adaptation in the methods; recompute the Secret Mafia leaderboard both with and without the correction on the released 29,571-game dataset; if rank order of top agents or correlation with rule-adherence metrics changes by more than 15%, the confound is not fully removed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that Colonel Blotto, IPD, Codenames and Secret Mafia together operationalize distinct ToM-relevant skills (belief attribution, opponent modeling, cooperative inference, sustained deception) and that the rating system isolates strategic ability from confounds. The abstract itself flags that failure-heavy environments reward error robustness, with Secret Mafia showing a pronounced error-survival confound. If the error-attribution procedure only post-hoc labels rather than prevents this (or if TrueSkill is applied without documented adjustments for variable player counts, team structure, or high-variance LLM play), then both the surfaced limitations and any agent rankings rest on an unverified separation of strategic reasoning from rule-following robustness and opponent-error tolerance.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript introduces Mindgames, a multi-game arena built on TextArena for evaluating social and strategic reasoning in multi-agent LLMs. It operationalizes theory-of-mind-relevant demands via four environments (Colonel Blotto, Iterated Prisoner's Dilemma, Codenames, Secret Mafia), reports results from a 2025 competition with 944 agents from 76 teams, identifies agent-level and evaluation-level limitations including brittle rule adherence and an error-survival confound, and releases a dataset of 29,571 games plus the MG-Ref deterministic offline tournament protocol.","tokens_in":1968,"tokens_out":488,"duration_ms":31283,"significance":"If the evaluation design is shown to isolate the targeted reasoning skills, the work supplies a useful open platform, large trajectory dataset, and reference protocol that could support reproducible progress on multi-agent LLM benchmarks beyond static vignettes. The explicit surfacing of evaluation confounds and the competition-scale data are concrete strengths for the field.","major_comments":[{"comment":"Abstract: the central claim that the four games plus TrueSkill rating and error-attribution lens supply a non-confounded measure of distinct ToM-relevant skills (belief attribution, opponent modeling, cooperative inference, sustained deception) is load-bearing, yet the abstract itself states that 'failure-heavy environments can reward robustness to opponent errors as much as strategic ability, with Secret Mafia exhibiting a pronounced error-survival confound'. It is unclear whether the rating system includes documented adjustments for variable player counts, team structure, or high-variance LLM play, or whether the error-attribution procedure prevents rather than post-hoc labels this confound.","section":"Abstract"},{"comment":"Abstract (analysis of leaderboard validity): the claim that top-performing systems rely on explicit structural scaffolding and that leaderboard validity differs sharply across environments rests on the separation of strategic ability from rule-following robustness; without explicit validation that the TrueSkill application and error lens achieve this separation, the surfaced limitations and agent rankings cannot be fully interpreted as measures of the intended reasoning demands.","section":"Abstract"}],"minor_comments":[{"comment":"The abstract contains a minor notation inconsistency with double backticks around 'theory of mind'; standard LaTeX or single quotes would improve readability.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive comments on the abstract. We address each point below, clarifying that the manuscript does not assert a fully non-confounded isolation of skills but instead describes the operationalization alongside explicitly surfaced limitations.","responses":[{"response":"The abstract does not advance a claim of supplying a non-confounded measure of the listed skills. It states that the four environments operationalize complementary ToM-relevant demands and then explicitly identifies the error-survival confound as an evaluation-level limitation observed in this cycle. TrueSkill is applied to observed outcomes without further documented adjustments for player counts, team structure, or LLM variance beyond the system's standard multi-player handling. The error-attribution procedure is a post-hoc labeling tool used to quantify and surface the confound, not to eliminate it. We will revise the abstract for added precision on this distinction.","revision_made":"partial","referee_comment":"[Abstract] Abstract: the central claim that the four games plus TrueSkill rating and error-attribution lens supply a non-confounded measure of distinct ToM-relevant skills (belief attribution, opponent modeling, cooperative inference, sustained deception) is load-bearing, yet the abstract itself states that 'failure-heavy environments can reward robustness to opponent errors as much as strategic ability, with Secret Mafia exhibiting a pronounced error-survival confound'. It is unclear whether the rating system includes documented adjustments for variable player counts, team structure, or high-variance LLM play, or whether the error-attribution procedure prevents rather than post-hoc labels this confound."},{"response":"The statements on structural scaffolding and differing leaderboard validity are empirical observations drawn from the 29k-game dataset after applying the error-attribution lens to separate rule violations from strategic actions. The manuscript presents these patterns and the resulting limitations without claiming that the lens or TrueSkill provides a formally validated separation of the targeted skills. A dedicated validation study would strengthen interpretation but is outside the scope of the current competition analysis.","revision_made":"no","referee_comment":"[Abstract] Abstract (analysis of leaderboard validity): the claim that top-performing systems rely on explicit structural scaffolding and that leaderboard validity differs sharply across environments rests on the separation of strategic ability from rule-following robustness; without explicit validation that the TrueSkill application and error lens achieve this separation, the surfaced limitations and agent rankings cannot be fully interpreted as measures of the intended reasoning demands."}],"tokens_in":1527,"tokens_out":508,"duration_ms":31505,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The paper's real addition is the Mindgames platform that puts LLM agents into four games with a shared interface, full logging, TrueSkill ratings, and a 2025 competition that drew 944 agents. They also ship the 29k-game dataset and the MG-Ref offline protocol that lets new agents be scored against a frozen reference pool. That combination moves past single-game vignettes and gives the community something concrete to build on.\n\nThe setup covers distinct demands: hidden information in Blotto, repeated interaction in IPD, asymmetric knowledge in Codenames, and deception in Mafia. Running at conference scale and releasing the trajectories is the part that actually helps other groups test agents without rebuilding the stack.\n\nThe soft spot is the error-survival confound they themselves note in Secret Mafia. Failure-heavy environments can reward agents that simply avoid crashing more than agents that model opponents well, and the post-hoc error attribution does not fully remove that from the rankings. TrueSkill is applied without visible adjustments for team structure or high variance in LLM play, so the leaderboard numbers in some games mix strategic ability with rule-following robustness. The abstract flags this, which is honest, but it still limits how strongly the results can be read as pure measures of theory-of-mind skills.\n\nThis is for researchers who need a logged, multi-agent testbed and are willing to work around the acknowledged gaps in one environment. The data release makes it worth engaging even if the live competition has noise.\n\nIt should go to peer review. The empirical work is grounded enough and the limitations are stated plainly, so referees can check the analysis details and the protocol.","headline":"Mindgames delivers a usable multi-game arena plus released dataset and offline protocol, but the error-survival issue in Secret Mafia undercuts clean separation of strategic skill from robustness.","tokens_in":2693,"tokens_out":410,"would_cite":false,"duration_ms":14309,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Mindgames provides four game environments that test LLM agents on belief attribution, opponent modeling, cooperative inference, and sustained deception while exposing brittle rule adherence and error confounds.","keywords":["multi-agent LLMs","social reasoning","strategic reasoning","theory of mind","game environments","evaluation benchmarks","deception","opponent modeling"],"falsifier":"Demonstrating that top agents reach high performance without explicit scaffolding or that Secret Mafia shows no measurable error-survival advantage would undermine the reported limitations and confounds.","tokens_in":2725,"feed_emoji":"🎮","tokens_out":691,"duration_ms":27219,"temperature":0.7,"pith_summary":"The paper introduces Mindgames as a live arena that runs LLM agents through four distinct games to capture sustained social and strategic reasoning demands. A 2025 competition evaluated 944 agents and found that even leading systems depend on explicit scaffolding and exhibit brittle rule following. One environment showed a clear error-survival effect that can reward robustness to mistakes rather than pure strategy. The work also releases a large dataset of logged trajectories and an offline scoring protocol for consistent future testing. These elements together aim to move evaluation beyond static vignettes toward interactive, multi-faceted settings.","feed_headline":"Four games expose LLM agents' rule and error limits","feed_subtitle":"Mindgames arena tests belief attribution and deception, finding brittle adherence and error-survival confounds in top systems.","key_machinery":"The four-game arena with TrueSkill-based rating and error-attribution lens that scores agents on belief attribution under hidden information, opponent modeling through repeated interaction, cooperative inference under knowledge asymmetries, and sustained deception.","core_discovery":"Mindgames operationalizes complementary reasoning demands relevant to theory of mind through Colonel Blotto, Iterated Prisoner's Dilemma, Codenames, and Secret Mafia, using a unified interface, TrueSkill ratings, and full trajectory logging. Analysis of the competition cycle surfaces agent-level limitations such as brittle rule adherence and reliance on structural scaffolding, along with evaluation-level issues including differing leaderboard validity across games and a pronounced error-survival confound in Secret Mafia.","pith_inferences":["Developers may need training methods that build intrinsic rule following rather than reliance on external prompts.","The error-survival pattern observed in one game could appear in other dynamic, failure-prone evaluation setups.","Extending the arena with additional games could test further reasoning demands not covered by the current four.","Real-world multi-agent deployments of LLMs may face similar confounds when opponents or teammates make mistakes."],"forward_implications":["Brittle rule adherence remains a major bottleneck for current LLM agents in multi-agent settings.","Top systems repeatedly depend on explicit structural scaffolding to succeed.","Leaderboard validity differs sharply across the four environments.","Failure-heavy games can reward robustness to opponent errors as much as strategic ability.","The released dataset of 29,571 games and the MG-Ref offline protocol enable consistent scoring of new agents against a frozen reference pool."],"fun_headline_variants":["Mindgames reveals LLM brittle rule adherence across games","Error survival confounds Secret Mafia leaderboard validity","LLM agents rely on structural scaffolding in Mindgames","Four games highlight theory of mind gaps in multi-agent LLMs","Mindgames surfaces differing evaluation validity by environment"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The four chosen games together with TrueSkill ratings and error-attribution analysis supply a valid, non-confounded measure of the targeted social and strategic reasoning skills.","fun_headline_variants_meta":{"raw":{"variants":["Mindgames reveals LLM brittle rule adherence across games","Error survival confounds Secret Mafia leaderboard validity","LLM agents rely on structural scaffolding in Mindgames","Four games highlight theory of mind gaps in multi-agent LLMs","Mindgames surfaces differing evaluation validity by environment"]},"model":"grok-4.3","cost_usd":0.002919,"raw_usage":{"total_tokens":1656,"prompt_tokens":762,"num_sources_used":0,"completion_tokens":71,"cost_in_usd_ticks":29187000,"prompt_tokens_details":{"text_tokens":762,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":823,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":762,"tokens_out":71,"duration_ms":6795,"temperature":1.0,"reasoning_tokens":823,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-29T07:12:59.993797+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Demonstrating that top agents reach high performance without explicit scaffolding or that Secret Mafia shows no measurable error-survival advantage would undermine the reported limitations and confounds.","supporting_citations":[],"review_version":1}