{"id":"32a837ec-ef62-4feb-b9c8-83187db85705","arxiv_id":"2506.12801","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A PPO agent with a Transformer encoder beats prompted LLMs and a history-limited Transformer baseline at Da Vinci Code, winning 58.5% of evaluation games.","lead":"The paper tests three AI approaches for the deduction game Da Vinci Code: a Transformer baseline, several large language models, and a PPO reinforcement learning agent. The PPO agent wins 58.5% of games against the baseline, beating all LLM agents, which max out below 39%.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim that PPO outperforms all LLM agents rests on win rates against a single uncalibrated Transformer baseline; since common-opponent win rates are not transitive in two-player zero-sum games, head-to-head PPO-versus-LLM matches are required to support it.","rationale":"I read the manuscript in good faith and identified the same load-bearing assumption as the reader: the comparative ranking of agents is inferred exclusively from win rates against a single common baseline, with no direct agent-versus-agent matches and no calibration of the baseline's strength. This is a genuine logical gap for a two-player zero-sum game because common-opponent win rates are not transitive. I considered other possible concerns—small LLM sample sizes, tiny human samples, placeholder figures, and the incomplete analyses admitted in Section 4—but these are secondary to the central inference. The paper's own results section explicitly frames all evaluations as 'against the baseline' (Section 4, Table 1), and the conclusion generalizes this to superiority over LLM counterparts. The proposed head-to-head test would settle whether the concern lands: if PPO wins head-to-head against every LLM, the comparative conclusion is robust; if not, it is unsupported. The reader's CONDITIONAL verdict is appropriate: the evidence as presented is not sufficient for ACCEPT, but the issue is addressable with additional experiments, so REJECT would be too harsh. Therefore, I recommend no change to the reader's verdict.","tokens_in":11975,"tokens_out":6299,"duration_ms":74548,"concrete_test":"Run a round-robin evaluation in which the PPO agent plays at least 1,000 games against each LLM agent (using the same environment, legal-action masking, and fallback rules), and also have the baseline play at least 1,000 games against a random-action agent as a calibration check. If PPO's win rate against every LLM exceeds 50% with a non-overlapping 95% confidence interval, and the baseline beats random by a substantial margin (e.g., >70%), the comparative claim is supported. If PPO does not win head-to-head, or if the baseline is near-random, the conclusion must be weakened to 'PPO beats the baseline more often than LLMs do,' which is a substantially narrower claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical conclusion in Section 4, Table 1 is that the PPO agent (58.5% ± 1.0%, n=10,000) significantly outperforms every LLM agent (best: DeepSeek R1 at 38.9% ± 15.9%, n=36). However, all reported win rates are measured only against the history-limited Transformer baseline; no direct PPO-versus-LLM games are reported anywhere in the manuscript. In a two-player zero-sum game, winning against a common opponent at rate r does not imply pairwise superiority: rock-paper-scissors cycles can invert transitive rankings. The baseline's own strength is never calibrated (e.g., against random play or against humans), and its decision policy is only described as 'predicting opponent tiles' (Section 3.2) without specifying the action-selection rule. If the baseline is weak, deterministic, or exploitable in a way that LLMs cannot exploit, the PPO's high win rate could reflect exploitation of baseline quirks rather than generally superior Da Vinci Code play. The paper's comparative claim ('significantly outperforming the LLM counterparts') thus goes beyond what the evidence supports. The human-comparison claim is similarly fragile given n=20 and n=25 games, but the load-bearing gap is the absence of any head-to-head evidence among the agents being compared.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper develops a Da Vinci Code game environment and implements three classes of agents: a history-limited Transformer baseline, several prompted LLM agents (Gemini, DeepSeek, GPT-4o, O3, Qwen), and a PPO-trained agent with a Transformer encoder. All agents are evaluated by win rate against the Transformer baseline over a varying number of games (Section 4, Table 1). The paper reports that the PPO agent achieves a 58.5% ± 1.0% win rate over 10,000 games, far exceeding all LLM agents, and interprets this as evidence that deep reinforcement learning with comprehensive history encoding outperforms prompted LLMs in this deductive game. The conclusion also states the PPO agent approaches the proficiency of skilled human players.","tokens_in":12380,"tokens_out":3358,"duration_ms":34618,"significance":"If the comparative claim were properly supported, this would be a useful empirical contribution: it provides an open-source game environment, benchmarks several state-of-the-art LLMs against a dedicated RL agent, and explores a relatively understudied imperfect-information deduction game. The authors are transparent about missing analyses (e.g., learning curves, game statistics) and make the code available. However, the central claim is currently supported only by win rates against a single, uncalibrated baseline, with no direct PPO-versus-LLM matches and very small LLM sample sizes; as it stands, the evidence does not justify the comparative conclusion. The manuscript also contains placeholder figures, which indicates it is not in a polished state.","major_comments":[{"comment":"The central claim that the PPO agent 'significantly outperforms all LLM agents' rests solely on win rates against the history-limited Transformer baseline. In a two-player zero-sum game, a high win rate against a common opponent does not imply pairwise superiority (rock-paper-scissors cycles are possible). No head-to-head PPO-versus-LLM games are reported anywhere in the manuscript, and the baseline's own strength is never calibrated against random play, an MCTS agent, or human players. To support the comparative claim, the authors need either direct pairwise matches or a calibrated common-opponent protocol that justifies transitivity.","section":"Section 4, Table 1"},{"comment":"The baseline model's action-selection rule is not specified. The text says it predicts opponent tiles 'token by token,' but it does not explain how these predictions translate into the environment's legal actions ('place' vs. 'guess' with a position and card value). Without knowing the baseline's policy, readers cannot assess whether it is a reasonable common opponent or a weak, deterministic policy that a PPO agent can exploit more easily than an LLM can. This missing detail is load-bearing for interpreting every win rate in Table 1.","section":"Section 3.2"},{"comment":"Several LLM results are based on very small samples: DeepSeek R1 has n=36 (CI ±15.9 percentage points) and Qwen3-235B-A22B has n=29 (CI ±16.3). The paper's ranking of LLMs and the claim that all are 'significantly' worse than the PPO agent would be more convincing with larger sample sizes and a proper statistical comparison that accounts for multiple testing and the transliteracy of the common-opponent design. The current evidence supports only a qualitative ordering, not a strong significance claim.","section":"Section 4, Table 1"},{"comment":"The human-comparison claim is weakly supported. 'Human Expert (Median)' and 'Human Expert (High)' are based on 20 and 25 games, respectively, with 95% confidence intervals of ±21.9 and ±18.8 percentage points. The criteria for selecting these experts and the conditions of play are not described. The statement that the PPO agent 'approaches the proficiency of skilled human players' goes beyond what this small, poorly specified sample can establish.","section":"Section 4 and Conclusion"}],"minor_comments":[{"comment":"The row group label 'LLMs - Ous Category Models' appears to be a typo; it should likely read 'Other Category Models' or 'OpenAI Category Models.'","section":"Table 1"},{"comment":"Figure 1 and Figure 2 are still placeholders ('Please replace this placeholder with figure1.png' and 'figure_2.png'). This makes the manuscript incomplete and prevents readers from seeing the intended visualizations.","section":"Section 4 and Appendix A.2"},{"comment":"The 'unspecified O3 model' is not described with a version, access date, or API details, which makes replication difficult. The same applies to the exact prompts for the other LLMs beyond the Gemini example.","section":"Section 3.3"},{"comment":"The Tree-of-Thought reference is listed as 'Shinn Yao et al.'; the correct first author for that paper is Shunyu Yao. The TRPO citation also lacks a complete author list and venue details.","section":"References"},{"comment":"The statement that 'A GUI for Transformer vs. human play was also partially developed' is a scope limitation that should be made explicit in the abstract or introduction, since the paper otherwise appears to present a complete evaluation environment.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"The paper is currently more suited to a workshop than a main conference, primarily because the experimental design does not yet support the headline comparative claim. The open-source environment and the breadth of LLMs tested are assets, but the authors should be asked to either add head-to-head matches or substantially strengthen the baseline calibration and LLM sample sizes before resubmission. I also note that the placeholder figures and typos suggest the manuscript was submitted before final polish."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing you should know: the headline claim rests on a comparison that the experiments don't actually make. Every win rate in Table 1 is against the same history-limited Transformer baseline, and win rates against a common opponent are not transitive in a two-player zero-sum game. The paper says PPO \"significantly outperforms\" the LLMs, but there are no PPO-vs-LLM games anywhere. The stress-test note has it right. If the baseline is weak or exploitable in a way the LLMs can't exploit, PPO's 58.5% could just mean it learned to beat that specific model.\n\nThat said, the paper does real work. It builds a clean Da Vinci Code environment, implements a PPO agent with a Transformer encoder and a sensible action space, releases the code, and runs 10,000 games for PPO. That is a reproducible, well-specified evaluation pipeline. The LLM prompting setup is also carefully described, including the fallback mechanism for invalid JSON. As a first systematic comparison of these agent families on this game, it's a legitimate data point.\n\nThe soft spots are real but addressable. The baseline is never calibrated: no win rate against random play or against humans, and its action-selection rule is only partially specified. LLM sample sizes are tiny (29 to 100 games), so the confidence intervals are huge and the ranking among LLMs is noise. The prompt encodes a specific median-candidate heuristic, so the LLM results partly test that heuristic rather than the raw reasoning ability of the models. The human comparison uses 20 and 25 games. The manuscript itself admits to missing learning curves and game statistics, and Figure 1 is a placeholder. The \"O3\" model is never fully identified.\n\nI'd send this to peer review, but with the clear expectation of major revision. The evaluation design needs to include head-to-head matches, a calibrated baseline, and larger LLM samples. The authors seem capable of doing this; the paper's own discussion acknowledges most of these gaps. As it stands, the empirical claim is plausible but not proven.\n\nFor a reader, the value is in the environment and the cautionary example: it shows how easy it is to overstate a comparative conclusion when all agents are measured against a single uncalibrated opponent. I wouldn't cite the comparative result, but I might cite the environment if I worked on small imperfect-information games.","headline":"The comparative PPO-vs-LLM claim doesn't follow from the data because all win rates are measured against a single uncalibrated baseline, but the paper's environment, code, and 10,000-game PPO evaluation are real and worth a serious referee's time.","tokens_in":12779,"tokens_out":1752,"would_cite":false,"duration_ms":23176,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A PPO-based agent with a Transformer encoder over full game history wins 58.5% ± 1.0% against a baseline, surpassing prompted LLM agents in Da Vinci Code.","keywords":["Da Vinci Code","imperfect-information games","reinforcement learning","Proximal Policy Optimization","large language models","logical deduction","game-playing agents"],"falsifier":"Run a round-robin tournament with enough games per pairing, for example 2,000 or more, in which the trained PPO agent plays directly against each LLM agent and against a random agent, and check whether the PPO agent's win rate stays above chance; a prompted LLM agent winning at or above 50% would contradict the paper's central comparative claim.","tokens_in":11816,"feed_emoji":"🎲","tokens_out":7528,"duration_ms":77603,"temperature":0.7,"pith_summary":"The paper argues that for the imperfect-information deduction game Da Vinci Code, a self-trained reinforcement-learning agent is the strongest of three agent families tested. The PPO-based agent, which encodes the full game history with a Transformer, wins 58.5% ± 1.0% of 10,000 games against a history-limited Transformer baseline. This is well above every prompted LLM agent, whose best rate is 38.9%, and it sits between the paper's median and high human-expert benchmarks of 50.0% and 64.0%. The authors read this as evidence that learning implicit deductive strategies from self-play outperforms prompting general-purpose language models, and that a history-aware state representation is a key ingredient.","feed_headline":"PPO agent beats LLMs at Da Vinci Code, 58.5% wins","feed_subtitle":"A self-trained reinforcement agent outperforms every prompted large language model against the same baseline.","key_machinery":"The central mechanism is a history-aware state encoder: the game state and chronological event log are serialized into a string, tokenized with a 64-token domain-specific vocabulary, and passed through a Transformer encoder whose first-token output becomes the state vector for the policy and value networks. Combined with the clipped PPO objective, generalized advantage estimation, legal-action masking, and a 339-action discrete space (one place action plus guesses over 13 opponent positions and 26 card values), this lets the agent learn from self-play while retaining long-range dependencies that the baseline's snapshot input lacks.","core_discovery":"The paper's discovery is that an agent trained entirely by self-play with Proximal Policy Optimization, using a Transformer to encode the full game history, becomes the strongest of the three agent families tested. Over 10,000 games against the same history-limited Transformer opponent, this agent wins 58.5% ± 1.0%, while the prompted LLM agents win between 17.0% and 38.9% depending on the model. The PPO agent approaches the paper's high-level human benchmark of 64.0% and sits above the median expert benchmark of 50.0%. The authors attribute the gap to the PPO agent's ability to learn implicit deductive strategies from experience and to track and update constraints over full game history, whereas the LLMs, despite structured prompts and a suggested minimum-candidate-set heuristic, sometimes issued logically inconsistent moves.","pith_inferences":["If the paper's ranking is correct, a direct head-to-head match between the PPO agent and the best LLM agent should replicate the margin; that match is the natural next experiment and would also control for baseline-specific exploitation.","The PPO agent's edge may depend on the paper's reward shaping, with +3.0 for a win, +0.2 for a correct guess, and −0.5 for an incorrect guess; ablating those weights would reveal whether the discovered strategy is robust or tuned to that objective.","The same history-encoding design should transfer to similar deduction games with hidden tiles, such as Mastermind variants; a positive transfer result would support the paper's broader claim about history-aware reinforcement learning."],"forward_implications":["For deduction games with hidden information, a history-aware self-play reinforcement-learning agent is a more reliable path to strong play than prompting a general-purpose LLM.","The PPO agent's 58.5% win rate against the baseline, between the median and high human benchmarks, suggests expert-level play is reachable with this architecture and modest training.","The baseline Transformer's poorer showing indicates that reasoning from a state snapshot without full game history loses strategic information that matters in this game.","The occasional logically inconsistent outputs from some LLMs imply that rule adherence, not just reasoning ability, is a bottleneck for using LLMs as game agents."],"supporting_citations":[{"why":"This supplies the Transformer architecture used for both the baseline model and the PPO agent's state encoder.","marker":"[Vaswani et al., 2017]"},{"why":"This supplies the Proximal Policy Optimization algorithm with clipped surrogate objective used to train the reinforcement-learning agent.","marker":"[Schulman et al., 2017]"},{"why":"This is cited for generalized advantage estimation in the PPO training loop.","marker":"[Schulman et al., 2015]"},{"why":"This defines the Gemini model used as one of the LLM agents.","marker":"[Team and Google, 2023]"},{"why":"This defines the GPT-4o model used as one of the LLM agents.","marker":"[OpenAI, 2024]"},{"why":"This defines the DeepSeek models used as LLM agents.","marker":"[Bi et al., 2024]"},{"why":"This defines the Qwen model used as one of the LLM agents.","marker":"[Bai et al., 2023]"},{"why":"This provides the DouZero result for DouDizhu that frames the prior deep reinforcement learning work this paper extends.","marker":"[Zha et al., 2021]"},{"why":"This provides Suphx, the Mahjong deep reinforcement learning system used as a domain benchmark for imperfect-information games.","marker":"[Li et al., 2020]"}],"fun_headline_variants":["Self-play PPO agent beats LLMs at Da Vinci Code","PPO agent wins 58.5% vs LLMs in Da Vinci Code","RL beats LLM agents at Da Vinci Code","Self-trained agent outwits LLMs in Da Vinci Code","Reinforcement learning wins at Da Vinci Code over LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking rests on win rates against a single history-limited Transformer baseline, and the paper treats those win rates as a transitive measure of relative agent strength; if that baseline is unrepresentative or exploitable in a way the compared agents exploit unevenly, the ranking would not generalize.","fun_headline_variants_meta":{"raw":{"variants":["Self-play PPO agent beats LLMs at Da Vinci Code","PPO agent wins 58.5% vs LLMs in Da Vinci Code","RL beats LLM agents at Da Vinci Code","Self-trained agent outwits LLMs in Da Vinci Code","Reinforcement learning wins at Da Vinci Code over LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00161,"raw_usage":{"total_tokens":6408,"prompt_tokens":938,"completion_tokens":5470,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":554,"completion_tokens_details":{"reasoning_tokens":5383}},"tokens_in":554,"tokens_out":5470,"duration_ms":36820,"temperature":1.0,"reasoning_tokens":5383,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:41:38.659355+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a round-robin tournament with enough games per pairing, for example 2,000 or more, in which the trained PPO agent plays directly against each LLM agent and against a random agent, and check whether the PPO agent's win rate stays above chance; a prompted LLM agent winning at or above 50% would contradict the paper's central comparative claim.","supporting_citations":[{"cited_title":"Trust region policy optimization","cited_arxiv_id":null,"evidence_quote":"This is cited for generalized advantage estimation in the PPO training loop."},{"cited_title":"Hello gpt-4o","cited_arxiv_id":null,"evidence_quote":"This defines the GPT-4o model used as one of the LLM agents."},{"cited_title":"DouZero: Mastering DouDizhu with Self-Play Deep Reinforcement Learning","cited_arxiv_id":"2106.06135","evidence_quote":"This provides the DouZero result for DouDizhu that frames the prior deep reinforcement learning work this paper extends."}],"review_version":1}