{"id":"eacb9f14-318b-4539-a394-22ba5d039dd1","arxiv_id":"2507.16196","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"In a new persuasion game, humans outperformed the LLM o1-preview when an opponent's preferences had to be inferred, while o1-preview outperformed humans when those preferences were disclosed.","lead":"This paper tests a planning version of theory of mind by having humans and large language models play a multi-turn persuasion game. Humans beat the reasoning model o1-preview when they had to infer the other player's hidden preferences, while o1-preview beat humans when those preferences were shown.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The unvalidated gpt-4o target classifier may misread o1-preview's longer, indirect questions as non-appeals, inflating the human-vs-o1 HIDDEN gap; replaying games with human-labeled ground truth would settle it.","rationale":"The reader's weakest assumption identifies precisely the same load-bearing concern: the gpt-4o classifier that implements the target's understanding of persuader messages is unvalidated and could bias the human-vs-o1 comparison. This is the right concern to elevate because the target classifier is the instrument through which every success or failure is measured. If the classifier's recall for appeals or disclosures differs between short human questions and long, indirect LLM messages, the central H5 finding (humans outperform o1-preview in HIDDEN) could be an artifact of the interface rather than a difference in planning theory of mind. The discrete-game variant strengthens this worry: when the need to parse natural language is removed and appeals are made explicit structured actions, o1-preview reaches 80% success in HIDDEN, comparable to its REVEALED performance. That result suggests the model may have the underlying PToM competence but fails to express it in the free-form dialogue format, or that the parser does not credit its expressions. The paper does not report any human validation of the classifier, so the condition is unmet. The recommended verdict remains CONDITIONAL as the reader set it: the study is well-designed and preregistered, but this specific instrumentation gap should be closed before the headline claim is treated as definitive. The concrete annotation-replay test is feasible with the linked code and data and would directly resolve whether the H5 gap survives ground-truth scoring.","tokens_in":25972,"tokens_out":5872,"duration_ms":67401,"concrete_test":"Annotate a random sample of 100 HIDDEN games (50 o1-preview, 50 human) at the message level with two independent raters, labeling each message for (a) disclosed proposal-attribute-utility facts and (b) appeals to motivational/informational/inferential states. Compute gpt-4o classifier recall/precision separately by persuader type. Then replay each game under the assumption that the target responds to the human-annotated ground truth (echoing all disclosures, answering all appeals), and recompute success rates for H5. If the human-o1 gap becomes non-significant (or drops by >10 percentage points), the measured PToM gap is largely a parser artifact. A complementary check: run a new cohort of o1-preview HIDDEN games against a target whose responses are manually verified/corrected by a human in real time, and compare success rates.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is that the gpt-4o-based target classifier (Section 3, 'Target Agent'; prompts in Figs. 15-16) labels persuader messages the same way for humans and o1-preview. The classifier is instructed to 'Ignore messages that do not make explicit appeals (almost always in a question form),' and it must extract exact proposal-attribute-utility triples from free text. No validation, inter-annotator agreement, or error analysis is reported. The failure mode is asymmetric: o1-preview messages are long, multi-clause, and often embed questions in persuasive argumentation (e.g., 'What are your thoughts on going with Proposal A?'), which may not be recognized as motivational/informational appeals, so the target does not answer and the persuader cannot learn the target's preferences. Human messages are typically short and directive ('what do you like and dislike?'), which the classifier handles well. If the classifier systematically under-recognizes LLM-style appeals, then o1-preview's low HIDDEN success (H5, p=0.006) reflects an interface artifact rather than a PToM deficit. The task-variant results are consistent with this reading: constraining o1-preview to explicit JSON actions (discrete-game) raises its HIDDEN success to 80%, comparable to REVEALED, which would not be expected if the model lacked the underlying PToM reasoning.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MindGames, a multi-turn persuasion game in which a persuader (human or LLM) interacts with a scripted, naively rational target and must disclose the right subset of hidden information to make the target choose the persuader's preferred proposal. In the HIDDEN condition the target's preferences and information are unknown and must be elicited; in the REVEALED condition they are shown. Using about 200 trials per condition for humans and several LLMs, the authors report that humans outperform o1-preview in HIDDEN (about 29% vs. 18% success; H5, p=0.006), while o1-preview outperforms humans in REVEALED (78% vs. 22%; H6). They also report that LLMs rarely ask about the target's mental states, and that o1-preview succeeds at 80% in a discrete-action variant of HIDDEN. The authors conclude that apparent LLM ToM abilities are largely spectatorial and that LLMs lack planning theory of mind.","tokens_in":26221,"tokens_out":7338,"duration_ms":81504,"significance":"If the headline results hold, MindGames is a valuable new benchmark for interactive, participatory theory-of-mind evaluation, and the paper is one of the few to compare humans and LLMs on a ToM task that requires planned intervention rather than passive prediction. The study has several genuine strengths: pre-registered hypotheses, a moderately large human sample, multiple LLMs, bootstrapped confidence intervals, a derived random-disclosure baseline, and a reproducibility statement with code and data. The success measure is the target's final choice, which is objective. However, the environment itself is mediated by an unvalidated gpt-4o classifier that decides whether the target answers a persuader's questions; this is the main threat to the central human-versus-LLM comparison and to the mechanistic appeal analysis. The paper's strong conclusion about absent PToM also sits uneasily with its own discrete-game result, in which o1-preview reaches 80% HIDDEN success when the action space is made explicit.","major_comments":[{"comment":"The target's recognition of appeals and disclosures is performed by gpt-4o using the prompts in Figs. 15-16, and no human-coded validation, inter-annotator agreement, or error analysis is reported. Because the target's responses determine whether a persuader obtains the information needed to win, the HIDDEN condition measures the interaction between the persuader and this unvalidated parser, not the persuader alone. The failure mode is asymmetric: the classifier is instructed to ignore messages that do not make 'explicit appeals (almost always in a question form),' and o1-preview's messages are longer, multi-clause, and often embed questions in argumentation. If the parser systematically under-recognizes LLM-style appeals, H5 and H3 could reflect an interface artifact rather than a PToM deficit. This is load-bearing for the paper's central claim and must be addressed, e.g., by human-labeled ground-truth labels on a sample of human and LLM messages, an error analysis by persuader type, or a robustness check that replays the main comparison using the structured discrete-game action space.","section":"Section 3, 'Target Agent'; Figs. 15-16"},{"comment":"The discrete-game variant, in which o1-preview is restricted to explicit JSON actions, yields 80% success in HIDDEN, comparable to its REVEALED performance. The Discussion concludes that 'PToM appears cleanly present in humans and absent in LLMs,' but this variant suggests that o1-preview can plan over mental states when the action space makes appeals and disclosures explicit. The default HIDDEN failure may therefore stem from natural-language generation or task ambiguity rather than from the absence of a causal model of the target. The conclusion should be substantially qualified: at most, the results show that default open-ended prompting does not elicit PToM-like planning in LLMs, not that the underlying capacity is absent.","section":"Section 4.2, Fig. 4"},{"comment":"The statistical models are described as 'binomial mixed effects models with random intercepts by scenario.' No random intercept for participant is reported, despite 124 participants each contributing up to five trials. Ignoring this non-independence can inflate test statistics for the human comparisons, including the headline H5 and the interaction H7. Please add participant-level random intercepts (or cluster-robust standard errors by participant) and confirm that H5 and H7 remain significant at the reported levels.","section":"Section 4, 'Hypotheses'"}],"minor_comments":[{"comment":"The reported H3 result (z=1.50, p=0.133) implies a HIDDEN success rate near 13%, while the text and abstract state that humans outperform o1-preview by 11 percentage points (about 29% vs. 18%). Please report the exact point estimates and model specification to resolve this apparent numerical inconsistency.","section":"Section 4.1, H3 and Fig. 2"},{"comment":"The text states that a 'generous baseline of 10%' is used for hypothesis testing, but Fig. 2 plots a grey dashed line at 0.075 corresponding to the random-disclosure maximum. Clarify which baseline is used for each hypothesis test and why the plotted value differs from the tested value.","section":"Section 4, 'Baseline' and Fig. 2"},{"comment":"The 'appeals to all mental states' measure is produced by the same gpt-4o classifier used in the target agent; the figure captions should state this dependency explicitly and point to the validation evidence (or its absence).","section":"Figure 3 and Fig. 13"},{"comment":"The random-disclosure baseline is derived for 'n pieces revealed with replacement,' but it is not stated how this maps onto the actual dialogue protocol, where persuaders send up to eight messages and may disclose multiple facts per turn. A short justification of the chosen n=6 would improve interpretability.","section":"Appendix A.4"},{"comment":"The prompts contain several typos ('value funciton', 'alwasys', 'consier', 'attept', 'Revlead' in Fig. 12) and the Acknowledgments contains 'We thanks.' A proofreading pass is needed before publication.","section":"Appendix D, prompts"},{"comment":"The text says that REVEALED was re-run 'due to an error' but does not describe the error. Please document it in the reproducibility section so readers can judge whether the two human samples are directly comparable.","section":"Section 4, 'Study details'"}],"recommendation":"major_revision","confidential_remarks":"The decisive issue is the unvalidated gpt-4o target classifier: the paper's main claim depends on a measurement instrument that is itself an LLM, and the failure mode is asymmetric between humans and o1-preview. If the authors can supply human-coded validation or demonstrate robustness via the discrete-game action space, the paper would be suitable. The participant-random-effects concern is also important but likely fixable with a reanalysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read it. The core facts: pre-registered, 124 human participants, ~200 trials per condition, mixed-effects models. Humans beat o1-preview in HIDDEN (29% vs 18%, p=0.006), and o1-preview crushes humans in REVEALED (78% vs 22%). That reversal is a nice result and the new benchmark (MindGames) is a genuine contribution: a controlled multi-turn persuasion environment with a scripted naively-rational target, open-sourced. Credit where due.\n\nThe soft spots are real. The target agent and the appeal/disclosure labels both come from a gpt-4o classifier with no human annotation, no inter-annotator agreement, no error analysis. The target's behavior is the dependent variable, so if the parser systematically misses o1-preview's long, indirect questions (e.g., 'What are your thoughts on going with Proposal A?'), the HIDDEN gap is inflated. The paper's own example dialogues show o1 wasting turns before landing on a direct question; the aggregate appeal counts could be partly a parser artifact. That needs to be tested with human-coded transcripts.\n\nThe discrete-game variant is the harder problem for the authors' interpretation. When o1-preview is given an explicit JSON action space, its HIDDEN success jumps to 80%—comparable to REVEALED. That says the model can plan over mental states when the actions are laid out, but doesn't spontaneously generate the right questions in open dialogue. The paper's conclusion 'PToM cleanly present in humans and absent in LLMs' is too strong; a fairer statement is that LLMs fail to initiate the right information-gathering in an unconstrained setting. That is still worth publishing, but the framing should change.\n\nThe 10% baseline is hand-set (they compute max 7.5% for random disclosure), so H3's null result is weak; no big deal. And the authors already flag working-memory/instruction-comprehension confounds for humans, which is honest.\n\nWho gets value: anyone building or evaluating LLM agents for social interaction, and cognitive scientists working on interactive ToM. It deserves a serious referee, but the revision needs classifier validation (a few hundred human-labeled messages, precision/recall by persuader type) and a careful rewrite of the PToM-absent claim in light of discrete-game.","headline":"Solid new task and a real reversal, but the unvalidated gpt-4o target parser and the discrete-game variant mean 'no planning ToM' overstates the case.","tokens_in":26824,"tokens_out":4284,"would_cite":true,"duration_ms":42388,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new persuasion game shows that models can plan with known preferences but fail to plan over hidden mental states.","keywords":["planning theory of mind","persuasion task","theory of mind evaluation","large language models","multi-turn dialogue","mental state inference","human-AI comparison"],"falsifier":"Recode the HIDDEN transcripts (202 human games and the corresponding 200 model games) with two independent human annotators labeling each message for appeals and disclosures, blind to persuader type; if the re-coded success rates no longer show a significant human advantage over o1-preview, the central dissociation would not survive.","tokens_in":25735,"feed_emoji":"🧠","tokens_out":6235,"duration_ms":65482,"temperature":0.7,"pith_summary":"MindGames tests a form of social intelligence that standard theory-of-mind evaluations skip: using beliefs and desires not just to predict someone's behavior, but to change it through multi-turn persuasion. Human participants significantly beat a top reasoning model when the target's mental states were hidden (29% vs. 18% success), while the model dramatically beat humans when those mental states were revealed (78% vs. 22%). The authors argue the dissociation is evidence that LLMs' apparent theory of mind is spectatorial and predictive rather than participatory and causal: o1-preview rarely asks the target what it knows or likes, whereas humans do about 40% of the time. If correct, the result draws a sharp line between planning with given mental states and planning to discover them, with practical stakes for LLMs deployed as negotiators, educators, and therapists.","feed_headline":"Humans beat o1-preview when the other mind is hidden","feed_subtitle":"A persuasion game splits planning from mind-reading: models win with revealed preferences, humans win when they must ask.","key_machinery":"The central object is the naively-rational target agent, a hard-coded bot that chooses among three proposals by maximizing a value function over the information it has been told, echoes back disclosures, and answers questions about its beliefs and preferences; a gpt-4o classifier determines whether each persuader message discloses information or appeals to the target's mental states. This makes the planning problem well-defined and observable: a persuader must reveal exactly the right two hidden facts and avoid revealing the two that would make another proposal optimal, and in the HIDDEN condition must first obtain the target's value function and knowledge through dialogue. The target's fully controlled outputs let the authors measure a single capacity—multi-step counterfactual planning over another agent's mental states—apart from general language skill.","core_discovery":"The paper's central claim is that planning theory of mind—using a causal model of another agent's beliefs and desires to select actions that change those states—is present in human performance but absent in current LLMs. In MindGames, a persuader must get a naively rational target to choose the persuader's preferred proposal by selectively disclosing hidden information. In the HIDDEN condition, the persuader must first elicit what the target knows and likes; in REVEALED, that information is supplied. The headline dissociation is that humans succeed more than o1-preview when mental states must be discovered (29% vs. 18%), while o1-preview succeeds far more than humans when they are given (78% vs. 22%). The paper attributes the failure to a missing behavior: LLMs do not appeal to all of the target's mental states—they rarely ask the target what it knows or likes—so they cannot plan over multiple steps in a partially observed social world.","pith_inferences":["One implication the authors leave implicit is that question-asking rate is a cheap, testable behavioral signature of a generative model of another agent; it could be measured in any dialogue agent and tracked across deployment.","A direct extension would vary the target's cooperativeness or the cost of asking, predicting that humans shift their question strategy adaptively while current LLMs do not.","Because the target's parser is itself a language model, a human-annotated re-coding of the HIDDEN transcripts would clarify how much of the 11-point gap is planning ability versus phrasing-dependent classification; this is a validation study, not a challenge to the task design."],"forward_implications":["Static false-belief and similar theory-of-mind scores should not be read as evidence that an LLM can use mental states to drive action in conversation.","In deployed settings such as assistants, educators, and negotiators, the binding constraint will be elicitation: asking what the user knows and wants, rather than answering with known preferences, and the HIDDEN results predict failures exactly there.","Reasoning models' extra inference-time computation helps them plan over explicit states but not discover hidden ones; adding hints or a discrete action space substantially closes the gap, pointing to the missing capacity rather than raw reasoning.","Evaluation suites for social AI should include participatory, multi-turn tasks with hidden mental states as a complement to prediction-only benchmarks."],"supporting_citations":[{"why":"Defines planning with theory of mind and motivates the claim that ToM functions include planning interventions, the theoretical target of MindGames.","marker":"Ho et al. (2022)"},{"why":"Reports LLMs passing false-belief tasks, the spectatorial ToM evidence the paper argues is insufficient.","marker":"Gandhi et al. (2023)"},{"why":"Claims LLMs display ToM on prediction tasks, one of the main claims the paper seeks to qualify.","marker":"Kosinski (2024)"},{"why":"Shows LLMs fail on trivial alterations to ToM tasks, supporting the brittleness interpretation.","marker":"Ullman (2023)"},{"why":"Stress-tests social reasoning in LLMs and suggests Clever-Hans-style shortcuts, used by the paper to argue that predictive ToM can be passed without causal mental-state models.","marker":"Shapira et al. (2024)"},{"why":"Argues ToM evaluations in LLMs are unreliable, used to justify a participatory planning task instead of prediction-only tests.","marker":"Hu et al. (2025)"}],"fun_headline_variants":["Humans outsmart o1-preview when persuasion requires mind-reading","LLMs struggle when planning needs hidden mental states","o1-preview beats humans only when preferences are given","Planning theory of mind: humans beat LLM in persuasion game","Hidden preferences flip the table: humans beat o1-preview"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The automated target's classifier is implemented with gpt-4o and is not human-validated, so the measured human-over-model gap in the hidden condition could be distorted if the classifier systematically misrecognizes appeals and disclosures in model phrasing.","fun_headline_variants_meta":{"raw":{"variants":["Humans outsmart o1-preview when persuasion requires mind-reading","LLMs struggle when planning needs hidden mental states","o1-preview beats humans only when preferences are given","Planning theory of mind: humans beat LLM in persuasion game","Hidden preferences flip the table: humans beat o1-preview"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000203,"raw_usage":{"total_tokens":1397,"prompt_tokens":968,"completion_tokens":429,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":346}},"tokens_in":584,"tokens_out":429,"duration_ms":5139,"temperature":1.0,"reasoning_tokens":346,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:16:05.244329+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recode the HIDDEN transcripts (202 human games and the corresponding 200 model games) with two independent human annotators labeling each message for appeals and disclosures, blind to persuader type; if the re-coded success rates no longer show a significant human advantage over o1-preview, the central dissociation would not survive.","supporting_citations":[],"review_version":1}