{"id":"87ff5f3d-6a16-46ae-bf57-cd0304b67b75","arxiv_id":"2507.14088","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A dual-process LLM agent with a three-stage theory-of-mind module outperforms baselines in real-time Overcooked human-AI collaboration.","lead":"This paper proposes DPMT, a framework that gives AI agents a 'theory of mind' about human partners in the game Overcooked, using a fast system for moment-to-moment actions and a slow system that infers the partner's knowledge, style, and intentions. The authors report higher collaboration scores and better subjective ratings than their baselines, but without released code or significance tests.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DPMT's headline claim of 'significantly enhances' is not backed by the reported statistics: the main baseline comparison (Table 1) has n=5 per map, no test, and Exp. 2 objective scores lack numerical values, so the central effect could still be within chance variation.","rationale":"I partially agree with the reader. The taxonomy-coverage concern is real: the ToM stages rely on hand-built corpora, and the paper does not release them, so the claimed generalization to 'unseen' partners cannot be checked as written. However, I think the more load-bearing point is upstream: the manuscript never establishes, with statistics, that DPMT beats the HLAgent baseline. The large DPMT w/o MsToM gap is credible evidence that the slow system contributes, and the dual-process framing is coherent, so I would not reject; but the central claim as worded ('significantly enhances') is stronger than the reported evidence. I would keep the CONDITIONAL verdict, with conditions: per-participant raw scores, an inferential analysis, and release of prompts and corpora. I also note an untranslated Chinese phrase left in the Slow System section and Figure 8 without numeric values, both of which reinforce reproducibility concerns without being the main attack.","tokens_in":9619,"tokens_out":7650,"duration_ms":548735,"concrete_test":"Obtain the raw per-trial scores behind Table 1 and Figure 8, plus the number of distinct participants per map, and fit a paired linear mixed-effects model with DPMT vs HLAgent as the fixed effect and participant and map as random effects; report adjusted p-values and 95% confidence intervals for all three maps and for the 30-person experiment. If DPMT is not significantly better than HLAgent on at least two maps after adjustment, change the abstract's 'significantly enhances' to a descriptive claim about the specific maps.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is empirical: DPMT 'significantly enhances human-AI collaboration.' The strongest numerical evidence is Table 1, which reports five repeated runs of a fixed-strategy human setup per map and no inferential statistics. A rough Welch t-test on the reported means and standard deviations gives p≈0.17 for Ring (121±13.56 vs 99±18.81), p≈0.03 for Bottleneck (101±13.56 vs 60±20.74), and p≈0.26 for Quick (104±16.73 vs 87±16.00); after correcting for three maps, none is robust. Human Experiment 2 has 30 participants, but its objective score comparison appears only in Figure 8 with no table of means, errors, or tests, and the tabulated evidence is a 5-point Likert self-report (Table 2). The text also does not say how many distinct human partners generated the five runs or whether the runs are paired. Thus the abstract's 'significantly' is either an unsupported statistical claim or an unsupported colloquial emphasis; even a perfect taxonomy would not fix the missing comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DPMT, a dual-process framework for real-time human-AI collaboration in Overcooked. A fast small-LLM system selects macro-actions, while a slow LLM-based theory-of-mind module reasons about a human partner's domain knowledge, cognitive style, and domain intentions using hand-authored corpora. The authors report experiments with fixed-strategy confederates and with 30 diverse human participants, plus ablations, and claim that DPMT significantly improves collaboration scores and subjective ratings over HLAgent and over a version without the multi-scale ToM module.","tokens_in":9880,"tokens_out":7331,"duration_ms":77648,"significance":"If the empirical claims were properly supported, the paper would make a useful contribution: it applies dual-process theory in a concrete way to LLM agents, separates fast action selection from slower partner modeling, and evaluates with real human partners in a recognized benchmark, which is relatively rare in the LLM-agent literature. The reported effect sizes on Bottleneck are large, and the ablation pattern is qualitatively sensible. However, the manuscript currently does not provide the statistical evidence needed for the 'significantly enhances' claim, and the dependence on undocumented hand-authored corpora makes the mechanism hard to evaluate. The significance is therefore conditional on a substantial revision of the experimental reporting.","major_comments":[{"comment":"The abstract's claim that DPMT 'significantly enhances' collaboration is not supported by the statistics reported in Table 1. The table gives means and standard deviations for five runs per map but no inferential tests, no information about whether the runs are paired or independent, and no statement about how many distinct human partners produced them. Recomputing Welch t-tests from the reported summary statistics gives p≈0.17 for Ring (121±13.56 vs 99±18.81), p≈0.03 for Bottleneck (101±13.56 vs 60±20.74), and p≈0.26 for Quick (104±16.73 vs 87±16.00); after adjusting for three comparisons, none is clearly significant. The authors should either add appropriate inferential statistics with a pre-specified multiple-comparison procedure and enough runs for power, or soften the language from 'significantly' to a descriptive claim.","section":"Human Experiment 1, Table 1"},{"comment":"The claim that DPMT adapts to diverse unseen human partners rests on Experiment 2, but the objective-score evidence appears only as a bar chart in Figure 8 with no numeric means, standard deviations, sample sizes per condition, or statistical tests, and Table 2 reports only 5-point Likert averages with no tests. Given the within-subject design with 30 participants, paired tests or mixed models should be reported for both objective scores and questionnaire responses; without them the reader cannot tell whether the visible differences are larger than participant noise.","section":"Human Experiment 2, Figure 8 and Table 2"},{"comment":"The core mechanism relies on hand-authored 'customized' corpora and a three-category taxonomy (domain knowledge, cognitive style, domain intention; field-dependent/independent; ingredient-preparation-oriented stable/random), but the manuscript gives no details on how these corpora were constructed, how categories were chosen, or how they were validated. Because the authors define the taxonomy and the corpora that are then used to evaluate the method, there is a circularity risk: the apparent adaptation may reflect that the test partners' behaviors were selected to fit the taxonomy. To make the central claim convincing, the authors should release the corpora and prompts, describe their construction and coverage, and provide at least one validation of the taxonomy, such as inter-rater agreement on style labels or a robustness check across taxonomy variants.","section":"Slow System (ToMknowledge, ToMstyle, ToMintention)"},{"comment":"No comparison is made with a prior ToM-based LLM agent, such as the mutual-ToM method of S. Zhang et al. (2024), or with a communication-enabled LLM agent, even though the introduction positions the work against these approaches. The w/o MsToM ablation shows that the slow system contributes within DPMT, but it does not show that the specific multi-scale mental-characteristic decomposition is better than a simpler single-stage ToM prompt or a prior ToM method. Adding such a baseline would make the contribution claim in the abstract more specific and would strengthen the paper's novelty argument.","section":"Experiments, Baselines"}],"minor_comments":[{"comment":"The caption contains the typo 'disicion-making' and should read 'decision-making'.","section":"Figure 1"},{"comment":"The figure caption includes an untranslated Chinese sentence ('心智模型建模智能体的心智特性—参考公共物品博弈任务中的心理变量'); all text in the paper should be in English.","section":"Figure 3"},{"comment":"The text says each participant collaborated with 'the DPMT and three baseline agents (A-D)', but four labels A-D are listed; the mapping from labels to specific agents should be stated explicitly.","section":"Human Experiment 2, Experiment Setup"},{"comment":"The bar chart lacks axis labels, error bars, and a statement of the score range; adding these would make the objective results interpretable.","section":"Figure 8"},{"comment":"The paper does not state whether the human experiments were approved by an ethics board or describe participant recruitment and exclusion criteria; such a statement is expected for studies with 30 human participants.","section":"Human Experiments, General"},{"comment":"There is no code, prompt, or corpus availability statement. Given that the method's behavior is determined largely by the custom prompts and corpora, releasing these artifacts or describing them in an appendix is important for reproducibility.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is within the journal's scope and could become a solid contribution if the empirical reporting is brought up to standard. In my view the main barrier is not the framework itself but the unsupported statistical language and the undocumented custom corpora. I would not reject outright, but I would ask the authors to provide full statistics for both human experiments, release or describe the corpora, and add at least one stronger ToM baseline before resubmission. I also note that several references are to the authors' own prior work; this is acceptable, but the novelty relative to their earlier LLM+ToM papers should be clarified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper's contribution is a dual-process architecture for LLM agents where a fast system makes quick decisions and a slow system reasons about a human partner's knowledge, cognitive style, and intention. That three-tier ToM hierarchy is a genuine organizational novelty, and the authors deserve credit for designing a system that separates fast action selection from deeper partner modeling. The ablation gap—DPMT roughly triples the score of the no-ToM variant on Bottleneck and Quick—is suggestive that the ToM module carries weight.\n\nThe problems are in the evidence. The abstract says 'significantly enhances' but Table 1 reports five runs per map with no tests. A rough t-test from the reported means and SDs gives uncorrected p≈0.17, 0.03, 0.26 across the three maps; none survives multiple-comparison correction. Experiment 2's objective scores appear only in a figure with no means, errors, or tests; the tabulated results are 5-point Likert self-reports. There is also no comparison against an existing ToM-based LLM agent—HLAgent is a non-ToM baseline, and mutual ToM or ProAgent would be the relevant reference points. The hand-built style taxonomy and corpora risk being tuned to the authors' own definitions, though that is a standard concern for this kind of work. Also, there is a stray Chinese line in the Figure 3 caption that looks like a draft note; that is minor but should be cleaned up.\n\nThe paper is worth reading for the framework, and the positive human-subject results are a decent start, but the central claim outruns the statistics. If this goes to review, the authors should be pushed to add inferential tests, a ToM-aware baseline, and the missing numerical details from Experiment 2.\n\nIt deserves a serious referee—the idea is good enough that the missing evidence can be supplied rather than being fatal.","headline":"The three-tier ToM hierarchy is a real organizational idea, but the 'significantly enhances' claim is not backed by the statistics as written.","tokens_in":10418,"tokens_out":2669,"would_cite":true,"duration_ms":29269,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a dual-process theory-of-mind framework, which pairs a fast small-LLM decision system with a slow multi-scale reasoning module that infers a human partner's domain knowledge, cognitive style, and intentions, improves…","keywords":["theory of mind","human-AI collaboration","dual process theory","large language models","multi-scale reasoning","Overcooked","partner modeling","real-time decision making"],"falsifier":"Have a human partner deliberately switch between style categories every few actions, or use behaviors not in the corpus, and compare DPMT to HLAgent on all three maps. If DPMT no longer scores higher, the claim that multi-scale ToM generalizes to unseen partners is refuted; if it still wins, the taxonomy assumption is not the source of the benefit.","tokens_in":9410,"feed_emoji":"🍳","tokens_out":5676,"duration_ms":61166,"temperature":0.7,"pith_summary":"The paper tries to establish that an AI agent can collaborate better with a human partner in real time if it maintains an explicit multi-scale theory of mind, inferring what the partner knows, how they tend to work, and what they intend to do next. This matters because current LLM agents often fail to adapt to unseen human behavior when direct communication is unavailable. DPMT implements the idea as a dual-process architecture: a fast small-LLM system picks macro-actions from token probabilities, while a slow large-LLM system reasons through three ToM stages. In Overcooked, DPMT scores 121, 101, and 104 on the Ring, Bottleneck, and Quick maps, compared with 99, 60, and 87 for the HLAgent baseline and 44, 23, and 9 when the ToM module is removed.","feed_headline":"Theory-of-mind AI out-scores baseline on all Overcooked maps","feed_subtitle":"A slow reasoning system models the partner's knowledge, style, and intentions; a fast system acts on it.","key_machinery":"The key machinery is the multi-scale ToM module, a three-stage LLM-based slow system: ToM_knowledge reasons about the partner's ingredient, order, and tool knowledge; ToM_style classifies the partner's cognitive style along dimensions such as field-dependent versus field-independent and stable versus random strategies; ToM_intention predicts the partner's short-term atomic actions and long-term macro-actions. Each stage is driven by a hand-authored corpus of mental characteristics and by the previous stage's prediction, and the resulting mental-state estimates kt, yt, and nt are injected into the fast system's prompt to shape action selection.","core_discovery":"The central claim is that the hierarchical ordering of three mental characteristics—domain knowledge, cognitive style, and domain intention—is what makes partner modeling effective for real-time human-AI collaboration. The slow system's three ToM stages each consume a different cue built from the language state and the partner's trajectory, and their outputs are passed to the fast system as text. The fast system then computes token probabilities over a predefined macro-action set and selects the action with the highest probability. The paper attributes the score improvements and the sharp ablation drops to this insertion of multi-scale ToM reasoning into the decision loop.","pith_inferences":["A testable extension the paper does not report is a latency-staleness experiment: refresh the slow system's reasoning only every N macro-actions and measure how much scores degrade as N grows, which would quantify how real-time the slow system must be.","A stronger generalization test would replace the hand-authored style corpus with a learned clustering of human trajectories; if the intention stage still helps, then the ToM mechanism rather than the specific taxonomy is the active ingredient.","The same three-stage mental-characteristic prompt could transfer to other real-time collaboration domains, such as shared warehouse robots or surgical teams, but the current corpora are Overcooked-specific, so transfer would require building new corpora for each domain."],"forward_implications":["DPMT should outperform an LLM agent that plans directly from environment state when interacting with partners whose behavior varies, because its intention predictions reduce redundant and blocking actions.","On maps with narrow passages or high congestion, the style-reasoning stage contributes the most; on maps with dense orders, the intention stage is the most load-bearing.","The fast system can run on a smaller, cheaper LLM without losing the collaboration benefit, as long as the slow system supplies accurate mental-state predictions.","Removing the multi-scale ToM module drops average scores to 44, 23, and 9, so the reasoning module rather than the base LLM alone drives the reported improvement.","The framework improves subjective judgments of game ability, collaboration fluidity, and perceived theory-of-mind ability compared with the HLAgent baseline."],"supporting_citations":[{"why":"Supplies the HLAgent baseline and the expanded Overcooked environment used in both human experiments.","marker":"Liu et al., 2023"},{"why":"Provides the mutual theory-of-mind collaboration setup and the subjective questionnaire template the paper adapts.","marker":"S. Zhang et al., 2024"},{"why":"Establishes machine theory of mind by predicting agents' future actions from trajectories, the foundation DPMT extends.","marker":"Rabinowitz et al., 2018"},{"why":"Gives the dual-process model of culture and action that motivates the fast/slow system split.","marker":"Vaisey, 2009"},{"why":"Defines theory of mind as understanding others' beliefs, the cognitive ability DPMT seeks to model.","marker":"Astington & Jenkins, 1995"},{"why":"Earlier partner-style modeling in multi-agent reinforcement learning that the paper contrasts with mental-characteristic modeling.","marker":"X. Li et al., 2023"}],"fun_headline_variants":["Hierarchical theory-of-mind boosts human-AI teamwork in real-time","Dual-process ToM: fast actions, slow reasoning for better collaboration","Three-level mind model improves real-time collaboration with humans","Slow system models partner at three scales; fast system acts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the hand-authored taxonomy of partner knowledge, cognitive styles, and intentions covers the behaviors of real human partners, so that any partner behavior outside those categories is invisible to the slow system and the claimed adaptation cannot happen.","fun_headline_variants_meta":{"raw":{"variants":["Hierarchical theory-of-mind boosts human-AI teamwork in real-time","Dual-process ToM: fast actions, slow reasoning for better collaboration","Three-level mind model improves real-time collaboration with humans","Slow system models partner at three scales; fast system acts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000862,"raw_usage":{"total_tokens":3665,"prompt_tokens":795,"completion_tokens":2870,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":411,"completion_tokens_details":{"reasoning_tokens":2798}},"tokens_in":411,"tokens_out":2870,"duration_ms":22858,"temperature":1.0,"reasoning_tokens":2798,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:01:17.231212+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a human partner deliberately switch between style categories every few actions, or use behaviors not in the corpus, and compare DPMT to HLAgent on all three maps. If DPMT no longer scores higher, the claim that multi-scale ToM generalizes to unseen partners is refuted; if it still wins, the taxonomy assumption is not the source of the benefit.","supporting_citations":[{"cited_title":", Perbet, F","cited_arxiv_id":null,"evidence_quote":"Establishes machine theory of mind by predicting agents' future actions from trajectories, the foundation DPMT extends."},{"cited_title":"APACrefauthors \\ 2009","cited_arxiv_id":null,"evidence_quote":"Gives the dual-process model of culture and action that motivates the fast/slow system split."},{"cited_title":"\\ Jenkins, J M","cited_arxiv_id":null,"evidence_quote":"Defines theory of mind as understanding others' beliefs, the cognitive ability DPMT seeks to model."}],"review_version":1}