{"id":"94a48441-4dcb-4a0a-8d05-7c97a04e3157","arxiv_id":"2505.20011","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A deep CNN+RNN model distinguishes human players from a specific game bot in a tactical game with F1=0.92, alongside a survey of 13 challenges in human-like game AI.","lead":"This paper surveys 13 challenges in building human-like game AI and trains a deep neural network to tell human players from bots in a tactical game, reaching an F1 score of 0.92. The practical upshot is a candidate baseline for bot detection in games and an automated tool for measuring how human-like an AI player is.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"F1=0.92 claim is uninterpretable without defining whether the classification unit is a match or a player; mixed AI-vs-human matches cannot have a single Human/Bot label.","rationale":"The survey portion is a reasonable synthesis and the paper is honest in framing the hypothesis as open. The empirical claim is the load-bearing element. The reader's weakest assumption is correct and central: the classification unit is never explicitly defined. The data section says '93,195 logs from ... matches' and the XGBoost baseline used '20 summary features per match', strongly implying each sample is a match. But the class labels are {Human, Bot}, which only makes sense at player level. In AI-vs-Human matches, a match-level label is ambiguous. If the authors labeled mixed matches as 'Human', the task becomes 'detect whether a match contains a human', which is easier and not the stated research question. The paper also omits the output layer and loss function, so it is not possible to infer the unit from the architecture. I also noticed an internal inconsistency: the defined macro-F1 cannot be 0.92 with the reported Precision(H)=0.87 and Recall(H)=0.81 unless the bot-class F1 is close to 1.0, which is not reported; this suggests the reported 'F1 Score' may be a different metric, further undermining the comparison to XGBoost's 0.58. The proposed concrete check—evaluating per-side predictions on mixed matches—would settle whether the model actually distinguishes individual players. This is a clarification/analysis request rather than a dismissal; the paper has a reproducible-looking setup and the survey is useful. Therefore the verdict remains conditional.","tokens_in":14978,"tokens_out":8328,"duration_ms":90369,"concrete_test":"Evaluate the trained model on the 2,190 AI-vs-Human matches only, treating each match as two player-level trajectories and labeling each side as Human or Bot. Report the per-side confusion matrix and macro-F1. If the model cannot separate the two sides within the same match (e.g., per-player F1 well below 0.92), the headline claim fails. As a secondary check, recompute the reported F1 from the published Precision(H) and Recall(H) using Eq. 2 to confirm the table is internally consistent.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that the CNN+RNN model distinguishes human players from bots with F1=0.92. The paper reports 93,195 logs, one per match, but the classes are {Human, Bot}. For the 2,190 AI-vs-Human matches, no single match-level ground-truth label exists: each match contains one human and one bot. If the label is instead 'contains at least one human', the model solves a match-level presence-detection task, not player-level discrimination, and F1=0.92 does not support the stated conclusion that individual players are distinguished. The paper does not describe the output layer, loss, or how mixed matches are labeled, so the unit is undefined. Additionally, the metric is suspect: Eq. 1 defines macro-F1, but with Precision(H)=0.87 and Recall(H)=0.81, F1(H)=0.84; a macro-F1 of 0.92 would require F1(Bot)≈1.0, which is not reported and is implausible given the minority class. The comparison to XGBoost is also confounded: the XGBoost model used 800 training matches while the DNN used 93,195 logs, so the F1 increase may reflect data volume rather than architecture.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper has two parts: a survey of thirteen challenges in creating human-like AI in games, and an empirical study in the tactical game Tactical Troops: Anthracite Shift. The empirical study trains a deep network combining convolutional and recurrent components on match logs to distinguish human players from bots, reporting a macro-F1 of 0.92, a substantial improvement over an earlier XGBoost model (F1=0.58). The authors frame the result as evidence that their architecture significantly outperforms the earlier approach and propose an inverse relationship between the difficulty of creating human-like agents and the ease of detecting them.","tokens_in":15270,"tokens_out":6067,"duration_ms":63272,"significance":"If the reported F1=0.92 is valid and the classification task is what the paper claims, the method would be a useful contribution to bot-detection and human-likeness evaluation in game environments. The survey, based on a literature search of 54 papers, organizes common challenges in a useful way and may serve as a reference for researchers. However, the empirical result is the load-bearing claim, and its current presentation has several ambiguities that prevent the reader from interpreting the number. The paper does offer a concrete falsifiable hypothesis and a planned public competition, which are strengths.","major_comments":[{"comment":"The classification unit is undefined. The text states that the dataset comprises 93,195 logs from 89,667 AI-vs-AI matches, 2,190 AI-vs-Human matches, and 1,338 Human-vs-Human matches. Since the classes are {Human, Bot}, an AI-vs-Human match contains both a human and a bot and cannot be assigned a single ground-truth label of either class. The paper must state whether each log is classified at match level or player level, and how the 2,190 mixed matches are labeled (e.g., as 'contains at least one human'). If the label is match-level human presence, then F1=0.92 does not support the conclusion that the model distinguishes individual human players from bots; it only detects whether a human participated in the match.","section":"Section 4.1, data description"},{"comment":"The claim that the deep learning model 'clearly indicates ... significantly outperformed' the XGBoost model is confounded by data size and feature set. The XGBoost model was trained on 800 matches with 20 summary features, while the deep model was trained on 93,195 logs with multimodal sequence inputs. The F1 increase from 0.58 to 0.92 could be due to the larger training set and richer features rather than the architectural choice. A controlled comparison using the same training data and comparable feature sets, or at least a discussion of this limitation, is needed before making the stronger claim.","section":"Section 4.1, comparison with XGBoost"},{"comment":"The reported numbers are internally inconsistent under the stated metric definition. With Precision(H)=0.87 and Recall(H)=0.81, Eq. (2) gives F1(H)=0.84. The reported Macro-F1 of 0.92 therefore requires F1(Bot)≈1.0, a per-class score that is not reported. Given the large majority of AI-vs-AI matches, a model that classifies almost all matches as bot would achieve high bot-class F1 while performing more poorly on humans. The authors should report per-class precision, recall, and F1 for both classes, together with the confusion matrix and fold-wise variance, so that the macro-F1 is interpretable.","section":"Section 4.1, Eqs. (1)-(4) and Table 1"},{"comment":"The construction of the cross-validation folds is not described. If individual players contribute multiple matches, a random split of logs can place logs from the same player into both training and test folds, leading to optimistic performance due to player-identity leakage. The paper should specify whether logs are grouped by player (or at least by match) before splitting, and how the 2,190 mixed matches are distributed across folds.","section":"Section 4.1, 5-fold cross-validation"},{"comment":"The neural network architecture is described only textually; essential details for reproducibility are omitted. The paper does not specify the output layer, loss function, optimizer, training epochs, batch size, number of layers, or activation functions, and Fig. 2 is referenced but not included in the text provided. Without these details, the reported F1 cannot be independently replicated or assessed for robustness to implementation choices.","section":"Section 4, model description"}],"minor_comments":[{"comment":"The notation in Eq. (1) uses 'Macro|F1', which is unconventional, and the text says 'TN_X, TN_X, TN_X' where it should list true negatives, false positives, and false negatives. Please correct the typography and the variable names.","section":"Section 4.1, Eq. (1) and surrounding text"},{"comment":"The phrase 'possess a certain level of skill' contains a spelling error: 'posses' should be 'possess'.","section":"Section 2, 'Avoiding Superhuman Behavior'"},{"comment":"The hypothesis about the inverse relationship between challenge of creation and ease of detection is stated as an open hypothesis, but the paper does not discuss how it might be empirically tested using the existing data or the planned competition. A brief operationalization would make the hypothesis more useful.","section":"Section 5"},{"comment":"The survey methodology is described briefly, but the final number of papers (54) is mentioned after the selection step. A short note on inclusion/exclusion criteria (e.g., reading abstracts and introductions) would strengthen the survey's transparency.","section":"Introduction"}],"recommendation":"major_revision","confidential_remarks":"The paper combines a survey and an empirical study. The survey is reasonable but not novel in a deep way. The empirical result, as currently presented, is not interpretable due to the undefined classification unit and the confounded baseline. These issues are fixable with clarification and additional experiments, but the central empirical claim must be re-framed or re-analyzed. If the classification task is actually match-level human presence, the contribution is significantly weaker than claimed. The paper would also benefit from releasing the data or providing more experimental detail, especially since the authors plan a public competition."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The survey part is a solid, well-organized synthesis of thirteen challenges in believable game agents; it reads like a good practitioner's map of the literature and includes concrete examples from the authors' own game. The empirical part, which is the headline, does not support the F1=0.92 claim as stated.\n\nWhat's actually new: the CNN+RNN application to Tactical Troops and the associated dataset are new, and the idea of combining spatial map layers with numeric sequences is sensible. The authors are also transparent about the severe class imbalance and report precision/recall for the minority human class.\n\nThe problems are real and load-bearing. First, the classification unit is never defined. The dataset is described as logs of matches (89,667 bot-vs-bot, 2,190 bot-vs-human, 1,338 human-vs-human). A bot-vs-human match has no single ground-truth label in a {Human, Bot} scheme. The paper doesn't say how these samples were labeled, what the output layer is, or whether the unit is a match, a player, or a state sequence. If the model is trained to predict 'match contains at least one human,' then the result is about presence detection, not about distinguishing individual players, and the abstract overstates it.\n\nSecond, the reported metrics cannot all be true for a binary classifier under the paper's own macro-F1 definition. Precision(H)=0.87 and Recall(H)=0.81 give F1(H)=0.84. Macro-F1=0.92 then forces F1(Bot)≈1.0, which in a binary setting would mean no bot sample is ever misclassified as human, which would make Precision(H)=1.0. The table is internally inconsistent. That is a referee-level arithmetic error, not a quibble.\n\nThird, the XGBoost comparison is confounded: 800 training matches versus 93,195 logs, and different feature sets. The improvement could be data volume or input richness, not the architecture. No error bars are given, and no code or data are released, though they promise a future competition.\n\nThe citation pattern looks fine; the heavy use of their earlier bot paper [49] is legitimate since it defines the environment and the baseline.\n\nWho gets value: practitioners wanting a digest of believability challenges, and students of game AI. The empirical result needs major revision before it can be used as a benchmark.\n\nMy recommendation: send it to peer review, but condition acceptance on a rewritten empirical section. The survey alone is publishable as a short paper; the detection result needs a clear definition of the sample unit, a consistent confusion matrix, a matched baseline, and released artifacts.","headline":"Useful believability survey, but the F1=0.92 detection result is unsupported because the classification unit is undefined and the reported metrics conflict.","tokens_in":15789,"tokens_out":7545,"would_cite":false,"duration_ms":72680,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A hybrid CNN-RNN model trained on map images and match statistics tells human players from game bots with an F1 score of 0.92, up from 0.58 for a feature-based baseline.","keywords":["Human-Like AI","Believable Agents","Bot Detection","Video Game AI","Convolutional Neural Network","Recurrent Neural Network","Behavioral Traces","Tactical Troops: Anthracite Shift"],"falsifier":"Re-run the network with the sample unit set to an individual player and with all states from one match kept inside the same cross-validation fold, then compare per-player precision and recall. If the F1 of 0.92 cannot be reproduced under that protocol, or if the model only detects matches that contain at least one human, the claim that individual humans are distinguished from bots is falsified.","tokens_in":14797,"feed_emoji":"🎮","tokens_out":10686,"duration_ms":114674,"temperature":0.7,"pith_summary":"This paper tries to establish that human-likeness in game agents is an empirical property that can be measured from gameplay data, at least in games with a sufficiently expressive action space. It makes the case in two ways: a survey that distills thirteen recurring challenges in building believable game AI, and an experiment in the tactical game Tactical Troops: Anthracite Shift where a hybrid convolutional-recurrent network is trained to label players as human or bot. On five-fold cross-validation the full network reaches an F1 score of 0.92, up from 0.58 for an earlier model built from twenty numerical features, with the recurrent component contributing most of the signal and the convolutional component adding precision. The paper closes by proposing an open hypothesis: the harder a game makes it to build human-like AI, the easier it should be to build a method that tells humans and that AI apart.","feed_headline":"Hybrid neural net tells human players from bots with F1 0.92","feed_subtitle":"The spatial-plus-temporal model beats the prior 0.58 baseline, making believability measurable from gameplay traces.","key_machinery":"The load-bearing mechanism is the hybrid deep network, fed with multimodal inputs over time. A convolutional subnetwork extracts spatial features from six two-dimensional map layers per game state, while a recurrent subnetwork processes a vector of per-player numerical features; the two streams are combined and the whole input is repeated over the last 250 states so the model can exploit both positioning and temporal dynamics. The ablation table shows the two branches are not redundant: the recurrent branch alone gives F1 0.88 but lower precision, and adding the convolutional branch raises precision sharply while nudging F1 to 0.92. The paper's conceptual machinery is the list of thirteen challenges, which explains why human-likeness is hard to build and why certain bot tells, such as superhuman precision, unrealistic reaction times, and repeated mistakes, make detection feasible.","core_discovery":"The central discovery reported here is that a deep network combining convolutional and recurrent layers can distinguish human players from AI-controlled players in Tactical Troops: Anthracite Shift using only logged game states. The network sees six spatial map layers that encode obstacles, rooftops, teleporters, control points, and the health of friendly and enemy units, alongside ten per-player scalar features such as turn number, damage dealt and received, friendly-fire ratio, grenade use, and status changes; these inputs are processed as sequences of up to 250 consecutive states so that temporal dynamics matter. The full model achieves a macro-F1 of 0.92, with human-class precision 0.87 and recall 0.81, compared with 0.58 for the earlier model; the recurrent branch alone scores 0.88 and the convolutional branch alone 0.59. The authors interpret this as evidence that believability has measurable behavioral signatures and that a detector can be built more easily in games where creating human-like agents is harder.","pith_inferences":["A testable consequence the paper leaves open: running the same detector across games with different action-space expressiveness should reproduce the inverse correlation, with near-chance performance in minimal games like rock-paper-scissors and higher scores in open-ended worlds.","The reported result is probably tied to this specific bot: a bot deliberately built to respect the paper's thirteen challenges, such as imperfect aiming, bounded reaction time, and non-repeated mistakes, could force the detector's F1 down, implying a natural adversarial loop between bot designers and detector builders.","The data contains only a few thousand matches with at least one human player among more than ninety thousand matches, so the human-class metrics rest on a small minority of the data; per-player labels and match-level cross-validation splits would settle whether the model separates individuals or merely detects the presence of a human in a match."],"forward_implications":["Human-likeness in an expressive game can be scored automatically from logged gameplay, without human judges, because the learned detector reaches F1 0.92 on held-out gameplay states.","The same architecture can serve as a quality-assurance loop: weak bots are flagged as non-human, then the bot is improved until the detector mistakes it for a human, and the detector is retrained on the new bot.","Bot detection in games and virtual environments that prohibit automated players can run on spatial and temporal traces alone, using the six map layers and numeric feature streams described here.","Believability scores will not transfer across games: the paper argues that a bot's detectability depends on the environment's expressiveness and on how the bot was built, so comparing human-likeness numbers between games is not meaningful."],"supporting_citations":[{"why":"Describes the game, its hybrid Utility AI and Monte Carlo Tree Search agent, and the earlier twenty-feature model whose F1 of 0.58 is the baseline this paper improves on.","marker":"[49]"},{"why":"Introduces the first competition that frames human-likeness as a human-versus-bot judgment task, defining the problem the detector is built for.","marker":"[18]"},{"why":"Reports an earlier human-versus-bot discrimination result in another game that the paper cites as the closest comparison.","marker":"[34]"},{"why":"Motivates automated human-likeness assessment by connecting it to the imitation-game tradition.","marker":"[47]"},{"why":"Documents concrete bot tells such as collecting items without looking and repeating mistakes, the kind of signals the learned model can exploit.","marker":"[43]"},{"why":"Reviews the Monte Carlo Tree Search algorithm that forms the bot's tactical layer, defining the behavior the detector must learn to recognize.","marker":"[48]"}],"fun_headline_variants":["Neural net spots humans in games with 0.92 F1","Game bot detector hits F1 0.92 using player logs","Why bots can't hide: deep net scores 0.92 F1","From game logs to bot detection: F1 0.92","Human-like bots exposed by 0.92 F1 detector"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result assumes that each training sample is labeled for an individual player rather than for a whole match, and that samples from the same match are never split between training and test folds; if either assumption fails, an F1 of 0.92 would not show that individual humans are being distinguished from bots.","fun_headline_variants_meta":{"raw":{"variants":["Neural net spots humans in games with 0.92 F1","Game bot detector hits F1 0.92 using player logs","Why bots can't hide: deep net scores 0.92 F1","From game logs to bot detection: F1 0.92","Human-like bots exposed by 0.92 F1 detector"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000403,"raw_usage":{"total_tokens":2121,"prompt_tokens":987,"completion_tokens":1134,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":1041}},"tokens_in":603,"tokens_out":1134,"duration_ms":10711,"temperature":1.0,"reasoning_tokens":1041,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:01:16.806426+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the network with the sample unit set to an individual player and with all states from one match kept inside the same cross-validation fold, then compare per-player precision and recall. If the F1 of 0.92 cannot be reproduced under that protocol, or if the model only detects matches that contain at least one human, the claim that individual humans are distinguished from bots is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the first competition that frames human-likeness as a human-versus-bot judgment task, defining the problem the detector is built for."},{"cited_title":"Sánchez-Ruiz-Granados, and Federico Peinado","cited_arxiv_id":null,"evidence_quote":"Reports an earlier human-versus-bot discrimination result in another game that the paper cites as the closest comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates automated human-likeness assessment by connecting it to the imitation-game tradition."},{"cited_title":"Karpov, and Risto Miikkulainen","cited_arxiv_id":null,"evidence_quote":"Documents concrete bot tells such as collecting items without looking and repeating mistakes, the kind of signals the learned model can exploit."}],"review_version":1}