REVIEW 4 major objections 5 minor 41 references
Human-AI Collaborative Bot Detection in MMORPGs
T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read An unsupervised pipeline embeds level-up intervals, clusters with DBSCAN, and uses an LLM to filter false positives, detecting auto-leveling bots with zero labels and an explainable trail.
desk verdict A practical, clearly-written bot detection pipeline undercut by a self-referential evaluation: the only outcome metric is also the tuning target, so the 'high accuracy' claim is unsubstantiated. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is the pair of a time-series embedding model and a density-based clusterer, with an LLM as a second reader. Each character $p$ is reduced to its level-up interval sequence $T(p)$, capped at level 50; a contrastive encoder (TS2Vec in the experiments) produces a latent vector $r(p)$, and DBSCAN with $\mathit{min\_sample}=3$ forms dense clusters while leaving irregular human play as noise. The DBSCAN radius $\varepsilon$ is picked as a quantile ($q=0.1$) of $k$-nearest-neighbor distances, following the prior framework that also supplies the evaluation metric. The LLM verification module takes each cluster's raw interval sequences, is prompted zero-shot with explicit sanction criteria and chain-of-thought steps, and returns which characters should be excluded; the growth-curve visualization shows the same data to human moderators.
What would settle it
Audit a random sample of accounts that the full pipeline flags with the lowest acc_info, using actual operator-side evidence such as payment method, device IDs, customer support tickets, and manual review. If a material share turn out to be legitimate players, such as shared family accounts or internet-cafe users with regular access patterns, then acc_info is not a valid bot signal and the paper's central claim fails; if instead nearly all audited accounts are confirmed automated farms, the claim survives.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that systematically repetitive level-up behavior is enough to separate bots from humans: a time-series representation model maps level-up interval sequences so that bot-like sequences land close together, DBSCAN finds dense clusters, and a GPT-4o reviewer filters the cluster using chain-of-thought reasoning over the raw intervals plus a growth-curve visualization. The authors claim this fully unsupervised process, with no labels and no per-title tuning beyond choosing $\varepsilon$, yields sanction candidate lists that are both accurate and explainable, as measured by lower access information homogeneity after LLM refinement across all three games. They position this as the first framework of its kind to replace the manual secondary judgment in prior work with an LLM, thereby cutting labeling and review costs.
Load-bearing premise
The load-bearing assumption is that a low access-information homogeneity score, meaning characters share near-identical login and access patterns, genuinely means the characters are bots run by the same player; if this proxy does not track real bot behavior, then neither the choice of $\varepsilon$ nor the reported improvement from the LLM filter establishes the central claim.
Editorial extensions
If this is right
- A game operator can run the full detection pipeline with zero annotated examples and receive daily lists of sanction candidates ready for an LLM-reviewed handoff to human moderators.
- Because the input is only level-up timestamps, the same pipeline should transfer to any MMORPG whose client logs level-up events, with no title-specific feature engineering.
- The LLM pass replaces the most repetitive part of a human game master's job, shifting human effort to higher-level decisions on the remaining clusters.
- The growth-curve plots give each candidate an audit trail, which addresses the explainability requirement that sanctions face in legal disputes.
Reading between the lines
- If access-information homogeneity is accepted as a valid bot signal, the same recipe of embedding plus density clustering plus LLM review could be applied to other repeated behavioral streams, such as quest completion logs or movement paths, not just level-up intervals.
- The current evaluation selects $\varepsilon$ to minimize the same acc_info score it later uses to judge success, so an independent ground-truth audit of flagged accounts would be needed to confirm that the LLM filter actually removes legitimate players.
- A self-hosted or smaller LLM would change the economics and latency of the second review; the paper's numbers were produced with a single commercial model and do not address how sensitive the filter is to prompt or model changes.
- One testable extension is to feed the LLM only the visualization rather than raw intervals, which would test whether the explainable artifact itself carries the verification signal.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes a fully unsupervised pipeline for detecting auto-leveling bots in MMORPGs. Level-up interval sequences are embedded with TS2Vec, clustered with DBSCAN, and then passed to GPT-4o as an auxiliary reviewer that is supposed to filter false positives from clusters. The authors also introduce a level-up interval visualization intended to support LLM and human moderation. The experiments use data from three commercial MMORPGs and report three evaluations: Kendall's Tau for representation robustness under synthetic perturbations, a clustering evaluation using access-information homogeneity (acc_info), and an LLM refinement evaluation using the same acc_info metric. The paper concludes that the method reduces labeling costs while achieving high accuracy.
Significance. If the central claim were supported, the framework would be practically valuable: an unsupervised, explainable bot-detection pipeline with LLM-assisted human oversight could reduce moderation costs in a commercial setting. The paper's orientation toward explainability and legal defensibility is appropriate for the domain, and the use of real operational data from three games is a strength. The perturbation-based robustness check for the representation model is a reasonable auxiliary validation. However, the evaluation never measures detection accuracy against known bot labels. The primary metric, acc_info, comes from the authors' prior work and is used both to select the DBSCAN epsilon and to claim that LLM refinement improves results, making the central quantitative claim self-referential. No human baseline is provided for the LLM verification step. As a result, the paper does not establish that the pipeline produces reliable bot sanctions.
major comments (4)
- [§4.3.1–4.4.2, Table 2] The evaluation is self-referential. The primary metric acc_info is introduced in the authors' prior work [13] and is used both to select the DBSCAN epsilon (q=0.1 is recommended because it gives lower acc_info) and to claim that LLM refinement improves the results. Because acc_info measures homogeneity of access/login patterns among characters in the same cluster, any filter that removes access-pattern outliers will tend to lower acc_info regardless of whether those outliers are human or bot. The manuscript provides no ground-truth labels, no precision/recall/F1, and no human-verification baseline, so the abstract's 'high accuracy' claim is not established.
- [§4.2, Table 1] The Kendall's Tau experiment only validates that TS2Vec preserves distance rankings between original and synthetically perturbed level-up sequences. This does not test whether bot representations are separable from human representations, nor whether clusters discovered by DBSCAN correspond to actual bot groups. Thus, Table 1 cannot support the clustering results in Section 4.3, and the perturbation experiment is not evidence of detection performance.
- [§3.3 and §4.4] The LLM verification component is evaluated only indirectly through the same acc_info metric. There is no evaluation of whether GPT-4o's inclusion/exclusion decisions agree with human moderators, no per-cluster decision outcomes, no error analysis, and no full prompt is provided. Consequently, the claim that the LLM 'effectively mimics secondary human judgment' is unsupported.
- [Table 2 and §4.3.2] The comparisons across epsilon settings and LLM on/off are reported as daily averages without standard deviations, confidence intervals, or significance tests. The #Det column contains decimal values despite being a count, which obscures the underlying variability. Without these, the practical magnitude of the LLM improvement and the robustness of the q=0.1 recommendation cannot be assessed.
minor comments (5)
- [§3.1.1] The heading contains a typo: 'Data preperation' should be 'Data preparation'.
- [§3.2.1] The heading contains a typo: 'Represntation' should be 'Representation'.
- [Table 2] The column header 'IP' is not defined; based on the table and text it appears to be a placeholder for the game dataset, but this should be stated explicitly.
- [§3.3.2] The prompt engineering section describes the strategy but does not include the actual prompt or a representative input/output example, which limits reproducibility of the LLM component.
- [§4.3.1] The statement that acc_info has a minimum value of 1 is referenced to [13] but no formula or derivation is provided here; a self-contained definition would help the reader understand what is being optimized.
Circularity Check
Evaluation of 'high accuracy' is self-referential: DBSCAN eps is tuned to minimize acc_info, and the same unvalidated acc_info metric from the authors' prior work is then used to validate the LLM filter.
-
fitted input called prediction
[Section 4.3.2 (Clustering results), Table 2; Section 4.4.2 (Effectiveness of LLM-based refinement)]
"The results indicate that setting q = 0.1 generally leads to lower acc_info scores across game datasets, indicating lower-risk clustering. ... As this study prioritizes minimizing false positives, we recommend q = 0.1. ... Experimental results show that the LLM effectively filtered out normal users from the sanction candidates in an appropriate direction, leading to a meaningful reduction in the access information homogeneity score across all three games."
The paper's accuracy claim rests on Table 2, but the DBSCAN epsilon is selected on that same table by minimizing acc_info ('we recommend q = 0.1'), and the LLM's benefit is then measured as a further reduction of the same acc_info. Acc_info is an unvalidated proxy for bot identity; no ground-truth labels or human baseline are used. Thus the reported 'high accuracy' is the value of the objective that was optimized, not an independent test of detection correctness.
-
self citation load bearing
[Section 4.3.1 (Evaluation metric)]
"Access information homogeneity, proposed in [13], quantifies behavioral similarity between characters based on their login/access patterns. Lower values indicate stronger similarity, suggesting that the characters may be controlled by the same player. The minimum value of access information homogeneity is 1, as noted in [13], and values closer to 1 are interpreted as better performance."
The interpretation that lower acc_info indicates bots is imported from [13], a paper co-authored by the present authors (Hyunsoo Kim and Jaeman Son). This paper provides no validation of acc_info against ground-truth bot labels, so the central claim that the pipeline achieves 'high accuracy' rests on an unverified metric whose authority is a self-citation.
full rationale
The paper's central claim is that the pipeline 'reduces labeling costs while achieving high accuracy.' The only quantitative support is Table 2, which reports access information homogeneity (acc_info). Section 4.3.2 chooses q = 0.1 because it gives lower acc_info, and Section 4.4.2 declares the LLM refinement successful because it further lowers acc_info. This is a fitted-input-called-prediction loop: the reported performance metric is the objective used to select the hyperparameter, evaluated on the same data. In addition, the interpretation of acc_info as indicating bots is imported from [13], a paper by the same authors, with no ground-truth validation in this paper. The Kendall's tau experiment in Section 4.2 validates representation robustness on synthetic perturbations, not bot detection, so it does not break the loop. The non-circular components (TS2Vec, DBSCAN, zero-shot LLM verification) are real, but the paper's accuracy claim reduces to the self-optimized proxy, warranting a score of 6.
Assumptions & free parameters
free parameters (3)
- DBSCAN min_sample =
3
- DBSCAN epsilon (via quantile q) =
q=0.1 (recommended); q=0.2 explored
- Level-up sequence cap =
50
assumptions (6)
- domain assumption Auto-leveling bots produce highly consistent level-up interval sequences due to optimized routes, while human players show more variation.
- domain assumption Similar level-up sequences map to similar representation vectors, allowing DBSCAN to separate bot clusters from human noise.
- domain assumption Access information homogeneity is a valid indicator of bot identity; lower values imply characters are likely controlled by the same player or bot.
- domain assumption GPT-4o can accurately distinguish auto-leveled bots from legitimate players in a zero-shot setting, given level-up interval lists.
- domain assumption Leveling behavior below level 50 is a stable signal, while higher levels are confounded by social and PvP activity even for bots.
- domain assumption Characters with missing level-up logs or paid-item-influenced level-ups should be excluded from analysis.
Cite this review
Pith. "Pith review of Human-AI Collaborative Bot Detection in MMORPGs." pith.science (2026). https://pith.science/paper/FXSTSC6V
@misc{pith2026250820578,
author = {Pith},
title = {Pith review of: Human-AI Collaborative Bot Detection in MMORPGs},
year = {2026},
howpublished = {\url{https://pith.science/paper/FXSTSC6V}},
note = {Machine review of arXiv:2508.20578}
}
read the original abstract
In Massively Multiplayer Online Role-Playing Games (MMORPGs), auto-leveling bots exploit automated programs to level up characters at scale, undermining gameplay balance and fairness. Detecting such bots is challenging, not only because they mimic human behavior, but also because punitive actions require explainable justification to avoid legal and user experience issues. In this paper, we present a novel framework for detecting auto-leveling bots by leveraging contrastive representation learning and clustering techniques in a fully unsupervised manner to identify groups of characters with similar level-up patterns. To ensure reliable decisions, we incorporate a Large Language Model (LLM) as an auxiliary reviewer to validate the clustered groups, effectively mimicking a secondary human judgment. We also introduce a growth curve-based visualization to assist both the LLM and human moderators in assessing leveling behavior. This collaborative approach improves the efficiency of bot detection workflows while maintaining explainability, thereby supporting scalable and accountable bot regulation in MMORPGs.
Figures
Reference graph
Works this paper leans on
-
[13]
Hyunsoo Kim, Jun Hee Kim, Jaeman Son, Jihoon Song, and Eunjo Lee. 2025. A Framework for Mining Collectively-Behaving Bots in MMORPGs. InInternational Conference on Pattern Recognition . Springer, 400–419
work page 2025
-
[1]
Kuan-Ta Chen, Andrew Liao, Hsing-Kuo Kenneth Pao, and Hao-Hua Chu. 2008. Game bot detection based on avatar trajectory. In International Conference on Entertainment Computing. Springer, 94–105
work page 2008
-
[2]
2023.{BotScreen}: Trust Everybody, but Cut the Aimbots Yourself
Minyeop Choi, Gihyuk Ko, and Sang Kil Cha. 2023.{BotScreen}: Trust Everybody, but Cut the Aimbots Yourself. In 32nd USENIX Security Symposium (USENIX Security 23). 481–498
work page 2023
-
[3]
Winnie Chow, Lauren Gardiner, Haraldur T Hallgrímsson, Maxwell A Xu, and Shirley You Ren. 2024. Towards time series reasoning with llms. arXiv preprint arXiv:2409.11376 (2024)
arXiv 2024
-
[4]
Manqing Dong, Hao Huang, and Longbing Cao. 2024. Can LLMs Serve As Time Series Anomaly Detectors? arXiv preprint arXiv:2408.03475 (2024)
arXiv 2024
-
[5]
Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. 1996. A density- based algorithm for discovering clusters in large spatial databases with noise. In kdd, Vol. 96. 226–231
1996
-
[6]
Nate Gruver, Marc Finzi, Shikai Qiu, and Andrew G Wilson. 2023. Large language models are zero-shot time series forecasters. Advances in Neural Information Processing Systems 36 (2023), 19622–19635
2023
-
[7]
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al . 2024. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594 (2024)
arXiv 2024
Show all 41 references
-
[8]
Jun-Sok Huhh. 2008. Simple economics of real-money trading in online games. A vailable at SSRN 1089307 (2008)
2008
-
[9]
Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y Zhang, Xiaoming Shi, Pin-Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, et al . 2023. Time-llm: Time series forecasting by reprogramming large language models. arXiv preprint arXiv:2310.01728 (2023)
2023 arXiv
-
[10]
Anssi Kanervisto, Tomi Kinnunen, and Ville Hautamäki. 2022. Gan-aimbots: Using machine learning for cheating in first person shooters. IEEE Transactions on Games 15, 4 (2022), 566–579
2022
-
[11]
Ah Reum Kang, Seong Hoon Jeong, Aziz Mohaisen, and Huy Kang Kim. 2016. Multimodal game bot detection using user behavioral characteristics.SpringerPlus 5 (2016), 1–19
2016
-
[12]
M. G. Kendall. 1938. A New Measure of Rank Correlation. Biometrika 30, 1/2 (1938), 81–93
1938
-
[14]
Yaxuan Kong, Yiyuan Yang, Yoontae Hwang, Wenjie Du, Stefan Zohren, Zhangyang Wang, Ming Jin, and Qingsong Wen. 2025. Time-MQA: Time Series Multi-Task Question Answering with Context Enhancement. arXiv preprint arXiv:2503.01875 (2025)
2025 arXiv
-
[15]
Hyukmin Kwon, Aziz Mohaisen, Jiyoung Woo, Yongdae Kim, Eunjo Lee, and Huy Kang Kim. 2016. Crime scene reconstruction: Online gold farming network analysis. IEEE Transactions on Information Forensics and Security 12, 3 (2016), 544–556
2016
-
[16]
Eunjo Lee, Jina Lee, and Janghwan Kim. 2011. Detecting the bank character in mmorpgs by analysis of a clustered network. In The 3rd International Conference on Internet
2011
-
[17]
Eunjo Lee, Jiyoung Woo, Hyoungshick Kim, and Huy Kang Kim. 2018. No silk road for online gamers! using social network analysis to unveil black markets in online games. In Proceedings of the 2018 World Wide Web Conference . 1825–1834
2018
-
[18]
Eunjo Lee, Jiyoung Woo, Hyoungshick Kim, Aziz Mohaisen, and Huy Kang Kim. 2016. You are a Game Bot!: Uncovering Game Bots in MMORPGs via Self-similarity in the Wild.. In Ndss. 1–15
2016
-
[19]
Jun Liu, Chaoyun Zhang, Jiaxu Qian, Minghua Ma, Si Qin, Chetan Bansal, Qingwei Lin, Saravan Rajmohan, and Dongmei Zhang. 2024. Large language models can deliver accurate and interpretable time series anomaly detection. arXiv preprint arXiv:2405.15370 (2024)
2024 arXiv
-
[20]
Yilun Liu, Shimin Tao, Weibin Meng, Jingyu Wang, Wenbing Ma, Yuhang Chen, Yanqing Zhao, Hao Yang, and Yanfei Jiang. 2024. Interpretable online log analysis using large language models with prompt strategies. In Proceedings of the 32nd IEEE/ACM International Conference on Progr...
2024
-
[21]
Yilun Liu, Shimin Tao, Weibin Meng, Feiyu Yao, Xiaofeng Zhao, and Hao Yang
-
[22]
Mike A Merrill, Mingtian Tan, Vinayak Gupta, Tom Hartvigsen, and Tim Althoff
-
[23]
Hsing-Kuo Pao, Kuan-Ta Chen, and Hong-Chung Chang. 2010. Game bot detec- tion via avatar trajectory analysis.IEEE Transactions on Computational Intelligence and AI in Games 2, 3 (2010), 162–175
2010
-
[24]
arXiv preprint arXiv:2404.11757 (2024)
Language models still struggle to zero-shot reason about time series. arXiv preprint arXiv:2404.11757 (2024)
2024 arXiv
-
[25]
Jiashu Pu, Jianshi Lin, Xiaoxi Mao, Jianrong Tao, Xudong Shen, Yue Shang, and Runze Wu. 2022. Unsupervised representation learning of player behavioral data with confidence guided masking. In Proceedings of the ACM Web Conference 2022 . 3396–3406
2022
-
[26]
José Pedro Pinto, André Pimenta, and Paulo Novais. 2021. Deep learning and multivariate time series for cheat detection in video games. Machine Learning 110, 11 (2021), 3037–3057
2021
-
[27]
Erich Schubert, Jörg Sander, Martin Ester, Hans Peter Kriegel, and Xiaowei Xu
-
[28]
Xianyang Qi, Jiashu Pu, Shiwei Zhao, Runze Wu, and Jianrong Tao. 2022. A GNN- Enhanced Game Bot Detection Model for MMORPGs. In Advances in Knowledge Discovery and Data Mining: 26th Pacific-Asia Conference, PAKDD 2022, Chengdu, China, May 16–19, 2022, Proceedings, Part II . Sp...
2022
-
[29]
Jianrong Tao, Jianshi Lin, Shize Zhang, Sha Zhao, Runze Wu, Changjie Fan, and Peng Cui. 2019. Mvan: Multi-view attention networks for real money trading detection in online games. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mini...
2019
-
[30]
Jianrong Tao, Jiarong Xu, Linxia Gong, Yifu Li, Changjie Fan, and Zhou Zhao
-
[31]
Yueyang Su, Di Yao, Jingwei Li, Baoli Wang, Jingping Bi, Shiwei Zhao, Runze Wu, Jianrong Tao, and Hao Deng. 2022. Trajectory-Based Mobile Game Bots Detection with Gaussian Mixture Model. InArtificial Neural Networks and Machine Learning– ICANN 2022: 31st International Conferen...
2022
-
[32]
Zhe Xie, Zeyan Li, Xiao He, Longlong Xu, Xidao Wen, Tieying Zhang, Jianjun Chen, Rui Shi, and Dan Pei. 2024. ChatTS: Aligning Time Series with LLMs via Synthetic Data for Enhanced Understanding and Reasoning. arXiv preprint arXiv:2412.03104 (2024)
2024
-
[33]
Jiarong Xu, Yifan Luo, Jianrong Tao, Changjie Fan, Zhou Zhao, and Jiangang Lu. 2020. Nguard+ an attention-based game bot detection framework via player behavior sequences. ACM Transactions on Knowledge Discovery from Data (TKDD) 14, 6 (2020), 1–24
2020
-
[34]
Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. 2022. Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI conference on artificial intelligence , Vol. 36. 8980–8987
2022
-
[35]
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824–24837
2022
-
[36]
Sha Zhao, Junwei Fang, Shiwei Zhao, Runze Wu, Jianrong Tao, Shijian Li, and Gang Pan. 2022. T-Detector: A Trajectory based Pre-trained Model for Game Bot Detection in MMORPGs. In 2022 IEEE 38th International Conference on Data Engineering (ICDE). IEEE, 992–1003
2022
-
[37]
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems 36 (2023), 46595–46623
2023
-
[39]
Haochuan Zhang, Chunhua Yang, Jie Han, Liyang Qin, and Xiaoli Wang. 2025. TempoGPT: Enhancing Temporal Reasoning via Quantizing Embedding. arXiv preprint arXiv:2501.07335 (2025)
2025 arXiv
-
[2017]
ACM Transactions on Database Systems (TODS) 42, 3 (2017), 1–21
DBSCAN revisited, revisited: why and how you should (still) use DBSCAN. ACM Transactions on Database Systems (TODS) 42, 3 (2017), 1–21
2017
-
[2018]
In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining
NGUARD: a game bot detection framework for NetEase MMORPGs. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining . 811–820
-
[2024]
In Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings
Logprompt: Prompt engineering towards zero-shot and interpretable log analysis. In Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings . 364–365
2024
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.