Show Me Your Account: Detecting MMORPG Game Bot Leveraging Financial Analysis with LSTM

Eunjo Lee; Huy Kang Kim; Kyung Ho Park

REVIEW 4 major objections 5 minor 14 references

Show Me Your Account: Detecting MMORPG Game Bot Leveraging Financial Analysis with LSTM

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a character's financial-status time series—cash, inventory, bank, and mailing balances—is enough for an LSTM to separate game bots from normal players, reporting an average F1-score of 0.9501 on Aion data.

desk verdict Sensible idea, unsupported numbers: the LSTM financial-status bot detector leaks character identity into its cross-validation, so the headline F1 is not credible without re-evaluation. read the letter →

arxiv 1908.03748 v1 pith:4I52A7BJ submitted 2019-08-10 cs.CR cs.LG

Kyung Ho Park , Eunjo Lee , Huy Kang Kim This is my paper · ORCID

classification cs.CRcs.LG

keywords gamebotdetectionMMORPGfinancialanalysisLSTMtime-seriesclassificationAionrealmoneytradingcross-validationleakage

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a bot's in-game financial status is a necessary condition of bot activity: bots exist to accumulate wealth, so their money flows cannot be disguised by changing movement or chat patterns. The authors train an LSTM on sliding time windows of nine financial features from Aion characters labeled by human inspectors, and report an average F1-score of 0.9501 across four test weeks. If the claim holds, bot detection becomes more sustainable and portable, because the same financial features appear in most MMORPG economies and do not need to be redesigned each time bot behavior changes. The reported score should be read alongside the cross-validation caveat described in the weakest assumption.

What carries the argument

The load-bearing object is the financial-status time series, represented as overlapping sliding windows of min-max normalized features: number of items, total cash, cash carried in inventory, cash in character bank, vendor cash, evaluated asset value, mailing asset value, and evaluated asset values in the two bank tiers. These windows are fed to an LSTM, a recurrent neural network whose hidden state carries temporal context, letting the model learn the pattern of wealth movement rather than just a static snapshot. The 'necessary condition' framing is what makes the machinery sustainable: a bot's behavior can be changed, but its ledger cannot.

What would settle it

Retrain the same LSTM with 10-fold cross-validation in which all windows from a given character are forced into the same fold, and compare the resulting F1-score with 0.9501; a large drop would confirm that the reported score partly comes from within-character leakage.

Watch

Extended reading notes

Core claim

The central claim is that financial status itself is a necessary condition of bot activity: because bots are built to accumulate wealth for real-money trading, their money and item balances must change in patterned ways, and those changes cannot be hidden by making movement or chat look human. To test this, the paper extracts nine financial features from Aion status logs, cuts each character's history into fixed-length sliding windows, min-max scales each feature, and trains an LSTM. On weekly test sets, the model reaches accuracy 0.9473, precision 0.9223, recall 0.9834, and average F1-score 0.9501. The paper reads the lower precision as acceptable because the intended use is to produce a list of doubtful accounts for human review, not to block accounts automatically.

Load-bearing premise

The evaluation assumes that overlapping time windows from the same character are independent samples, so randomly splitting windows into training and test folds does not leak a character's identity into both sides of the test.

Editorial extensions

If this is right

Detection should keep working when bots imitate human movement or chat, since the financial accumulation pattern itself is the signal.
The same financial features should transfer to other MMORPGs with cash-and-item economies, letting one model serve many games.
The model classifies individual characters, so an operator can generate a review list instead of blocking accounts outright.
Because the LSTM is a black box, bot developers cannot easily reverse-engineer the thresholds the detector uses.
Ensembling this financial-status detector with other necessary-condition features, such as transaction networks, should reduce false positives.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The reported F1 is likely inflated: the sliding-window design creates many overlapping windows per character, and random mixing before 10-fold cross-validation can let the same character appear in both training and test folds; a character-disjoint split would give a more honest score.
A direct test of sustainability would be to train on one in-game economic period and evaluate on a later one after bot software updates, rather than only within the same week.
An adversarial countermeasure suggested by the ledger framing is wealth dispersal: if a bot farm splits gains across many small accounts, each account's financial pattern may look closer to a normal player, so the method's robustness to account-level laundering remains unexamined.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First thing: the idea has legs, but the reported F1 is a leaky number. The paper applies LSTM to a windowed time-series of in-game financial status features to distinguish bots from normal players in Aion. That specific combination is new compared to the cited work, which uses transaction networks or coordinates. The motivation—using 'necessary conditions' that bot developers can't easily change—is sensible, and the feature set in Table IV (cash, item counts, evaluated asset values) is reasonable. The ground truth is from NCSOFT inspectors and described carefully.

The soft spot is fatal for the evaluation. Section III-B2 generates training windows by sliding a fixed-length time window along each character's log, so a single character produces many overlapping windows. Then Section IV says 'We randomly mixed data, and performed 10-fold cross validation.' That means windows from the same character land in both training and test folds. The LSTM can memorize per-character financial dynamics rather than learning a general bot/normal distinction, so the average F1 of 0.9501 is an optimistic, leakage-inflated estimate. This is not a minor caveat; it is the central experimental claim. Grouped cross-validation by character (or leave-one-character-out) is the minimum fix. Also absent: baselines (a logistic regression on window statistics would tell you whether the LSTM is adding anything), error bars across folds, and details on window length, batch size, and architecture.

The paper is not incoherent. The writing is clear, the related work is placed honestly, and the limitation paragraphs in the conclusion are about generality and ensembling rather than evaluation validity—so the authors may simply not have noticed the leakage. But the quantitative conclusion is not supported.

Who should read it: people in game security who want a short list of financial-status features and a reminder of how easy it is to leak identity when doing sliding-window cross-validation. I would not cite the performance number, and I would not bring it to a reading group except as a cautionary example.

For peer review: I would send it to referees—not to accept, but because the underlying question is real and the evaluation is repairable. A serious editor should not desk-reject outright; a referee should ask for grouped CV, baselines, and architectural details. If the authors redo the experiment, there might be a modest result worth reporting.

Referee Report

4 major / 5 minor

Summary. The paper proposes a server-side MMORPG bot detection method based on financial-status time series. The authors extract nine financial features from Aion character logs, generate training windows by sliding a fixed-length time window over each character's log, and train an LSTM classifier to distinguish bots from normal users. On a dataset collected during the first week of May 2010, the model reportedly achieves an average F1 score of 0.9501 under 10-fold cross-validation. The authors argue that financial status is a necessary condition of bot behavior, making detection sustainable against changing bot activity patterns and applicable to other MMORPGs.

Significance. If the reported results were valid, the work would make a useful practical contribution: financial-status features are plausible and generally available, and an LSTM approach to individual-character classification is a sensible extension of prior transaction-network analyses. The paper also honestly discusses limitations such as false positives and the need for ensembling. However, the significance is severely undercut by the evaluation methodology, as the cross-validation procedure leaks information between training and test folds, and no baselines or error bars are provided. The central empirical claim therefore is not currently supported.

major comments (4)

[Section IV with Section III-B2] The 10-fold cross-validation is invalid because the data are mixed randomly before splitting, while the training data are generated by sliding a fixed-length time window over each character's log. Consecutive windows from a single character overlap in time, and all windows from a character share that character's financial dynamics. Randomly assigning these windows to folds means windows from the same character appear in both training and test sets, allowing the LSTM to memorize character-specific patterns rather than learning general bot-versus-normal behavior. The reported F1 of 0.9501 is thus an optimistic, leakage-inflated estimate. The authors should use grouped cross-validation by character (or account) and, ideally, leave-one-character-out evaluation, and report the resulting performance.
[Section IV] No baseline methods are compared. The paper claims LSTM with financial features achieves 'reasonable performance,' but without comparing to simpler classifiers (e.g., logistic regression, random forest, or a non-temporal model using the same features), the value added by the LSTM architecture and by the sliding-window construction is not demonstrated. The literature review in Section II describes prior detection methods, yet none of them is evaluated on this dataset, so the claimed improvement over existing approaches is not established.
[Sections III-C and IV] The manuscript omits essential experimental details needed for reproducibility and for assessing the leakage issue: the sliding time-window length and stride, the LSTM architecture (number of layers, hidden units, dropout), the optimizer, learning rate, batch size, number of epochs, and the normalization procedure (in particular, whether min-max statistics are computed on the full dataset before splitting). The paper also reports only point estimates without standard deviations or per-fold results, so the stability of the 0.9501 F1 cannot be evaluated.
[Section V] The 'sustainability' and 'generality' claims are not supported by the experiments. The evaluation uses data from a single week (May 2010) of a single game; there is no temporal split, no test of robustness to changing bot behavior, and no validation on another MMORPG. These claims are stated as contributions but are only argued qualitatively.

minor comments (5)

[Index terms] The term 'bot detetion' contains a typo; it should be 'bot detection.'
[Section V] In the first paragraph of the Discussion, 'nsembling' should be 'ensembling.'
[Section III-B2] The normalization equation uses per-feature min and max, but the paper does not state whether these are computed on the training set only; if they are computed on the full mixed dataset, this introduces additional information leakage.
[Section IV] The term 'monthly dataset' is confusing because the data collection described in Section III-A covers only 'the first week of May, 2010.' Please clarify how a monthly dataset was obtained and why weekly splits are used.
[Figure 1] The paper references 'Figure 1: Chosen boxplots' but the figure is not included in the text. Either include the figure or remove the reference.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: supervised LSTM classification with human-labeled ground truth; the reported F1 score is an empirical result, not a quantity forced by construction.

full rationale

The paper's central claim is that financial status time-series data can classify Aion characters as bots or normal users. This is supported by a supervised machine learning experiment: the ground-truth labels come from NCSOFT human inspectors (Section III-A2), while the features are financial status values extracted from logs (Table IV), defined independently of the bot label. The LSTM is trained on labeled data and evaluated with 10-fold cross-validation (Section IV), so the reported F1 score is an empirical estimate rather than a result equivalent to the model's inputs by definition. Self-citations to prior bot-detection work appear only in the literature review (Section II) and are not load-bearing for the new model's derivation. A reader may question the experiment because random 10-fold CV over overlapping per-character sliding windows (Section III-B2 versus Section IV) likely leaks character identity into training folds, but data leakage is a methodological validity concern, not circularity: the prediction does not reduce to the input by construction, and no fitted parameter is renamed as a prediction. Accordingly, no circular step is present.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new physical or conceptual entities are introduced. The central claim rests on the domain assumption that financial status is a necessary condition of bot behavior, on the reliability of human-provided labels, on the (invalid) assumption that randomly shuffled sliding windows are independent, and on the representativeness of old data.

free parameters (3)

Sliding time window length
Not reported; the length of the sliding window determines the input sequence length for the LSTM and is chosen by the authors (Section III-B2).
LSTM architecture hyperparameters
Number of layers, hidden units, batch size, and regularization details are not provided in the paper (Section III-C).
Min-max normalization statistics
Normalization is done with the global Min(Xi) and Max(Xi) (Section III-B2); if computed over the entire dataset including test data, this leaks information.

assumptions (4)

domain assumption Financial status is a necessary condition of bot activity and cannot be evaded by bot developers.
Invoked in Abstract and Sections I and II to justify the sustainability claim; not empirically validated against adversarial bots.
domain assumption Ground-truth labels from NCSOFT human inspectors are accurate.
Section III-A2; the model relies entirely on these labels without a formal audit.
ad hoc to paper Sliding windows from the same character are independent after random mixing.
Section IV random shuffling before cross-validation; this assumption is false for overlapping time windows and invalidates the evaluation.
domain assumption The 2010 dataset remains representative of current bot and user behavior.
The paper uses data from May 2010 but makes claims about sustainability and generality (Sections V and VI).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Show Me Your Account: Detecting MMORPG Game Bot Leveraging Financial Analysis with LSTM." pith.science (2026). https://pith.science/paper/4I52A7BJ

@misc{pith2026190803748,
  author       = {Pith},
  title        = {Pith review of: Show Me Your Account: Detecting MMORPG Game Bot Leveraging Financial Analysis with LSTM},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4I52A7BJ}},
  note         = {Machine review of arXiv:1908.03748}
}

read the original abstract

With the rapid growth of MMORPG market, game bot detection has become an essential task for maintaining stable in-game ecosystem. To classify bots from normal users, detection methods are proposed in both game client and server-side. Among various classification methods, data mining method in server-side captured unique characteristics of bots efficiently. For features used in data mining, behavioral and social actions of character are analyzed with numerous algorithms. However, bot developers can evade the previous detection methods by changing bot's activities continuously. Eventually, overall maintenance cost increases because the selected features need to be updated along with the change of bot's behavior. To overcome this limitation, we propose improved bot detection method with financial analysis. As bot's activity absolutely necessitates the change of financial status, analyzing financial fluctuation effectively captures bots as a key feature. We trained and tested model with actual data of Aion, a leading MMORPG in Asia. Leveraging that LSTM efficiently recognizes time-series movement of data, we achieved meaningful detection performance. Further on this model, we expect sustainable bot detection system in the near future.

Figures

Figures reproduced from arXiv: 1908.03748 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 14 canonical work pages

[1]

Global MMO Games Market 2018-2022,

Technavio Research, “Global MMO Games Market 2018-2022,” ”https: //www.apnews.com/9e7c20b7267841efb0fb22b2bd9398e3”, 2018, [On- line; Last Access 27 May 2019]

work page 2018
[2]

Virtual worlds: A ﬁrst-hand account of market and society on the cyberian frontier,

E. Castronova, “Virtual worlds: A ﬁrst-hand account of market and society on the cyberian frontier,” CESinfo Working Paper Series , 2001

work page 2001
[3]

Simple economics of real-money trading in online games,

J.-S. Huhh, “Simple economics of real-money trading in online games,” 2008

work page 2008
[4]

You are a game bot!: Uncovering game bots in mmorpgs via self-similarity in the wild

E. Lee, J. Woo, H. Kim, A. Mohaisen, and H. K. Kim, “You are a game bot!: Uncovering game bots in mmorpgs via self-similarity in the wild.” in NDSS, 2016

work page 2016
[5]

Mining for gold farmers: Automatic detection of deviant players in mmogs,

M. A. Ahmad, B. Keegan, J. Srivastava, D. Williams, and N. Contractor, “Mining for gold farmers: Automatic detection of deviant players in mmogs,” in 2009 International Conference on Computational Science and Engineering, vol. 4. IEEE, 2009, pp. 340–345

work page 2009
[6]

Survey and research direction on online game security,

J. Woo and H. K. Kim, “Survey and research direction on online game security,” in Proceedings of the Workshop at SIGGRAPH Asia . ACM, 2012, pp. 19–25

work page 2012
[7]

In-game action sequence analysis for game bot detection on the big data analysis platform,

J. Lee, J. Lim, W. Cho, and H. K. Kim, “In-game action sequence analysis for game bot detection on the big data analysis platform,” in Proceedings of the 18th Asia Paciﬁc Symposium on Intelligent and Evolutionary Systems-Volume 2. Springer, 2015, pp. 403–414

work page 2015
[8]

Detection of mmorpg bots based on behavior analysis,

R. Thawonmas, Y . Kashifuji, and K.-T. Chen, “Detection of mmorpg bots based on behavior analysis,” in Proceedings of the 2008 Interna- tional Conference on Advances in Computer Entertainment Technology . ACM, 2008, pp. 91–94

work page 2008

Show all 14 references

[9]

Chatting pattern based game bot detection: do they talk like us?

A. R. Kang, H. K. Kim, and J. Woo, “Chatting pattern based game bot detection: do they talk like us?” KSII Transactions on Internet & Information Systems, vol. 6, no. 11, 2012

2012
[10]

Online game bot detection based on party-play log analysis,

A. R. Kang, J. Woo, J. Park, and H. K. Kim, “Online game bot detection based on party-play log analysis,” Computers & Mathematics with Applications, vol. 65, no. 9, pp. 1384–1395, 2013

2013
[11]

Game-bot detection based on clustering of asset-varied location coordinates,

H. M. Song and H. K. Kim, “Game-bot detection based on clustering of asset-varied location coordinates,” Journal of the Korea Institute of Information Security and Cryptology , vol. 25, no. 5, pp. 1131–1141, 2015

2015
[12]

No silk road for online gamers!: Using social network analysis to unveil black markets in online games,

E. Lee, J. Woo, H. Kim, and H. K. Kim, “No silk road for online gamers!: Using social network analysis to unveil black markets in online games,” in Proceedings of the 2018 World Wide Web Conference on World Wide Web. International World Wide Web Conferences Steering Committee,...

2018
[13]

Recurrent neural networks and time series prediction,

J. Connor and L. Atlas, “Recurrent neural networks and time series prediction,” in IJCNN-91-Seattle International Joint Conference on Neural Networks, vol. 1. IEEE, 1991, pp. 301–306

1991
[14]

Crime scene reconstruction: Online gold farming network analysis,

H. Kwon, A. Mohaisen, J. Woo, Y . Kim, E. Lee, and H. K. Kim, “Crime scene reconstruction: Online gold farming network analysis,” IEEE Transactions on Information Forensics and Security, vol. 12, no. 3, pp. 544–556, 2016

2016

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Global MMO Games Market 2018-2022,

Technavio Research, “Global MMO Games Market 2018-2022,” ”https: //www.apnews.com/9e7c20b7267841efb0fb22b2bd9398e3”, 2018, [On- line; Last Access 27 May 2019]

work page 2018

[2] [2]

Virtual worlds: A ﬁrst-hand account of market and society on the cyberian frontier,

E. Castronova, “Virtual worlds: A ﬁrst-hand account of market and society on the cyberian frontier,” CESinfo Working Paper Series , 2001

work page 2001

[3] [3]

Simple economics of real-money trading in online games,

J.-S. Huhh, “Simple economics of real-money trading in online games,” 2008

work page 2008

[4] [4]

You are a game bot!: Uncovering game bots in mmorpgs via self-similarity in the wild

E. Lee, J. Woo, H. Kim, A. Mohaisen, and H. K. Kim, “You are a game bot!: Uncovering game bots in mmorpgs via self-similarity in the wild.” in NDSS, 2016

work page 2016

[5] [5]

Mining for gold farmers: Automatic detection of deviant players in mmogs,

M. A. Ahmad, B. Keegan, J. Srivastava, D. Williams, and N. Contractor, “Mining for gold farmers: Automatic detection of deviant players in mmogs,” in 2009 International Conference on Computational Science and Engineering, vol. 4. IEEE, 2009, pp. 340–345

work page 2009

[6] [6]

Survey and research direction on online game security,

J. Woo and H. K. Kim, “Survey and research direction on online game security,” in Proceedings of the Workshop at SIGGRAPH Asia . ACM, 2012, pp. 19–25

work page 2012

[7] [7]

In-game action sequence analysis for game bot detection on the big data analysis platform,

J. Lee, J. Lim, W. Cho, and H. K. Kim, “In-game action sequence analysis for game bot detection on the big data analysis platform,” in Proceedings of the 18th Asia Paciﬁc Symposium on Intelligent and Evolutionary Systems-Volume 2. Springer, 2015, pp. 403–414

work page 2015

[8] [8]

Detection of mmorpg bots based on behavior analysis,

R. Thawonmas, Y . Kashifuji, and K.-T. Chen, “Detection of mmorpg bots based on behavior analysis,” in Proceedings of the 2008 Interna- tional Conference on Advances in Computer Entertainment Technology . ACM, 2008, pp. 91–94

work page 2008