{"id":"f68468fb-15a1-40a3-9db4-5472b8c7a8c5","arxiv_id":"2501.10461","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":9,"one_line_summary":"BotTRep uses contrastive trajectory representation learning and DBSCAN clustering to mine collectively-behaving bot groups in MMORPGs without any labels.","lead":"This paper introduces BotTRep, an unsupervised framework that learns trajectory embeddings for MMORPG players and clusters them with DBSCAN to surface groups of players who move in coordinated ways, which the authors argue are likely bots. It is tested on eight days of real gameplay data from NCSOFT's Lineage W and is designed to be fast enough to train within a nine-hour operational window.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central 'high accuracy' claim is unsupported without ground-truth bot labels, since all clusters are defined as bots and the proxy metrics do not distinguish bot farms from benign groups that happen to share networks and routes.","rationale":"The framework's technical contribution -- contrastive trajectory representation with zone/cell tokens, masked cell prediction, fast training, and visualization -- is credible and in principle reproducible, but the load-bearing claim is not the architecture; it is the assertion that the resulting clusters are actual bot groups. Without ground truth, every number in Tables 1 and 2 is consistent with a system that finds tight trajectory clusters, some of which share access information, but that never demonstrates that these are bots and not benign players with similar routes or shared networks. The contextual-similarity metric is partly circular because positive pairs are defined by the clusters being evaluated. The access-information metric is stronger but still a proxy, since shared IP or device is not equivalent to automation. A direct comparison to game-master-confirmed or subsequently banned accounts would settle whether the clustering recovers real bot groups and would also allow the epsilon parameter to be chosen on a validation split rather than on the reported test data. This does not invalidate the method or require rejection; it means the 'high accuracy' conclusion is conditional on validation that the paper does not provide.","tokens_in":14125,"tokens_out":5263,"duration_ms":62283,"concrete_test":"Compute precision and recall (or F1) of BotTRep's DBSCAN clusters from the July 9-15, 2023 downstream set against gold-standard labels for that period: accounts confirmed as bots by game masters and/or subsequently banned for botting, with a defined lag window. Count an account as a true positive if it is in a cluster, a false positive if a non-bot account is clustered, and report the same numbers for the Bi-GRU, Bi-LSTM, and Transformer baselines; if gold labels cannot be shared, an independent blinded manual audit of a random sample of clustered and noise accounts by game masters would settle the same question.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's operational definition in Section 3.5 ('we decided to define all the clustered groups ... as collectively-behaving groups') makes the target class coextensive with the model's own clusters. The first evaluation metric, contextual similarity (Section 4.2, Eqs. 9-11), builds positive pairs from those same cluster labels and measures whether members of a cluster have similar raw trajectories; this is an internal-consistency check, not evidence that the cluster is a bot group. The second metric, access-information homogeneity, is independent of the representation, and Table 1's value of 1.0079 is suggestive. But sharing IP/device/network is also consistent with benign multi-accounting, shared households, internet cafes, or guild members using a common VPN; without labels for automation, it cannot certify 'bot groups.' The single YouTube example (average Jaccard ~0.3 and access homogeneity 1.0) is anecdotal. The paper reports no precision/recall against known bot accounts, no banned-player validation, and no manual audit by game masters with stated criteria. Consequently, the central sentence -- 'our model has effectively detected the bot groups we aimed to identify with high accuracy' -- is not supported by any outcome measure that can detect false positives. The epsilon quantile q is also tuned on the same downstream data used to report results, but the gold-standard gap is the primary issue.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BotTRep, an unsupervised framework for mining collectively-behaving bot groups in MMORPGs. Trajectories are binned into zone and cell tokens, embedded with a Transformer trained jointly on triplet margin contrastive learning and masked cell prediction, and then clustered with DBSCAN. The method is evaluated on seven days of unlabeled Lineage W trajectories using two metrics: time-aware Jaccard contextual similarity within detected clusters versus random pairs, and access-information homogeneity across cluster members. The authors report that BotTRep achieves the best contextual similarity and lowest homogeneity value, and conclude that it detected the targeted bot groups with high accuracy.","tokens_in":14418,"tokens_out":3920,"duration_ms":43578,"significance":"If its central claim were properly supported, BotTRep would be a valuable industrial contribution: it addresses label scarcity, long-sequence training constraints, and explainability for game-master review, and it is evaluated on real production data with a useful ablation study. The explicit treatment of the nine-hour training deadline, the lightweight contrastive design, and the visualization tool are genuine strengths. However, the manuscript's load-bearing assertion of \"high accuracy\" is not supported by the evidence presented, because the evaluation lacks ground-truth bot labels and the primary metric is partly circular with the clustering itself. The paper is an interesting system description, but it needs external validation or substantially tempered claims before it can be accepted as a scientific claim about bot detection accuracy.","major_comments":[{"comment":"The evaluation is partially circular. In Section 3.5 the authors state 'we decided to define all the clustered groups (i.e. DBSCAN did not classify as noise) as collectively-behaving groups,' and in Section 4.2 positive pairs are constructed from those same cluster labels via Eq. (9), with contextual similarity measured as time-aware Jaccard similarity of raw trajectories. Since DBSCAN groups trajectories whose learned embeddings are close, and the model was trained to make contextually similar trajectories closer, high within-cluster contextual similarity is an internal-consistency check rather than evidence that the members are bots. To support the claim that the framework 'effectively detected the bot groups we aimed to identify with high accuracy' (Section 4.4), the authors need an external outcome measure: precision/recall against known bot accounts or banned-player lists, a manual audit by game masters with stated criteria, or at minimum a comparison against a random-clustering baseline to show that the observed similarity is not a trivial artifact of clustering.","section":"Section 3.5 and Section 4.2 (Eqs. 9-11)"},{"comment":"Access-information homogeneity is independent of the learned representation, which is a strength, but it cannot by itself certify that a cluster is a bot group. Identical IP, device, or network information is also consistent with benign multi-accounting, shared households, internet cafés, or guild members using a common VPN. The reported value of 1.0079 in Table 1 is suggestive, but without a base-rate comparison for randomly formed groups or for known benign clusters, it does not establish that the clusters are automated bot groups. The sentence in Section 4.4 that the access information of all players in a cluster is 'related, signifying our model has effectively detected the bot groups' therefore overstates what the metric can show.","section":"Section 4.2 (Access information homogeneity)"},{"comment":"The clustering parameter q (the epsilon quantile) is selected on the same downstream dataset used to report final results. The rows labelled (c) in Table 1 vary q and the paper chooses q=0.05 because it yields the best contextual similarity and lowest access-information homogeneity; this is test-set tuning and can inflate apparent performance. The authors should use a separate validation period or a nested selection procedure for q and for the other hyperparameters, or report all settings without claiming the best one as the final model.","section":"Section 4.3 and Table 1"},{"comment":"The only evidence connecting clusters to actual bot behavior is anecdotal: the YouTube example in Section 4.3 (average Jaccard around 0.3, access homogeneity 1.0) and the heatmap visualizations in Fig. 5. No systematic manual audit of detected clusters is reported, and the paper does not provide any measure that can detect false positives, such as the proportion of clusters later confirmed by game masters or the proportion of cluster members with no automation evidence. Without such an outcome, the central claim remains an assertion rather than a demonstrated result.","section":"Section 4.3 and Section 4.5"}],"minor_comments":[{"comment":"The heading 'Traning dataset' contains a typo and should be 'Training dataset'.","section":"Section 3.2"},{"comment":"The game name is misspelled as 'Linage W' in the paragraph on local coordinate systems; it should be 'Lineage W'.","section":"Section 2.1"},{"comment":"The caption says 'The thrid image illustrates clustering results'; 'thrid' should be 'third'.","section":"Fig. 4 caption"},{"comment":"The notation in Eq. (7) could be clarified: it applies Linear(·) to f^A_j for j in δ, but the text first says 'among these 16 extracted results, Linear(·) is applied to the tokens that had been masked'; make explicit that j indexes positions in the anchor sequence, not token identities.","section":"Eq. (7) and surrounding text"},{"comment":"The 'Training time Minutes (per epoch)' column is awkwardly formatted; consider splitting into total training time and minutes per epoch, and indicate the hardware configuration in the caption.","section":"Table 2"},{"comment":"The YouTube link is not a citable evaluation artifact; if it is intended to support the claim about detected bot examples, it should be replaced with supplementary material or a persistent repository.","section":"Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The paper's header states 'Published in ICPR 2024'; if this is a journal submission of a prior conference paper, the editor should verify that the extension is substantial and clearly disclosed. The principal scientific issue is the absence of ground-truth validation for the central 'high accuracy' claim; I would not reject the manuscript outright, because the system and the access-information signal are interesting, but the authors must either add external validation or substantially weaken the claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely useful part of this paper is the engineering: a contrastive Transformer trajectory representation trained with masked cell prediction, followed by DBSCAN, explicitly adapted to teleportation and overlapping local coordinate systems in MMORPGs. Training completes in about 8.5 hours, which is a real operational constraint, and the visualization tool is a sensible way to give game masters something they can act on. The access-information homogeneity metric is an independent signal, and the value of 1.0079 for the proposed setting is suggestive that clusters are not random.\n\nThe soft spots are concentrated in evaluation. The primary metric, contextual similarity, is computed on clusters produced by the model that was trained to make contextually similar trajectories closer. That is partly circular: it checks internal consistency, not whether the clusters are actually bots. The paper also defines all DBSCAN clusters as collectively-behaving groups, which means false positives are structurally invisible. A group of benign players who share a hunting route, or a guild on a common VPN, would likely pass the contextual similarity test. Access-information homogeneity helps, but shared IP or device is consistent with benign multi-accounting or internet cafés, not just automation. There is no validation against known bot accounts, no banned-player follow-up, and no manual audit with stated criteria. The epsilon quantile is also tuned on the same downstream data used for reporting results. These are not fatal to the engineering contribution, but they do mean the paper's conclusion that the model detects bot groups 'with high accuracy' is not supported by the current evidence.\n\nThat said, the authors are clear about their operational constraints and the design choices are well motivated. The related work is handled fairly, and the method is described in enough detail to reproduce. The central claim is plausible; it just needs a better yardstick. A serious referee should ask for a validation set with known bots or a structured manual audit, and a comparison to a published detector, even if only on a subset.\n\nWho is this for? Practitioners building bot-detection systems for live games, and researchers working on trajectory representation learning in domains with teleportation or discontinuous spaces. It deserves peer review, with the expectation of meaningful revision. I would not desk-reject it, and I would put it on a reading group list if anyone in the group cares about applied ML for game security.","headline":"A solid applied framework for mining bot groups from unlabeled MMORPG trajectories, but the central 'high accuracy' claim outruns the evidence because the main metric is partly circular and there is no ground-truth validation.","tokens_in":14943,"tokens_out":1545,"would_cite":true,"duration_ms":18469,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An unsupervised framework, BotTRep, mines collectively-behaving bot groups in MMORPGs from unlabeled trajectory data by learning context-aware embeddings and clustering them with DBSCAN, and the paper shows it outperforms autoencoder…","keywords":["game bot detection","MMORPG","trajectory representation learning","contrastive learning","DBSCAN","unsupervised learning","collective behavior","access information homogeneity"],"falsifier":"A direct falsifier would be to obtain a set of accounts later confirmed as bots (e.g., from bans or real-money-trading investigations) and check whether they fall disproportionately inside the mined clusters. If confirmed bot accounts appear as often in the noise class as inside clusters, or if clusters contain many legitimate players with coincidentally overlapping routes, the central claim would fail.","tokens_in":13913,"feed_emoji":"🎮","tokens_out":9493,"duration_ms":81345,"temperature":0.7,"pith_summary":"BotTRep is an unsupervised framework for mining collectively-behaving bot groups in MMORPGs from completely unlabeled in-game trajectory data. It learns to embed each player's daily movement so that contextually similar trajectories end up close together, then applies DBSCAN to isolate dense clusters as suspected bot groups. The paper claims this is the first trajectory-based bot mining approach that is both label-free and practical for industrial service, training in about 8.5 hours so that monitoring can restart after daily game patches. If correct, it lets game companies detect coordinated bot groups without a labeling infrastructure and with visual evidence that can support legal ban decisions.","feed_headline":"Unlabeled movement reveals coordinated MMORPG bot groups","feed_subtitle":"A contrastive Transformer plus DBSCAN flags bot clusters overnight, with evidence human moderators can review.","key_machinery":"The central machinery is the BotTRep encoder, a Transformer with contrastive triplets. Each minute-level location is tokenized into a zone (256-by-256 coordinate block) and a cell (8-by-8 block), and each token embeds as the sum of a zone embedding, a cell embedding, and a timestamp encoding; this design intentionally replaces raw Euclidean distance with contextual region co-occurrence, so teleportation between a village and a hunting ground is treated as 'close' when players regularly visit both. The encoder is trained jointly with triplet margin loss (pulling anchor and positive trajectory segments together while pushing a random negative apart) and masked cell prediction (predicting masked cell tokens in the anchor), which together force the model to capture spatiotemporal context. During inference, entire daily trajectories are embedded and clustered with DBSCAN, using an epsilon chosen from the 4-nearest-neighbor distance distribution; all non-noise clusters are defined as collectively-behaving groups. The design addresses the two MMORPG-specific problems identified in the paper: teleportation breaking spatial-distance assumptions, and multiple instance dungeons sharing the same local coordinate system.","core_discovery":"The paper's central claim is that contrastive trajectory representation learning followed by DBSCAN clustering accurately mines collectively-behaving bot groups from raw, unlabeled MMORPG movement logs. On the authors' own evaluation, the mined clusters show a time-aware Jaccard similarity of about 0.36 against under 0.001 for random pairs, and an access-information homogeneity of 1.0079, meaning that players inside a cluster share the same IP or device network—the signature of a single operator controlling multiple avatars. These numbers beat three autoencoder baselines (Bi-GRU, Bi-LSTM, and a plain Transformer) on both metrics, while completing training in 8.5 hours versus more than 24 hours for the baselines. The authors therefore conclude that the framework has effectively detected the bot groups it aimed to identify, and that the clusters are genuinely coordinated rather than coincidental overlaps.","pith_inferences":["A testable extension is to run BotTRep on a game that lacks access-information logs; if clusters still show high time-aware Jaccard similarity, the method would generalize to environments without IP or device linkage data.","The contextual co-occurrence principle suggests a broader design rule for trajectory representation: when movement includes long-range jumps or portals, semantic region proximity matters more than raw spatial distance, a lesson that could transfer to delivery fleets, transit networks, or any domain with discontinuous movement.","The paper evaluates clusters against proxies rather than a true bot gold standard; augmenting the framework with a small set of confirmed banned accounts as a validation anchor would turn the proxy metrics into calibrated precision and recall.","The definition of collectively-behaving groups as 'all non-noise DBSCAN clusters' is a strong prior; an inference worth testing is whether clusters that persist across multiple consecutive days correspond to the same underlying bot operators, which would strengthen the claim that the clusters are stable entities."],"forward_implications":["Game companies can deploy BotTRep without a labeling pipeline, since the framework mines bot groups from unlabeled movement logs alone.","The 8.5-hour training time fits the paper's stated service constraint, allowing daily monitoring to start at 9 AM after each game patch and to finish within the 9-hour window.","The visualization heatmap gives game masters a human-checkable record of each cluster's synchronized movement, which the paper argues lowers false-banning risk and provides evidence if a ban is challenged.","The access-information homogeneity of about 1.0 implies that players inside the same mined cluster are typically operated by the same person or network, supporting the interpretation that these are real bot farms.","The two-task training (contrastive triplet loss plus masked cell prediction) yields better contextual similarity and homogeneity than autoencoder-based baselines, suggesting direct similarity supervision is more effective than reconstruction for trajectory representation in this domain."],"supporting_citations":[{"why":"Supplies the DBSCAN density-based clustering algorithm that turns learned trajectory representations into collectively-behaving groups.","marker":"[5]"},{"why":"Provides the triplet margin loss used to pull contextually similar trajectory segments together and push dissimilar ones apart.","marker":"[17]"},{"why":"Supplies the masked-language-model objective that the paper adapts as masked cell prediction to refine cell-token learning.","marker":"[11]"},{"why":"The Transformer encoder architecture underlying the BotTRep representation model.","marker":"[22]"},{"why":"The k-nearest-neighbor distance method the paper adapts for selecting DBSCAN's epsilon parameter.","marker":"[18]"},{"why":"Justifies the choice of contrastive representation learning, which the paper cites for robustness and faster training.","marker":"[26]"}],"fun_headline_variants":["Contrastive learning mines bot gangs from unlabeled MMORPG logs","DBSCAN on trajectory embeddings flags coordinated bot clusters","Unlabeled trajectories reveal coordinated bot groups in MMOs","Contrastive trajectories spot bot clusters without training labels","Clustering movement paths unmasks coordinated bot groups in MMORPGs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a DBSCAN cluster of players with contextually similar trajectories and overlapping access information is actually a group of bots, an assumption never validated against a gold standard of known bot accounts or bans.","fun_headline_variants_meta":{"raw":{"variants":["Contrastive learning mines bot gangs from unlabeled MMORPG logs","DBSCAN on trajectory embeddings flags coordinated bot clusters","Unlabeled trajectories reveal coordinated bot groups in MMOs","Contrastive trajectories spot bot clusters without training labels","Clustering movement paths unmasks coordinated bot groups in MMORPGs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001536,"raw_usage":{"total_tokens":6123,"prompt_tokens":896,"completion_tokens":5227,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":512,"completion_tokens_details":{"reasoning_tokens":5142}},"tokens_in":512,"tokens_out":5227,"duration_ms":34174,"temperature":1.0,"reasoning_tokens":5142,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:19:15.116425+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct falsifier would be to obtain a set of accounts later confirmed as bots (e.g., from bans or real-money-trading investigations) and check whether they fall disproportionately inside the mined clusters. If confirmed bot accounts appear as often in the noise class as inside clusters, or if clusters contain many legitimate players with coincidentally overlapping routes, the central claim would fail.","supporting_citations":[{"cited_title":"ACM Transactions on Database Systems (TODS) 42(3), 1–21 (2017) 18 H","cited_arxiv_id":null,"evidence_quote":"The k-nearest-neighbor distance method the paper adapts for selecting DBSCAN's epsilon parameter."}],"review_version":1}