{"id":"cf6010b9-4d43-437e-8def-ea4cabdc0b4e","arxiv_id":"2508.20578","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper proposes an unsupervised pipeline of TS2Vec, DBSCAN, and a GPT-4o reviewer for MMORPG bot detection, but validates it only with a proxied access-pattern metric.","lead":"A short industry paper combines an existing time-series encoder with density clustering and GPT-4o review to find auto-leveling bots in online games, claiming zero-label training. The evidence depends on a self-defined proxy metric instead of confirmed bot labels, so the reported gains are not yet established.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'high accuracy' claim rests on acc_info, an unvalidated proxy from the authors' prior work that is also used to tune DBSCAN eps; no ground-truth labels or human baseline support the conclusion.","rationale":"The reader's weakest assumption correctly identifies acc_info as the load-bearing proxy. My stress-test concurs and sharpens the issue: acc_info is not merely one metric among several; it is the objective used to select epsilon (Section 4.3.2) and the outcome measure used to validate the LLM (Section 4.4.2). This creates a self-confirmation risk that no other experiment resolves. The Section 4.2 embedding evaluation is orthogonal, and no human baseline is reported. Under the paper's own framing, a game operator would be asked to sanction characters based on this pipeline, so the absence of ground-truth validation is decisive. I do not see a reason to change the reader's REJECT verdict; the concern is methodological rather than stylistic. No ad hominem is intended: the issue is that the evidence presented does not establish the central claim.","tokens_in":8991,"tokens_out":4494,"duration_ms":41686,"concrete_test":"Obtain independent labels for a sample of characters from the three games: have game masters, blind to all model outputs, classify 500 characters per game drawn from (a) DBSCAN q=0.1 candidates before LLM filtering, (b) the same candidates after LLM filtering, (c) characters the LLM removed, and (d) a random sample of non-clustered characters. Compute precision, recall, and F1 of the pipeline against these labels, and compare acc_info distributions between human-labeled bots and humans. If precision is not high (e.g., below 90%) or acc_info does not separate the two label classes, the central 'high accuracy' claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim (abstract and conclusion) is that the pipeline 'reduces labeling costs while achieving high accuracy.' For this to hold, the clusters and the LLM filter must correspond to actual bot identity. The only quantitative evidence in Table 2 is the access-information homogeneity (acc_info) metric from the authors' prior BotTRep paper [13]. As defined in Section 4.3.1, acc_info measures similarity of login/access patterns and is interpreted as indicating characters controlled by the same player; it is not validated anywhere in this paper against ground-truth bot labels. Section 4.3.2 selects DBSCAN's epsilon by minimizing acc_info, and Section 4.4.2 uses the same acc_info to claim that LLM refinement improves the results. Tuning and evaluating on the same unvalidated proxy makes the 'high accuracy' claim self-confirming: any filter that removes access-pattern outliers will improve acc_info regardless of whether those outliers are actual humans. The Section 4.2 perturbation experiment measures Kendall's Tau on synthetic corruptions of level-up sequences; that validates representation robustness, not bot detection. No human baseline is provided for the LLM verification step, so the claim that the LLM 'effectively mimics secondary human judgment' is unsupported. Consequently, an operator could not infer from this paper that the pipeline produces reliable, explainable bot sanctions.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a fully unsupervised pipeline for detecting auto-leveling bots in MMORPGs. Level-up interval sequences are embedded with TS2Vec, clustered with DBSCAN, and then passed to GPT-4o as an auxiliary reviewer that is supposed to filter false positives from clusters. The authors also introduce a level-up interval visualization intended to support LLM and human moderation. The experiments use data from three commercial MMORPGs and report three evaluations: Kendall's Tau for representation robustness under synthetic perturbations, a clustering evaluation using access-information homogeneity (acc_info), and an LLM refinement evaluation using the same acc_info metric. The paper concludes that the method reduces labeling costs while achieving high accuracy.","tokens_in":9269,"tokens_out":3900,"duration_ms":36699,"significance":"If the central claim were supported, the framework would be practically valuable: an unsupervised, explainable bot-detection pipeline with LLM-assisted human oversight could reduce moderation costs in a commercial setting. The paper's orientation toward explainability and legal defensibility is appropriate for the domain, and the use of real operational data from three games is a strength. The perturbation-based robustness check for the representation model is a reasonable auxiliary validation. However, the evaluation never measures detection accuracy against known bot labels. The primary metric, acc_info, comes from the authors' prior work and is used both to select the DBSCAN epsilon and to claim that LLM refinement improves results, making the central quantitative claim self-referential. No human baseline is provided for the LLM verification step. As a result, the paper does not establish that the pipeline produces reliable bot sanctions.","major_comments":[{"comment":"The evaluation is self-referential. The primary metric acc_info is introduced in the authors' prior work [13] and is used both to select the DBSCAN epsilon (q=0.1 is recommended because it gives lower acc_info) and to claim that LLM refinement improves the results. Because acc_info measures homogeneity of access/login patterns among characters in the same cluster, any filter that removes access-pattern outliers will tend to lower acc_info regardless of whether those outliers are human or bot. The manuscript provides no ground-truth labels, no precision/recall/F1, and no human-verification baseline, so the abstract's 'high accuracy' claim is not established.","section":"§4.3.1–4.4.2, Table 2"},{"comment":"The Kendall's Tau experiment only validates that TS2Vec preserves distance rankings between original and synthetically perturbed level-up sequences. This does not test whether bot representations are separable from human representations, nor whether clusters discovered by DBSCAN correspond to actual bot groups. Thus, Table 1 cannot support the clustering results in Section 4.3, and the perturbation experiment is not evidence of detection performance.","section":"§4.2, Table 1"},{"comment":"The LLM verification component is evaluated only indirectly through the same acc_info metric. There is no evaluation of whether GPT-4o's inclusion/exclusion decisions agree with human moderators, no per-cluster decision outcomes, no error analysis, and no full prompt is provided. Consequently, the claim that the LLM 'effectively mimics secondary human judgment' is unsupported.","section":"§3.3 and §4.4"},{"comment":"The comparisons across epsilon settings and LLM on/off are reported as daily averages without standard deviations, confidence intervals, or significance tests. The #Det column contains decimal values despite being a count, which obscures the underlying variability. Without these, the practical magnitude of the LLM improvement and the robustness of the q=0.1 recommendation cannot be assessed.","section":"Table 2 and §4.3.2"}],"minor_comments":[{"comment":"The heading contains a typo: 'Data preperation' should be 'Data preparation'.","section":"§3.1.1"},{"comment":"The heading contains a typo: 'Represntation' should be 'Representation'.","section":"§3.2.1"},{"comment":"The column header 'IP' is not defined; based on the table and text it appears to be a placeholder for the game dataset, but this should be stated explicitly.","section":"Table 2"},{"comment":"The prompt engineering section describes the strategy but does not include the actual prompt or a representative input/output example, which limits reproducibility of the LLM component.","section":"§3.3.2"},{"comment":"The statement that acc_info has a minimum value of 1 is referenced to [13] but no formula or derivation is provided here; a self-contained definition would help the reader understand what is being optimized.","section":"§4.3.1"}],"recommendation":"reject","confidential_remarks":"The self-referential evaluation is the deciding issue. The central 'high accuracy' claim rests on a proxy metric that is used both for parameter selection and for evaluation, with no ground-truth labels. This is not a presentation issue; it requires a new evaluation design. I would not consider this paper for acceptance until the authors validate acc_info against known bot/human labels or provide an alternative ground-truth-based evaluation, ideally with a human baseline for the LLM verification step."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nQuick take: this is an industry paper from NCSOFT that combines TS2Vec embeddings, DBSCAN clustering, and a GPT-4o verification pass to find auto-leveling bots in three mobile MMORPGs. The genuinely new piece is the LLM-as-a-second-judge step; everything else is borrowed, including the authors' own BotTRep clustering recipe and the acc_info evaluation metric.\n\nWhat the paper does well: it is clearly written, the problem is real, and the data is genuine deployment-scale (about a million training sequences across three titles). The perturbation experiment in Section 4.2 is a legitimate sanity check: corrupt the level-up sequences and measure whether the embedding distances rank the corruptions correctly. That gives you some confidence the representation captures structure. The growth-curve visualization is a sensible way to make cluster membership interpretable to a human moderator.\n\nThe soft spot is not stylistic; it is load-bearing. The evaluation never measures detection accuracy against known bot labels. The primary outcome, acc_info, is a proxy from the authors' earlier paper that measures login-pattern homogeneity. In Section 4.3.2 they pick the DBSCAN epsilon by minimizing acc_info; in Section 4.4.2 they use the same acc_info to claim the LLM refinement helps. That is a closed loop. Any filter that removes characters with less homogeneous access patterns will improve acc_info, whether or not those characters are human. The Kendall's Tau result is about embedding robustness, not about whether the clusters contain bots. The LLM verification has no human baseline, so the statement that it 'effectively mimics secondary human judgment' is unsupported. There are also no released artifacts to reproduce Table 2.\n\nGiven that, the conclusion's 'high accuracy' is not supported by the evidence. The paper shows a plausible pipeline with internal consistency, but not that it works.\n\nThe reader's REJECT is fair. I would send it to peer review rather than desk reject, because the problem is real and a revision with ground-truth labels, a human baseline, or at least a comparison against BotTRep's manual verification results could make the paper genuinely useful. As it stands, treat it as an interesting work-in-progress, not a validated method.\n\nBest,\n[Name]","headline":"A practical, clearly-written bot detection pipeline undercut by a self-referential evaluation: the only outcome metric is also the tuning target, so the 'high accuracy' claim is unsubstantiated.","tokens_in":9794,"tokens_out":2354,"would_cite":false,"duration_ms":22320,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An unsupervised pipeline embeds level-up intervals, clusters with DBSCAN, and uses an LLM to filter false positives, detecting auto-leveling bots with zero labels and an explainable trail.","keywords":["game bot detection","MMORPG","unsupervised learning","representation learning","DBSCAN clustering","LLM-assisted verification","level-up interval sequences","access information homogeneity"],"falsifier":"Audit a random sample of accounts that the full pipeline flags with the lowest acc_info, using actual operator-side evidence such as payment method, device IDs, customer support tickets, and manual review. If a material share turn out to be legitimate players, such as shared family accounts or internet-cafe users with regular access patterns, then acc_info is not a valid bot signal and the paper's central claim fails; if instead nearly all audited accounts are confirmed automated farms, the claim survives.","tokens_in":8782,"feed_emoji":"🎮","tokens_out":4713,"duration_ms":44421,"temperature":0.7,"pith_summary":"The paper claims that auto-leveling bots in MMORPGs can be detected without any labeled training data by first embedding each character's level-up interval sequence, clustering the embeddings with DBSCAN, and then asking an LLM to review each cluster and remove normal players who were mistakenly grouped in. The authors argue this replaces the labor-intensive human inspection of bot candidates with machine review while keeping an explainable record for sanctions. They report experiments on data from three commercial MMORPGs showing lower access-information homogeneity, a proxy for same-player control, after the LLM filter, which they interpret as fewer false positives. A sympathetic reader should care because bot regulation requires both scale and explainability, and this pipeline claims to deliver both without annotation cost.","feed_headline":"Bot detection goes label-free with an LLM reviewer","feed_subtitle":"Unsupervised clustering of level-up patterns plus LLM review cuts false positives in three MMORPGs.","key_machinery":"The load-bearing machinery is the pair of a time-series embedding model and a density-based clusterer, with an LLM as a second reader. Each character $p$ is reduced to its level-up interval sequence $T(p)$, capped at level 50; a contrastive encoder (TS2Vec in the experiments) produces a latent vector $r(p)$, and DBSCAN with $\\mathit{min\\_sample}=3$ forms dense clusters while leaving irregular human play as noise. The DBSCAN radius $\\varepsilon$ is picked as a quantile ($q=0.1$) of $k$-nearest-neighbor distances, following the prior framework that also supplies the evaluation metric. The LLM verification module takes each cluster's raw interval sequences, is prompted zero-shot with explicit sanction criteria and chain-of-thought steps, and returns which characters should be excluded; the growth-curve visualization shows the same data to human moderators.","core_discovery":"On the paper's own terms, the central discovery is that systematically repetitive level-up behavior is enough to separate bots from humans: a time-series representation model maps level-up interval sequences so that bot-like sequences land close together, DBSCAN finds dense clusters, and a GPT-4o reviewer filters the cluster using chain-of-thought reasoning over the raw intervals plus a growth-curve visualization. The authors claim this fully unsupervised process, with no labels and no per-title tuning beyond choosing $\\varepsilon$, yields sanction candidate lists that are both accurate and explainable, as measured by lower access information homogeneity after LLM refinement across all three games. They position this as the first framework of its kind to replace the manual secondary judgment in prior work with an LLM, thereby cutting labeling and review costs.","pith_inferences":["If access-information homogeneity is accepted as a valid bot signal, the same recipe of embedding plus density clustering plus LLM review could be applied to other repeated behavioral streams, such as quest completion logs or movement paths, not just level-up intervals.","The current evaluation selects $\\varepsilon$ to minimize the same acc_info score it later uses to judge success, so an independent ground-truth audit of flagged accounts would be needed to confirm that the LLM filter actually removes legitimate players.","A self-hosted or smaller LLM would change the economics and latency of the second review; the paper's numbers were produced with a single commercial model and do not address how sensitive the filter is to prompt or model changes.","One testable extension is to feed the LLM only the visualization rather than raw intervals, which would test whether the explainable artifact itself carries the verification signal."],"forward_implications":["A game operator can run the full detection pipeline with zero annotated examples and receive daily lists of sanction candidates ready for an LLM-reviewed handoff to human moderators.","Because the input is only level-up timestamps, the same pipeline should transfer to any MMORPG whose client logs level-up events, with no title-specific feature engineering.","The LLM pass replaces the most repetitive part of a human game master's job, shifting human effort to higher-level decisions on the remaining clusters.","The growth-curve plots give each candidate an audit trail, which addresses the explainability requirement that sanctions face in legal disputes."],"supporting_citations":[{"why":"Supplies the density-based clustering algorithm DBSCAN that groups the embedded level-up sequences.","marker":"[5]"},{"why":"Provides Kendall's Tau, the rank-correlation metric used to evaluate embedding quality under perturbation.","marker":"[12]"},{"why":"Supplies the access information homogeneity metric, the epsilon-selection method, and the prior human-in-the-loop framework this work automates.","marker":"[13]"},{"why":"Supplies the DBSCAN parameter-recommendation scheme behind the quantile-based epsilon selection.","marker":"[27]"},{"why":"Co-source for the access information homogeneity metric used to evaluate clustering and LLM refinement.","marker":"[29]"},{"why":"Provides chain-of-thought prompting, the strategy used to structure the LLM verification prompt.","marker":"[31]"},{"why":"Supplies TS2Vec, the time-series representation model whose latent vectors are clustered.","marker":"[34]"}],"fun_headline_variants":["Label-free bot detection with LLM second opinion","Unsupervised clustering plus LLM review for bot bans","No labels, no manual review: LLM checks bot clusters","LLM-validated bot clusters from level-up curves","GPT-4o as bot-reviewer in unsupervised detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that a low access-information homogeneity score, meaning characters share near-identical login and access patterns, genuinely means the characters are bots run by the same player; if this proxy does not track real bot behavior, then neither the choice of $\\varepsilon$ nor the reported improvement from the LLM filter establishes the central claim.","fun_headline_variants_meta":{"raw":{"variants":["Label-free bot detection with LLM second opinion","Unsupervised clustering plus LLM review for bot bans","No labels, no manual review: LLM checks bot clusters","LLM-validated bot clusters from level-up curves","GPT-4o as bot-reviewer in unsupervised detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00114,"raw_usage":{"total_tokens":4688,"prompt_tokens":857,"completion_tokens":3831,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":473,"completion_tokens_details":{"reasoning_tokens":3751}},"tokens_in":473,"tokens_out":3831,"duration_ms":24673,"temperature":1.0,"reasoning_tokens":3751,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:42:52.044722+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Audit a random sample of accounts that the full pipeline flags with the lowest acc_info, using actual operator-side evidence such as payment method, device IDs, customer support tickets, and manual review. If a material share turn out to be legitimate players, such as shared family accounts or internet-cafe users with regular access patterns, then acc_info is not a valid bot signal and the paper's central claim fails; if instead nearly all audited accounts are confirmed automated farms, the claim survives.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides Kendall's Tau, the rank-correlation metric used to evaluate embedding quality under perturbation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the access information homogeneity metric, the epsilon-selection method, and the prior human-in-the-loop framework this work automates."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Co-source for the access information homogeneity metric used to evaluate clustering and LLM refinement."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides chain-of-thought prompting, the strategy used to structure the LLM verification prompt."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies TS2Vec, the time-series representation model whose latent vectors are clustered."}],"review_version":2}