{"id":"73e211bb-98dc-4e42-915a-11bc95969cd0","arxiv_id":"2604.10175","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new labeled dataset of 15k League of Legends match messages enables a custom NLP toxicity detector that outperforms existing general and state-of-the-art models.","lead":"This paper collects a new dataset of toxic and non-toxic chat messages from League of Legends matches and trains an NLP model that flags toxicity better than general-purpose detectors. A smart generalist might read it to understand how tailored AI tools could reduce harassment in online games.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Labeling by only 8 expert players lacks reported inter-annotator agreement or validation against live-match ground truth, weakening the outperformance claim.","rationale":"The reader's weakest assumption directly matches the load-bearing point for the empirical outperformance claim. Full-text review would be needed to confirm absence of agreement metrics, but the abstract alone already flags this as the least secure precondition. No other internal inconsistency (e.g., dataset size or extension claims) appears more critical.","tokens_in":1831,"tokens_out":339,"duration_ms":25171,"concrete_test":"Extract the annotation section (likely §3 or §4) and check whether inter-annotator agreement statistics are reported for the 1.4k toxic labels; if absent or below 0.6, recompute detector performance after simulating 10-20% label flips on the toxic class to quantify sensitivity.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim requires that L2DTnH labels accurately reflect toxicity as it occurs in matches and that the resulting detector generalizes. The abstract states the dataset was created by partnering with 8 expert LoL players, but provides no details on annotation protocol (e.g., independent labeling, majority vote, definition of toxicity), inter-rater reliability (Cohen/Fleiss kappa), or any external validation. With only 1.4k toxic examples, even modest label noise or player-specific bias would directly affect the empirical comparison to general-purpose and SOTA detectors. The paper's later claim of testing on data beyond L2DTnH does not address whether the training labels themselves are reliable.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper conducts a systematic literature review (n=1,039) showing scarcity of NLP/ML-based toxicity detectors for live video-game matches. It introduces the L2DTnH dataset of 15.2k League of Legends chat messages (1.4k toxic) labeled by 8 expert players, trains an NLP detector claimed to outperform general-purpose and SOTA toxicity detectors, validates it on external game data, releases a local browser extension for flagging toxic content, and makes all resources public.","tokens_in":1969,"tokens_out":537,"duration_ms":38631,"significance":"If the outperformance and generalization claims are supported by reliable labels and transparent evaluation, the work supplies a domain-specific dataset and practical deployment artifact for toxicity mitigation during active matches, directly addressing the gap identified in the literature review. The public release of the dataset, detector, and serverless extension is a clear strength enabling reproducibility.","major_comments":[{"comment":"Dataset construction section: the labeling procedure by the 8 expert players is described only at high level. No annotation protocol, definition of toxicity, independence of labels, majority-vote rule, or inter-annotator agreement statistic (Cohen/Fleiss kappa) is reported. Because the central outperformance claim rests on the quality of these 1.4k toxic labels, missing reliability metrics constitute a load-bearing gap.","section":"Dataset construction"},{"comment":"Empirical results section: the abstract asserts that the L2DTnH-trained detector outperforms general-purpose and SOTA NLP toxicity detectors, yet no concrete metrics (precision, recall, F1, AUC), baseline models, train/test split details, or cross-validation procedure are supplied in the provided text. With only 1.4k positive examples, these omissions prevent assessment of whether the reported gains are robust or artifactual.","section":"Empirical results"}],"minor_comments":[{"comment":"The title's use of quotation marks around 'bot lane noob' is stylistically unclear and does not immediately convey the paper's focus on deployable detectors.","section":"Title"},{"comment":"The literature-review methodology would benefit from explicit search strings, databases queried, and inclusion/exclusion criteria to substantiate the n=1,039 count.","section":"Literature review"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is archived under cs.CR yet centers on NLP and HCI concerns; the editor should verify alignment with the journal's primary scope before proceeding."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their careful and constructive review of our manuscript. The comments identify important gaps in transparency that we have addressed through targeted revisions. We respond to each major comment below.","responses":[{"response":"We agree that the original Dataset construction section provided only a high-level description and omitted key details on the labeling process. In the revised manuscript we have expanded this section to include: the complete annotation protocol followed by the eight expert players, the operational definition of toxicity used during labeling, confirmation that annotations were performed independently, the majority-vote aggregation rule applied to produce final labels, and the inter-annotator agreement statistic (Fleiss' kappa). These additions directly improve transparency and allow readers to assess the reliability of the 1.4k toxic labels that underpin our detector.","revision_made":"yes","referee_comment":"[Dataset construction] Dataset construction section: the labeling procedure by the 8 expert players is described only at high level. No annotation protocol, definition of toxicity, independence of labels, majority-vote rule, or inter-annotator agreement statistic (Cohen/Fleiss kappa) is reported. Because the central outperformance claim rests on the quality of these 1.4k toxic labels, missing reliability metrics constitute a load-bearing gap."},{"response":"We acknowledge that the Empirical results section lacked the concrete methodological details needed for proper evaluation. In the revised version we have added: the full set of performance metrics (precision, recall, F1, and AUC) for both our detector and all baselines, the identities of the baseline models (general-purpose and SOTA toxicity detectors), the train/test split procedure, and the cross-validation protocol employed. We also describe how class imbalance was handled. These changes enable readers to assess the robustness of the reported outperformance given the limited number of positive examples.","revision_made":"yes","referee_comment":"[Empirical results] Empirical results section: the abstract asserts that the L2DTnH-trained detector outperforms general-purpose and SOTA NLP toxicity detectors, yet no concrete metrics (precision, recall, F1, AUC), baseline models, train/test split details, or cross-validation procedure are supplied in the provided text. With only 1.4k positive examples, these omissions prevent assessment of whether the reported gains are robust or artifactual."}],"tokens_in":1527,"tokens_out":504,"duration_ms":42239,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main thing to know is that they built and released L2DTnH, a dataset of real in-match League of Legends messages with 1.4k toxic and 13.8k non-toxic examples, then trained a detector on it that outperforms off-the-shelf and state-of-the-art alternatives in their experiments. They also ship a local browser extension and test the model on extra game data outside the main set. That package is the concrete output here. They start with a literature review that shows most existing toxicity work skips live match chat, which justifies the data collection effort. Partnering with eight expert players to label actual game logs is a direct response to the gap they identified, and releasing everything publicly makes the contribution usable by others. The extension that flags content without calling external servers is a practical touch for deployment questions. The soft spot is the labeling step. Eight annotators is a narrow base, and the write-up gives no numbers on agreement between them or any cross-check against actual match reports or player complaints. At 1.4k toxic examples, any consistent bias or noise in those labels would directly affect how much the outperformance claim holds up against broader detectors. The extra testing beyond L2DTnH helps a little but does not fix the training labels themselves. This is for people doing applied work on game moderation, domain-specific NLP, or online community tools. A reader who needs a ready dataset for toxicity in competitive gaming or wants an end-to-end example from collection to a working extension would get something usable. It deserves a serious referee because the data fills a documented gap and is public, even if the evaluation needs more scrutiny on annotation quality. I would send it to review rather than desk reject.","headline":"The paper releases a new dataset of live League of Legends chat messages labeled by eight expert players and shows a detector trained on it beats general toxicity tools, but the labeling process has thin validation.","tokens_in":2430,"tokens_out":430,"would_cite":false,"duration_ms":31480,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"A dataset of messages labeled by expert League of Legends players produces a toxicity detector that outperforms general-purpose NLP tools.","keywords":["toxicity detection","video games","League of Legends","natural language processing","harassment","datasets","machine learning","browser extension"],"falsifier":"Independent expert labeling of a new set of recent League of Legends match messages shows the detector performing no better than general-purpose NLP toxicity detectors.","tokens_in":2736,"feed_emoji":"🎮","tokens_out":544,"duration_ms":34953,"temperature":0.7,"pith_summary":"The paper seeks to fill the gap in tools for detecting toxic messages sent during live video game matches, where most prior work has focused on broader social media rather than in-game chat. A literature review of over one thousand works confirms that few efforts have built and tested NLP-based detectors specifically for match-time harassment. The authors address this by recruiting eight expert players to label more than fifteen thousand League of Legends messages, creating the L2DTnH dataset of toxic and non-toxic examples. They train a detector on this data and show through experiments that it exceeds the performance of both general-purpose toxicity detectors and state-of-the-art NLP models on game content. They also test the detector on additional game-related texts, build a browser extension that flags toxic text locally without external servers, and release all resources for others to use.","feed_headline":"LoL chat dataset builds toxicity detector better than general NLP","feed_subtitle":"Expert labeling of 15,000 in-game messages allows a local browser tool to flag harassment more accurately than broad social media detectors.","key_machinery":"The L2DTnH dataset of expert-labeled in-game chat messages from League of Legends, used to train a specialized NLP toxicity detector that is then compared against general models.","core_discovery":"Through a systematic review of 1,039 works, the paper finds limited prior efforts on ML/NLP detectors for toxicity in live matches. It introduces L2DTnH, a dataset of 1.4k toxic and 13.8k non-toxic messages labeled by eight expert LoL players. A detector trained on L2DTnH is shown to outperform general-purpose and state-of-the-art NLP toxicity detectors. The approach is validated on additional game-related data, and a web browser extension is developed to flag toxic content locally without third-party AI servers. All resources are released publicly to support further applied research against toxicity in video games.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["LoL chats dataset outperforms general NLP toxicity detectors","15k labeled LoL messages train game toxicity detector","Detector for live LoL toxicity uses expert labeled data","Public release of LoL toxicity dataset for applied NLP"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The labels assigned by eight expert players accurately capture what counts as toxicity in live matches and the detector generalizes to new game data without overfitting.","fun_headline_variants_meta":{"raw":{"variants":["LoL chats dataset outperforms general NLP toxicity detectors","15k labeled LoL messages train game toxicity detector","Detector for live LoL toxicity uses expert labeled data","Public release of LoL toxicity dataset for applied NLP"]},"model":"grok-4.3","cost_usd":0.005664,"raw_usage":{"total_tokens":2801,"prompt_tokens":856,"num_sources_used":0,"completion_tokens":60,"cost_in_usd_ticks":56637000,"prompt_tokens_details":{"text_tokens":856,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1885,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":856,"tokens_out":60,"duration_ms":13892,"temperature":1.0,"reasoning_tokens":1885,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-10T15:48:05.867012+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Independent expert labeling of a new set of recent League of Legends match messages shows the detector performing no better than general-purpose NLP toxicity detectors.","supporting_citations":[],"review_version":1}