{"id":"155bf75a-6159-436e-89ee-e1d1a40dfb3b","arxiv_id":"2412.00609","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Cyberbullying detection models trained on one Twitter dataset lose on average 0.222 Macro F1 when tested on another dataset.","lead":"This paper tests whether machine learning models trained to detect cyberbullying on one Twitter dataset still work on other Twitter datasets. Across three datasets the models lost about 0.22 in Macro F1 score, showing that dataset-specific bias limits real-world use.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 0.222 Macro F1 drop is not robust because Dataset 2's 'Offensive' tweets are mapped to Not Cyberbullying, conflating offensive language with non-cyberbullying; a sensitivity analysis of this mapping is needed before the cross-dataset conclusion can be accepted.","rationale":"The reader and I identify the same weakest assumption. The paper's theoretical setup (Section 2.2.1) is sound: models approximate human labeling functions, and differences in H across datasets should cause transfer failures. The six cross-dataset experiments are the only direct evidence for the central claim. However, Dataset 2's binary conversion is an unvalidated equivalence. The paper itself admits that this is not the only feasible way to convert the labels. A sensitivity analysis is cheap and decisive. Because the reader already made the paper conditional on this issue, I do not recommend changing the verdict. No ad hominem; the issue is an unstated assumption, not misconduct. The DQE contamination estimate is a secondary concern because it affects only Dataset 1 and could be checked by re-analyzing sample counts, but it is not as fundamental as the label semantics.","tokens_in":7953,"tokens_out":4005,"duration_ms":37314,"concrete_test":"Re-run the six cross-dataset experiments with Dataset 2 labels changed in two ways: (i) exclude all Offensive tweets from Dataset 2 in both training and test sets; (ii) map Offensive to Cyberbullying, or better, independently re-annotate a random sample of Offensive tweets to obtain a gold-standard binary cyberbullying label. Report the average Macro F1 drop over the same six train/test pairs and per-experiment values. If the average shifts by more than about 0.05 or the pattern of drops reverses, the paper's headline 0.222 is not established as dataset bias; if the average remains near 0.222, the concern is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim depends on treating the three datasets as comparable after a binary relabeling (Section 2.2.2). For Dataset 2, the original three-label scheme from Davidson et al. [2] is Hate Speech / Offensive / Non-Offensive. The paper maps only Hate Speech to Cyberbullying and maps all Offensive tweets to Not Cyberbullying. This is not the original dataset's semantics: 'offensive' is a distinct category from hate speech, not from harassment or cyberbullying, and offensive language can be part of cyberbullying when directed or repeated. Because Dataset 2 participates in four of the six experiments (Exp. 1, 3, 4, 6), this mapping directly influences the average drop of 0.222. If Offensive tweets are predominantly abusive, models trained on Dataset 2 will learn to suppress abusive-language features as negative, inflating drops on Datasets 1 and 3; models tested on Dataset 2 will have abusive tweets labeled negative, inflating drops for Exp. 1 and 6. The authors themselves state the conversion is 'not the only feasible way,' but no alternative mapping is tested, so the headline drop may be an artifact of the relabeling rather than of genuine dataset shift.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript performs cross-dataset evaluation of machine-learning models for cyberbullying detection on three public Twitter corpora. The authors relabel all datasets into a binary Cyberbullying/Not Cyberbullying scheme, train CatBoost and XGBoost models with hyperparameter search and cross-validation, then test each dataset's best models on the other two datasets. They report an average Macro F1 drop of 0.222 across six experiments and interpret this as evidence that current datasets are only useful for same-corpus prediction. The paper also formulates a critique of automated dataset-expansion methods such as Dynamic Query Expansion (DQE).","tokens_in":8234,"tokens_out":4007,"duration_ms":38086,"significance":"If the headline drop is robust, the study provides a useful, concrete cautionary demonstration of dataset shift in cyberbullying detection, with implications for model deployment and dataset curation. The experimental protocol is simple and transparent, code is publicly available, and the authors are unusually honest about limitations, explicitly stating in Section 5 that they do not provide causal evidence linking labeling differences to the observed drop. However, the central quantitative claim currently rests on a contested relabeling of one dataset's 'Offensive' class, and on averages reported without any measure of variability or significance testing; both issues must be addressed before the claim can be considered established.","major_comments":[{"comment":"The mapping of Dataset 2's 'Offensive' class to Not Cyberbullying is a load-bearing modeling choice. Dataset 2 participates in four of the six experiments (Exp. 1, 3, 4, and 6), so this mapping directly influences the average drop of 0.222 reported in Section 4. The authors themselves note that the conversion is 'not the only feasible way,' and the original Davidson et al. scheme treats Offensive as a distinct category from both Hate Speech and Non-Offensive, not as non-cyberbullying. A sensitivity analysis is needed: for example, re-run the pipeline with Offensive mapped to Cyberbullying, or with Dataset 2 excluded, and report how the average drop changes. Without this, the cross-dataset conclusion cannot be separated from the relabeling artifact.","section":"Section 2.2.2, Table 1"},{"comment":"The abstract and Section 4 describe the drop as 'significant,' but no measure of variability or statistical test is provided. Each experiment averages over the top 10 models, so the manuscript should report the standard deviation across those models, a paired test comparing cross-validation and cross-dataset Macro F1 per model (e.g., Wilcoxon signed-rank), or a bootstrap confidence interval. This is particularly important because Exp. 6 has a negative drop in F1 Weighted (-0.0108), demonstrating substantial experiment-level variability that the current reporting hides.","section":"Section 3.2, Table 1"}],"minor_comments":[{"comment":"The claim that Dataset 1 is 'likely overtly tainted' by DQE is supported only by an expected value (25,358.20) with no confidence interval or variance estimate. Since the DQE hypothesis is not directly tested in the experiments, this language should be softened or supplemented with a sensitivity analysis.","section":"Section 2.1.1"},{"comment":"The text cites 'Van Hee et al.' for a definition of cyberbullying, but reference [8] is Van Cleemput et al., 'The development of a serious game on cyberbullying.' Please correct the citation or use the intended source.","section":"Introduction, reference [8]"},{"comment":"There is a typo: 'different different learning rates' should read 'different learning rates.'","section":"Section 2.2.3"},{"comment":"The terms 'Macro Mean F1,' 'F1 Macro Mean,' 'Macro F1 Score,' and 'F1 Weighted' are used inconsistently. Please standardize terminology to match Table 1.","section":"Section 4"},{"comment":"The admission that the paper does 'not explicitly provide evidence that differing labeling schemes or definitions affect the overall performance drop' is important and should be reflected in the abstract and introduction, which currently frame the drop as evidence of bias from definitions and collection methods.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a straightforward empirical study that could be suitable for publication after the sensitivity analysis and uncertainty quantification are added. The label-mapping issue is the main technical risk; if a robustness check shows the drop persists across reasonable relabelings, the paper's central message will be much stronger. The absence of error bars or significance tests is surprising given that the top-10-model protocol makes such analyses easy to compute."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid, honest empirical demonstration that cyberbullying models trained on one public Twitter corpus lose a lot of Macro F1 on another, with code and data shipped. The headline 0.222 average drop is believable as a qualitative claim, but the specific number is weaker than it looks because of how Dataset 2's 'Offensive' labels were folded into 'Not Cyberbullying.'\n\nWhat's actually new: not the general point—cross-dataset generalization failures are well known in NLP, and the paper says so itself. The contribution is the three-dataset measurement set and the explicit formalization of Dynamic Query Expansion (DQE) as a labeling function that a model might approximate instead of human labelers. That formalism is simple but useful, and it makes the data collection critique concrete. The OOV correlation analysis is a nice negative result; it rules out one simple explanation for the drop.\n\nSoft spots, in order of importance. The label mapping. Section 2.2.2 maps Hate Speech to Cyberbullying and Offensive plus Non-Offensive to Not Cyberbullying. Davidson et al. treat Offensive as a separate category from hate speech, and offensive language can be part of cyberbullying. Because Dataset 2 appears in four of the six experiments, this choice directly drives the average drop. The authors admit the conversion is not the only feasible way, but they never test an alternative mapping. This is fixable, not fatal: even if you remapped Offensive as Cyberbullying, the drops are large across all pairs, so the qualitative conclusion would likely survive. But the specific 0.222 number needs either a sensitivity analysis or a caveat that it depends on a contested mapping.\n\nSecond, there are no error bars or significance tests on the cross-dataset drops. Table 1 gives averages but no variance across the ten models per experiment, and the abstract's 'significant' is informal. Minor relative to the effect size.\n\nThird, the DQE contamination estimate in Section 2.1.1 is unverifiable: the hypergeometric calculation uses quantities that the original paper doesn't document. This doesn't affect the cross-dataset results.\n\nWho should read this: anyone working on cyberbullying detection or cross-dataset evaluation of text classifiers. It's a good case study for a seminar. It deserves a real referee: the empirical work is reproducible and the label mapping issue is easily fixed in revision.\n\nRecommendation: send to peer review with a request for sensitivity analysis on the label mapping and either error bars or a more modest claim.","headline":"A useful, honest cross-dataset evaluation with a debatable label mapping that weakens the headline number; the paper is worth refereeing.","tokens_in":8729,"tokens_out":3513,"would_cite":false,"duration_ms":29275,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Cross-dataset tests show cyberbullying models lose 0.222 Macro F1","keywords":["cyberbullying detection","cross-dataset evaluation","dataset bias","label mapping","dynamic query expansion","Macro F1 score","machine learning generalization","Twitter datasets"],"falsifier":"Re-run the six cross-dataset experiments after moving Dataset 2's Offensive class from Not Cyberbullying to Cyberbullying (or excluding it entirely). If the average Macro F1 drop falls well below 0.222, the headline result is an artifact of the label mapping; if the drop remains, the conclusion of corpus-level bias is supported.","tokens_in":7766,"feed_emoji":"📉","tokens_out":4621,"duration_ms":39786,"temperature":0.7,"pith_summary":"This paper tries to establish that cyberbullying detection models trained on one Twitter dataset do not transfer to other Twitter datasets, even when training uses strong anti-overfitting measures. It reports an average drop of 0.222 in Macro F1 when the best models from cross-validation are tested on unseen corpora, and argues the gap comes from differences in how cyberbullying is defined, how tweets are collected, and how labels are assigned. The paper also singles out Dynamic Query Expansion (DQE), an automated method that labels new tweets using keywords from already-labeled ones, as a source of bias that can make a model approximate the expansion algorithm instead of human judgment. If correct, the findings imply that reported cyberbullying detection accuracy on a single dataset overstates real-world usefulness.","feed_headline":"Cyberbullying models drop 0.222 in F1 on unseen datasets","feed_subtitle":"Testing three Twitter corpora shows that detectors trained on one dataset do not transfer to another.","key_machinery":"The load-bearing machinery is the cross-dataset evaluation protocol: stratified k-fold cross-validation selects the top ten models from two gradient-boosting classifiers per dataset, then each is retrained on its own dataset and tested on the other two, with the Macro F1 drop relative to cross-validation as the outcome. Alongside this, the paper defines Dynamic Query Expansion (DQE) as a function DQE(D_y) = (x̂, y) that pulls a new tweet x̂ from outside the dataset and immediately assigns it the label y of the tweets that generated the query. This formulation shows how automated expansion can silently replace the human labeling function H with the DQE rule, and it motivates the prediction that models trained on such data will fail on datasets labeled by H.","core_discovery":"The central claim is that the usefulness of a cyberbullying model is dataset-bound: a model that scores well in cross-validation on its training corpus drops substantially when evaluated on another corpus. In six experiments pairing three Twitter-based datasets, the average Macro F1 drop is 0.222 and the average weighted F1 drop is 0.253, with the largest single Macro F1 drop at 0.3912. The paper argues this drop is evidence of bias in data collection and labeling, and it formalizes DQE as a function that both fetches and labels tweets in one step, so that a model trained on DQE-expanded data may be approximating DQE rather than the human labeling process.","pith_inferences":["If the label mapping is the real driver of the drop, then the same experimental setup could be reused to quantify definitional disagreement: by remapping labels under alternative definitions and measuring how much the average drop changes, one could turn a negative result into a measurement of labeling bias.","The critique of DQE extends beyond cyberbullying to any semi-supervised or synthetic-data augmentation pipeline in NLP: the danger is not only keyword bias but that the model learns the labeling shortcut, so a testable extension is to compare models trained on augmented versus human-labeled subsets of the same corpus while controlling for size.","The OOV correlation results suggest that vocabulary mismatch is not the main cause of the drop; a direct next experiment would be to measure whether the drop persists when all three datasets are filtered to a common vocabulary, isolating label-definition effects from lexical effects.","A practical application of the finding is that dataset health can be scored by cross-dataset transferability, giving dataset curators a cheap proxy for annotation quality without additional human labeling."],"forward_implications":["Cyberbullying models should not be deployed on new platforms or populations without first being tested on a locally labeled sample; a strong within-dataset score is not evidence of real-world readiness.","Dataset curation choices, especially lexicon-based collection and automated expansion, need to be reported alongside accuracy numbers, because they determine whether the model learned human labeling or the collection rule.","Cross-dataset evaluation, not just held-out splits of one dataset, should become a standard reporting requirement for cyberbullying detection research.","Models trained on DQE-expanded data may be approximating the expansion algorithm instead of human labelers, so their apparent performance is not a measure of cyberbullying detection ability.","An average drop of 0.222 in Macro F1 means a detector that looks reliable in research settings can be materially worse in practice."],"supporting_citations":[{"why":"Supplies Dataset 1 and the Dynamic Query Expansion method whose bias is central to the paper's critique.","marker":"[3]"},{"why":"Supplies Dataset 2, with lexicon-based collection and Hate/Offensive/Non-Offensive labels that create the mapping issue.","marker":"[2]"},{"why":"Supplies Dataset 3, the online harassment corpus collected with alt-right-related search terms.","marker":"[10]"},{"why":"Provides a prior example of strong within-dataset results but limited cross-dataset hate speech detection.","marker":"[14]"},{"why":"Documents how lexicon-dependent models fail on non-explicit cyberbullying in different contexts.","marker":"[4]"},{"why":"Shows a keyword-based expansion method that propagates collection bias.","marker":"[12]"},{"why":"Supports the general need for cross-dataset evaluation when assessing model generalizability.","marker":"[17]"}],"fun_headline_variants":["Cyberbullying AI fails on new datasets, F1 drops 0.222","Cross-dataset test: Cyberbullying models lose 0.222 F1","Bias in cyberbullying data: Models drop 0.222 F1 on new data","Cyberbullying detectors don't generalize: 0.222 F1 drop","Dataset bias trims cyberbullying model F1 by 0.222"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the three datasets can be compared after mapping each original label to Cyberbullying or Not Cyberbullying, in particular treating Dataset 2's Offensive tweets as Not Cyberbullying; if that mapping is wrong, the reported 0.222 drop could be an artifact of the relabeling rather than of dataset bias.","fun_headline_variants_meta":{"raw":{"variants":["Cyberbullying AI fails on new datasets, F1 drops 0.222","Cross-dataset test: Cyberbullying models lose 0.222 F1","Bias in cyberbullying data: Models drop 0.222 F1 on new data","Cyberbullying detectors don't generalize: 0.222 F1 drop","Dataset bias trims cyberbullying model F1 by 0.222"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000606,"raw_usage":{"total_tokens":2780,"prompt_tokens":856,"completion_tokens":1924,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":472,"completion_tokens_details":{"reasoning_tokens":1817}},"tokens_in":472,"tokens_out":1924,"duration_ms":19550,"temperature":1.0,"reasoning_tokens":1817,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:10:26.162463+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the six cross-dataset experiments after moving Dataset 2's Offensive class from Not Cyberbullying to Cyberbullying (or excluding it entirely). If the average Macro F1 drop falls well below 0.222, the headline result is an artifact of the label mapping; if the drop remains, the conclusion of corpus-level bias is supported.","supporting_citations":[{"cited_title":"Sosnet: A graph convolutional network approach to fine-grained cyberbullying detection,","cited_arxiv_id":null,"evidence_quote":"Supplies Dataset 1 and the Dynamic Query Expansion method whose bias is central to the paper's critique."},{"cited_title":"Effective hate-speech detection in twitter data using recurrent neural networks,","cited_arxiv_id":null,"evidence_quote":"Provides a prior example of strong within-dataset results but limited cross-dataset hate speech detection."},{"cited_title":"ID-XCB: Data-independent Debiasing for Fair and Accurate Transformer-based Cyberbullying Detection","cited_arxiv_id":"2402.16458","evidence_quote":"Documents how lexicon-dependent models fail on non-explicit cyberbullying in different contexts."},{"cited_title":"Hateful symbols or hateful people? predictive features for hate speech detection on Twitter,","cited_arxiv_id":null,"evidence_quote":"Shows a keyword-based expansion method that propagates collection bias."},{"cited_title":"Generalizability of Machine Learning Models: Quantitative Evaluation of Three Methodological Pitfalls","cited_arxiv_id":"2202.01337","evidence_quote":"Supports the general need for cross-dataset evaluation when assessing model generalizability."}],"review_version":1}