REVIEW 3 major objections 4 minor 1 cited by
Offensive Language and Hate Speech Detection for Danish
T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read First Danish dataset for offensive-language detection, annotated for type and target, supports classifiers reaching macro F1 of 0.70.
desk verdict First Danish offensive-language dataset, but the weak annotation reliability means the headline F1 scores should be read as provisional until the labels are cleaned or re-checked. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is a three-level annotation scheme for offensive language: first decide whether a post is offensive, then whether the offence is targeted at someone, then whether the target is an individual, a group, or something else. This scheme comes from the English-language task that the paper adopts, and the paper's Danish corpus instantiates it for the first time. The argument is carried by a suite of classifiers — a logistic regression baseline and three BiLSTM variants using learned embeddings, fixed pre-trained embeddings, or augmented features such as sentiment scores and n-grams — evaluated on both the English and Danish datasets under the same three subtasks.
What would settle it
Take a random sample of roughly 300 Danish posts from the corpus, have several independent Danish-speaking annotators label them with the same guidelines, and measure agreement with the published labels; if the original labels cannot be reproduced or agreement is far below the pilot Jaccard values, the reported F1 scores do not reflect real detection quality.
Extended reading notes
Core claim
The paper's contribution is the first Danish dataset for offensive-language detection, together with evidence that detectors built on it work at levels close to English. The corpus consists of 3,600 user-generated comments gathered from Danish Reddit and Facebook, labeled according to three subtasks: offensive versus not offensive, targeted versus untargeted, and target type (individual, group, other). The best Danish systems reach macro F1 scores of 0.70, 0.73, and 0.63 on these three subtasks, while the English systems reach 0.74, 0.62, and 0.56. The result is presented as showing that a shared annotation scheme, combined with pre-trained word embeddings and a mix of lexical and linguistic features, supports automatic offensive-language detection in Danish despite a small, imbalanced training set.
Load-bearing premise
The dataset labels are assumed to be reliable enough to train and evaluate the detectors, but only 100 posts were double-annotated with low agreement, and the remaining 3,500 were annotated by a single person after a guideline change.
Editorial extensions
If this is right
- Danish content moderation can use these models to flag offensive comments at accuracy levels comparable to English systems, with a concrete best macro F1 of 0.70 for the binary task.
- The released corpus gives Danish NLP a benchmark for future work on offensive and hateful language, including cross-platform tests since the data comes from two different platforms.
- The finding that a fixed pre-trained embedding model is best for English but worst for Danish suggests no single architecture dominates, so practical deployments should tune per language and task.
- Class imbalance is the main bottleneck: with only about 12% offensive posts in Danish, recall for the minority classes is low, so methods that rebalance training data are likely to matter more than model choice.
- The same annotation scheme and model family used for English yields workable Danish results, supporting the broader idea that this task can be exported to other under-resourced languages.
Reading between the lines
- We infer that the reported F1 scores should be treated as upper bounds until the labels themselves are validated: only 100 posts were double-annotated, with Jaccard agreement of 39–43%, and the remaining 3,500 posts were labeled by one person after the guidelines were revised.
- The same low agreement on the pilot set suggests the annotation guidelines are ambiguous for Danish, especially for context-dependent posts; a follow-up with a larger annotator pool and adjudication would clarify how much of the measured performance is learning the task versus learning one annotator's interpretation.
- Because Danish is closely related to Norwegian and Swedish, the dataset could seed cross-lingual transfer for offensive-language detection in other low-resource Scandinavian settings, though this is beyond what the paper tests.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper constructs a Danish dataset of 3,600 user-generated comments from Reddit and Facebook, annotated following the OffensEval schema for three sub-tasks: offensive language identification (A), categorization of offensive language type (B), and offensive language target identification (C). It compares logistic regression and several BiLSTM-based classifiers on this dataset and on the English OLID dataset, reporting macro-averaged F1 scores of 0.70 for Danish sub-task A, 0.73 for sub-task B, and 0.63 for sub-task C. The paper also analyzes misclassifications and claims to provide the first Danish dataset of its kind.
Significance. If the dataset and reported results are reliable, the contribution is useful for under-resourced Danish NLP: it provides a new resource, follows an established annotation schema, compares multiple standard models, and includes a data statement. The availability of the classifiers and dataset under CC-BY is a strength, and the error analysis in Section 7 is a useful addition. However, the central quantitative claims rest on gold labels whose reliability is not established and that are, by the authors' own admission, inconsistent with the final annotation guidelines in at least some test instances. The contribution is therefore significant in intention but the specific performance numbers are not currently well supported.
major comments (3)
- [Section 7, Sub-task A] Section 7 states that the classifier labels profanity-bearing content as offensive while the test set labels such posts NOT, giving 'Are you fucking serious?' and 'Fuck I cried in this scene' as examples that 'according to annotation guidelines should be classed as offensive.' This directly contradicts the final annotation rule in Section 3.1 that any post containing any form of profanity should automatically be labeled offensive. Since these test labels are the gold labels used to compute Table 4 and the class-wise scores in Table 5, the reported Danish sub-task A macro F1 of 0.699 is computed against a test set that the authors themselves identify as containing wrong labels. The reported number is therefore not a trustworthy estimate of performance under the final annotation schema.
- [Section 3.1] Inter-annotator agreement was measured only on the first 100 posts, with Jaccard indices of 41.9% for sub-task A, 39.1% for sub-task B, and 42.8% for sub-task C. After the guidelines were refined, the remaining approximately 3,500 posts were annotated by a single author, and no final agreement study, second pass, or re-annotation of the warm-up posts is reported. Given the low agreement on the initial set, the reliability of the final labels is not established. The classifier scores in Tables 4, 7, and 10 should be interpreted with this fundamental uncertainty in mind.
- [Section 7, Sub-task B] In the error analysis for sub-task B, the authors report that a test-set sample containing 'HillaryForPrison' is labeled as untargeted, although they characterize it as a clear targeted insult. This is another direct example of a test-set label violating the authors' own understanding of the annotation schema. It affects the Danish sub-task B evaluation reported in Table 7 and indicates that label noise is not confined to sub-task A, further undermining the reliability of the reported F1 scores.
minor comments (4)
- [Section 6, Sub-task B (English)] The text says 'Recall and precision scores are lower for UNT than TIN (Table 5)', but Table 5 reports sub-task A results; the relevant table is Table 8.
- [Throughout] The terms 'macro averaged F1-score', 'F1macro', and 'macro F1' are used interchangeably; please standardize the notation.
- [Section 4] There are several typos and possible OCR artifacts, such as 'T op-level features', 'the the' in the Background section, and 'Futher' in Section 6. A careful proofreading pass is needed.
- [Tables 3 and 4] The baseline rows labeled 'All NOT' report macro F1 values; it would be helpful to state explicitly how the macro average is computed for a one-class baseline.
Circularity Check
No significant circularity: the Danish classifier scores are standard train/test evaluations, and the paper's own Section 7 flags only a label-consistency problem.
full rationale
The paper's central quantitative claims are empirical macro-F1 scores obtained by training four classifiers on a training split and evaluating them on a held-out test split. No fitted parameter, coefficient, or lexicon-derived quantity is renamed as a prediction. The annotation schema is adopted from external OffensEval work ([1], [18]), not derived from the present model outputs. The novelty claim (first Danish offensive-language dataset) is supported by an external resource survey [5]; the fact that one author appears on both that survey and the present paper does not make the evaluation circular, because the F1 numbers do not depend on that survey. The paper's own Section 7 admits a label-consistency defect: profanity-bearing test posts such as "Are you fucking serious?" and "Fuck I cried in this scene" are labeled NOT although the final annotation rule says any post containing profanity should be offensive. This is a data-quality and correctness risk, not a circularity, because the classifier outputs are compared against fixed test labels rather than derived from those labels, and no equation in the paper reduces a claimed prediction to its input. Likewise, the low Jaccard indices (41.9%, 39.1%, 42.8%) on the initial 100 double-annotated posts and single-author annotation of the remaining 3,500 posts concern annotation reliability, not circular derivation. No self-definitional step, no fitted-input-as-prediction step, and no self-citation chain that forces a central result was found.
Assumptions & free parameters
assumptions (2)
- domain assumption The OLID annotation scheme from [1] is applicable to Danish and the categories (NOT/OFF, TIN/UNT, IND/GRP/OTH) have the same meaning in both languages.
- domain assumption The dataset, annotated primarily by one annotator after an initial low-agreement warm-up, is of sufficient quality to support the reported results.
Cite this review
Pith. "Pith review of Offensive Language and Hate Speech Detection for Danish." pith.science (2026). https://pith.science/paper/25CH75FR
@misc{pith2026190804531,
author = {Pith},
title = {Pith review of: Offensive Language and Hate Speech Detection for Danish},
year = {2026},
howpublished = {\url{https://pith.science/paper/25CH75FR}},
note = {Machine review of arXiv:1908.04531}
}
abstract
The presence of offensive language on social media platforms and the implications this poses is becoming a major concern in modern society. Given the enormous amount of content created every day, automatic methods are required to detect and deal with this type of content. Until now, most of the research has focused on solving the problem for the English language, while the problem is multilingual. We construct a Danish dataset containing user-generated comments from \textit{Reddit} and \textit{Facebook}. It contains user generated comments from various social media platforms, and to our knowledge, it is the first of its kind. Our dataset is annotated to capture various types and target of offensive language. We develop four automatic classification systems, each designed to work for both the English and the Danish language. In the detection of offensive language in English, the best performing system achieves a macro averaged F1-score of $0.74$, and the best performing system for Danish achieves a macro averaged F1-score of $0.70$. In the detection of whether or not an offensive post is targeted, the best performing system for English achieves a macro averaged F1-score of $0.62$, while the best performing system for Danish achieves a macro averaged F1-score of $0.73$. Finally, in the detection of the target type in a targeted offensive post, the best performing system for English achieves a macro averaged F1-score of $0.56$, and the best performing system for Danish achieves a macro averaged F1-score of $0.63$. Our work for both the English and the Danish language captures the type and targets of offensive language, and present automatic methods for detecting different kinds of offensive language such as hate speech and cyberbullying.
Forward citations
Cited by 1 Pith paper
-
A Survey on Automatic Online Hate Speech Detection in Low-Resource Languages
A survey cataloging datasets, features, and machine-learning methods for automatic hate speech detection in low-resource languages, organized by world region, with an overview of open challenges.
Reference graph
Works this paper leans on
-
[1]
Predicting the type and target of offensive posts in social media
Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara R osenthal, Noura Farra, and Ritesh Kumar. Predicting the type and target of offensive posts in social media. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Ling uistics: Human Language T echnologies, V olume 1 (Long and Short Papers), pages 141...
work page 2019
-
[2]
Fanning the flames of ha te: Social media and hate crime
Karsten Müller and Carlo Schwarz. Fanning the flames of ha te: Social media and hate crime. Available at SSRN 3082972, 2018
work page 2018
-
[3]
Automated hate speech detection and the problem of offensive language
Thomas Davidson, Dana Warmsley, Michael Macy, and Ingma r Weber. Automated hate speech detection and the problem of offensive language. In Eleventh International AAAI Conference on W eb and Social Media, 2017
work page 2017
-
[4]
A survey on hate speech detection using natural language processing
Anna Schmidt and Michael Wiegand. A survey on hate speech detection using natural language processing. In Proceedings of the Fifth International W orkshop on Natural Language Processing for Social Media , pages 1–10, 2017
work page 2017
-
[5]
The Lacunae of Danish Natural Language Processing
Andreas Kirkedal, Barbara Plank, Leon Derczynski, and N atalie Schluter. The Lacunae of Danish Natural Language Processing. In Proceedings of the Nordic Conference on Computational Ling uistics (NODALIDA). Northern European Association for Language Technology, 20 19
-
[6]
Kvinder nedgøres oftere end mænd i politiske debatter på sociale medier
Leon Derczynski et al. Kvinder nedgøres oftere end mænd i politiske debatter på sociale medier. TjekDet / Mandag Morgen, 2019
work page 2019
-
[7]
Guide- lines for the fine-grained analysis of cyberbullying
Cynthia V an Hee, Ben V erhoeven, Els Lefever, Guy De Pauw, Véronique Hoste, and Walter Daelemans. Guide- lines for the fine-grained analysis of cyberbullying. Techn ical report, Language and Translation Technology Team, Ghent University, 2015
work page 2015
-
[8]
Cyberstalking: Dangers on the informa tion superhighway
Trudy M Gregorie. Cyberstalking: Dangers on the informa tion superhighway. National Center for Victims of crime, 2001. 10 OFFENSIVE LANGUAGE AND HATE SPEECH DETECTION FOR DANISH
work page 2001
Show all 30 references
-
[9]
Detection and fine-grained classificat ion of cyberbullying events
Cynthia V an Hee, Els Lefever, Ben V erhoeven, Julie Menne s, Bart Desmet, Guy De Pauw, Walter Daelemans, and Véronique Hoste. Detection and fine-grained classificat ion of cyberbullying events. In Proceedings of the international conference recent advances in natural langu age ...
2015
-
[10]
The international covenant on civil and political rights: c ases, materials, and commentary
Sarah Joseph and Melissa Castan. The international covenant on civil and political rights: c ases, materials, and commentary. Oxford University Press, 2013
2013
-
[11]
https://danskelove.dk/straffeloven/266b
Straffeloven § 266 b. https://danskelove.dk/straffeloven/266b. Accessed: 2019-05-29
2019
-
[12]
https://eur-lex.europa.eu/legal-content/EN/TXT/?uri =LEGISSUM%3Al33178
EU Council Framework Decision 2008/913/JHA. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri =LEGISSUM%3Al33178. Accessed: 2019-05-29
2008
-
[13]
Regulating hate speech online
James Banks. Regulating hate speech online. International Review of Law, Computers & T echnology, 24(3):233– 239, 2010
2010
-
[15]
Understanding abuse: A typology of abusive language detection subtasks
Zeerak Waseem, Thomas Davidson, Dana Warmsley, and Ing mar Weber. Understanding abuse: A typology of abusive language detection subtasks. In Proceedings of the First W orkshop on Abusive Language Onlin e, pages 78–84, V ancouver, BC, Canada, August 2017. Association for Comput...
2017
-
[16]
Abusive language detection in online user content
Chikashi Nobata, Joel Tetreault, Achint Thomas, Y asha r Mehdad, and Yi Chang. Abusive language detection in online user content. In Proceedings of the 25th international conference on world w ide web, pages 145–153. International World Wide Web Conferences Steering Committ ee, 2016
2016
-
[17]
Hateful symbols or hateful people? Predictive features for hate speech detection on twitter
Zeerak Waseem and Dirk Hovy. Hateful symbols or hateful people? Predictive features for hate speech detection on twitter. In Proceedings of the NAACL student research workshop , pages 88–93, 2016
2016
-
[18]
Semeval- 2019 task 6: Identifying and categorizing offensive langua ge in social media (offenseval)
Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. Semeval- 2019 task 6: Identifying and categorizing offensive langua ge in social media (offenseval). In Proceedings of SemEval, 2019
2019
-
[19]
Similarity measures in scientometr ic research: The jaccard index versus salton’s cosine formula
Lieve Hamers et al. Similarity measures in scientometr ic research: The jaccard index versus salton’s cosine formula. Information Processing and Management , 25(3):315–18, 1989
1989
-
[20]
Efficient estimation of word representations in vector space
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean . Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 , 2013
2013 arXiv
-
[21]
Distributed representation s of sentences and documents
Quoc Le and Tomas Mikolov. Distributed representation s of sentences and documents. In International confer- ence on machine learning , pages 1188–1196, 2014
2014
-
[22]
Deep learning for hate speech de- tection in tweets
Pinkesh Badjatiya, Shashank Gupta, Manish Gupta, and V asudeva V arma. Deep learning for hate speech de- tection in tweets. In Proceedings of the 26th International Conference on W orld W ide W eb Companion, pages 759–760, 2017
2017
-
[23]
GloV e: Global V ectors for word representation
Jeffrey Pennington, Richard Socher, and Christopher M anning. GloV e: Global V ectors for word representation. In Proceedings of the 2014 conference on empirical methods in n atural language processing (EMNLP) , pages 1532–1543, 2014
2014
-
[24]
Advances in pre- training distributed word representations
Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Chris tian Puhrsch, and Armand Joulin. Advances in pre- training distributed word representations. In Proceedings of the International Conference on Language Resources and Evaluation (LREC) , 2018
2018
-
[25]
V ader: A parsimonious rule-based model for sentiment analysis of social media text
Clayton J Hutto and Eric Gilbert. V ader: A parsimonious rule-based model for sentiment analysis of social media text. In Proceedings of the international AAAI conference on weblog s and social media (ICWSM) , 2014
2014
-
[26]
A new anew: Evaluation of a word list f or sentiment analysis in microblogs
Finn Årup Nielsen. A new anew: Evaluation of a word list f or sentiment analysis in microblogs. arXiv preprint arXiv:1103.2903, 2011
2011 arXiv
-
[27]
Scikit-learn: Machine learning in Python
Fabian Pedregosa, Gaël V aroquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Math- ieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubour g, et al. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12(Oct):2825–2830, 2011
2011
-
[28]
François Chollet et al. Keras. https://keras.io, 2015
2015
-
[29]
Adam: A method for stocha stic optimization
Diederik P Kingma and Jimmy Ba. Adam: A method for stocha stic optimization. arXiv preprint arXiv:1412.6980, 2014
2014 arXiv
-
[30]
Bender and Batya Friedman
Emily M. Bender and Batya Friedman. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguis tics, 6:587– 604, 2018. 11 OFFENSIVE LANGUAGE AND HATE SPEECH DETECTION FOR DANISH
2018
-
[31]
Offensive & hate speec h detection
Guðbjartur Ingi Sigurbergsson. Offensive & hate speec h detection. Master’s thesis, IT University of Copenhagen, 2019. 12 OFFENSIVE LANGUAGE AND HATE SPEECH DETECTION FOR DANISH A Data statement Curation rationale Examples of offensive language and hate speech, in Danish Langu...
2019
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.