Beyond Dataset Creation: Critical View of Annotation Variation and Bias Probing of a Dataset for Online Radical Content Detection

Arij Riabi; Djam\'e Seddah; Menel Mahamdi; Virginie Mouilleron; Wissam Antoun

REVIEW 3 major objections 6 minor 12 references

Beyond Dataset Creation: Critical View of Annotation Variation and Bias Probing of a Dataset for Online Radical Content Detection

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Radical-content detection is shaped by who labels and who tests the model.

desk verdict The COUNTER dataset and annotation-aggregation experiments are the real contribution; the synthetic bias analysis is confounded and should be reframed as exploratory. read the letter →

arxiv 2412.11745 v2 pith:24BV6JBA submitted 2024-12-16 cs.CL

Arij Riabi , Virginie Mouilleron , Menel Mahamdi , Wissam Antoun , Djamé Seddah This is my paper · ORCID

classification cs.CL

keywords radicalcontentdetectionmultilingualdatasetannotationdisagreementlabelaggregationsyntheticpersonadatabiasprobingfairnessmetricspseudonymization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Detecting radical content online is not a stable labeling task, the paper argues: different human annotators disagree enough that the same model can look noticeably better or worse purely because of how labels were aggregated and which test slices were used. To make that case, the paper presents COUNTER, a pseudo-anonymized dataset of English, French, and Arabic posts from platforms such as Twitter, Telegram, and 4chan, annotated for radicalization level, call for action, and named entities. The authors show that contractor annotations, majority vote, and MACE-derived labels disagree substantially, and that training on one gold standard and testing on another drops French Macro-F1 by up to about 15 points. They then generate synthetic posts from hand-crafted socio-demographic profiles and report systematic performance gaps across nationality, ethnicity, political view, religion, and place of living, with the largest fairness differences for place of living, ethnicity, and nationality. If these findings hold, evaluation and fairness audits of radical-content detectors must be reported as conditional on annotation and demographic choices.

What carries the argument

The load-bearing machinery is the COUNTER resource itself plus two controlled perturbations of it: (1) the multiple annotation views — prescriptive contractor labels, descriptive annotator labels, and MACE and majority aggregations — used to show that label construction changes what a model learns and how it scores; and (2) the synthetic persona-prompting setup, in which Wizard-Vicuna-13B-Uncensored generates posts from hand-written profiles varying gender, age, ethnicity, nationality, political view, religion, job, language register, and place of living, used to probe group-level performance with demographic parity and equalized odds.

What would settle it

Collect genuine extremist or radical posts whose authors' demographic attributes are known or reliably inferred, annotate them with the same guidelines, and measure the same Macro-F1 gaps across nationality, ethnicity, religion, and political view. If the real-data gaps do not reproduce the synthetic gaps — for example, if performance is roughly equal across groups once topic is controlled — the claimed sociodemographic influence on model outcomes would be an artifact of the generation process rather than a property of radical-content detection.

Watch

Extended reading notes

Core claim

The central claim is that the label a radical-content detector learns is co-produced by the annotation protocol and the annotators' socio-demographic positions, and that this propagates into model evaluation. The paper demonstrates this with an English, French, and Arabic corpus whose Call for Action and Radicalization Level labels come from a prescriptive contractor pass plus a descriptive double-annotation pass; inter-annotator agreement ranges from slight (English Call for Action, Fleiss' kappa 0.13) to moderate (French Radicalization Level, 0.50). Models trained and evaluated on different aggregation views of the same posts show large gaps, and in the synthetic persona experiment all tested models show uneven Macro-F1 across demographic attribute values, with XLM-T showing both the best overall performance and the largest disparities. The authors conclude that subjectivity in human annotations and sociodemographic variations measurably influence data and model outcomes, so robust radical-content detection requires reporting and accounting for these choices.

Load-bearing premise

The fairness conclusion rests on treating posts that an uncensored language model wrote from hand-crafted demographic profiles as representative of how real online radical content varies across demographic groups.

Editorial extensions

If this is right

Reported performance on radical-content detection is partly an artifact of label aggregation: models trained and tested on the same aggregation look best, and switching the test gold standard can drop French Macro-F1 by roughly 15 points.
Multi-task auxiliary signals such as radicalization level, ideology prediction, and named-entity recognition do not reliably improve Call for Action detection, while separate per-language classifiers consistently beat a shared classifier.
A regression formulation on the ordinal Call for Action scale avoids distant misclassifications but earns lower Macro-F1, making the classification-versus-regression choice a trade-off between discrete accuracy and error severity.
Fairness audits on synthetic demographic profiles reveal large demographic-parity and equalized-odds differences for place of living, ethnicity, and nationality, with English nationality reaching an equalized-odds difference of 0.66 for XLM-T.
Synthetic bias patterns are model-dependent: XLM-T attains higher overall scores but larger demographic disparities than XLM-R and mBERT, suggesting that social-media pretraining amplifies attribute-correlated errors.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

My reading: the synthetic-data evidence should be treated as an upper bound on sociodemographic bias, because the generator's own stereotypes may produce the attribute-correlated language; validating against real posts with known author demographics is the decisive test of the fairness claim.
My reading: the kappa asymmetry between English (0.13–0.26) and French (0.43–0.50) may reflect differences in the annotators' cultural proximity to the two corpora; a controlled study varying annotator cultural background would separate task ambiguity from annotator subjectivity.
My reading: for deployment, these results argue for reporting a performance range over aggregation methods and demographic slices, and for training on disagreement-aware objectives rather than a single aggregated gold label.
My reading: the paper's release of multi-view labels invites predicting full label distributions, which would make moderation decisions more transparent about uncertainty than thresholding a single class.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Bottom line: the dataset is the contribution; the bias analysis is not. I would send this to reviewers, but with a clear note that the fairness claims need reframing or re-analysis.

What is new: COUNTER is a genuinely useful trilingual dataset—English, French, Arabic—with radicalization level, call-for-action, NER, annotator disagreement, guidelines, and pseudonymization. That fills a real gap. The annotation-variation experiments are the strongest part: three labels for French, aggregation via MACE, majority vote, repeated labeling, and an annotator-classifier, with results that honestly show how much the choice of gold standard changes measured performance. The regression-versus-classification comparison is a nice, modest addition; the adjacent-class error pattern is worth having.

The soft spot is Section 5.2. The synthetic profiles in Table 10 include political view and religion/culture as generation conditions. That means the demographic variable is not varied independently of the ideology and content of the generated post. The Macro-F1 gaps across far-right versus far-left and across religions are as easily read as per-class difficulty as model bias. On top of that, demographic parity is computed on data where the positive-class base rate varies across groups; even a perfectly calibrated model would show parity differences under that label shift. The authors are transparent that the synthetic data may be stereotyped and not real-world, and the Limitations section acknowledges this, but the conclusion still leans on the bias result. I would want the paper to either re-analyze with ideology fixed, or report per-attribute results conditional on political view and religion, and to state plainly that the fairness numbers are not identified.

Minor: no code is shipped and the dataset is on request, which is reasonable for sensitive data but lowers reproducibility below what the phrase "publicly available" implies. The annotation section could also say more about the small number of annotators and the narrow demographic range.

Overall, this is an honest paper with a good resource and known limitations acknowledged. The reader's CONDITIONAL verdict is right, and the stress-test note lands once you actually read the appendix. This is not a desk reject; it deserves referee time and likely major revision on the bias section.

Referee Report

3 major / 6 minor

Summary. This paper introduces COUNTER, a pseudo-anonymized multilingual dataset (English, French, Arabic) annotated for radicalization level, call-for-action, and named entities, and uses it to study how annotation choices and socio-demographic variation affect radical-content detection. The paper reports (i) an annotation-variation experiment comparing contractor labels, MACE, majority voting, repeated labeling, and an annotator-classifier; (ii) multi-task training results; (iii) a synthetic bias analysis in which Wizard-Vicuna posts generated from persona profiles are used to measure model performance and fairness metrics across socio-demographic groups; and (iv) a comparison of classification and regression for the ordinal call-for-action label. The central conclusion is that the subjectivity of annotations and sociodemographic variations materially influence measured performance and fairness. The annotation-variation and dataset contributions are substantial, but the synthetic bias analysis has internal-validity problems that bear on the main fairness claims.

Significance. The dataset and annotation-variation experiments are valuable: COUNTER is a substantial multilingual resource with multiple annotations, inter-annotator agreement statistics, pseudonymization checks, and a transparent data statement. The label-aggregation comparison in Section 5.1 is carefully executed and gives a useful demonstration that gold-label choices change model evaluation. If the synthetic bias analysis were valid, the fairness findings would be a significant contribution. As written, however, the bias analysis confounds demographic attributes with the ideological content used to generate the test data and relies on a fairness metric that is not identifiable under group-dependent base rates, so its central fairness conclusions are not yet established. The regression-versus-classification analysis is a useful auxiliary result.

major comments (3)

[Section 5.2, Appendix E.1 (Table 10)] The main fairness conclusion is not identified because the synthetic generation protocol varies the tested attribute jointly with the textual topic and ideology. Table 10 lists "Political view (e.g., Far-Right)" and "Religion/Culture" (e.g., "Islam (Jihadism)") as profile variables that are "crucial for producing radical content," so the generated posts for different political-view or religion groups are about different ideologies by construction. The Macro-F1 gaps in Figure 1 and the fairness gaps in Table 4 therefore conflate model bias with per-ideology classification difficulty. This confound is not removed by the acknowledged stereotypical generation limitation in Section 3.4; it is an internal-validity problem. To support the claim that socio-demographic attributes, rather than ideology or topic, drive the gaps, the authors need to control for ideology across groups (e.g., generate the same ideology/topic under different demographic profiles) or re-scope the claims to model sensitivity to generation prompts.
[Section 5.2 and Appendix D] The demographic-parity calculation in Table 4 is not a valid bias diagnostic under the experimental setup. The authors treat class 0 as the favorable outcome and compute selection-rate differences across groups. If the base rate P(Y=0 | A=a) differs by group, even a perfectly calibrated Bayes-optimal classifier will exhibit nonzero demographic-parity difference, so the reported values cannot be read as evidence of model bias. The paper should report per-group base rates and either condition on the true label (as equalized odds does) or use a metric that accounts for label shift, and it should state that demographic parity is confounded with label distribution when base rates vary across groups.
[Section 3.4, Limitations] The bias analysis rests on an unvalidated synthetic proxy: posts generated by Wizard-Vicuna-13B-Uncensored from hand-crafted persona profiles. The authors acknowledge in Section 3.4 that the generated posts "sometimes rely heavily on stereotypical keywords" and in the Limitations that synthetic data "might not accurately represent the complexity and diversity of radical content in the actual world," yet the conclusion in Section 6 that "sociodemographic variations influence data and model outcomes" depends on this proxy. There is no external validation that the generated posts have a label distribution, error structure, or demographic signal resembling real online radical content. Without such validation, or a re-scoping of the conclusions to "performance of models on LLM-generated persona-conditioned content," the fairness findings remain a property of the generation procedure as much as of the models.

minor comments (6)

[Section 3.3] "Pseudononymization" is misspelled; it should be "Pseudonymization."
[Figure 1 and Figures 8-9] The labels are extremely small and some are truncated (e.g., "T aoism"), making it difficult to assess the claimed differences; a zoomable or table-based version with per-group standard deviations would be clearer.
[Section 5.1] The human-label-variation analysis is reported only for French (Table 3), but Section 3.3 states that a large English sample was also double-annotated; the authors should explain why English is excluded from this analysis and whether the same patterns hold.
[Section 5.2, Table 4] No confidence intervals or significance tests are reported for the fairness differences, and some synthetic categories are very small (e.g., African American at 1% in Table 11), so the reported gaps may be unstable.
[Section 5.3, Table 5] The regression models are compared using rounded predictions for Macro-F1, but the rounding threshold and the treatment of predictions outside the [0,4] range are not specified.
[Appendix E] The full prompt template used for synthetic generation is not included; Section 3.4 refers to it, but the appendix only provides a screenshot and variable descriptions, which is insufficient for reproducibility.

Circularity Check

2 steps flagged · score 6.0 of 10

Synthetic bias analysis is self-referential: political-view and religion/culture are prompt conditions that generate the text, so per-group Macro-F1 and fairness gaps are built into the test set rather than independently measured model bias.

self definitional [Section 3.4 (Synthetic Data for Bias Analysis), Section 5.2 (Bias Analysis), Appendix E.1, Table 10]
"Our approach is based on creating user profiles that include socio-demographic variables... These profiles were used to prompt an uncensored LLM called Wizard-Vicuna-13B-Uncensored... We compared the average Macro-F1 score per value for each socio-demographic attribute using our generated dataset to assess model bias. ... Political view (e.g., Far-Right) This variable is crucial for producing radical content and is often specified."

The attribute being probed (political view, religion/culture) is the prompt condition that generates the radical content. Table 10 states that political view is 'crucial for producing radical content' and that religion/culture is 'used to indicate if the person is radicalized.' Therefore, the generated texts differ across groups in topic, ideology, and label distribution by design. Reporting per-group Macro-F1 (Figure 1) and fairness disparities (Table 4) as evidence of model bias is a self-referential comparison: the variation in X across A is an input to generation, so any model with nonzero accuracy will show performance differences across A.
other [Section 5.2 (Bias Analysis), Appendix D (Metrics Definitions)]
"We have a multi-class classification, so we considered class zero as the positive outcome. We use the demographic parity difference, also called disparate impact; it measures the ratio of favorable outcomes between different groups to assess whether the model treats different groups equally. ... Results in Table 4 show that overall Demographic Parity is generally smaller than Equalized Odds..."

Demographic parity requires h(X) to be independent of the protected attribute A, but the synthetic data were generated so that the distribution of the target label Y varies with A (e.g., far-right vs. far-left profiles are prompted to produce different radical content). With P(Y|A) varying by construction, any classifier h that predicts Y from X will have selection rates E[h|A] that differ across A; a nonzero demographic-parity difference is thus mathematically implied even for a Bayes-optimal model. Interpreting the Table 4 disparities as 'the model's predictive performance is less balanced across these groups' (Section 5.2) therefore conflates construction-induced label shift with model bias.

full rationale

The dataset contribution and the annotation-variation analysis (Section 5.1) are independent and empirically grounded: they rely on real annotations, inter-annotator agreement, and standard aggregation methods, and they do not reduce to their inputs. The self-citations (Riabi et al. 2024; Montariol et al. 2022) are methodological and not load-bearing. However, the synthetic bias analysis (Sections 3.4 and 5.2) is partially circular by design. The profiles used to prompt the LLM include the very attributes that are later tested for bias, and Table 10 explicitly says political view is 'crucial for producing radical content' and religion/culture is 'used to indicate if the person is radicalized.' Consequently, the per-group performance gaps and fairness-metric disparities reflect properties of the stimulus construction, not an independently measured model bias. The Limitations section acknowledges that synthetic data may not represent real-world complexity, but it does not address this internal confound: even a perfectly accurate model would exhibit nonzero demographic-parity differences on data where the label distribution varies across groups by construction. Therefore, the central 'sociodemographic variations influence model outcomes' claim is partly forced by the evaluation design, while the annotation-variation findings remain independent.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim about sociodemographic bias rests on the synthetic data assumption, and the annotation-variation claims rest on the two annotators' reliability and the contractor gold labels. The only hand-set numeric input affecting the headline bias results is the synthetic profile attribute distribution.

free parameters (1)

Synthetic profile attribute distributions = Hand-set percentages in Table 11 (e.g., far-right 45%, left 12% in English; France 72% place-of-living in French)
The composition of the synthetic bias test set is chosen by the authors, not derived from real population data. These choices determine group sizes and therefore the computed demographic parity and equalized odds gaps.

assumptions (4)

ad hoc to paper Synthetic posts from Wizard-Vicuna-13B-Uncensored with persona profiles are a valid proxy for real online radical content.
Invoked in Section 3.4 and used throughout Section 5.2 to infer model bias. The paper's Limitations section weakens this by noting the synthetic data may not represent real-world complexity.
domain assumption The two linguist annotators' labels on synthetic data are reliable enough to estimate model bias.
Cohen's Kappa for the 300-example sample is moderate (0.40 to 0.54), and the authors report the generated posts sometimes rely on stereotypical keywords.
domain assumption The contractor's annotations are treated as expert gold standard.
Used as one of the three gold labels in Table 3; no information about the contractor's process or demographics is available (Sections 3.2 and A.4).
domain assumption Models trained on COUNTER generalize to the synthetic distribution sufficiently for bias diagnostics.
Section 5.2 reports lower performance on synthetic data due to distribution shift, yet the gaps are interpreted as bias.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Beyond Dataset Creation: Critical View of Annotation Variation and Bias Probing of a Dataset for Online Radical Content Detection." pith.science (2026). https://pith.science/paper/24BV6JBA

@misc{pith2026241211745,
  author       = {Pith},
  title        = {Pith review of: Beyond Dataset Creation: Critical View of Annotation Variation and Bias Probing of a Dataset for Online Radical Content Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/24BV6JBA}},
  note         = {Machine review of arXiv:2412.11745}
}

read the original abstract

The proliferation of radical content on online platforms poses significant risks, including inciting violence and spreading extremist ideologies. Despite ongoing research, existing datasets and models often fail to address the complexities of multilingual and diverse data. To bridge this gap, we introduce a publicly available multilingual dataset annotated with radicalization levels, calls for action, and named entities in English, French, and Arabic. This dataset is pseudonymized to protect individual privacy while preserving contextual information. Beyond presenting our freely available dataset, we analyze the annotation process, highlighting biases and disagreements among annotators and their implications for model performance. Additionally, we use synthetic data to investigate the influence of socio-demographic traits on annotation patterns and model predictions. Our work offers a comprehensive examination of the challenges and opportunities in building robust datasets for radical content detection, emphasizing the importance of fairness and transparency in model development.

Figures

Figures reproduced from arXiv: 2412.11745 by the authors.

**Figure 2.** Confusion Matrix for Call for Action. classes, but the regression model maintains better consistency, typically misclassifying into adjacent classes rather than across distant ones. 6 Conclusion In this paper, we presented COUNTER, a novel, multilingual, pseudo-anonymized dataset for detecting online radical content, accompanied by an in-depth analysis of the dataset’s biases and the impact of human label variation… view at source ↗

**Figure 3.** Data source distributions for English, French, [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗

Figures from the paper (6 more)

**Figure 4.** Figure 4: Data length distributions for English, French, [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]

**Figure 5.** Figure 5: Data distributions for English, French, and Arabic [PITH_FULL_IMAGE:figures/full_fig_p020_5.png]

**Figure 6.** Figure 6: Examples from the synthetic data English XLM-T XLM-R mBERT French XLM-T XLM-R mBERT [PITH_FULL_IMAGE:figures/full_fig_p021_6.png]

**Figure 7.** Figure 7: Confusion Matrix for Call for Action prediction averaged over five seeds on the generated data for bias analysis for different models [PITH_FULL_IMAGE:figures/full_fig_p021_7.png]

**Figure 8.** Figure 8: Average Macro-F1 variations for the various attributes for XLM-R and mBERT for English. [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]

**Figure 9.** Figure 9: Average Macro-F1 variations for the various attributes for all the models for French generated data. [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 11 canonical work pages

[2]

Political Analysis, 31(3):337–351

Out of one, many: Using language mod- els to simulate human samples. Political Analysis, 31(3):337–351. Michael Ashcroft, Ali Fisher, Lisa Kaati, Enghin Omer, and Nico Prucha. 2015. Detecting jihadist messages on twitter. In 2015 European intelligence and secu- rity informatics conference, pages 161–164. IEEE. Eytan Bakshy, Solomon Messing, and Lada A. Adamic

work page 2015
[4]

IEEE Access , 11:120463–120491

A comprehensive survey of detection and pre- vention approaches for online radicalization: Iden- tifying gaps and future directions. IEEE Access , 11:120463–120491. James Bisbee, Joshua Clinton, Cassy Dorff, Brenton Kenkel, and Jennifer Larson. 2023. Synthetic re- placements for human survey data? the perils of large language models. Lorraine Bowman-Griev...

arXiv 2023
[9]

Identifying Morality Frames in Political Tweets using Relational Learning

Understanding the radical mind: Identifying signals to detect extremist content on twitter. 2019 IEEE International Conference on Intelligence and Security Informatics (ISI), pages 98–103. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. CAMeL tools: An op...

work page Pith review arXiv 2019
[10]

Automatic Detection of Online Jihadist Hate Speech

Automatic detection of online jihadist hate speech. Preprint, arXiv:1803.04596. Anna Sotnikova, Yang Trista Cao, Hal Daumé III, and Rachel Rudinger. 2021. Analyzing stereotypes in generative text inference tasks. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021, pages 4052–4065, Online. Association for Computational Linguisti...

work page Pith review arXiv 2021
[12]

Interpretable Multi-Modal Hate Speech Detection

Learning from disagreement: A survey. J. Artif. Int. Res., 72:1385–1470. Rob van der Goot, Ahmet Üstün, Alan Ramponi, Ibrahim Sharaf, and Barbara Plank. 2021. Massive choice, ample tasks (MaChAmp): A toolkit for multi- task learning in NLP. In Proceedings of the 16th Conference of the European Chapter of the Associa- tion for Computational Linguistics: Sy...

work page Pith review arXiv 2011
[2003]

A survey on extremism analysis using Natural Language Processing

Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142– 147. Javier Torregrosa, Gema Bello Orgaz, Eugenio Martínez Cámara, Javier Del Ser, and David Camacho. 2021. A survey on extremism analysis using natural lan- guage pr...

work page Pith review arXiv 2003
[2015]

Science, 348(6239):1130– 1132

Exposure to ideologically diverse news and opinion on facebook. Science, 348(6239):1130– 1132. Francesco Barbieri, Luis Espinosa Anke, and Jose Camacho-Collados. 2022. XLM-T: Multilingual language models in Twitter for sentiment analysis and beyond. In Proceedings of the Thirteenth Lan- guage Resources and Evaluation Conference, pages 258–266, Marseille, ...

work page 2022
[2016]

Public Opinion Quarterly, 80(S1):298–320

Filter Bubbles, Echo Chambers, and On- line News Consumption. Public Opinion Quarterly, 80(S1):298–320. Eve Fleisig, Rediet Abebe, and Dan Klein. 2023. When the majority is wrong: Modeling annotator disagree- ment for subjective tasks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing, pages 6715–6726, Singapore. ...

work page 2023

Show all 12 references

[2018]

In Proceedings of the 10th ACM Conference on Web Science, WebSci ’18, page 1–10, New York, NY , USA

Understanding the roots of radicalisation on twitter. In Proceedings of the 10th ACM Conference on Web Science, WebSci ’18, page 1–10, New York, NY , USA. Association for Computing Machinery. Louis Fink. 2014. Understanding radicalisation and dynamics of terrorist networks thr...

2014
[2019]

Preprint, arXiv:1907.12368

Detecting radical text over online media using deep learning. Preprint, arXiv:1907.12368. Raúl Lara-Cabrera, Antonio González Pardo, Karim Be- nouaret, Noura Faci, Djamal Benslimane, and David Camacho. 2017. Measuring the radicalisation risk in social networks. IEEE Access, 5:...

1907 arXiv
[2022]

CrimRxiv

Linguistic radicalisation of right-wing and salafi jihadist groups in social media: a corpus-driven lexicometric analysis. CrimRxiv. Shynar Mussiraliyeva, Milana Bolatbek, Batyrkhan Omarov, Zhanar Medetbek, Gulshat Baispay, and Ruslan Ospanov. 2020. On detecting online radical...

2020
[2023]

In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 337–371

Using large language models to simulate mul- tiple humans and replicate human subject studies. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 337–371. PMLR. Hind S. Alatawi, Areej M. Alhoth...

2021

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [2]

Political Analysis, 31(3):337–351

Out of one, many: Using language mod- els to simulate human samples. Political Analysis, 31(3):337–351. Michael Ashcroft, Ali Fisher, Lisa Kaati, Enghin Omer, and Nico Prucha. 2015. Detecting jihadist messages on twitter. In 2015 European intelligence and secu- rity informatics conference, pages 161–164. IEEE. Eytan Bakshy, Solomon Messing, and Lada A. Adamic

work page 2015

[2] [4]

IEEE Access , 11:120463–120491

A comprehensive survey of detection and pre- vention approaches for online radicalization: Iden- tifying gaps and future directions. IEEE Access , 11:120463–120491. James Bisbee, Joshua Clinton, Cassy Dorff, Brenton Kenkel, and Jennifer Larson. 2023. Synthetic re- placements for human survey data? the perils of large language models. Lorraine Bowman-Griev...

arXiv 2023

[3] [9]

Identifying Morality Frames in Political Tweets using Relational Learning

Understanding the radical mind: Identifying signals to detect extremist content on twitter. 2019 IEEE International Conference on Intelligence and Security Informatics (ISI), pages 98–103. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. CAMeL tools: An op...

work page Pith review arXiv 2019

[4] [10]

Automatic Detection of Online Jihadist Hate Speech

Automatic detection of online jihadist hate speech. Preprint, arXiv:1803.04596. Anna Sotnikova, Yang Trista Cao, Hal Daumé III, and Rachel Rudinger. 2021. Analyzing stereotypes in generative text inference tasks. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021, pages 4052–4065, Online. Association for Computational Linguisti...

work page Pith review arXiv 2021

[5] [12]

Interpretable Multi-Modal Hate Speech Detection

Learning from disagreement: A survey. J. Artif. Int. Res., 72:1385–1470. Rob van der Goot, Ahmet Üstün, Alan Ramponi, Ibrahim Sharaf, and Barbara Plank. 2021. Massive choice, ample tasks (MaChAmp): A toolkit for multi- task learning in NLP. In Proceedings of the 16th Conference of the European Chapter of the Associa- tion for Computational Linguistics: Sy...

work page Pith review arXiv 2011

[6] [2003]

A survey on extremism analysis using Natural Language Processing

Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142– 147. Javier Torregrosa, Gema Bello Orgaz, Eugenio Martínez Cámara, Javier Del Ser, and David Camacho. 2021. A survey on extremism analysis using natural lan- guage pr...

work page Pith review arXiv 2003

[7] [2015]

Science, 348(6239):1130– 1132

Exposure to ideologically diverse news and opinion on facebook. Science, 348(6239):1130– 1132. Francesco Barbieri, Luis Espinosa Anke, and Jose Camacho-Collados. 2022. XLM-T: Multilingual language models in Twitter for sentiment analysis and beyond. In Proceedings of the Thirteenth Lan- guage Resources and Evaluation Conference, pages 258–266, Marseille, ...

work page 2022

[8] [2016]

Public Opinion Quarterly, 80(S1):298–320

Filter Bubbles, Echo Chambers, and On- line News Consumption. Public Opinion Quarterly, 80(S1):298–320. Eve Fleisig, Rediet Abebe, and Dan Klein. 2023. When the majority is wrong: Modeling annotator disagree- ment for subjective tasks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing, pages 6715–6726, Singapore. ...

work page 2023