{"id":"ac8e65c7-da6d-4af3-b70c-0d4c3c6698ce","arxiv_id":"2501.04820","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A unified 11-factor psychosocial model, derived from text similarity to existing extremism scales, distinguishes future incel forum joiners from general Reddit users months before they join.","lead":"This paper uses sentence embeddings to score social media posts against 89 psychology survey statements, then uses factor analysis to summarize them into 11 extremist traits. It reports that these trait scores can flag future incel-forum members several months before they join, with accuracy climbing as the join date approaches.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The early-warning AUC is not yet evidence of trait-based prediction: control users are not topic/activity matched and the EFA is fit on the same users later classified, so the reported separation may reflect dating/loneliness topics or representation leakage rather than extremism traits.","rationale":"The reader's conditional verdict is sound. The paper contributes a useful synthesis, releases code, and tests a plausible construct, but the central early-warning claim requires stronger validation. The most load-bearing weakness is that the reported AUC may reflect topic and engagement differences rather than extremist traits, because the control users are not matched on pre-join content and the factor model is fit on the same users later classified. I considered whether there were internal inconsistencies; the main issue is instead external validity of the prediction experiment. A topic baseline is the decisive comparator, since the factor scores are cosine similarities to survey sentences and can be driven by lexical overlap. The EFA leakage is secondary but real and easy to fix via nested cross-validation. If both fixes preserve the AUC, the paper's claim should be accepted; otherwise the abstract's predictive claim should be softened. Because the reader already identified these risks and chose CONDITIONAL, my assessment does not change the verdict.","tokens_in":21403,"tokens_out":7507,"duration_ms":79890,"concrete_test":"Rerun the Section 7 user-level prediction with two changes: (1) fit the EFA on the training folds only, or on a separate external corpus, so held-out users never influence factor loadings; and (2) add a non-trait baseline, such as logistic regression on TF-IDF unigrams/bigrams or LDA topic proportions, using the same user aggregation, cross-validation folds, and month offsets. Compare AUCs at 10, 8, 4, and 3 months before joining. The trait interpretation is supported only if the leakage-free factor model beats the topic baseline by a nontrivial margin and retains AUC >0.6 at 8-10 months; if the baseline matches or exceeds it, or if out-of-sample EFA collapses the AUC, the central claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline RQ3 claim is that the Extremist Eleven scores detect would-be incel joiners 8-10 months out (AUC >0.6, rising to ~0.9 at 3-4 months). For this to be a claim about extremism traits, the pre-joining separation must come from the 11 factors and not from incidental properties of the two comparison groups. Two properties of the design leave this unsecured. First, the control event in Section 7 is a general Reddit user's first post in any new subreddit; the 100 general users are not matched to the 82 incel users on pre-join topics, posting volume, or time period. Users who later join incel forums plausibly spend their earlier Reddit history in dating and loneliness subreddits, and cosine similarity to first-person scale items can be driven by lexical and topical overlap rather than by the intended trait; the paper itself shows this failure mode in the r/Femcel example in Section 7. No bag-of-words, topic-model, or LIWC baseline is reported, so the AUC does not establish that the signal is extremist traits rather than talking about relationships and frustration. Second, the exploratory factor analysis in Section 6 is estimated once on the pooled corpus that includes the incel and general users used in the prediction experiment; the 5-fold cross-validation in Section 7 is applied only to the logistic-regression stage, not to factor estimation. Factor loadings can therefore encode corpus-level differences that are not independent measurements of the held-out users, inflating apparent predictive accuracy. Both issues are testable, and the construct-level interpretation is currently conditional on these confounds being absent.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a unified, unsupervised framework for measuring extremism in online text. It embeds 89 items from 10 established extremism-related psychological scales, scores each post by cosine similarity to those items, and applies exploratory factor analysis to the pooled post-level scores to derive 11 orthogonal factors, the \"Extremist Eleven.\" The framework is used to characterize White Supremacist, Incel, ISIS, Politosphere, and General Reddit discourse (RQ1, RQ2), and to predict whether users will later join an incel subreddit based on their pre-join language (RQ3). The reported prediction experiment uses the average Extremist Eleven scores in 100-word chunks, followed by 5-fold cross-validated logistic regression, and reports AUCs of about 0.6 at 8-10 months before joining, rising to about 0.9 by 3-4 months before joining. The paper also reports a LOESS-based comparison of extremism scores before and after joining, and a qualitative case study of one user's trajectory.","tokens_in":21694,"tokens_out":3516,"duration_ms":35101,"significance":"If the early-warning result holds, the paper would provide a valuable demonstration that theory-grounded, language-based psychosocial scores can signal radicalization risk before active engagement with an extremist forum. The methodological combination of established questionnaires, contextualized embeddings, and factor analysis is creative, and the authors share their code, which supports reproducibility. The use of multiple ideological communities (white supremacist, incel, ISIS) is a genuine strength, as is the explicit attempt to find cross-ideological common factors. However, the central RQ3 claim currently rests on two unsecured assumptions: that the factor representation is independent of the users being predicted, and that the comparison between incel-bound and general users is not driven by topic or activity differences. The paper's own r/Femcel false-positive example indicates that cosine similarity can reflect lexical overlap rather than the intended trait, so the predictive AUCs, as presented, do not yet establish trait-based early warning.","major_comments":[{"comment":"The control event for the 100 general users is defined as \"the time when they post on any new forum (except the incel community),\" while the positive event is first engagement with an incel subreddit; the two groups are not matched on pre-event topic distribution, posting volume, or calendar period. Because cosine similarity to first-person survey items can be driven by lexical and topical overlap (as the r/Femcel example in Table 3 demonstrates), the reported AUC could reflect differences in how often users discuss dating, loneliness, or frustration before joining rather than differences in the 11 extremism factors. A bag-of-words, topic-model, or LIWC baseline, or a topic/activity-matched control group, is needed to support the abstract's trait-based early-warning claim.","section":"Section 7, User-level Analysis (Fig. 6)"},{"comment":"The exploratory factor analysis is estimated once on the pooled corpus of 882,042 data points that includes the posts of the 82 incel and 100 general users later used in the prediction experiment, and the 5-fold cross-validation is applied only to the logistic-regression stage. Factor loadings are therefore not independent of the users being predicted; representation leakage can inflate the AUC. Please re-estimate the factor model inside the cross-validation, or estimate it on a training corpus excluding all target users, and report the difference.","section":"Section 6 (EFA) and Section 7 (Fig. 6)"},{"comment":"The prediction experiment uses only 82 positive and 100 negative users, but the paper does not report the number of posts per user-month, the fraction of users with posts in each pre-join month, or confidence intervals for the monthly AUC values. With this sample size, an AUC of \">0.6\" at 8-10 months may not be distinguishable from chance; please report exact counts, per-month confidence intervals, and the effect of requiring a minimum number of posts per month.","section":"Section 7, Figure 6"},{"comment":"The LOESS comparison assigns T0 randomly for general users, whereas for incel users T0 is chosen as first engagement with an incel subreddit. Since the incel users already have elevated factor scores before T0, the pre-/post-joining comparison conflates selection (who joins) with amplification (what joining does). Please analyze within-user change around T0 with matched event times and report uncertainty in the LOESS trajectories.","section":"Section 7, Figure 5"}],"minor_comments":[{"comment":"\"Morever\" should be \"Moreover\" in the first paragraph of the User-level Analysis.","section":"Section 7, User-level Analysis"},{"comment":"The x-axis label appears garbled (\"Months before joining the forum612 ...\"); the month tick labels should be rendered cleanly from 12 down to 1.","section":"Figure 6"},{"comment":"The figure caption states that the quoted spans are \"paraphrased extracts\"; because these are presented as evidence of a user's trajectory, either use verbatim quotes (with anonymization) or clearly mark them as paraphrases in both the caption and the main text.","section":"Figure 7"},{"comment":"The inclusion criteria for the 100 general Reddit users (active in at least 20 of 24 months) and the 82 incel users (engaged for at least 10 out of 12 consecutive months) are described in different places; a consolidated table of inclusion criteria and corpus sizes would improve reproducibility.","section":"Section 4"},{"comment":"The ethics checklist states that compute details appear in \"§C,\" but the paper body does not reference this appendix; please add a cross-reference.","section":"Ethics Checklist"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The Extremist Eleven is a real synthesis: combining ten psychometric scales into an embedding-similarity scoring pipeline and factor-analyzing the pooled result is a legitimate, reproducible contribution, and the public code plus the detailed factor-loading appendix are concrete assets. The pre-joining incel AUC is striking, but the current evidence doesn't convincingly show the signal comes from extremist traits rather than topic or engagement differences.\n\nWhat's genuinely new: the 11-factor model itself, the cross-ideological comparison of Stormfront, incel, and ISIS text against general Reddit, and the claim that users headed to incel forums can be flagged months in advance. The factor loadings in Table 4 are informative and the face-validity examples are useful. The paper also earns credit for honesty: it openly discusses the femcel false positive in Table 3 and the limitations in Section 9.\n\nNow the soft spots, which are real but addressable. First, the control users in the RQ3 experiment are not matched on pre-join topics, posting volume, or time period. Users who later join incel forums plausibly spend earlier Reddit history in dating and loneliness subreddits, and cosine similarity to first-person scale items can be driven by lexical overlap rather than the intended trait. The femcel example demonstrates exactly this failure mode, and no bag-of-words, topic-model, or LIWC baseline is reported, so the AUC doesn't isolate traits. Second, the EFA in Section 6 is estimated once on the pooled corpus that includes the 82 incel and 100 general users later predicted; the 5-fold cross-validation is applied only to the logistic-regression stage, not to factor estimation. That means the factor loadings are not independent of the held-out users, which can inflate apparent predictive accuracy. The sample is also small and selected.\n\nThese are not fatal: the prediction is not definitionally forced, since the join labels are not used in factor extraction, and the core idea is plausible. But the abstract's 'accurately predicts' claim is stronger than what the current design supports. A serious reviewer should ask for separate-corpus factor estimation, matched controls, and simple text baselines.\n\nThis paper deserves a serious referee. The framework is useful, the code is out, and the issues are testable. I'd send it to review with major-revision expectations. I'd also bring it to reading group, because it's a clean example of how representation leakage can threaten a downstream prediction claim.","headline":"A genuinely useful synthesis of psychometric scales into 11 factors, with public code and honest limitations, but the headline incel early-warning AUC is not yet evidence of trait-based prediction because of factor leakage and unmatched controls.","tokens_in":22276,"tokens_out":1903,"would_cite":true,"duration_ms":19305,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Language scores predict extremist forum entry up to 10 months early","keywords":["extremism","radicalization prediction","psycholinguistics","factor analysis","online communities","incel community","natural language processing","early warning"],"falsifier":"Re-run the incel prediction experiment with a factor model estimated only on pre-entry training posts and a control group matched on posting frequency, subreddit topics, and activity timing; if the pre-entry AUC drops to chance, the signal is an artifact of event definition or factor leakage rather than a marker of radicalization.","tokens_in":21145,"feed_emoji":"📈","tokens_out":10481,"duration_ms":91304,"temperature":0.7,"pith_summary":"This paper argues that radicalization leaves a measurable verbal trace before a person ever joins an extremist community. The authors combine ten established psychosocial survey scales with language models to score hundreds of thousands of posts across white supremacist, incel, ISIS, general, and political forums for 89 extremism-related items, then use exploratory factor analysis to reduce those scores to eleven interpretable factors they call the Extremist Eleven. In a case study, the factor scores of ordinary Reddit users who later joined incel subreddits rise above those of other Reddit users months before the join, yielding AUC above 0.6 as early as ten months out and about 0.9 three to four months out. If the finding holds, language-based trait profiles could serve as an early-warning signal for radicalization, and the eleven factors would give a common vocabulary for comparing very different extremist ideologies.","feed_headline":"Language scores predict extremist forum entry up to 10 months early","feed_subtitle":"An unsupervised model scores ordinary posts on 11 psychosocial factors, reaching about 0.9 AUC just before users join incel forums.","key_machinery":"The Extremist Eleven factor model is the load-bearing object. It is built by encoding both posts and questionnaire items with a pretrained embedding model, scoring each 100-word chunk of a post by cosine similarity to each of 89 self-report items, mean-aggregating chunk scores to post scores, and then applying exploratory factor analysis across the pooled corpus; the eleven resulting orthogonal factors give each post, user, and forum an eleven-dimensional extremism profile. Those profiles are what power both the cross-ideology comparisons and the logistic-regression forecasts of future forum joining.","core_discovery":"The paper's central claim is that a single unsupervised factor model, estimated from language alone, captures psychosocial dimensions shared across ideologically different forms of extremism and can detect them in individuals before they affiliate. The pipeline scores each post by cosine similarity against 89 first-person items from ten psychometric scales, aggregates the item scores, and runs exploratory factor analysis on the pooled corpus; Horn's parallel analysis selects eleven factors, named Revolutionary Attitude, Radical and Violent Intent, Social Dominance Orientation, Cold and Calculating, State Control, Moral Detachment, Nationalism, Dogmatism, Anti-capitalism, Xenophobia, and National Self-interest. Applied to 82 users who later joined incel forums and 100 general Reddit users, the average factor scores feed a logistic regression that separates future incel joiners from general users with AUC greater than 0.6 eight to ten months before the first incel post, rising to about 0.9 three to four months before. The same scores show that incel users' extremism levels climb in the months before joining and then plateau, rather than escalating inside the forum.","pith_inferences":["A direct extension the paper leaves open: apply the same pipeline to users who first join a non-extremist subreddit, matched on activity and topic, to test whether the advanced signal is specific to radicalization or common to any new-community entry.","The factor model is estimated on the full pooled corpus, including the users later scored for prediction; re-estimating the factors on training folds only would show how much of the reported AUC depends on that in-sample measurement model.","If the plateau after entry generalizes, the practical window for intervention is the pre-entry phase, because the model sees rising scores before the first contact and little further escalation afterward.","A further testable consequence is that the same eleven factors should transfer to other radicalization contexts, such as first-time engagement with banned political subreddits or pro-ISIS material, with similar advance warning; if the transfer fails, the model may be capturing incel-specific language conventions rather than general extremism."],"forward_implications":["The reported AUC trajectory implies that ordinary, non-extremist post history contains detectable radicalization signals months before a user first participates in an extremist forum.","The finding that extremism scores plateau after entry suggests that extremist forums are more likely gathering places for people already moving toward extremism than amplifiers that push members further once inside.","Because the same eleven factors describe white supremacist, incel, and ISIS texts, researchers can compare ideologies on shared psychosocial dimensions rather than separate trait-specific models.","Elevated radical-and-violent-intent and xenophobia scores in banned political subreddits indicate the factor model can also characterize communities, not just individuals."],"supporting_citations":[{"why":"Supplies the theoretical premise that authoritarian susceptibility can be identified before overt radicalization, which the prediction experiment is framed to extend.","marker":"Adorno et al. (1950)"},{"why":"Supplies the archetype scoring approach: cosine similarity between contextualized embeddings of text and theory-based items.","marker":"Varadarajan et al. (2024)"},{"why":"Establishes contextualized construct representation, the method for scoring posts against questionnaire items without direct surveys.","marker":"Atari, Omrani, and Dehghani (2023)"},{"why":"Contributes the 14-item Extremism Scale used among the 89 items.","marker":"Ozer and Bertelsen (2018)"},{"why":"Contributes the Social Dominance Orientation scale items.","marker":"Ho et al. (2015)"},{"why":"Contributes Radicalism Intention items used to define factors.","marker":"Moskalenko and McCauley (2009)"},{"why":"Contributes the three survey scales for general, left-wing, and right-wing radicalism that supply many of the 89 items.","marker":"Jungkunz, Helbling, and Osenbrügge (2024)"},{"why":"Contributes the short Right-Wing Authoritarianism scale.","marker":"Zakrisson (2005)"},{"why":"Contributes the Dirty Dozen items behind the cold-and-calculating factor.","marker":"Jonason and Webster (2010)"},{"why":"Supplies the Stormfront white supremacist corpus used for factor discovery and forum-level validation.","marker":"van der Vegt et al. (2021)"}],"fun_headline_variants":["Eleven language factors forecast extremist entry months ahead","Unsupervised model spots future extremists 10 months early","Language-based model detects radicalization before it happens","The Extremist Eleven: language scores flag incel recruits early"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The early-warning result rests on treating a user's first post in an incel subreddit as the moment of joining and treating a general Reddit user's first post in any new subreddit as a comparable 'join' event; if those two events are not comparable in topic, activity level, or timing, the model's separation could be about those differences rather than about extremism.","fun_headline_variants_meta":{"raw":{"variants":["Eleven language factors forecast extremist entry months ahead","Unsupervised model spots future extremists 10 months early","Language-based model detects radicalization before it happens","The Extremist Eleven: language scores flag incel recruits early"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000228,"raw_usage":{"total_tokens":1516,"prompt_tokens":1029,"completion_tokens":487,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":645,"completion_tokens_details":{"reasoning_tokens":422}},"tokens_in":645,"tokens_out":487,"duration_ms":4732,"temperature":1.0,"reasoning_tokens":422,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:25:12.117238+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the incel prediction experiment with a factor model estimated only on pre-entry training posts and a control group matched on posting frequency, subreddit topics, and activity timing; if the pre-entry AUC drops to chance, the signal is an artifact of event definition or factor leakage rather than a marker of radicalization.","supporting_citations":[{"cited_title":"W.; Frenkel-Brunswik, E.; Levinson, D","cited_arxiv_id":null,"evidence_quote":"Supplies the theoretical premise that authoritarian susceptibility can be identified before overt radicalization, which the prediction experiment is framed to extend."},{"cited_title":"V.; Dey, G.; Mangalik, S.; Bucur, A.-M.; Soni, N.; Rao, R.; Lanning, K.; Vallejo, I.; Flek, L.; Schwartz, H","cited_arxiv_id":null,"evidence_quote":"Supplies the archetype scoring approach: cosine similarity between contextualized embeddings of text and theory-based items."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes contextualized construct representation, the method for scoring posts against questionnaire items without direct surveys."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the 14-item Extremism Scale used among the 89 items."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes Radicalism Intention items used to define factors."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the three survey scales for general, left-wing, and right-wing radicalism that supply many of the 89 items."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the short Right-Wing Authoritarianism scale."},{"cited_title":"K.; and Webster, G","cited_arxiv_id":null,"evidence_quote":"Contributes the Dirty Dozen items behind the cold-and-calculating factor."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Stormfront white supremacist corpus used for factor discovery and forum-level validation."}],"review_version":1}