{"id":"1bf268da-9960-4b1b-aba8-0cc0fc0391cf","arxiv_id":"1910.01212","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":8,"one_line_summary":"iRadical combines community detection and particle swarm optimization for influence maximization with keyword counting over radicalization criteria to score Twitter users, but the evaluation is limited and partly circular.","lead":"This thesis presents iRadical, a social media analytics pipeline that finds influential Twitter users and estimates how radicalized they are by counting words from expert-defined criteria. The authors compare their optimization method to a prior algorithm and report a radicalization score for ISIS-supporting accounts, but the evaluation is too weak to support the claims.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Context-analytics score in Eq. (3.12) is unvalidated and criterion selection is in-sample; the 8.097% figure is not a demonstrated measure of radicalization.","rationale":"The reader's weakest assumption is exactly the load-bearing point: the Context Analytics algorithm measures radicalization criteria by keyword counts in tweets, and the resulting scores, including the 8.097% figure, are never validated against ground truth. My stress-test confirms this is the most consequential weakness. The pipeline's two halves are influence maximization and context analytics; even if the PSO-IM comparison were perfect, the radicalization claim would still stand or fall on whether Eq. (3.12) and Algorithm 5 produce meaningful scores. The in-sample selection of the seven criteria in Eq. (3.12) is a concrete circularity: the criteria are chosen using the same ISIS-proponent dataset that then yields the headline average. The all-zero Table 4.3 result sharpens the concern because it is not explained and suggests the detectors may be insensitive rather than correctly finding no radicalization in a general corpus. A direct discriminative test against the labeled Kaggle data would settle whether the score carries signal; absent that, the central claim is unsupported. I therefore recommend keeping the reader's REJECT verdict unchanged.","tokens_in":34078,"tokens_out":4015,"duration_ms":44812,"concrete_test":"Run Algorithm 5 and Eq. (3.12) on the labeled Kaggle ISIS-proponent users and on a matched sample of general Twitter users, then compute the AUC of the resulting scores for separating the two groups. Also re-select the seven criteria on one random half of the ISIS-proponent data and recompute the average radicalization rate on the other half. If the AUC is near 0.5, or if the out-of-sample average differs materially from 8.097%, the radicalization score is not a valid measure and the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2.2 defines each radicalization criterion Cj through Algorithm 5's getrate function: the fraction of words in a manually assembled knowledge base that appear in a seed user's tweets. This equates psychological constructs such as C11 (personality traits) with lexical overlap, but no evidence is provided that the knowledge bases measure the intended constructs. Eq. (3.12) then averages only seven of the thirteen criteria, and the text says these seven were chosen 'based on our experiments' on the same 17,000-tweet ISIS-proponent corpus used to produce the 8.097% rate in Table 4.4. No held-out validation, cross-validation, or comparison against the dataset's known labels is reported. Table 4.3 is especially telling: on a large general Australian Twitter corpus, every criterion is 0.0 across all seed sizes and corpus sizes. That uniform zero is implausible for constructs like introversion and discrimination unless the knowledge bases are extremely narrow or the matching is degenerate, yet the paper does not investigate which is the case. The Context Analytics output is the only component of iRadical that connects influence maximization to radicalization, so if Eq. (3.12) does not measure radicalization, the central claim that iRadical enables analysts to explore the potential for online radicalization is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript presents iRadical, a social data analytics pipeline that combines influence maximization with context analytics to identify influential users at risk of online radicalization. The proposed system constructs a Social Personality Graph, clusters users with the BGLL algorithm, selects seed nodes with a PSO-based algorithm (PSO-IM) using a 2-hop influence fitness function, and then scores those users with a Context Analytics module that counts keyword matches against thirteen expert-defined radicalization criteria. Experiments compare PSO-IM with a Memetic algorithm on the Dolphin and Twitter networks and report context-analytics scores on a large Australian Twitter corpus and on a 17,000-tweet Kaggle corpus of ISIS proponents, including a headline average radicalization rate of 8.097%.","tokens_in":34519,"tokens_out":5942,"duration_ms":56323,"significance":"The publication of the GitHub repository, the comparison against Memetic, and the systematic survey of the influence-maximization literature are useful starting points. If the radicalization metric were validated, the pipeline could become a practical tool for exploratory social-media analysis. However, as submitted, the central claim is not established: the criteria used in the headline rate are selected from the same data that produce the rate, the keyword-overlap measure is never validated against ground truth, and the component that connects influence maximization to radicalization fails on a large general corpus.","major_comments":[{"comment":"The selection of the seven criteria in Eq. (3.12) is explicitly described as 'based on our experiments' (Section 3.2.2), and those experiments use the same Kaggle ISIS corpus that later yields the 8.097% average in Table 4.4. Table 4.4 shows that exactly these seven criteria (C2, C3, C4, C5, C8, C11, C12) are the only ones with any nonzero entries across seed sizes; the other six are identically zero. Consequently, the 8.097% figure is a fitted statistic on the same data, not a validated measurement of radicalization risk. The manuscript should report a pre-specified criterion set or provide held-out and cross-validated results before the headline number is used.","section":"Section 3.2.2, Eq. (3.12)"},{"comment":"The getrate function equates each psychological construct (e.g., C2 Introversion, C5 negative thoughts about the West) with the fraction of words from a manually assembled knowledge base that appear in a user's tweets. There is no evidence that the knowledge bases are valid indicators of these constructs, no comparison with labeled radicalization ground truth, and no assessment of precision, recall, or inter-rater agreement. Because this is the sole bridge between influence maximization and radicalization in iRadical, the central claim that iRadical enables exploration of online radicalization is unsupported without construct validation.","section":"Section 3.2.2, Algorithm 5"},{"comment":"Table 4.3 reports 0.0 for every radicalization factor on the 3-month Australian Twitter dataset across all sample sizes. The paper does not diagnose this uniform zero. The likely explanations, such as extremely narrow keyword lists or degenerate string matching, would undermine the general applicability of the Context Analytics component; the paper should report dictionary sizes, matching statistics, and tests on non-ISIS corpora to rule out a degenerate implementation.","section":"Table 4.3"},{"comment":"The PSO implementation is under-specified and the reported comparison is not statistically grounded. Table 4.2 sets Soccoe to 100, which contradicts the constraint 'soccoe <= 4' stated in Algorithm 1 and Eq. (3.11); the roles of theta and lambda in Eq. (3.3) are also swapped between the text and the table. Moreover, the PSO-vs-Memetic plots (Figures 4.1-4.4) show single runs without error bars, multiple trials, or significance tests, even though PSO is a stochastic algorithm, so the conclusion that PSO 'is a decisive factor' is not supported.","section":"Section 3.2.1 and Table 4.2"},{"comment":"Section 3.1 defines the Social Personality Graph as a rich entity-relationship model with affective, cognitive, and personal-concern features, but the experimental graph in Section 3.2.1 is built only from word co-occurrence similarities between users. No personality features enter the influence-maximization experiments, so the claimed novelty of the Social Personality Graph is not actually tested or demonstrated.","section":"Section 3.1 vs. Section 3.2.1"}],"minor_comments":[{"comment":"The column heading '100,000,000' in Table 4.3 is inconsistent with the dataset description of 14,976,862 tweets; clarify whether this is a typo or a different sample.","section":"Section 4.1, Table 4.3"},{"comment":"The notation 'n' in Eq. (3.13) is defined as 'the maximum number of seed size', but the computation of the 8.097% average uses n=5; state the value explicitly and justify it.","section":"Section 3.2.2, Eq. (3.13)"},{"comment":"Algorithm 2 uses an undefined variable 'xyz' for population size; this should be named consistently with 'population_size' or 'pop'.","section":"Algorithm 2"},{"comment":"The 'if wrd in Text' operation matches substrings rather than tokens, and no tokenization or stemming is described; this may inflate the reported rates and should be documented.","section":"Algorithm 5"},{"comment":"The figures lack axis labels and error bars, and the captions contain the typo 'Comparsion'; please correct the spelling and add descriptive axis titles.","section":"Figures 4.1-4.4"},{"comment":"Reference [16] appears both as [11] and [16], and several references have inconsistent formatting (e.g., mixed arXiv identifiers and missing DOIs); please unify the bibliography.","section":"References"}],"recommendation":"reject","confidential_remarks":"The manuscript reads as an unrevised master's thesis rather than a refereed conference or journal paper. The central experimental claim is circular, and the construct validation is absent; substantial new experiments, not just editing, would be needed to make the radicalization metric credible. The paper's topic fits the cs.CY scope, but the current version is not suitable for publication as a standalone research article."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is a master's thesis that assembles a plausible pipeline—community detection, PSO with a 2-hop fitness, and keyword-based radicalization scoring—but the radicalization component is circular and unvalidated. The 8.097% figure in the abstract is a fitted number, not a measured rate.\n\nThe genuinely new thing is the iRadical combination itself, plus the Social Personality Graph as a descriptive model. The influence-maximization variant (BGLL + PSO with RHD initialization and local search) is a reasonable engineering contribution, and the code is on GitHub. The comparison against Memetic on the Dolphin and Twitter networks is a legitimate sanity check, though it lacks error bars and uses an approximate fitness function.\n\nThe soft spots are load-bearing. Eq. (3.12) selects seven criteria from thirteen \"based on our experiments\"—the same Kaggle ISIS dataset that later produces the headline rate. That's in-sample fitting presented as a result. The criteria themselves are measured by keyword overlap, with no evidence that lexical frequency tracks psychological constructs like introversion or discrimination. Table 4.3 shows all zeros for every criterion on the general Australian dataset, which is implausible for constructs like mental health and racism; the paper never explains why. The parameter table has inconsistencies (soccoe listed as 100 while the text says ≤4; θ and λ appear swapped), and the C2 arithmetic in the worked example is sloppy. None of these are fatal to the influence-maximization part, but they gut the radicalization claim.\n\nWho it's for: someone working on influence-maximization heuristics might skim the IM section; anyone doing security analytics should not rely on the context scoring. As a thesis it's an honest literature review plus an attempt, but as a paper it isn't ready for peer review. I'd desk reject with an invitation to resubmit if the author adds holdout validation, explains the zeros, and fixes the parameter inconsistencies.","headline":"Reasonable IM variant, but the radicalization score is fitted in-sample; the headline 8.097% rate is not evidence.","tokens_in":34909,"tokens_out":3479,"would_cite":false,"duration_ms":30945,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a two-stage social data pipeline, iRadical, can rank influential users and score their tweets for radicalization risk, reporting an average radicalization rate of 8.097% on ISIS-supporting Twitter users.","keywords":["influence maximization","particle swarm optimization","radicalization","social network analysis","social data analytics","context analytics","Twitter","community detection"],"falsifier":"Have expert coders label a random sample of the 17,000 ISIS-proponent tweets as radicalizing or not, run iRadical's Context Analytics on the same users, and check whether the keyword-based radicalization score separates those users from a matched control group of ordinary Twitter users; if the scores do not separate the groups, the 8.097% average and the claimed ability to rank users by radicalization risk collapse.","tokens_in":33879,"feed_emoji":"📊","tokens_out":9862,"duration_ms":81851,"temperature":0.7,"pith_summary":"This paper tries to show that the risk of online radicalization can be explored with a social data analytics pipeline, called iRadical, rather than by manual review alone. The pipeline combines influence maximization with context analytics: it first finds a small set of influential users in a social network, then scores those users' tweets against expert-defined radicalization criteria. On a large Twitter network, the paper reports that its PSO-based influence maximization algorithm spreads influence further than a memetic baseline; on a small dolphin social network the two are comparable. Applying the context scoring to 17,000 tweets from ISIS proponents yields an average radicalization rate of 8.097%. A sympathetic reader would take the contribution to be an extensible architecture that makes both the influence dimension and the personality dimension of radicalization first-class citizens.","feed_headline":"PSO influence search wins on large Twitter networks","feed_subtitle":"Two-stage iRadical pipeline also scores ISIS-proponent tweets, estimating an 8.097% radicalization rate.","key_machinery":"The load-bearing mechanism is the two-stage iRadical pipeline. Stage one builds a Social Personality Graph, clusters it with the BGLL community-detection algorithm, prunes candidates with a resemblance-based high-degree (RHD) heuristic, and runs a particle swarm optimizer (PSO-IM) whose fitness function (Eq. 3.4) estimates 2-hop influence spread under the Independent Cascade model; a local-search step refines the seed set. Stage two, Context Analytics, reads lists of words assembled into knowledge bases for each of thirteen expert-defined radicalization criteria and computes each seed user's rate as the fraction of knowledge-base words appearing in the user's tweets; the paper's Eq. (3.12) averages seven selected criteria into one radicalization rate.","core_discovery":"The central claim is that radicalization potential on social media can be assessed by the iRadical pipeline, which treats radicalization as having two measurable dimensions: influence and personality. The influence dimension is handled by PSO-IM, a particle swarm optimization algorithm that selects k seed nodes by maximizing a 2-hop influence spread fitness function, after BGLL community detection and a resemblance-based high-degree heuristic reduce the candidate pool. The personality dimension is handled by Context Analytics, which counts how many words from thirteen expert-derived knowledge bases (introversion, discrimination, disappointment, and others) appear in a seed user's tweets and averages seven of those criteria, via Eq. (3.12), into a single radicalization rate. The paper reports that PSO-IM outperforms Memetic on a large Twitter dataset while performing comparably on the small Dolphin dataset, and that the average radicalization rate among ISIS-supporting users in the labeled Twitter dataset is 8.097%.","pith_inferences":["The validity of the 8.097% number depends entirely on whether word-count proxies for introversion, discrimination, and the other criteria really track the psychological constructs named by experts; a reader should not treat the number as a measured prevalence of radicalized individuals.","Because the composite in Eq. (3.12) is a plain average of seven criterion scores and the negative-thoughts-about-Western-society criterion (C5) contributes the largest values in Table 4.4, the composite is likely dominated by anti-Western sentiment; reweighting or dropping C5 would probably change the headline number substantially.","The pipeline's modularity suggests a testable extension: replace the keyword-count scorers with a supervised classifier trained on labeled radicalized and non-radicalized users, then compare the resulting rankings with iRadical's."],"forward_implications":["If PSO-IM's advantage over Memetic holds up on larger networks, the algorithm offers a practical seed-selection method for influence-maximization tasks at social-media scale.","If keyword-count scores for the seven radicalization criteria measure what they claim, the Context Analytics stage can turn raw tweets into per-user radicalization risk scores without human annotation.","The reported 8.097% average radicalization rate for ISIS-supporting users implies that most tweets from such users do not match the selected criteria words, which could inform how monitoring thresholds are set.","The publicly available iRadical implementation lets other researchers reproduce the pipeline's outputs on the same datasets."],"supporting_citations":[{"why":"Defines the influence maximization problem and the Independent Cascade and Linear Threshold models that PSO-IM builds on.","marker":"[2]"},{"why":"Supplies the definition of radicalization and its two dimensions, personality and influence, that iRadical is designed to analyze.","marker":"[4]"},{"why":"Supplies the memetic algorithm baseline that PSO-IM is compared against for influence spread and convergence.","marker":"[11]"},{"why":"Provides the particle swarm optimization formulation (velocity, cognitive and social coefficients) that PSO-IM adapts.","marker":"[17]"},{"why":"Supplies the BGLL community detection used to shrink the candidate seed set before the PSO stage.","marker":"[94]"},{"why":"Provides the labeled ISIS-proponent Twitter dataset (17,000 tweets) from which the radicalization rates in Table 4.4 are computed.","marker":"[95]"},{"why":"Supplies the expert-defined radicalization criteria and keyword-based measurement approach that Context Analytics scores.","marker":"[102]"},{"why":"Provides the large Twitter dataset used to test PSO-IM against Memetic on a real-world network.","marker":"[113]"},{"why":"Provides the small Dolphin social network used to show that the two algorithms perform similarly on a small graph.","marker":"[114]"}],"fun_headline_variants":["iRadical pipeline reveals 8.097% radicalization in ISIS tweets","PSO-IM beats Memetic on large Twitter networks","8.097% radicalization rate in ISIS-supporting tweets","Two-stage analytics detects radicalization on social media","Influence + personality: iRadical pipeline for extremism"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole radicalization score rests on the assumption that counting how often words from hand-built lists appear in someone's tweets accurately measures psychological states like introversion, discrimination, or hostility to the West, and that the seven chosen criteria capture a user's radicalization risk.","fun_headline_variants_meta":{"raw":{"variants":["iRadical pipeline reveals 8.097% radicalization in ISIS tweets","PSO-IM beats Memetic on large Twitter networks","8.097% radicalization rate in ISIS-supporting tweets","Two-stage analytics detects radicalization on social media","Influence + personality: iRadical pipeline for extremism"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001237,"raw_usage":{"total_tokens":5077,"prompt_tokens":939,"completion_tokens":4138,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":4053}},"tokens_in":555,"tokens_out":4138,"duration_ms":28970,"temperature":1.0,"reasoning_tokens":4053,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:14:58.476051+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have expert coders label a random sample of the 17,000 ISIS-proponent tweets as radicalizing or not, run iRadical's Context Analytics on the same users, and check whether the keyword-based radicalization score separates those users from a matched control group of ordinary Twitter users; if the scores do not separate the groups, the 8.097% average and the claimed ability to rank users by radicalization risk collapse.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the BGLL community detection used to shrink the candidate seed set before the PSO stage."},{"cited_title":"Kaggle (2018)","cited_arxiv_id":null,"evidence_quote":"Provides the labeled ISIS-proponent Twitter dataset (17,000 tweets) from which the radicalization rates in Table 4.4 are computed."},{"cited_title":"Lara-Cabrera, A","cited_arxiv_id":null,"evidence_quote":"Supplies the expert-defined radicalization criteria and keyword-based measurement approach that Context Analytics scores."},{"cited_title":"Beheshti, B","cited_arxiv_id":null,"evidence_quote":"Provides the large Twitter dataset used to test PSO-IM against Memetic on a real-world network."},{"cited_title":"Lusseau, K","cited_arxiv_id":null,"evidence_quote":"Provides the small Dolphin social network used to show that the two algorithms perform similarly on a small graph."}],"review_version":1}