{"id":"710512e2-63dc-4a22-8838-b33edcc9b314","arxiv_id":"2608.05259","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A multimodal inductive classifier for risky social network users that combines text embeddings, follow graphs, and spatial closeness, evaluated on Twitter data with machine-generated labels.","lead":"IMMENSE is a machine-learning system that flags 'risky' social media users by combining what they post, who they follow, and where they are located. The authors say it beats five existing methods and can label new users without retraining, but the test labels were generated from the same text and network signals the system uses.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The synthetic ground truth is generated from the same Word2Vec content and follow-graph signals that IMMENSE consumes, and no trivial labeling-rule baseline is compared; reported superiority may only reflect re-learning the labeling rule.","rationale":"The paper is clearly written and the implementation is open source, but the central claim that experiments showed superiority over five state-of-the-art competitors rests entirely on the constructed labels. The weakest assumption is not just that the labels are noisy; it is that the labels are a deterministic function of the very feature spaces the method uses. This makes the comparison to methods using other representations, such as bag-of-words or XLNet, structurally biased. A simple oracle baseline based on Eq. (11) would settle the issue immediately, and its absence is the single most load-bearing gap. If that baseline matches or beats IMMENSE, then the headline claim should be rejected or substantially narrowed. The reader's rejection is therefore justified, and I would not change the verdict. I agree with the reader's identification of the synthetic ground truth as the load-bearing concern, though I would sharpen it by emphasizing the missing labeling-rule baseline rather than relying only on general circularity language.","tokens_in":23165,"tokens_out":4295,"duration_ms":40952,"concrete_test":"Add a baseline that exactly reproduces the Section 4.1 labeling rule on the test split: (i) compute v_u as the sum of Word2Vec embeddings of each test user's posts, using the same pre-trained model and preprocessing; (ii) predict risky iff cosine_similarity(v_u, v_R) >= 0.88; (iii) additionally relabel as risky any predicted-safe user with more than 10% of follow relationships to predicted-risky users, using the same directed adjacency matrix as Section 3.2. Evaluate this baseline with the same metrics and the same 80/20 split used in Tables 2 through 4. If its F1 on risky users is at or above IMMENSE's reported values (0.870 with class weighting and 0.940 with focal loss at k=512), the claimed superiority is an artifact of the synthetic ground truth.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The evaluation's ground truth is generated by the same signals IMMENSE consumes, so the reported superiority may only show that IMMENSE can re-learn the labeling rule. In Section 4.1, Eq. (11) defines labels as cosine_similarity(v_u, v_R) >= 0.88, using summed Word2Vec embeddings of user posts; Section 3.1's content module uses exactly the additive Word2Vec embedding of the same posts. The additional social relabeling rule (more than 10% of relationships to risky users) uses the same directed follow graph that Section 3.2 feeds to GraphSAGE. Thus the target labels are not an independent external proxy for risky-user status; they are a deterministic function of the exact content and relational representations the method was built to exploit. The paper never compares against the trivial baseline that simply applies Eq. (11) plus the social relabeling rule. Since the test labels are produced by that rule, such a baseline should obtain near-perfect F1; IMMENSE's reported risky F1 of 0.870 to 0.940 would then be below the oracle, not superior. Consequently, the abstract's claim that experiments showed superiority over five state-of-the-art competitors is unsupported as evidence of real-world detection capability; it only supports the weaker statement that IMMENSE fits a synthetic labeling procedure better than models using different representations.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes IMMENSE, an inductive multimodal classifier for identifying 'risky' users in social networks. It combines a content module (summed Word2Vec embeddings plus one-class autoencoders), a relational module (GraphSAGE on the follow graph), a spatial module (GraphSAGE on geodetic closeness), and an MLP fusion. The authors evaluate on a Twitter/X dataset and report that IMMENSE outperforms five state-of-the-art competitors, including the transductive SAIRUS, while also reducing inference time.","tokens_in":23368,"tokens_out":5772,"duration_ms":51803,"significance":"If the evaluation were valid, the paper would make a useful contribution: an inductive, reusable model for law-enforcement monitoring, with open-source code and a clear modular design. The three-perspective architecture is coherent, and the paper includes informative ablations and explanation analyses. However, the empirical claim rests on a ground-truth definition that is manufactured from the same signals the method consumes, so the reported superiority cannot be interpreted as evidence of real-world detection capability. The method itself is not invalidated, but the central evidence for it is.","major_comments":[{"comment":"The ground-truth labels are defined as cosine_similarity(v_u, v_R) ≥ δ, where v_u is the sum of Word2Vec embeddings of the user's posts and v_R is the sum over known risky posts. This is exactly the additive Word2Vec representation used by the content module in Section 3.1. The subsequent social relabeling rule (safe users with more than 10% of relationships to risky users are relabeled risky) uses the same directed follow graph that the relational module in Section 3.2 feeds to GraphSAGE. The evaluation therefore measures how well IMMENSE can re-learn the authors' own labeling rule, not how well it detects risky users under an independent definition. A trivial baseline that applies Eq. (11) plus the social relabeling rule to the test set would be expected to achieve near-perfect F1, and the paper never compares against such a baseline. This is a load-bearing threat to the central claim that IMMENSE outperforms five state-of-the-art competitors.","section":"§4.1, Eq. (11)"},{"comment":"The dataset is described as 'real-world' because it is built from Twitter/X posts and follow edges, but the risky/safe labels are entirely synthetic: they are produced by the threshold δ = 0.88, chosen after inspecting the similarity distribution, together with the 10% social relabeling rule. There is no manual annotation, no external benchmark, and no evaluation against an independent notion of riskiness. Consequently, the experimental results can only support the statement that IMMENSE fits this particular synthetic labeling procedure better than models using different representations; they do not support the abstract's claim that IMMENSE is superior for identifying malicious users in practice. To support the central claim, the authors would need an independent gold standard (e.g., expert annotation or a pre-existing dataset) and a baseline consisting of the labeling rule itself.","section":"§4.1"},{"comment":"The comparison with competitors is not sufficient to establish superiority even internally to the synthetic labels. MrSBC and Re3py classify all users as safe (risky recall 0) across every configuration, which suggests their hyperparameters or input representations may not have been properly adapted to this task; HGT is given a different text encoder (XLNet) than IMMENSE, and Rio-GNN is only compared at three embedding dimensions. Moreover, the reported improvements over SAIRUS are based on a single train/test split with no variance or significance testing, and the paper itself notes the comparison is 'inherently unfair in favor of SAIRUS' because of its transductive access. These issues do not invalidate the method by themselves, but they mean the claimed margins of 9.41% and 20.83% are not established as statistically meaningful.","section":"§4.3, Tables 4–6"}],"minor_comments":[{"comment":"The weighting rule appears reversed: since label 0 denotes safe (Section 3.1), the condition 'if y_i = 0' should assign the safe weight, not the risky weight. Please check.","section":"§3.2, Eq. (4)"},{"comment":"The mean and standard deviation of pairwise distances sum over ordered pairs (a,b) with a≠b but divide by |N| rather than by the number of ordered pairs |N|(|N|−1). This makes the z-scores scale-dependent and should be corrected.","section":"§3.3, Eq. (8)"},{"comment":"The rows for MrSBC and Re3py are identical across all configurations; please clarify whether the reported configurations actually correspond to different inputs, and if so why the outputs do not change.","section":"Table 4"},{"comment":"The superscript 'a' used to denote 'without the fusion module' is only explained in the body text and not in the table captions; please define it in each caption.","section":"Tables 2–5"},{"comment":"The text says 'four state-of-the-art inductive competitors' while the abstract and Section 4.2 list five competitors (including SAIRUS); please reconcile the count.","section":"Conclusions"},{"comment":"The claim of 'very neat separation' is based on a visual inspection of t-SNE plots; a quantitative cluster-separation measure would be more informative.","section":"Fig. 3"}],"recommendation":"reject","confidential_remarks":"The circularity of the ground truth is the decisive issue. I do not see how the manuscript can be repaired without a new evaluation on independently labeled data, which is effectively a new empirical study. The method may be worth pursuing, but the present evidence cannot support the claims made. I would be willing to review a substantially revised version with such an evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: IMMENSE is a coherent engineering contribution — GraphSAGE on follow and spatial graphs, one-class autoencoders on summed Word2Vec, and an MLP fusion — and the inductive framing is genuinely useful for LEA-style screening. Code is public. But the evaluation as packaged cannot support the headline claim, and the reason is exactly the one in the stress-test note.\n\nThe ground truth in Section 4.1 is built by thresholding cosine similarity between a user's summed Word2Vec embedding and a global risky vector v_R (Eq. 11). The content module in Section 3.1 uses that same additive embedding space. The social relabeling rule — safe users with more than 10% risky neighbors become risky — operates on the same follow graph that feeds the relational GraphSAGE. So the labels are a deterministic function of the very representations the method is built to exploit. The authors never compare against the trivial rule itself: apply Eq. 11 plus the relabeling threshold. On the test split, that rule is the ground-truth generator, so it should score near-perfect F1. If it does, IMMENSE's 0.87–0.94 risky F1 is not \"superiority\"; it is approximation of a labeling procedure. If it doesn't, then something is inconsistent in their split or feature pipeline, which would be equally important to see.\n\nThat is a load-bearing problem, not a cosmetic one. The comparison to SAIRUS and four inductive competitors is also less informative than it looks: competitors use bag-of-words or different representations, so part of the gap may just be representation alignment with the labeling rule. There are no error bars or repeated runs, and the spatial module is evaluated on 1,043 users with spatial info. The claim about \"entire new networks\" is not actually tested.\n\nCredit where due: the paper is well-written, the ablations are systematic, the runtime comparison is concrete, and the code is public. The t-SNE/LIME analysis is illustrative rather than evidential, and they don't oversell it. The architecture itself is plausible and could be solid with independent labels.\n\nMy take: the current submission should not be accepted because the central empirical claim is unsupported. But this is fixable, and worth fixing. A revision with human-annotated or otherwise externally validated labels, a trivial-rule baseline, and repeated trials would give a genuinely useful paper. I would not cite it in its current form, but I'd send it back for serious review rather than desk-reject outright.","headline":"Useful inductive multi-perspective framing, but the evaluation labels are generated from the method's own features, so the superiority claim is unsupported as stated.","tokens_in":23996,"tokens_out":1842,"would_cite":false,"duration_ms":17739,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"IMMENSE is an inductive hybrid classifier that identifies risky social-media users by fusing content semantics, social topology, and spatial closeness, and it is reported to outperform five state-of-the-art competitors even when the…","keywords":["risky user identification","inductive learning","social network analysis","text embeddings","graph neural networks","spatial closeness","class imbalance","multi-perspective classification"],"falsifier":"Re-run the same comparison on a dataset whose risky/safe labels come from outside the labeling rule, for example expert-annotated users or accounts later suspended by the platform for harmful content, and check whether IMMENSE's F1 advantage over the five competitors persists; if the margin vanishes or reverses, the reported superiority is an artifact of training and testing on labels generated by the same text-similarity and follow-graph signals the model uses.","tokens_in":22854,"feed_emoji":"🕵️","tokens_out":8607,"duration_ms":72143,"temperature":0.7,"pith_summary":"The paper introduces IMMENSE, a machine-learning method for flagging users who spread hateful, discriminatory, or violent content on social networks. Its central claim is that a hybrid model combining three perspectives—the semantics of a user's posts, their follow relationships, and their geographic closeness to other users—classifies risky users more accurately than any single perspective, and does so inductively: once trained, it can label users it has never seen without retraining. On a Twitter/X dataset of 37,945 users with roughly 7% risky users, IMMENSE is reported to outperform five competitors, including SAIRUS, even though SAIRUS is run in a transductive setting that lets it see test users during training. If the result holds, law-enforcement agencies get a reusable tool that can score a batch of new users in seconds rather than retraining a model from scratch.","feed_headline":"Inductive IMMENSE beats five rivals in risky-user detection","feed_subtitle":"Post text, follow networks, and geolocation combine to flag risky users faster than a transductive rival.","key_machinery":"The machinery is a per-perspective encoder plus a fusion layer. The text module exploits the additive compositionality of word embeddings to turn a user's chronological post stream into one vector, and uses stacked autoencoders to produce per-class reconstruction errors. The relational and spatial modules both use GraphSAGE—the first on the directed follow graph initialized with the semantic embeddings, the second on a graph whose edge weights are normalized geographic closeness—to output safe/risky probabilities. An MLP with one hidden layer learns to combine the three modules' seven outputs. The inductive property comes from GraphSAGE and the autoencoders operating on local information: a new user's representation depends only on their own posts and immediate neighborhood, so the trained model can be applied to unseen nodes or a whole new network without retraining.","core_discovery":"The central claim is that a three-module inductive architecture can outperform both content-only classifiers and a transductive method that enjoys an information advantage. IMMENSE represents each user's text as the summed Word2Vec embeddings of their posts, then trains two stacked autoencoders, one per class, whose reconstruction errors become the content signal. It feeds the same embeddings plus user labels into a supervised GraphSAGE model over the follow graph, and a second GraphSAGE model over a spatial-closeness graph built from z-normalized geodetic distances, producing safe/risky probabilities from each. A small MLP fuses these seven features into a final prediction. In the best reported configuration (focal loss, embedding dimension 512), IMMENSE reaches F1 0.967 overall; against SAIRUS it improves F1 on all users by up to 9.41% under class weighting and 14.12% under focal loss, and F1 on risky users by up to 20.83% and 30.54%, respectively, while reducing inference time by up to 76.93%.","pith_inferences":["The synthetic ground truth uses the same textual embedding space and follow graph that the model consumes, so an independent-label evaluation, such as expert annotation or platform enforcement actions, would show whether the reported margins reflect genuine detection or self-consistency with the labeling rule.","Because the architecture treats each perspective as a plug-in encoder, swapping Word2Vec for contextual embeddings such as BERT-style encoders is a natural testable extension, at higher computational cost.","The spatial module's usefulness despite sparse geolocation hints that coarser proxies, such as timezone, IP-derived region, or interaction locality, could extend the same design to networks where precise coordinates are unavailable.","If the labels do track real-world risk, the model's low inference cost makes it usable as an online triage system that surfaces candidate users for human review rather than as an autonomous enforcement tool."],"forward_implications":["A trained IMMENSE model can be applied directly to new users or new networks, removing the need to retrain node embeddings on every new graph; reported inference on about 7,500 users takes at most 12.4 seconds.","All three perspectives contribute: adding spatial closeness improves F1 in 5 of 6 evaluated configurations even though only 1,043 of 37,945 users have location data, and the full fusion gives the best risky-class performance.","The inductive setting is harder than the transductive one, yet IMMENSE reports higher F1 than the transductive SAIRUS while also cutting inference time by roughly three-quarters.","Class imbalance can be handled in two ways, class weighting or focal loss, and both yield balanced precision and recall on the risky class rather than collapsing to a safe-only classifier."],"supporting_citations":[{"why":"Defines the risky-user task and the three-perspective SAIRUS baseline; IMMENSE's main comparison and the transductive method it must beat.","marker":"[1]"},{"why":"Supplies the inductive GraphSAGE embeddings used in both the social-relationship and spatial modules, giving IMMENSE the ability to generalize to unseen nodes.","marker":"[23]"},{"why":"Provides the pre-trained Word2Vec embeddings that form user content vectors and also anchor the synthetic ground-truth labeling rule.","marker":"[40]"},{"why":"Establishes the additive compositionality property that justifies representing a user by the sum of their words' embeddings.","marker":"[41]"},{"why":"Underpins the two stacked autoencoders whose reconstruction errors serve as the content perspective's classification signal.","marker":"[44]"},{"why":"Supplies the focal loss used to handle class imbalance in the relational, spatial, and fusion modules.","marker":"[47]"},{"why":"One of the four inductive competitors; a multi-relational naive Bayes classifier that the paper compares IMMENSE against.","marker":"[39]"},{"why":"One of the inductive competitors; an ensemble of relational decision trees used as a baseline.","marker":"[38]"},{"why":"One of the inductive competitors; a heterogeneous graph transformer baseline with XLNet-derived node features.","marker":"[24]"},{"why":"One of the inductive competitors; a multi-relational GNN baseline used for embeddings plus an MLP classifier.","marker":"[26]"}],"fun_headline_variants":["Inductive IMMENSE beats five rivals in risky-user F1","Three views, one model: IMMENSE finds malicious users","No retraining, better F1: IMMENSE fuses text, social, spatial","IMMENSE: Inductive multi-perspective classification outperforms transductive","Faster and sharper: IMMENSE identifies harmful users via 3 angles"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the synthetic ground truth—users labeled risky when the cosine similarity between their summed word-vector posts and a risky-post vector is at least 0.88, with safe users relabeled risky when more than 10% of their follow relationships point to risky users—captures real-world risky status independently of the model's own features.","fun_headline_variants_meta":{"raw":{"variants":["Inductive IMMENSE beats five rivals in risky-user F1","Three views, one model: IMMENSE finds malicious users","No retraining, better F1: IMMENSE fuses text, social, spatial","IMMENSE: Inductive multi-perspective classification outperforms transductive","Faster and sharper: IMMENSE identifies harmful users via 3 angles"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000285,"raw_usage":{"total_tokens":1688,"prompt_tokens":962,"completion_tokens":726,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":631}},"tokens_in":578,"tokens_out":726,"duration_ms":6863,"temperature":1.0,"reasoning_tokens":631,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T16:54:47.882263+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same comparison on a dataset whose risky/safe labels come from outside the labeling rule, for example expert-annotated users or accounts later suspended by the platform for harmful content, and check whether IMMENSE's F1 advantage over the five competitors persists; if the margin vanishes or reverses, the reported superiority is an artifact of training and testing on labels generated by the same text-similarity and follow-graph signals the model uses.","supporting_citations":[{"cited_title":"Pellicani, G","cited_arxiv_id":null,"evidence_quote":"Defines the risky-user task and the three-perspective SAIRUS baseline; IMMENSE's main comparison and the transductive method it must beat."},{"cited_title":"Hamilton, Z","cited_arxiv_id":null,"evidence_quote":"Supplies the inductive GraphSAGE embeddings used in both the social-relationship and spatial modules, giving IMMENSE the ability to generalize to unseen nodes."},{"cited_title":"Mikolov, K","cited_arxiv_id":null,"evidence_quote":"Provides the pre-trained Word2Vec embeddings that form user content vectors and also anchor the synthetic ground-truth labeling rule."},{"cited_title":"Mikolov, I","cited_arxiv_id":null,"evidence_quote":"Establishes the additive compositionality property that justifies representing a user by the sum of their words' embeddings."},{"cited_title":"Rumelhart, J.L","cited_arxiv_id":null,"evidence_quote":"Underpins the two stacked autoencoders whose reconstruction errors serve as the content perspective's classification signal."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the focal loss used to handle class imbalance in the relational, spatial, and fusion modules."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"One of the four inductive competitors; a multi-relational naive Bayes classifier that the paper compares IMMENSE against."},{"cited_title":"Petkovic, M","cited_arxiv_id":null,"evidence_quote":"One of the inductive competitors; an ensemble of relational decision trees used as a baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"One of the inductive competitors; a heterogeneous graph transformer baseline with XLNet-derived node features."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"One of the inductive competitors; a multi-relational GNN baseline used for embeddings plus an MLP classifier."}],"review_version":1}