{"id":"522098e6-2810-4d9a-98b8-d17141b60e95","arxiv_id":"1908.03146","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"User stances toward five topics are detected from Twitter network signals (follows, likes, interactions) with accuracy comparable to text-based models, and combining networks with text gives the highest reported F1 on SemEval 2016.","lead":"This paper shows that a Twitter user's stance on a topic can be predicted from who they follow, what they like, and which accounts they interact with, even without reading their posts about the topic. The authors report that combining these network signals with text reaches the best F1 score yet on the SemEval 2016 stance benchmark, raising privacy concerns.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"If the SemEval train/test split is not user-disjoint, the headline 72.49% F1 may largely reflect the classifier memorizing users, because all tweets from the same user share identical network features; the paper never rules this out.","rationale":"The reader's weakest assumption — possible user overlap between train and test — is also the most load-bearing concern I find. The paper's own Section 3.1 shows that multiple tweets from the same user will receive identical user-level network features, but it never establishes that those users are confined to a single split. The justification that stance is stable per user addresses label consistency, not feature-vector leakage: a test tweet from a user already seen in training gives the classifier an exact copy of a training example, which is a memorization shortcut rather than evidence of generalization. The headline SOTA claim and the privacy-oriented conclusion both depend on generalizing to users whose network footprint was not seen during training. The proposed check is cheap and decisive: compute the user-ID overlap and, if present, re-evaluate on a user-disjoint split. I do not push the verdict to REJECT because the flaw is conditional on the split's structure and may be absent; CONDITIONAL already captures this. I also note the comparison to prior SOTA is complicated by the reduced test subset, but the user-overlap question is the more fundamental threat to the quantitative claims.","tokens_in":21338,"tokens_out":7121,"duration_ms":80356,"concrete_test":"Download the user IDs for the retrieved train and test sets (the authors state the tweet/network data will be released). Compute the intersection of user ID sets between train and test, per topic and overall. If the intersection is non-empty, remove all test tweets whose author appears in training, retrain the binary TXT+IN@+INDM model and the TXT baseline on the reduced user-disjoint training set, and report macro-F1 on the disjoint test set. If the F1 drop is small, the concern does not land; if 72.49% falls to or below the 70.03% prior SOTA, the headline claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 acknowledges that 167 users produced multiple tweets and that network features are user-level, so all tweets from one user receive identical IN, PN, and CN vectors. The paper does not state whether the SemEval train/test split, or the reduced subset in Table 2, keeps every user in only one split. If any user appears in both splits, test tweets contain exact copies of training feature vectors, and the SVM can exploit a direct identity shortcut rather than learning a generalizable stance signal. This would inflate the headline 72.49% in Table 5(B) and also weaken the broader claim that stance can be inferred from passive network footprint for unseen users. The original SemEval split may in fact be user-disjoint, so this is a testable assumption rather than an established flaw, but the paper should have verified and reported it explicitly.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether Twitter users' stances toward five SemEval-2016 topics can be inferred from user-level network features rather than solely from on-topic tweet text. It defines four feature groups: on-topic content (TXT), interaction network (IN: mentions and domains in home timeline), preference network (PN: mentions and domains in liked tweets), and connection network (CN: friends/followers), and trains linear SVMs in three-class and two-class (favor/against) settings. On the subset of users whose accounts could be retrieved, it reports that network features alone are comparable to or better than a text-only baseline, and that combining TXT with IN achieves an average F-measure of 72.49%, which it calls the highest reported performance to date on the SemEval dataset. It then analyzes feature overlap, most influential accounts and domains per topic, and qualitative examples to argue that stance is strongly embedded in users' online social footprint, with implications for privacy.","tokens_in":21471,"tokens_out":4687,"duration_ms":48477,"significance":"If the results hold, the paper makes a useful contribution to stance detection and privacy research: it broadens the feature space beyond text to include preferences and connections that can apply to silent users, tests generalization across five topics from different domains, and provides an extensive analysis of which network signals matter. The public release of the collected network data and the careful comparison of feature groups are strengths. The main caveat is that the headline comparison is not yet demonstrated on the full SemEval test set or on a split that is shown to be user-disjoint, so the quantitative claims need verification before the 'highest reported' statement can be accepted.","major_comments":[{"comment":"The headline claim of 72.49% F-measure is computed on the reduced test subset of 906 tweets (Table 2), whereas the published state-of-the-art values to which it is compared (70.03% in [46], 68.98% in [40]) are reported on the full 1,249-tweet SemEval test set. Because the 343 excluded tweets come from accounts that were deleted or suspended, the subset is not a random sample, and the paper does not show that performance on this subset is representative of performance on the full test set. To support the 'highest reported' claim, the authors should either evaluate their models on the full test set or, at minimum, re-run the published baselines on the same 906-tweet subset and explicitly report the comparison as subset-restricted rather than dataset-level.","section":"§5.1, Table 5(B) and Table 2"},{"comment":"The paper does not state whether the SemEval train/test split, or the reduced existing-users split in Table 2, is user-disjoint. Section 3.1 reports that 167 users authored multiple tweets, and because IN, PN, and CN features are user-level, every tweet from the same user receives identical feature vectors. If any of these users appears in both train and test, the test set would contain feature vectors that are exact copies of training vectors, allowing the SVM to exploit an identity shortcut rather than learning a generalizable stance signal. Please report the number of users appearing in both splits for both the full and reduced datasets, and if any overlap exists, re-run the evaluation with a user-disjoint split or leave-one-user-out validation.","section":"§3.1, Table 2"},{"comment":"The claim that the combined model is statistically significantly better than the baseline is supported only by 'two-tailed t-test with p-value < 0.05', with the Mann-Whitney U test not significant. No test statistic, degrees of freedom, or per-topic scores are given, and with only five topic-level F-measures the power of such a test is extremely limited. Please provide the full test details and either report the p-values for both tests or soften the significance claim; as written, the significance statement is not checkable.","section":"§5.1"}],"minor_comments":[{"comment":"The column headers contain garbled characters such as 'Ff avour' and 'Faдainst'; the Cyrillic 'д' should be 'd', and the headers should be regenerated.","section":"Tables 3 and 4"},{"comment":"The text says 'cross-validation with k=5' but does not specify what is tuned or whether the test set is used only once; please clarify the model selection procedure.","section":"§3.3"},{"comment":"The captions and axis labels appear corrupted with Unicode placeholder characters and cannot be read; please regenerate the figures.","section":"Figures 2 and 3"},{"comment":"Reference [11] appears to carry the same DOI as reference [10]; please verify and correct the DOI.","section":"References"},{"comment":"Footnote 1 provides a GitHub URL for the data, while footnote 6 says the list of ids and network information 'would be made available'; please make the availability statement consistent and explicit.","section":"Footnotes 1 and 6"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's my read on Aldayel & Magdy's stance detection paper. The real contribution is the preference network: using the accounts and domains in a user's likes timeline as a stance signal, and comparing it head-to-head with interaction network, connection network, and text on five SemEval topics. That comparison, plus the feature-level analysis showing which accounts and domains separate favor from against per topic, is genuinely useful and not in the prior literature. The qualitative examples in Section 6.3 are a strength: they show the mechanism—users reveal stance by following, liking, and retweeting topically unrelated accounts. The privacy angle is well argued.\n\nThe softer spots are all around the headline number. The 72.49% is computed on a reduced test subset (906 of 1249 tweets) because 25% of accounts were deleted or suspended. The paper compares that number against published F1 on the full SemEval test set. That is apples to oranges. Their own text baseline on the reduced set is 69.80 vs 68.98 on the full set, so the reduced set is not neutral to the comparison. The stress-test concern about user leakage is also legitimate. Since network features are user-level and 167 users have multiple tweets, if any user appears in both train and test, the SVM sees exact feature copies and can memorize identity. The paper never states that the split is user-disjoint. It might well be—SemEval likely was designed that way—but it needs to be verified and reported. If there is overlap, the SOTA claim collapses and the 'unseen users' claim weakens. The significance test is also a bit shaky: the t-test is significant but the Mann-Whitney U test is not, and they still lean on the t-test.\n\nNone of this kills the paper. The central direction—network-only features comparable to text, and the new PN feature—is plausible and the analysis is careful. The quantitative claims just need to be re-baselined on the same split, and the leakage question answered explicitly.\n\nFor peer review: I'd send it out. It's a worthwhile empirical study that will be of interest to the CSCW and social media analysis community. The revisions are clear: verify user-disjoint split, report baselines on the same test subset, and either remove or caveat the SOTA claim. If those are addressed, it's a solid contribution. If not, it's still a useful qualitative study but the headline should be withdrawn.","headline":"Network-based stance detection with a new likes timeline feature is a real contribution, but the headline F1 claim rests on an unreported split-leakage check and an apples-to-oranges comparison.","tokens_in":21999,"tokens_out":2451,"would_cite":true,"duration_ms":23578,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A user's stance toward a topic can be read from their online network footprint alone, and combining network signals with tweet text reaches 72.49% F1, the highest reported on the SemEval stance dataset.","keywords":["stance detection","social media","Twitter","network features","homophily","user privacy","silent users","online footprint"],"falsifier":"Check whether any user ID appears in both the SemEval training and test sets; if so, rebuild the splits so that no user spans both and re-run the SVM experiments to see whether the 72.49% F1 drops toward the text-only baseline.","tokens_in":21118,"feed_emoji":"🐦","tokens_out":8379,"duration_ms":76458,"temperature":0.7,"pith_summary":"This paper asks whether a social media user's stance on a topic, in favor or against, can be read from signals other than what they post. On the five-topic SemEval Twitter benchmark, it shows that network-based features alone, such as whom the user interacts with, follows, likes, and which web domains they link, detect stance about as well as text-only models and often better. It then reports that combining these network signals with tweet text reaches an F-measure of 72.49% on the benchmark, the highest figure reported for that dataset at the time. The broader claim is that stance is embedded in a user's online footprint, so even users who never mention a topic can be profiled.","feed_headline":"Your likes and follows reveal your stance, even if you never post","feed_subtitle":"Who you follow and like predicts stance as well as text, and combining both hits 72.49% F1 on the SemEval test set.","key_machinery":"The central mechanism is homophily, the tendency of users with shared beliefs to touch overlapping accounts and domains, turned into three user-level feature sets: the interaction network, the accounts or domains a user retweets, replies to, mentions, or links; the preference network, the accounts or domains appearing in tweets the user liked; and the connection network, the accounts the user follows or is followed by. Each user is represented as a Boolean presence vector over these sets and fed to a linear SVM, whose weights also make the influential accounts and domains interpretable. Because the three networks overlap by only about 10 to 20 percent, the argument is that each supplies an independent proxy for the same underlying stance signal.","core_discovery":"The discovery, on the paper's own terms, is that on the SemEval 2016 stance dataset, user-level network features outperform the benchmark's strongest text-based system and that the two kinds of signal are additive. Using a linear SVM with Boolean presence vectors over sets of accounts and domains, the interaction network reaches an F1 of 71.56% in the two-class favor/against setting, above the 69.80% text baseline, and adding text to those network features yields 72.49%. The preference network built from liked tweets and the connection network built from followees also beat the text baseline, implying that a user's likes and follow relationships are sufficient evidence of stance even when the user publishes no topical content.","pith_inferences":["If the near-equivalence of three largely disjoint networks generalizes, then stance signal is distributed across every kind of social contact, and a user cannot hide their stance by avoiding one interaction type.","A natural stress test is to re-split the benchmark so no user appears in both training and testing; if the 72.49% figure drops toward the text-only baseline, part of the reported gain would come from user identity rather than from network signal.","The same feature sets could be ported to other platforms and languages; if the result holds there, stance profiling of non-posting users becomes a general privacy concern rather than a Twitter-specific one.","An adversarial implication is that users could try to camouflage their footprint by following or liking balanced sets of accounts, turning stance leakage into an arms race between profilers and users."],"forward_implications":["Stance can be inferred for silent users who never post about a topic, as long as their likes or follow relationships overlap with other users' networks.","Content-free network features are a viable substitute for text models when topical text is scarce, unavailable, or deliberately absent.","Combining text with the interaction-network features gives the highest reported F1 on the SemEval stance benchmark, indicating that content and network carry complementary signals.","Forcing a binary favor/against decision outperforms allowing a neutral class, even though it misclassifies neutral tweets, because the gain in recall on polarized stances outweighs the added false positives.","Public social-media footprints, including likes and follows, are enough to profile users' political, religious, and social leanings, which has direct privacy implications."],"supporting_citations":[{"why":"This reference supplies the SemEval 2016 stance dataset, its five topics, the evaluation script, and the text-only SVM baseline that the paper's network models are measured against.","marker":"[40]"},{"why":"This reference reports the previous highest F1 (70.03%) on the dataset, the figure the paper's 72.49% result is designed to surpass.","marker":"[46]"},{"why":"This reference shows that interaction networks can predict unspoken views of users who never discussed a topic, the direct precedent for network-based stance inference.","marker":"[36]"},{"why":"This reference introduces network-distance user similarity for stance prediction, an antecedent of the connection-network feature set used here.","marker":"[14]"},{"why":"This reference provides a strong text-based stance classifier whose reported performance serves as a comparative baseline for the network-only models.","marker":"[55]"},{"why":"This reference supplies the homophily assumption that users with similar beliefs tend to be connected to similar accounts, the theoretical basis for using network overlap as a stance signal.","marker":"[2]"},{"why":"This reference argues that there is no completely neutral stance, motivating the two-class favor/against experimental setting.","marker":"[32]"}],"fun_headline_variants":["Likes and follows predict stance as well as your posts do","Stance leaks through your network even before you post","Network features match text for stance; combined sets new best","Your social connections betray your stance without a word","Stance detection: your follow graph is a telltale"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central load-bearing premise is that the retrieved subset of users and the train/test split are representative and user-disjoint, since 167 users authored multiple tweets and identical network features for a repeated user would give the classifier an identity shortcut that could inflate the reported F1.","fun_headline_variants_meta":{"raw":{"variants":["Likes and follows predict stance as well as your posts do","Stance leaks through your network even before you post","Network features match text for stance; combined sets new best","Your social connections betray your stance without a word","Stance detection: your follow graph is a telltale"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000412,"raw_usage":{"total_tokens":2151,"prompt_tokens":985,"completion_tokens":1166,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":1086}},"tokens_in":601,"tokens_out":1166,"duration_ms":12904,"temperature":1.0,"reasoning_tokens":1086,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:21:55.092879+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Check whether any user ID appears in both the SemEval training and test sets; if so, rebuild the splits so that no user spans both and re-run the SVM experiments to see whether the 72.49% F1 drops toward the text-only baseline.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"This reference supplies the SemEval 2016 stance dataset, its five topics, the evaluation script, and the text-only SVM baseline that the paper's network models are measured against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"This reference reports the previous highest F1 (70.03%) on the dataset, the figure the paper's 72.49% result is designed to surpass."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"This reference shows that interaction networks can predict unspoken views of users who never discussed a topic, the direct precedent for network-based stance inference."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"This reference introduces network-distance user similarity for stance prediction, an antecedent of the connection-network feature set used here."},{"cited_title":"MITRE at SemEval-2016 Task 6: Transfer Learning for Stance Detection","cited_arxiv_id":"1606.03784","evidence_quote":"This reference provides a strong text-based stance classifier whose reported performance serves as a comparative baseline for the network-only models."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"This reference supplies the homophily assumption that users with similar beliefs tend to be connected to similar accounts, the theoretical basis for using network overlap as a stance signal."}],"review_version":1}