{"id":"9d98ab77-cfc0-4731-872b-86604a665f74","arxiv_id":"2506.13343","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"TwiUSD is a new manually annotated user-level stance benchmark with follower links, and MRFG, which filters followee tweets with an LLM and routes features by graph usefulness, reports top in-target performance.","lead":"The paper introduces TwiUSD, a manually labeled dataset of about 16,000 Twitter users and 48,000 tweets about the 2020 U.S. election, together with follow relationships between users. It also presents MRFG, a framework that filters social-network text with an LLM and splits features between a graph network and a simple network, and MRFG reports the best in-target accuracy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's cross-target claim is contradicted by Table 4: MRFG is not the best method in either cross-target direction, and Section 6.2 concedes this.","rationale":"The reader's weakest assumption about annotation circularity is a genuine methodological threat to the benchmark's validity: since 73.29% of users are followers whose labels are inferred partly from followee stances, a model consuming followee tweets could learn to replicate the annotation rule. That concern deserves the proposed reannotation check and supports a conditional verdict. However, it requires data access and a judgment about whether the benchmark intentionally defines follower stance via social influence. The cross-target contradiction is more decisive because it is internal: the paper's own Table 4 and Section 6.2 contradict the abstract's and conclusion's claim that MRFG achieves state-of-the-art performance in cross-target settings. No new experiments are needed to see that MRFG is not the best method in either direction on Favg. This matters because the central claim explicitly bundles cross-target performance; a reader acting on the abstract alone would be misled. It is also a fixable presentation issue: in-target results remain strong, and the benchmark contribution is independent of the overstatement. I therefore keep the reader's CONDITIONAL verdict rather than moving to REJECT, and I partially agree with the reader: the annotation-circularity concern is not the same as mine, but the reader did note the cross-target overstatement in the rationale. A corrected manuscript should either remove or qualify the cross-target claim and report statistical comparisons on both cross-target directions.","tokens_in":18105,"tokens_out":5782,"duration_ms":55905,"concrete_test":"Recompute the cross-target comparison directly from Table 4: verify that in Trump→Biden, MRFG (Favg 45.16, Acc 43.24) is lower than BERT (47.49, 49.13) and JointCL (48.38, 44.86), and in Biden→Trump MRFG Favg 46.08 is lower than BERT-GCN 49.03. If confirmed, run paired significance tests over the three seeded runs comparing MRFG against BERT, RoBERTa, JointCL, and BERT-GCN on the cross-target splits; unless MRFG is significantly better on a majority of comparisons, revise the abstract and conclusion to claim only in-target superiority and to report cross-target performance as competitive with a known transfer limitation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that MRFG 'consistently outperforms' strong baselines in both in-target and cross-target evaluation is internally contradicted by Table 4. In Trump→Biden, MRFG achieves Favg 45.16 and Acc 43.24, below BERT (47.49, 49.13) and JointCL (48.38, 44.86). In Biden→Trump, MRFG's Favg 46.08 is below BERT-GCN's 49.03, although MRFG has the best F_against (59.55). Section 6.2 explicitly acknowledges this, stating that 'BERT and JoinCL achieve the best Favg or accuracy in at least one direction' and that TFI-based feature selection 'may not transfer well.' The abstract and conclusion nonetheless claim state-of-the-art performance across both settings, so the headline empirical claim is overstated as written. This is a straightforward verifiability failure, not a matter of interpretation: the point estimates in the paper's own results contradict the claim. Because the benchmark's in-target results and the resource itself may still be valuable, the paper can be repaired by limiting the claim to in-target performance and describing cross-target results as competitive with target-specific best metrics, but the claim must be corrected before acceptance.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TwiUSD, a user-level stance detection benchmark built from TwiBot-22 and filtered for the 2020 U.S. presidential election, with 16,211 users, 47,757 tweets, explicit follow relationships, and favor/against/none labels obtained through a manual annotation protocol. It also proposes MRFG, a structure-aware framework that uses an LLM-based relevance filter to select followee tweets and a TFI-based feature splitting mechanism to route graph-favored features through an RGCN and graph-disfavored features through an MLP. Experiments report strong in-target results for MRFG over PLMs, graph-based models, and LLM prompting baselines, as well as ablation studies, sensitivity analysis over the feature selection ratio r, and an error analysis.","tokens_in":18286,"tokens_out":4379,"duration_ms":44109,"significance":"If the in-target results hold, TwiUSD would be a valuable resource for user-level stance detection because it is large, manually annotated, and explicitly incorporates follow relationships, with high inter-annotator agreement (kappa 0.90 for Biden and 0.91 for Trump). The MRFG framework is also a sensible engineering contribution: the ablation studies isolating LLM-FU and the S-TFI-based dual-path design, the comparison against cosine-similarity filtering, and the error analysis are useful and generally credible. However, the benchmark's annotation rule for follower users, the overstatement of the cross-target results, and the absence of variance reporting currently prevent acceptance. The code and dataset are not yet released, so the reproducibility claims cannot be independently verified.","major_comments":[{"comment":"The abstract and conclusion claim that MRFG 'consistently outperforms strong baselines' in both in-target and cross-target evaluation, but Table 4 contradicts this. In Trump→Biden, MRFG achieves Favg 45.16 and Acc 43.24, below BERT (47.49, 49.13) and JointCL (48.38, 44.86); in Biden→Trump, MRFG's Favg 46.08 is below BERT-GCN's 49.03. Section 6.2 itself concedes that 'BERT and JoinCL achieve the best Favg or accuracy in at least one direction' and that TFI-based feature selection 'may not transfer well.' The headline empirical claim must be revised to state that MRFG is best in in-target evaluation and competitive, but not universally best, in cross-target evaluation.","section":"Abstract, Section 6.2, Table 4, Conclusion"},{"comment":"The benchmark construction has a circularity problem that bears directly on the validity of the reported in-target gains. According to Section 3.3, labels for follower users (73.29% of all users) are 'inferred by jointly considering their own content and the stances of their followees,' while MRFG receives followee tweets TF(u) as model input in Eq. (1) and propagates over the follow graph. A model can therefore achieve high accuracy on the follower portion by learning to reproduce the annotation rule rather than by independently detecting the user's stance. The paper provides no check that follower labels agree with a purely content-based reannotation of the same users, and no breakdown of performance by user category. At minimum, the authors should report results separately for followees, followers, and isolated users, and provide evidence that the benchmark measures stance beyond annotation-rule replication.","section":"Section 3.3, Figure 1, Eq. (1)"},{"comment":"The paper states that 'the reported results are averaged over three runs with different random seeds' but reports no standard deviations, confidence intervals, or significance tests for most comparisons. The paired t-test in Section 6.1 is only reported against JoinCL and TPDG, not against the strongest baselines (BERT and RoBERTa) or against cross-target baselines. Because the cross-target gaps are small and the central claim depends on point estimates, the authors should report variance and perform significance tests for the main in-target and cross-target comparisons.","section":"Section 6, Tables 3 and 4"},{"comment":"The dataset size is internally inconsistent. Table 1 reports Stage-2 counts of 8,348 users for Biden and 10,837 users for Trump, which sum to 19,185 users, while Section 3.4 and Table 2 state a total of 16,211 users. The paper does not explain whether users overlap across the two target subsets or how such overlap is handled in the 70/15/15 splits. The authors should clarify whether a user can appear in both target subsets and, if so, how train/test leakage across targets is prevented.","section":"Table 1, Table 2, Section 3.4"}],"minor_comments":[{"comment":"The computation of Cohen's kappa is described as using only the 'Favor' and 'Against' classes, following Li et al. (2021), but the task has three classes including 'None.' The paper should clarify how 'None' instances are treated in the agreement calculation.","section":"Section 3.3"},{"comment":"The axis labels in Figure 3 appear as unicode escape artifacts (e.g., '/uni00000013/uni00000011/...'), making the figure unreadable; the figure needs to be regenerated.","section":"Figure 3"},{"comment":"The paper says bot accounts were filtered out but does not describe the filtering procedure or its accuracy; since the benchmark's credibility depends on using human users only, a brief description of the bot classifier or criteria should be added.","section":"Section 3.1"},{"comment":"The sensitivity analysis in Figure 3 suggests that performance peaks at r in [0.3, 0.4], but Tables 3 and 4 only report results for r = 0.3. The authors should state whether r = 0.3 was chosen before seeing the test set or report results across the full range with variance.","section":"Section 6.5"},{"comment":"There is a typo, 'termss,' and the disclosure that ChatGPT was used for writing is not related to the scientific content; the sentence should be corrected or removed.","section":"Ethics Statement"},{"comment":"The claim that TwiUSD 'surpasses the diversity of most existing user-level stance datasets' is vague and unsupported; a concrete comparison of user counts, tweet counts, and label distributions with prior datasets would be more informative.","section":"Section 3.4"}],"recommendation":"major_revision","confidential_remarks":"The paper's central in-target result may be sound, but the benchmarking circularity and the overstated cross-target claim are significant enough that acceptance should wait for a revised version. I would also encourage the editor to ask the authors to provide the dataset and code (or at least an anonymized version) to reviewers, because the resource itself is a major part of the claimed contribution and is currently unavailable for verification."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nTwo things to know. TwiUSD is a genuinely useful new resource: a user-level stance benchmark on 16,211 users with explicit follow edges and manual annotation at a scale that existing datasets don't offer. And the paper's headline claim about cross-target performance is contradicted by its own Table 4, which has to be fixed before anyone trusts the results.\n\nWhat's new is the dataset. Prior user-level stance datasets lean on hashtag heuristics or retweet patterns; TwiUSD uses blind double annotation with kappa 0.90/0.91 and records follow relationships. The authors also show convincingly (Table 6) that hashtag heuristics from UUSDT and DoubleH misclassify most users on this data, which is a strong justification for manual labeling. The MRFG framework is a sensible assembly of existing pieces—LLM relevance filtering, S-TFI feature splitting, RGCN and MLP routing—but the ablations show each component earns its keep, and the in-target numbers are internally consistent with large gains.\n\nThe soft spots. First, the abstract and conclusion claim MRFG 'consistently outperforms' baselines in both in-target and cross-target. Table 4 shows the opposite for cross-target: BERT and JointCL beat MRFG in Trump→Biden, and BERT-GCN beats it in Biden→Trump. Section 6.2 admits this. The claim must be limited to in-target, with cross-target described as competitive. That's not a matter of interpretation; the paper's own numbers refute the abstract.\n\nSecond, the label circularity. 73.29% of users are followers, and their gold labels were inferred by jointly considering their own content and their followees' stances. The model consumes followee tweets as input. So the strong in-target results may partly reflect learning to reproduce the annotation rule rather than inferring an independently held stance. The kappa statistic tells us annotators agreed with each other, not that the rule is valid. The paper needs a reannotation check: re-label a sample of followers from content alone and report agreement. The error analysis (Cases 2–4) actually shows the model over-relying on followee stances, which makes the circularity concrete.\n\nMinor issues: three runs are averaged but no standard deviations are reported; the feature selection ratio r is tuned per target; and some collection hashtags (#Trump2020LandSlide, #TrumpCrimeFamily) are not ideologically neutral, which may skew the label distribution. All fixable.\n\nWho this is for: anyone building or evaluating user-level stance models. The dataset, once released, could become a standard testbed. I would not trust the cross-target claims, and I'd want the reannotation check before treating the benchmark as clean.\n\nRecommendation: send it to peer review—it's a serious empirical paper with a fixable overclaim and an addressable validity gap. As it stands, conditional accept at best.","headline":"A useful new user-level stance dataset with a fixable overclaim; the follower-label circularity needs a reannotation check.","tokens_in":18870,"tokens_out":4423,"would_cite":false,"duration_ms":39629,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new benchmark and model show that user-level stance detection gains from filtering followee tweets and routing features by graph structure.","keywords":["user-level stance detection","TwiUSD benchmark","followee relationships","social graph","LLM relevance filtering","feature routing","graph neural networks","2020 U.S. election"],"falsifier":"Reannotate a random sample of follower users from their own tweets and profiles alone, with annotators blind to followee stances, and measure agreement with TwiUSD's labels; if agreement is far below the reported kappa values of 0.90 and 0.91, the follower labels are substantially determined by followee stances. A complementary check: train MRFG without any followee tweets as input and compare follower accuracy; a large drop would indicate the model is copying followee stance rather than detecting an independent user stance.","tokens_in":17858,"feed_emoji":"🗳️","tokens_out":9153,"duration_ms":78897,"temperature":0.7,"pith_summary":"User-level stance detection—deciding a Twitter user's overall position on a target such as a presidential candidate—has been held back by noisy labels produced from hashtags or retweets and by datasets that ignore social links. This paper introduces TwiUSD, which it describes as the first large-scale manually annotated user-level stance benchmark with explicit followee relationships: 16,211 users and 47,757 tweets about the 2020 U.S. election, labeled toward Joe Biden and Donald Trump. On that resource the paper builds MRFG, a framework that first lets a large language model discard followee tweets irrelevant to the user, then splits the user's feature dimensions by a graph-informativeness score and routes the structure-sensitive ones through a relational graph network and the rest through an MLP. The paper reports that MRFG clearly beats content-only pre-trained models and graph baselines in in-target tests—Biden Favg 84.19 versus RoBERTa's 79.21—and stays competitive when training and test targets are swapped. If the benchmark and method hold up, the field gains a realistic testbed that joins language with social structure instead of relying on hashtag heuristics.","feed_headline":"LLM-filtered followee tweets lift stance detection to 84% F1","feed_subtitle":"A new benchmark and a graph-aware model beat content-only baselines on 16,211 annotated users.","key_machinery":"The load-bearing machinery is the pairing of the user-followee graph with S-TFI feature ranking. For user features $X$ and normalized adjacency matrix $\\hat{A}$, the informativeness of dimension $m$ is $TFI_m = I(Y; \\tilde{X}_{:,m})$ with $\\tilde{X} = \\hat{A}X$, meaning the mutual information between that feature dimension and stance labels after one step of graph propagation. Features with high TFI are treated as structure-sensitive and processed by a relational graph convolutional network; low-TFI features are treated as content-sensitive and processed by an MLP, and the two encoded halves are concatenated before classification. Before feature routing, a large language model scores each followee tweet against the user's own content and deletes tweets scored as irrelevant, so the social context entering the model is selected rather than raw. This design is what lets the model choose context and delegate features to the encoder that can use them.","core_discovery":"The paper's central claim has two parts. First, TwiUSD supplies a user-level stance benchmark whose labels are assigned not only from a user's own tweets and profile but, for the 73.29% of users who follow others, from the stances of the users they follow; this is meant to mirror stance propagation in real social networks and to make structure usable as supervision. Second, MRFG outperforms strong baselines by treating social context as noisy rather than aggregating all neighbors: an LLM filtering unit scores each followee tweet as 1, 2, or 3 for relevance to the target user, a structural topological feature informativeness (S-TFI) score ranks each embedding dimension by the mutual information it carries about labels after graph smoothing, and the top-ranked dimensions go through a relational graph convolutional network while the rest go through an MLP. The headline empirical results are in-target Favg of 84.19 on Biden and 81.27 on Trump, above RoBERTa's 79.21 and 76.34, with the paper reporting statistically significant gains over its strongest baselines. In cross-target evaluation the paper's model is competitive, and the paper notes a trade-off between target-specific feature selection and transfer.","pith_inferences":["A content-only reannotation of follower users would settle whether TwiUSD measures stance a user holds independently or stance inferred from social influence; the paper does not report such a check.","The paper's own cross-target numbers suggest the TFI ranking, computed on training-target labels, is the main obstacle to transfer; reranking features on the target side or learning a transferable routing policy is a natural next step.","The filter-and-route recipe should transfer to other user-level social tasks, such as bot detection or preference inference, whenever noisy neighborhood text accompanies a user graph.","Error analysis singles out single-tweet users and users with conflicting followees as the failure frontier, so increasing tweets per user or adding a conflict-resolution step would likely raise the ceiling."],"forward_implications":["Manual, followee-aware annotation can replace noisy hashtag heuristics: applying two published hashtag labeling rules to TwiUSD mislabels more than half of users, so the benchmark raises the evaluation bar.","LLM-based relevance filtering pays off: removing the filter drops Biden Favg from 84.19 to 81.33, and cosine-similarity filtering is consistently worse.","Feature routing is the active ingredient: forcing all features through either the RGCN or the MLP alone reduces performance, so splitting by graph-informativeness is what carries the gain.","The benchmark's 70/15/15 splits for two targets make both in-target and cross-target evaluation reproducible, letting models be tested for transfer between candidates.","Because follower labels are inferred jointly with followee stances, any model trained on TwiUSD is being rewarded for using social propagation signals, not just text."],"supporting_citations":[{"why":"Supplies the large Twitter graph dataset from which the TwiUSD user pool is built.","marker":"(Feng et al., 2022)"},{"why":"Provides the Cohen's kappa statistic used to report annotation reliability.","marker":"(McHugh, 2012)"},{"why":"Introduces the TFI feature-informativeness measure that MRFG adapts as S-TFI for feature routing.","marker":"(Zheng et al., 2025)"},{"why":"Provides the relational graph convolutional network used on structure-sensitive features.","marker":"(Schlichtkrull et al., 2017)"},{"why":"Supplies BERT, the text encoder inside MRFG and the strongest content-only baseline.","marker":"(Devlin et al., 2019)"},{"why":"Representative unsupervised user stance method whose hashtag-based labels are shown to misclassify on TwiUSD.","marker":"(Darwish et al., 2020)"},{"why":"DoubleH's hashtag labeling rules and bipartite graph method used in the unsupervised comparison.","marker":"(Zhang et al., 2024b)"},{"why":"Provides the P-Stance evaluation conventions and kappa computation used in experiments.","marker":"(Li et al., 2021)"}],"fun_headline_variants":["Structure-aware LLM framework with followee filtering hits 84% F1 on stance","TwiUSD: Large-scale social graph benchmark for user stance detection","Followee stance propagation and LLM routing boost user stance detection","New user stance benchmark TwiUSD adds social structure to LLM pipelines","Graph-aware LLM routing with relevance filtering raises stance F1 to 84%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is the annotation rule that a follower user's gold label is inferred by jointly considering the follower's own content and the stances of the users they follow; if that rule does not capture a stance the user would hold independently, nearly three-quarters of the benchmark's labels partly encode the annotation rule itself.","fun_headline_variants_meta":{"raw":{"variants":["Structure-aware LLM framework with followee filtering hits 84% F1 on stance","TwiUSD: Large-scale social graph benchmark for user stance detection","Followee stance propagation and LLM routing boost user stance detection","New user stance benchmark TwiUSD adds social structure to LLM pipelines","Graph-aware LLM routing with relevance filtering raises stance F1 to 84%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000936,"raw_usage":{"total_tokens":4009,"prompt_tokens":953,"completion_tokens":3056,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":2958}},"tokens_in":569,"tokens_out":3056,"duration_ms":19427,"temperature":1.0,"reasoning_tokens":2958,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:04:46.704645+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Reannotate a random sample of follower users from their own tweets and profiles alone, with annotators blind to followee stances, and measure agreement with TwiUSD's labels; if agreement is far below the reported kappa values of 0.90 and 0.91, the follower labels are substantially determined by followee stances. A complementary check: train MRFG without any followee tweets as input and compare follower accuracy; a large drop would indicate the model is copying followee stance rather than detecting an independent user stance.","supporting_citations":[{"cited_title":"Aupetit, and Preslav Nakov","cited_arxiv_id":null,"evidence_quote":"Representative unsupervised user stance method whose hashtag-based labels are shown to misclassify on TwiUSD."}],"review_version":2}