{"id":"1aff5bf8-6a69-437a-98dc-5e364618223c","arxiv_id":"2412.01825","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":8,"one_line_summary":"An ensemble of text and author-graph embeddings reaches 82.5-89.5% F1 on binary true/false subsets of Twitter15 and Twitter16, but the state-of-the-art comparison is not run under the same evaluation protocol.","lead":"GETAE is a deep learning ensemble that combines tweet text with user graph embeddings to detect fake news on Twitter. The authors report gains over several published detectors, but the comparison mixes a binary task with the original four-class benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SOTA comparison likely invalid: GETAE is evaluated on a binary true/false subset of Twitter15/16 while Table 10 baselines are quoted from the full 4-class task, so the headline outperformance claim is not established.","rationale":"The abstract's central claim is that the approach 'proves' improvement and outperforms state-of-the-art models. The strongest experimental evidence is Table 10, where GETAE reports F1 82.7% on Twitter15 and 89.6% on Twitter16, above GCAN's 82.5% and 75.9%. However, Section 4.1 explicitly states that only source tweets labeled true and false are used because GETAE is binary, while the comparison numbers in Table 10 are quoted from papers that evaluate on the full 4-class Twitter15/16 rumor-detection benchmark (false/true/non-rumor/unverified). GCAN's reported 4-class performance is not a valid baseline for a binary subset; the class distribution, difficulty, and metric (macro-F1 over 4 classes) differ. The paper does not describe re-running any baseline on the binary subset. This is not an internal inconsistency but a correctness risk in the evaluation protocol, and it directly undermines the headline claim. The reader's weakest_assumption identified the Propagation Branch's use of author node embeddings as the key weakness. I see that as a related but secondary issue: even if the propagation encoding were conceptually sound, the SOTA claim would still be unsupported by the mismatched comparison. Conversely, if the SOTA claim were repaired by matched baselines, the author-embedding concern would then become the central validity question. The transductive training of node embeddings (Algorithm 1 line 10) is also serious and could explain part of the propagation-branch gains; it compounds the evaluation problem. But the single most load-bearing point is that the paper's main comparative result is not established under a fair protocol. The concrete test is straightforward because the authors provide public code on GitHub. Re-evaluating GCAN and other Table 10 baselines on the exact binary subset with matched CV and metric would settle whether GETAE actually outperforms them. If it does, the headline claim survives on that axis, though the transductive and author-embedding concerns would remain. If it does not, the paper should be revised with either matched baselines or a properly scoped claim. I therefore recommend REJECT, consistent with the reader's verdict, but with the emphasis on protocol mismatch rather than the propagation-branch design alone.","tokens_in":24775,"tokens_out":5526,"duration_ms":55499,"concrete_test":"Re-run the comparison of Section 4.5 under a matched protocol: take the released code for GCAN (and at least GCAN-G and DANES) and evaluate them on exactly the same binary true/false subset of Twitter15/16 used for GETAE, with the same 10-fold cross-validation and macro-F1 metric. If GETAE's F1 (82.7% on Twitter15, 89.6% on Twitter16) is not higher than the re-evaluated baselines, the 'outperforms state-of-the-art' claim in the abstract and Section 4.5 is refuted.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's central claim is that GETAE outperforms state-of-the-art models on Twitter15/Twitter16 (Section 4.5, Tables 10-11). This comparison is not apples-to-apples: Section 4.1 states that, because GETAE is a binary classifier, only source tweets labeled true and false are used for training and evaluation. However, the GCAN, GCAN-G, DANES, and other baseline numbers in Table 10 are taken from papers that evaluate on the full 4-class rumor-detection task (false/true/non-rumor/unverified). A binary true-vs-false macro-F1 of 0.827/0.896 is not comparable to a 4-class macro-F1 of 0.825/0.759; the tasks, class priors, and metric definitions differ. The paper does not describe re-running any baseline on the same binary subset. Thus the headline 'outperforms state-of-the-art' is unsupported before considering further protocol issues. This is load-bearing because the architecture's novelty (the Propagation Branch) is justified by the claimed SOTA gains; if the comparison is invalid, the paper's main contribution is unverified. A compounding issue is that Algorithm 1 (line 10) trains node embeddings on the full graph before the train/test split, so the Propagation Branch may exploit transductive information from test tweets' propagation edges, meaning the ablation gains attributed to the Propagation Branch may reflect leakage rather than genuine diffusion modeling.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GETAE, an ensemble architecture for fake news detection that combines a Text Branch (word/transformer embeddings followed by recurrent layers) with a Propagation Branch (author node embeddings from Node2Vec or DeepWalk followed by a dense layer), concatenating the two into a Propagation-Enhanced Content Embedding for binary classification. The authors evaluate GETAE on Twitter15 and Twitter16 using only the true and false labeled tweets, perform ablation and hyperparameter studies, and compare the best configurations against previously published state-of-the-art results, reporting F1 scores of 0.827 on Twitter15 and 0.895-0.896 on Twitter16. The paper claims that GETAE outperforms state-of-the-art models on these datasets.","tokens_in":25075,"tokens_out":4733,"duration_ms":49684,"significance":"Fake news detection that jointly models textual content and social propagation is a relevant research direction, and the GETAE architecture is a reasonable combination of established components. The paper's strengths include a clearly described architecture, publicly available code, extensive ablation over word embeddings, node embeddings, and recurrent cell types, and the use of standard Twitter15/Twitter16 datasets. If the state-of-the-art comparison were valid, the reported improvements would be of interest to the community. However, as written, the evaluation protocol does not establish the headline claim, so the paper's main contribution is not yet verified.","major_comments":[{"comment":"The central claim that GETAE outperforms state-of-the-art models is not supported because the comparison is not apples-to-apples. Section 4.1 states that only source tweets labeled true and false are used for training and evaluation, so GETAE is trained and tested on a binary subset of Twitter15/Twitter16. However, the baseline results in Tables 10 and 11 (DTC, SVM-TS, mGRU, RFC, tCNN, CRNN, CSI, dEFEND, DANES, GCAN, GCAN-G, HiMaP, SSGE) are quoted from papers that evaluate on the full four-class rumor detection task (false, true, non-rumor, unverified). No baseline is re-run on the same binary subset. A binary macro-F1 of 0.827 on Twitter15 is not comparable to a four-class macro-F1 of 0.825, since the class priors, task difficulty, and metric definitions differ. The abstract and Section 5 therefore overstate the findings. The authors must either re-run all baselines on the same binary subsets or substantially weaken the claims to 'competitive on a binary true/false subset'.","section":"Section 4.1 and Section 4.5, Tables 10-11"},{"comment":"The Propagation Branch may exploit transductive information from the test set. Algorithm 1 trains node embeddings on the full graph G (line 10) before any train/test split is applied, and the dataset's graph includes the propagation edges for all tweets, including those in the test fold. Because each test tweet's author node embedding is learned from the full graph, the embedding can carry information derived from the test tweet's own propagation structure. This can inflate the apparent contribution of the Propagation Branch. The authors should train node embeddings on the training subgraph only, or explicitly adopt a transductive setting and compare with baselines that also have access to transductive graph information.","section":"Algorithm 1, line 10"},{"comment":"The ablation results do not consistently support the claim that the Propagation Branch improves detection. For example, on Twitter15 with BERT embeddings and BiLSTM, adding Node2Vec raises F1 from 0.794 to 0.825, but with RNN it lowers F1 from 0.784 to 0.755, and with Word2Vec and BiRNN it lowers F1 from 0.751 to 0.656. Similar reversals appear throughout Table 4 and Table 5 (e.g., Word2Vec+Node2Vec on Twitter15 reduces F1 relative to text-only for nearly every recurrent layer). The paper's conclusion in Section 5 that the Propagation-Enhanced Content Embedding 'performs better' is therefore only true for a subset of configurations. The authors should report significance tests (e.g., paired tests across folds) and discuss the conditions under which the Propagation Branch helps or hurts.","section":"Section 4.3, Tables 4-5"},{"comment":"The hyperparameter selection procedure appears to use the test set repeatedly, which can produce optimistic results. Sections 4.3 and 4.4 state that the same 80-20 train-test split and 10-fold cross-validation are used, and the best configurations (e.g., p=1, q=1, d=100 on both datasets) are chosen by comparing test-set metrics across many settings. No separate validation set or nested cross-validation is described, so the reported best numbers are likely subject to selection bias. The paper should either introduce a proper validation split for model selection or explicitly acknowledge and correct for the multiple-testing problem.","section":"Section 4.4, Tables 6-9"}],"minor_comments":[{"comment":"There are several typos and inconsistent notations: 'Algortihm 1' (Algorithm 1), 'reccurent' (recurrent), 'ensamble' (ensemble), 'Wor2Vec' (Word2Vec), and 'ber-base-uncased' should be 'bert-base-uncased'.","section":"Throughout"},{"comment":"The GETAE row in Table 10 does not match the values reported in Tables 4-6: Table 10 lists Twitter15 Precision/Recall/F1 as 0.827/0.831/0.827, while Table 4 reports F1 of 0.825 for the best BERT+Node2Vec+BiLSTM configuration and Section 5 reports an F1 of 82.7% for a different configuration. The discrepancies should be reconciled.","section":"Table 10"},{"comment":"The description of DeepWalk's objective function is imprecise: Equation (3) writes a softmax over f(u) and f(v), but DeepWalk's objective is typically a hierarchical softmax over context nodes; the equation as written is not the standard DeepWalk objective and should be corrected or clarified.","section":"Section 3.1.2"}],"recommendation":"major_revision","confidential_remarks":"The self-citation pattern is noticeable (several references to the authors' own prior work, including DANES, which is a close relative of GETAE), but it is not egregious. The main concern is strictly scientific: the SOTA comparison is invalid due to the binary-versus-four-class mismatch, and the transductive node-embedding step creates leakage. Both are fixable with additional experiments, so I recommend major revision rather than outright rejection, but the revisions would need to be substantial and would likely require re-running all baselines and the embedding procedure."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe headline claim here is not supported. The abstract says GETAE outperforms state-of-the-art on Twitter15/16, but GETAE is trained and evaluated on a binary true/false subset (Section 4.1), while the baselines in Tables 10-11 are quoted from the original four-class rumor task. Binary macro-F1 and four-class macro-F1 are different tasks with different priors. No baseline is re-run on the same binary subset, so the “outperforms” claim is unverified. That is the load-bearing issue.\n\nWhat the paper does well: it is a clean, fully specified architecture with code released on GitHub. The ablation is extensive — three word embeddings, six recurrent variants, two node embedding models, and a node2vec hyperparameter sweep. The discussion is honest in that it answers its main research question with a qualified “no” and lists dataset limitations. That is real engineering work.\n\nSoft spots beyond the protocol mismatch. First, the Propagation Branch does not actually model per-tweet diffusion. Algorithm 1 (lines 28-31) feeds the node embedding of the author into a dense layer; the graph embedding is trained on the full user graph before any train/test split. So the ablation gains attributed to propagation may reflect transductive leakage from test tweets’ edges rather than a genuine diffusion signal. Second, the best configuration is selected using test-set performance rather than a validation split, which inflates the reported numbers. Third, the architecture is a close relative of the authors’ own DANES, and there is no systematic comparison against DANES on the same binary subset. That is a surprising omission given the citation.\n\nThe novelty is modest: the “novel embeddings” are hidden states of standard RNNs and dense layers over standard node2vec/DeepWalk embeddings. That is not a flaw by itself, but it means the paper’s contribution is the empirical combination, and the empirical claim currently fails.\n\nWho is this for? Someone looking for a worked example of text-plus-graph ensemble training for fake news, or a cautionary tale about evaluation protocols. I would not cite it for the SOTA claim. A revised version with matched-protocol baselines, validation-based model selection, and a genuine per-tweet propagation encoding could be worth another look. As is, I would send it back for major revision rather than accept, but it deserves a serious referee to force those fixes.","headline":"Solid ablation study, but the SOTA claim rests on a binary-vs-4-class comparison and possible transductive leakage; the headline does not hold.","tokens_in":25646,"tokens_out":3347,"would_cite":false,"duration_ms":31854,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fake news detector that reads tweet text and the social graph hits 89% F1.","keywords":["fake news detection","information diffusion","node embeddings","ensemble architecture","word embeddings","propagation tree","social network analysis"],"falsifier":"Train GETAE on Twitter16 twice, once with node embeddings learned only from the training fold's subgraph and once with node embeddings learned from the full graph; if the F1 gap over the text-only baseline disappears in the first setting, the reported propagation gain is an artifact of graph leakage, not of diffusion information.","tokens_in":24533,"feed_emoji":"🕸️","tokens_out":7656,"duration_ms":71454,"temperature":0.7,"pith_summary":"GETAE is an ensemble architecture for detecting fake news as a binary true/false problem on Twitter. The paper's central claim is that detection improves when the classifier sees both the tweet's text and the social-network position of the tweet's author, rather than text alone. The Text Branch turns word or Transformer embeddings into a Text Content Embedding via recurrent layers; the Propagation Branch turns Node2Vec or DeepWalk author embeddings into a Propagation Embedding via a dense layer; the two are concatenated into a Propagation-Enhanced Content Embedding used for classification. On Twitter15 the best configuration (BERT, Node2Vec, BiLSTM) reaches 82.7% F1, and on Twitter16 the best configuration (BERT, Node2Vec, RNN) reaches 89.5-89.6% F1, which the paper reports as outperforming the state-of-the-art models it compares against. The paper also concludes, in a qualified way, that no single architecture is a definitive solution and that mixtures of models are needed.","feed_headline":"Fake news detector that reads text and social graph hits 89% F1","feed_subtitle":"GETAE pairs tweet embeddings with author-network embeddings and beats compared state-of-the-art on F1.","key_machinery":"The central object is the Propagation-Enhanced Content Embedding: the concatenation of two branch outputs — a Text Content Embedding produced by a recurrent layer (RNN, LSTM, or GRU, unidirectional or bidirectional) over word or Transformer embeddings, and a Propagation Embedding produced by a dense ReLU layer over a Node2Vec or DeepWalk node embedding of the source tweet's author. The node embedding methods are the machinery that turns the social graph into vectors: Node2Vec uses biased second-order random walks with return/in-out parameters (p, q), and DeepWalk uses uniform random walks, both trained with Skip-Gram objectives. The concatenated vector is passed through a ReLU dense layer and then a softmax output layer, so the classification decision depends jointly on lexical content and the author's structural position in the network.","core_discovery":"On its own terms, GETAE claims that fusing textual content with information-diffusion structure yields better fake news detection than text-only or graph-only models. The novel object is the Propagation-Enhanced Content Embedding, built by concatenating a recurrent Text Content Embedding with a dense-network Propagation Embedding and passing the result through a ReLU dense layer before softmax classification. The paper reports that this combination, with BERT as the text encoder and Node2Vec as the graph encoder, achieves 82.7% F1 on Twitter15 and 89.5-89.6% F1 on Twitter16, and that on F1, Precision, and Recall these scores exceed the compared state-of-the-art models, while GCAN retains the higher Accuracy on both datasets. The authors further conclude that the best recurrent layer differs by dataset (BiLSTM on Twitter15, RNN on Twitter16), that balanced Node2Vec random walks with 100-dimensional embeddings worked best, and that no single model is a definitive answer to fake news detection.","pith_inferences":["The claimed 'diffusion' signal is carried only by the source author's node embedding, so the graph branch is really testing whether an author-level structural feature predicts veracity; a per-tweet propagation-subgraph embedding would be a stricter test of the information-diffusion story.","Node embeddings are trained on the full graph before the train/test split, so some transductive leakage is possible; evaluating with a temporal or user-disjoint split would show whether the gains persist for unseen users.","The Twitter16 advantage over GCAN is large, but the comparison is on a binary true/false subset; re-running on the original four-class labels would show whether the advantage is specific to binary veracity classification."],"forward_implications":["If the reported results hold, adding author-level graph embeddings to a text classifier is enough to lift F1 from roughly 85% (text-only BERT on Twitter16) to about 89.6% on the same dataset.","The best configuration differs by dataset, so the paper's conclusion is that embedding models and recurrent layers must be selected per dataset rather than assumed to transfer.","Balanced Node2Vec random walks (p=1, q=1) with 100-dimensional node embeddings gave the strongest results, indicating that neither pure breadth-first nor pure depth-first bias was best.","Because the architecture is modular, any word embedding and any node embedding can be substituted without restructuring the ensemble, so the same pipeline can be pointed at other social platforms or other content encoders.","The paper explicitly concludes that no single architecture is a definitive solution and that a mixture of models (e.g., Mixture of Experts) is the likely route to robust fake news detection."],"supporting_citations":[{"why":"Supplies the Twitter15 and Twitter16 datasets with source tweets, labels, and propagation trees on which all experiments and comparisons run.","marker":"[47]"},{"why":"Provides Node2Vec, the biased random-walk node embedding method used by the Propagation Branch.","marker":"[45]"},{"why":"Provides DeepWalk, the alternative random-walk node embedding method used by the Propagation Branch.","marker":"[46]"},{"why":"Supplies the trainable Word2Vec Skip-Gram word embedding used as one of the three Text Branch encoders.","marker":"[42]"},{"why":"Supplies the BERT transformer embedding used in the best-performing GETAE configurations.","marker":"[43]"},{"why":"Supplies BERTweet, the Twitter-tuned transformer embedding used as the third Text Branch encoder.","marker":"[44]"},{"why":"Defines GCAN and the other comparison models and datasets used as the state-of-the-art baseline, and supplies the Accuracy figures that GETAE does not surpass.","marker":"[21]"},{"why":"DANES is the closest prior ensemble architecture combining social and textual context and is the strongest non-GCAN F1 baseline in the comparison.","marker":"[8]"},{"why":"HiMaP is the higher-scoring graph-embedding baseline GETAE is compared with on Twitter15/16 Accuracy.","marker":"[58]"},{"why":"SSGE is the propagation-embedding baseline in the node-embedding comparison.","marker":"[59]"}],"fun_headline_variants":["GETAE fuses tweet text and social spread to beat fake news detectors","Text plus propagation graph lifts fake news F1 to 89.5% on Twitter16","Graph-aware text embedding outdoes text-only fake news models","Fake news detector merges content and spread network, beats SOTA on F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the source author's Node2Vec or DeepWalk embedding, learned from the full user graph before the train/test split, actually represents the diffusion pattern of each individual tweet rather than just a user-level constant.","fun_headline_variants_meta":{"raw":{"variants":["GETAE fuses tweet text and social spread to beat fake news detectors","Text plus propagation graph lifts fake news F1 to 89.5% on Twitter16","Graph-aware text embedding outdoes text-only fake news models","Fake news detector merges content and spread network, beats SOTA on F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000621,"raw_usage":{"total_tokens":2940,"prompt_tokens":1065,"completion_tokens":1875,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":681,"completion_tokens_details":{"reasoning_tokens":1792}},"tokens_in":681,"tokens_out":1875,"duration_ms":14461,"temperature":1.0,"reasoning_tokens":1792,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:54:06.314863+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train GETAE on Twitter16 twice, once with node embeddings learned only from the training fold's subgraph and once with node embeddings learned from the full graph; if the F1 gap over the text-only baseline disappears in the first setting, the reported propagation gain is an artifact of graph leakage, not of diffusion information.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Twitter15 and Twitter16 datasets with source tweets, labels, and propagation trees on which all experiments and comparisons run."},{"cited_title":"Mikolov, K","cited_arxiv_id":null,"evidence_quote":"Supplies the trainable Word2Vec Skip-Gram word embedding used as one of the three Text Branch encoders."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"HiMaP is the higher-scoring graph-embedding baseline GETAE is compared with on Twitter15/16 Accuracy."}],"review_version":1}