{"id":"9259fa98-ad54-452f-b10a-1b1ab30a4f04","arxiv_id":"1909.01720","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A sifted multi-task learning model with gated and attention-based shared feature selection reports state-of-the-art F1 on RumourEval and PHEME for fake news detection.","lead":"This paper proposes a multi-task neural model that filters shared features between stance detection and fake news detection using gates and attention. On two public rumour datasets it reports small F1 improvements over prior state-of-the-art methods, but it does not release code or significance tests.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim rests on baseline comparability that the paper never establishes; internal table inconsistencies make the reported F1 margins unverifiable.","rationale":"The reader's weakest assumption identifies the experimental pipeline and dataset counts as the soft spot, which matches my central concern. I mark agreement as partial rather than full because the most decisive piece of evidence is not the Table 1 count mismatch alone but the combination of that mismatch with the Table 2 versus Table 3 accuracy discrepancy and the complete absence of a stated baseline-reproduction protocol. Since the claimed improvements are small, the comparison protocol is the linchpin: a modest difference in label filtering or evaluation split could plausibly change the ranking between the proposed model and Bayesian-DL. The paper provides no code, no data splits, and no per-fold statistics, so the central SOTA claim cannot currently be verified. This keeps the manuscript in CONDITIONAL status: the concern is addressable by releasing the exact preprocessing and evaluation pipeline and rerunning baselines under it, but without that evidence the headline F1 boosts should not be taken at face value.","tokens_in":13715,"tokens_out":4315,"duration_ms":46638,"concrete_test":"Obtain the filtered PHEME and RumourEval binary subsets as described in Section 4.1, fix random seeds, and rerun the proposed model and at least the two strongest baselines (Bayesian-DL and TRNN) under exactly the same 10%-holdout plus 5-fold cross-validation protocol, reporting per-fold F1 with mean and standard deviation plus a paired significance test (e.g., McNemar or bootstrap). Also check whether the same model reproduces Table 2's 81.48 accuracy or Table 3's 82.10 accuracy on RumourEval; if both values appear across runs, the paper must disclose the variance and reconcile the discrepancy. If the baseline F1 values under the shared protocol exceed the reported 78.65 and 80.09, or if the 0.87 and 1.31 margins fall within one standard deviation, the SOTA claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the sifted multi-task model outperforms prior state-of-the-art methods by 0.87% and 1.31% F1 on RumourEval and PHEME. That claim requires that every method in Table 2 was evaluated on the same binary true/false subset and under the same 10%-holdout plus 5-fold cross-validation protocol described in Section 4.1. The paper does not state how baseline numbers were obtained, and its own Table 1 undermines the assumption: PHEME is described as 6,425 threads and 105,354 tweets, but only 2,402 items have veracity labels (1,067 true + 638 false + 697 unverified), while the stance-label counts sum to 4,434. It is therefore unclear whether the fake-news task uses thread-level labels, tweet-level labels, or a mixture, and whether stance supervision covers a different subset. A further internal inconsistency appears between Table 2 and Table 3: the same model, MT-trans-G-A, reports accuracy 81.48 in Table 2 but 82.10 in Table 3 with identical precision, recall, and F1. Without code, random seeds, per-fold results, or error bars, the claimed 0.87 and 1.31 percentage-point F1 gains cannot be distinguished from run-to-run variance or from differences in data filtering. The manuscript also contains unfinished Chinese drafting notes, consistent with an unreviewed draft. None of this is evidence of fabrication, but it is a missing-evidence problem that blocks the SOTA claim as stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a sifted multi-task learning model for fake news detection, trained jointly with stance detection. The model uses transformer encoders in a shared-private architecture and adds a 'selected sharing layer' composed of a gated sharing cell and an attention sharing cell to filter and select shared features for each task. Experiments on RumourEval and PHEME report state-of-the-art accuracy and F1, with F1 gains of 0.87 and 1.31 percentage points over the best baseline, and ablations show incremental gains from the gate and attention cells. Case-study visualizations illustrate task-specific selection of shared tokens.","tokens_in":14029,"tokens_out":6944,"duration_ms":62876,"significance":"If the empirical claims are reproducible, the selected sharing layer is a simple and potentially useful component for multi-task veracity and stance learning, and the ablation study provides evidence for the mechanism rather than only an end-task gain. The use of public datasets is a strength, and the visualizations are a useful addition. However, the contribution is empirical and the reported margins are small: the absence of error bars, the unclear baseline protocol, and the inconsistent dataset statistics substantially weaken the state-of-the-art claim as it stands. The paper does not include code or per-fold results, so verification currently rests on the tables alone.","major_comments":[{"comment":"The dataset statistics are internally inconsistent and block the interpretation of the evaluation. For PHEME, the thread and tweet counts (6,425 and 105,354) do not match the veracity-label counts (1,067 true + 638 false + 697 unverified = 2,402) or the stance-label counts (891 + 335 + 353 + 2,855 = 4,434), while for RumourEval the veracity labels sum to the thread count (325) but the stance labels sum to the tweet count (5,568). The paper must state, per dataset, whether the fake-news task is evaluated at thread level or tweet level, how the filtered binary subset is formed, and how many instances are used for each task.","section":"Section 4.1, Table 1"},{"comment":"The state-of-the-art claim relies on baseline numbers whose protocol is never established. The paper does not state whether the reported SVM, CNN, TE, DeClarE, MTL-LSTM, TRNN, and Bayesian-DL results were produced by reimplementation under the same 10% holdout plus 5-fold cross-validation on the same filtered true/false subset, or copied from original papers with potentially different splits, label sets, and evaluation units. Without this, the 0.87 and 1.31 F1-point margins cannot be attributed to the method rather than to protocol differences.","section":"Section 4.3.2, Table 2"},{"comment":"No estimate of variance is given, and the internal accuracy inconsistency makes the reported gains unverifiable. MT-trans-G-A reports A=81.48 in Table 2 but A=82.10 in Table 3 with identical precision, recall, and F1; if the tables come from different runs, the authors should report per-fold results and error bars. With margins under 1.5 F1 points, the claimed state-of-the-art improvement cannot be distinguished from run-to-run variation without such information.","section":"Tables 2 and 3"},{"comment":"The loss for the stance task is underspecified. The formula given is binary cross-entropy, but stance detection has four classes (support, deny, query, comment); if y_i and y-hat_i are one-hot vectors, this is not the standard multinomial cross-entropy. The paper should give the exact per-task loss and explain how the task weight lambda_i is applied to each task.","section":"Section 3.4, Eq. (12)"}],"minor_comments":[{"comment":"The dimension of H_shared is written as R^{1 x l(dp+dw)}, while Eq. (6) treats the same quantity as R^{l x (dp+dw)}; please make the notation consistent.","section":"Section 3.3, Eq. (4)"},{"comment":"Please define G and A explicitly as the vectorized outputs of the gated and attention cells before the concatenation, since the tensor shapes are otherwise ambiguous.","section":"Section 3.3, Eq. (8)"},{"comment":"The transformer encoder is said to use 6 attention heads and 2 blocks, while Section 3.3 reports best performance with h=2 in the selected sharing layer; clarify to which module each h refers.","section":"Section 4.2 and Section 3.3"},{"comment":"The claim that MT-trans outperforms MT-lstm is only true in accuracy; in F1, MT-lstm is better on both datasets (67.10 vs. 68.15 on RumourEval and 71.73 vs. 75.29 on PHEME). Please qualify the claim.","section":"Section 4.4.1, Table 3"},{"comment":"The text first lists 'gunmen, hostages, Sydney, ISIS' as the tokens receiving more attention and then says 'gunmen' and 'ISIS' obtain the highest weights; please resolve the inconsistency.","section":"Section 4.5.2"},{"comment":"The manuscript contains untranslated Chinese planning notes and placeholder text, including in Section 4.5.2 and in duplicated figure captions in the body text; these should be removed before any resubmission.","section":"Full text"}],"recommendation":"major_revision","confidential_remarks":"The paper appears to be an early draft: the inserted Chinese notes and duplicated figure captions suggest it has not yet been through a careful review cycle. This does not by itself affect the technical assessment, but it is consistent with the missing experimental details that currently block the state-of-the-art claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The kernel here is worth taking seriously: the selected sharing layer, a gate plus attention on top of a transformer shared-private extractor, is a sensible way to filter shared features for each task. The ablations are well designed and suggest both cells contribute. That is the real contribution, and it might transfer to other multi-task NLP problems.\n\nBut the paper as it stands is a rough draft, and the central empirical claim does not hold up. Most damning, Table 2 reports accuracy 81.48 for the full model on RumourEval while Table 3 reports 82.10 for the same model with identical precision, recall, and F1. That is not a typo you can wave away; it is a direct contradiction in the numbers that ground the SOTA claim. The dataset statistics in Table 1 are also misleading: veracity labels are thread-level (325 for RumourEval, matching threads), while stance labels are tweet-level (5,568 and 105,354, matching tweets). The text says they 'filter unverified tweets,' but the table lumps both levels together, so the exact training set and evaluation unit are unclear.\n\nThe comparison to previous work is fragile beyond that. F1 gains of 0.87 and 1.31 points come with no error bars, no significance tests, and no code. The paper does not say whether baseline numbers were reimplemented under the same 10%-holdout-plus-5-fold protocol or taken from original papers. Without that, the gains are within run-to-run variance.\n\nThe manuscript also contains Chinese editorial notes, including a reminder to cite 'Learning What to Share: Leaky Multi-Task Network' and 'Gated Multi-Task Network for Text Classification'—two closely related works that the authors know about but do not cite. That is a citation integrity problem, not just a polish issue.\n\nMy take: the architectural idea is publishable in principle, but this draft is not. The internal numerical contradiction alone blocks it. I would desk-reject and invite resubmission after a major rewrite, with corrected tables, error bars or per-fold results, a clear statement of the evaluation granularity, and proper citations. It might be worth a reading group as a cautionary example, but not as a source of results.","headline":"Plausible multi-task feature-selection idea, but the paper is an unfinished draft with numerical inconsistencies that invalidate the SOTA claim as stated.","tokens_in":14548,"tokens_out":3386,"would_cite":false,"duration_ms":34758,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Sifting shared features with a gate-plus-attention layer lifts fake-news detection F1 by 0.87 percentage points on RumourEval and 1.31 on PHEME.","keywords":["fake news detection","multi-task learning","stance detection","shared-private model","selected sharing layer","gate mechanism","attention mechanism","transformer encoder"],"falsifier":"Re-run the exact filtered datasets under the paper's 10%-holdout plus 5-fold protocol, verifying that every veracity label is a thread-level label and that PHEME's class counts sum to the number of labeled threads; then recompute F1 for the baselines and the proposed model. If the baseline F1 numbers change or the proposed model no longer beats the previous best by roughly 0.87 and 1.31 percentage points, the central claim fails.","tokens_in":13520,"feed_emoji":"📰","tokens_out":5301,"duration_ms":49505,"temperature":0.7,"pith_summary":"The paper argues that in multi-task models for fake news detection, sending every shared feature to every task hurts performance: some shared features are useless or misleading for a particular task. To fix this, it inserts a selected sharing layer between the shared representation and each task, made of a gate that filters features and an attention mechanism that highlights features relevant to that task. The layer is trained jointly with stance detection and veracity detection using transformer encoders. On RumourEval and PHEME the resulting model posts the best results among the compared methods, raising F1 by more than 0.87 and 1.31 percentage points over the previous best. The point is that the same shared pool of features can serve each task better when absorption, not just sharing, is task-specific.","feed_headline":"F1 up 0.87 and 1.31 points when tasks sift shared features","feed_subtitle":"A gate-plus-attention layer lets stance and veracity tasks absorb only useful features, beating prior state of the art.","key_machinery":"The selected sharing layer, placed after the shared layer and before each task's output. It combines a gated sharing cell, $G = \\sigma(W H_{\\text{shared}} + b) \\odot H_{\\text{shared}}$, which filters useless shared features, with an attention sharing cell, $A = \\text{MultiHead}(E_{\\text{task}}, K_{\\text{shared}}, V_{\\text{shared}})$, which focuses on shared features relevant to the specific task. The two are integrated as $[G; |G-A|; G\\odot A; A]$ and concatenated with the task's private features before a softmax output. Transformer encoders carry out the shared and private feature extraction.","core_discovery":"The central discovery is that 'different absorption from the same sharing' is learnable: a selected sharing layer can make two related tasks read different, task-appropriate subsets of one shared feature representation. Previously, shared-private models passed the entire shared output to both tasks, mixing useless and even adversarial features into each task's prediction. The paper shows, through ablations, that the combined gate and attention cells outperform either component alone, and through case studies that the gate keeps key tokens while the attention weights differ by task, so the same shared representation is absorbed differently by fake news detection and stance detection.","pith_inferences":["A natural extension the paper does not run: applying the same selected sharing layer to other multi-task pairs, such as sentiment and sarcasm detection or claim verification and stance, to see whether the different-absorption benefit is a general property or specific to rumor tasks.","The paper's choice of one-hot position embeddings over sinusoidal embeddings is motivated by a small-dataset experiment; a direct comparison with learned position embeddings on these datasets would clarify whether the conclusion is about position encoding or about model capacity.","The visualizations suggest an interpretability distinction: the gate behaves like a keyword detector while the attention cell tracks task-specific evidence; that distinction could be turned into a diagnostic for whether the model is using spurious cues."],"forward_implications":["If the method is right, multi-task NLP systems can stop treating shared layers as a common pool and instead learn per-task selection, which should transfer to other task pairs with partially conflicting signals.","The reported results imply that for RumourEval and PHEME, stance detection and veracity detection share enough structure that a transformer-based shared-private model with the selected sharing layer outperforms previous LSTM- and Bayesian-based veracity detectors.","The ablation results imply that filtering alone and selection alone each add value, and the two together perform best; future designs should treat them as complementary rather than interchangeable.","Because accuracy and F1 are both highest on both datasets, the claim that selective sharing is beneficial does not depend on a single evaluation metric."],"supporting_citations":[{"why":"Defines the RumourEval dataset and task, providing one of the two evaluation benchmarks.","marker":"(Derczynski et al., 2017)"},{"why":"Provides the PHEME dataset of Twitter conversation threads used as the second evaluation benchmark.","marker":"(Zubiaga et al., 2016b)"},{"why":"Supplies the transformer encoder used for shared, private, and attention-cell feature extraction.","marker":"(Vaswani et al., 2017)"},{"why":"Inspires the gate mechanism in the gated sharing cell.","marker":"(Hochreiter and Schmidhuber, 1997)"},{"why":"Supplies the integration formula $[G; |G-A|; G\\odot A; A]$ used to combine the two sharing cells.","marker":"(Mou et al., 2016)"},{"why":"Is the MTL-LSTM multi-task baseline that jointly trains veracity, rumor detection, and stance, against which the method is compared.","marker":"(Kochkina et al., 2018)"},{"why":"Is the Bayesian-DL baseline, the previous best comparator that the proposed method claims to surpass on both datasets.","marker":"(Zhang et al., 2019)"},{"why":"Provides an earlier neural multi-task approach for detecting rumor and stance jointly, framing the shared-feature problem this paper addresses.","marker":"(Ma et al., 2018a)"}],"fun_headline_variants":["Sifted shared features boost fake news F1 by up to 1.31 points","Tasks absorb only useful features via gated attention, lifting F1","Gate+attention lets each task pick its own features from shared layer","Selective sharing layer filters features improving fake news detection","Same sharing, different absorption: sifted multitask for fake news"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The experimental comparison is fair: the filtered class counts in Table 1 are correct, the 10% holdout plus 5-fold cross-validation is applied identically to every method, and the baseline numbers were produced under the same protocol; if PHEME's 105,354 tweets do not correspond to the 2,402 veracity-labelled threads or the baselines used different splits, the state-of-the-art comparison collapses.","fun_headline_variants_meta":{"raw":{"variants":["Sifted shared features boost fake news F1 by up to 1.31 points","Tasks absorb only useful features via gated attention, lifting F1","Gate+attention lets each task pick its own features from shared layer","Selective sharing layer filters features improving fake news detection","Same sharing, different absorption: sifted multitask for fake news"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001162,"raw_usage":{"total_tokens":4744,"prompt_tokens":810,"completion_tokens":3934,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":426,"completion_tokens_details":{"reasoning_tokens":3842}},"tokens_in":426,"tokens_out":3934,"duration_ms":28328,"temperature":1.0,"reasoning_tokens":3842,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:09:15.531228+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the exact filtered datasets under the paper's 10%-holdout plus 5-fold protocol, verifying that every veracity label is a thread-level label and that PHEME's class counts sum to the number of labeled threads; then recompute F1 for the baselines and the proposed model. If the baseline F1 numbers change or the proposed model no longer beats the previous best by roughly 0.87 and 1.31 percentage points, the central claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the RumourEval dataset and task, providing one of the two evaluation benchmarks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the integration formula $[G; |G-A|; G\\odot A; A]$ used to combine the two sharing cells."},{"cited_title":"All-in-one: Multi-task Learning for Rumour Verification","cited_arxiv_id":"1806.03713","evidence_quote":"Is the MTL-LSTM multi-task baseline that jointly trains veracity, rumor detection, and stance, against which the method is compared."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is the Bayesian-DL baseline, the previous best comparator that the proposed method claims to surpass on both datasets."}],"review_version":1}