{"id":"d5b8e436-536a-4788-b7ca-28289f29065e","arxiv_id":"2505.23812","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"SPLAENet claims state-of-the-art stance detection on RumourEval, SemEval, and P-Stance, but the reported 'average gains' are over the mean of all baselines, not the best baseline.","lead":"SPLAENet, a new neural model, uses emotion, attention, and label information to detect whether a social media reply supports, denies, queries, or comments on a source post. It reports higher accuracy and F1 on three stance detection benchmarks, but the headline gains are measured against the average of many weak baselines, not against the strongest published model.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RumourEval results may not be comparable: Table 5 reports a modified test split (1,746 vs official 1,827), and the claimed gains are averaged over weak baselines instead of actual SOTA; central claim is untested until rerun on the official split with real top systems.","rationale":"I read the paper as making an empirical claim: a single architecture, SPLAENet, substantially improves stance detection on three benchmarks with different imbalance characteristics. The architecture itself is coherent, and the ablation studies suggest the proposed components help on the reported splits. However, the central claim is only as strong as the comparison protocol, and the manuscript contains internal evidence that the protocol is not the standard benchmark protocol. Table 5 explicitly reports a post-preprocessing RumourEval test set of 1,746 posts versus the official 1,827, meaning the paper evaluates on a different test set from the published systems it compares against. This alone invalidates direct comparison to published SOTA numbers. Additionally, the headline gains in the abstract are average over all listed baselines, not against the best baseline or published SOTA; when viewed against the best listed baseline, the gains shrink to around one point of accuracy on SemEval and P-Stance, which is well within typical run-to-run variance. The reader's weakest assumption correctly identified the baseline-comparison protocol as load-bearing; my concern sharpens it by locating the exact discrepancy in Table 5 and the exact averaging mechanism in Table 8. I am not alleging misconduct; the issue could be an artifact of reporting. The concrete test above would settle it: rerun on the official test split and compare against the actual best published systems. If the margin persists, the paper's claim is supported; if not, the central claim fails. Because the reader already marked the paper CONDITIONAL and my proposed check is precisely the condition that would resolve the uncertainty, I do not change the verdict, but I emphasize that the condition is mandatory before the claim can be accepted.","tokens_in":32991,"tokens_out":5597,"duration_ms":63484,"concrete_test":"Download the official SemEval-2019 Task 7 RumourEval data, train SPLAENet on the official train/dev split, and evaluate on the untouched official test split (1,827 posts) without removing any posts during cleaning. Also include the published top SemEval-2019 Task 7 systems (e.g., the cited BUT-FIT system) and the actual best published numbers for SemEval-2016 Task 6A and P-Stance in the comparison table, and report macro-F1 with standard deviations over 3-5 seeds. If the resulting margin over the best published system is below the reported 17.36 F1 points, or within one standard deviation, the 'significant SOTA improvement' claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that SPLAENet 'significantly outperforms existing state-of-the-art methods' rests on Table 8, but Table 8 is not a fair SOTA comparison. First, RumourEval is not the official SemEval-2019 Task 7 test set: Table 5 says preprocessing reduced the dataset from 8,529 to 8,083 posts, including shrinking the test split from 1,827 to 1,746. Removing 81 test posts (and more train/val posts) changes the evaluation set, so numbers reported by published systems on the official test are not directly comparable. Second, the abstract's 8.92/17.36 gains are computed against the arithmetic mean of the 16 listed systems (e.g., 51.52 - 34.16 = 17.36 on F1), not against the best published system; that mean includes zero-shot LLMs (Mistral/GPT-3.5/LLaMa3), StanceBERTa, and BERT with macro-F1 as low as 22.91. Against the best listed baseline, SPLAENet's advantage is +1.19 accuracy and +8.83 F1 on RumourEval, +0.36 accuracy and +1.16 F1 on SemEval, and +1.06 accuracy and +1.19 F1 on P-Stance. The gap to the actual state of the art is therefore much smaller than the abstract suggests. The internal count inconsistency in Table 4 (6,165+819+567+553=8,104, not 8,083) reinforces concern that the dataset pipeline is not the standard benchmark and that the reported 'significant SOTA improvement' is not yet established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SPLAENet, a stance-detection architecture that combines RoBERTa textual features with a dual cross-attention mechanism, a hierarchical attention network, NRCLex-based emotion features, feature-closeness distance metrics, and a label-fusion module. The model is evaluated on RumourEval, SemEval-2016 Task 6A, and P-Stance, with reported average gains over baseline systems of up to 8.92% accuracy and 17.36% F1 on RumourEval. The main claimed contribution is that jointly modeling inter/intra-textual attention, emotional alignment, and label proximity yields consistent improvements across highly imbalanced, moderately imbalanced, and balanced benchmark datasets.","tokens_in":33407,"tokens_out":3658,"duration_ms":39029,"significance":"If the reported results are validated on the official benchmark splits and against a fair set of published state-of-the-art systems, SPLAENet would be a meaningful contribution: the dual cross-attention design is well motivated, the emotion-synthesis module is a plausible mechanism for stance discrimination, and the label-fusion idea is cleanly described. The paper also provides public code and a relatively extensive ablation study across three datasets, which are strengths. However, the central significance claim currently rests on a comparison protocol that inflates the reported gains, and on a modified RumourEval test split that prevents direct comparison with prior published results.","major_comments":[{"comment":"The RumourEval evaluation is run on a modified test split (1,746 posts after preprocessing) rather than the official SemEval-2019 Task 7 test set (1,827 posts), with the total dataset reduced from 8,529 to 8,083 posts. Because the test set is different, the numbers previously published by state-of-the-art systems on the official split are not directly comparable, so the claimed superiority over those systems is not established. The authors should either rerun the model on the official test split or clearly frame the results as a new, differently preprocessed benchmark and avoid claiming state-of-the-art comparison with published RumourEval numbers.","section":"§5.1.2, Table 5"},{"comment":"The headline average gains (8.92% accuracy and 17.36% F1 on RumourEval, and similar figures on the other datasets) are computed against the arithmetic mean of all listed baselines, including zero-shot LLMs, StanceBERTa, and BERT variants with macro-F1 scores as low as 22.91. Against the strongest listed baseline the gains are much smaller: on RumourEval, +1.19 accuracy and +8.83 F1; on SemEval, +0.36 accuracy and +1.16 F1; on P-Stance, +1.06 accuracy and +1.19 F1. The \"significant improvements over state-of-the-art\" claim should be based on pairwise comparisons with the best available published systems, not on the mean of a heterogeneous baseline set.","section":"Abstract and Table 8"},{"comment":"The dataset counts in Table 4 are internally inconsistent: the RumourEval label counts sum to 6,165 + 819 + 567 + 553 = 8,104, not 8,083 as reported, and the percentages do not correspond to the listed counts. Table 5 also mixes label percentages with split percentages, showing train/val/test rows of 75.11/11.23/7.04 before preprocessing and 50.68/25.39/23.93 after, the latter matching the label distribution of Table 4 rather than the actual split proportions. These inconsistencies undermine the reproducibility of the dataset pipeline and should be corrected and reconciled with the official benchmark statistics.","section":"§5.1.2, Table 4"},{"comment":"The Friedman test results are reported as evidence that SPLAENet \"significantly outperforms\" competing methods, but the test only establishes that not all systems perform equally; it does not identify SPLAENet as significantly better than the best baseline. The p-value for P-Stance is 0.0477, which is borderline, and no post-hoc pairwise comparisons or corrections for multiple testing are provided. To support the central claim, the authors should report pairwise significance tests against the strongest baseline on each dataset.","section":"§5.3.5, Statistical Analysis"}],"minor_comments":[{"comment":"The sentence \"SPLAENet also outperforms StanceBERTa, BERT, FLAN-T5, and RoBERTa [21]\" appears to cite the wrong reference for RoBERTa; the correct citation is [20]. Additionally, DeBERTa is mentioned in the surrounding text but is not listed in this sentence.","section":"§5.2.1"},{"comment":"Equation (3a) contains mismatched parentheses in the softmax argument: \"softmax( Qs(Hs))Kr(Hr)T)\" should be rewritten to make clear that the softmax is applied to Q_s K_r^T / sqrt(d_k). The same formatting issue affects later attention equations.","section":"§4.2.1, Eq. (3a)"},{"comment":"The text says \"As seen in the Table 11\" when discussing P-Stance performance against existing state-of-the-art methods, but the comparative results are in Table 8; Table 11 reports ablations.","section":"§5.2.3"},{"comment":"In the ablation discussion, the sentence \"integrating DCA mechanism leads to a 1.82% increase in accuracy compared to the proposed method\" reverses the direction of the comparison; the increase should be relative to the variant without DCA, not relative to the full SPLAENet. The same direction reversal appears in the HAN ablation discussion.","section":"§5.3.1"},{"comment":"Table 12 lists LKI-BART twice in the predictions block, and one row is labeled \"ZSSD [61]\" while the correct reference for ZSSD is [45]. The qualitative discussion also states that Post 3's reply \"is categorized as a Query\" while the ground truth is Deny, which contradicts the table and the surrounding text; this should be clarified.","section":"§5.3.4, Table 12"},{"comment":"The P-Stance dataset description says it contains 30,000 annotated tweets, but Table 4 reports a total of 21,574. The authors should reconcile these numbers with the P-Stance paper.","section":"§5.1.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's core architecture is interesting and the ablation study is reasonably thorough, but the evaluation protocol needs substantial revision before the central claim can be accepted. My main concern is not the method itself but the comparability of the reported numbers: the modified RumourEval split and the averaging over weak baselines make the headline gains misleading. The authors should be asked to rerun on the official split and to reframe the comparison around the best available published systems. I do not see evidence of circular reasoning or fabrication, but the internal inconsistencies in Tables 4 and 5 need to be fixed as part of the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a plausible architecture paper with a real reproducibility burden. The new composition (dual cross-attention with key/value modes, label-fusion DML, emotion synthesis) is worth looking at, and the ablations suggest each piece contributes. But the central claim — 'significant improvements over SOTA' — does not survive contact with the comparison protocol. The 17.36-point F1 gain on RumourEval is computed against the mean of 16 systems, several of which are zero-shot LLMs scoring in the 20s. Against the best listed baseline, the F1 advantage drops to ~8.8 points on RumourEval, and to ~1.2 points on SemEval and P-Stance. That is a materially different claim.\n\nThe bigger problem is the RumourEval test split. Table 5 reports a modified test set (1,746 posts vs the official 1,827). Removing 81 test posts makes comparisons with published results on the official split non-comparable. The paper needs to either rerun on the official split or clearly classify its results as 'modified setting' and stop claiming direct SOTA improvements. Also Table 4's counts don't add up (6165+819+567+553 = 8104, not 8083), which does not help confidence. No error bars, and the Friedman test over four metrics is an odd way to compare models.\n\nThe architecture itself is described in enough detail to be reimplemented, code is claimed, and the limitations section is honest about language coverage and noisy text. I don't see circularity or invented entities.\n\nThe bottom line: with a revised evaluation protocol this could be a solid paper. As written, the headline is overclaimed. It deserves a serious referee — the model is non-trivial and the problem is important — but the referee should insist on official splits, best-baseline comparisons, and error bars before acceptance.\n\nWho this is for: people working on stance detection in social media, especially those combining affective signals with attention. I would bring it to a reading group as a case study in baseline choices, but I would not cite it as evidence of SOTA until the numbers are reworked.\n\nRecommendation: send to peer review, but make clear it needs major revision before the stated contributions can be credited.","headline":"A plausible new architecture whose headline 'SOTA' gains evaporate once you compare against the best baseline rather than the average of weak ones; worth a revise-and-resubmit, not a desk reject.","tokens_in":33919,"tokens_out":2221,"would_cite":false,"duration_ms":24546,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SPLAENet reports higher accuracy and F1 on three stance benchmarks by fusing emotion, attention, and label distance.","keywords":["stance detection","dual cross-attention","emotion synthesis","label fusion","distance metric learning","misinformation","social media","RoBERTa"],"falsifier":"Run SPLAENet and the top officially reported systems for RumourEval, SemEval, and P-Stance under identical train, validation, and test splits with the same preprocessing and hyperparameter search; if SPLAENet's macro-F1 advantage over the strongest published system for each dataset is close to zero or negative, the paper's central significance claim is falsified.","tokens_in":32840,"feed_emoji":"💬","tokens_out":8820,"duration_ms":86509,"temperature":0.7,"pith_summary":"The paper aims to show that stance detection in misinformation-heavy social media improves when a model explicitly relates a source post to its reply in both directions, rather than encoding each text alone. To do this it introduces SPLAENet, which combines a two-stage cross-attention module with hierarchical attention, an emotion-alignment signal, and a label-fusion step that measures how close learned features sit to each stance label. On three public benchmarks with different class balance, it reports average accuracy gains between roughly 7 and 10 points and macro-F1 gains between about 11 and 17 points over the compared systems. The reason to care is that stance, whether a reply supports, denies, queries, or comments on a claim, is a practical signal for detecting how misinformation spreads and polarises opinion.","feed_headline":"Emotion-aware attention lifts stance detection accuracy by up to 8.9%","feed_subtitle":"SPLAENet pairs dual cross-attention with emotion alignment and label fusion to beat prior systems on three benchmarks.","key_machinery":"The load-bearing mechanism is the dual cross-attention module: stage one uses each text's own queries and values but the other text's keys, and stage two uses each text's own queries and keys but the other text's values, with self-attention applied after each stage. This is followed by a hierarchical attention network that pools the attended token vectors into context vectors. Complementing it are three distance-based signals: the L2-normalized absolute difference of [CLS] representations, the absolute difference of top-K emotion embeddings produced by NRCLex and RoBERTa, and a label-fusion step that computes absolute differences between a projected feature vector and each stance-label embedding, concatenates them, and feeds the result into the classification head.","core_discovery":"The central claim is that modelling the source-reply pair with a dual cross-attention pass, first letting each text attend to the other's keys and then to the other's values, with self-attention between the passes, extracts the inter-textual dependencies that single-encoder classifiers miss. On top of that, SPLAENet treats emotions as a relational signal: it embeds the top three NRCLex emotions of each text and takes their absolute difference as an emotion-divergence vector. It also computes an L2-normalized difference between the [CLS] representations of source and reply, and a label-fusion module that concatenates feature-to-label distances before classification. With these components the model reaches 86.50% accuracy and 51.52% macro-F1 on RumourEval, 75.26% and 72.50% on SemEval, and 85.67% and 85.58% on P-Stance, which the paper presents as the best overall performance among all systems it compares against on all three datasets.","pith_inferences":["The reported average gain is computed against the mean of the systems in Table 8, so the headline margins are a property of that comparison set; a comparison limited to the strongest published system for each dataset would almost certainly show smaller differences.","The label-fusion idea transfers naturally to other pairwise-text classification tasks, such as claim verification, argument mining, or reply-quality ranking, where the relationship between two texts rather than the content of either alone determines the label.","A cheap testable extension is to replace the one-word stance labels with paraphrased or descriptive label embeddings in the fusion module; the paper's own analysis says label proximity works best when labels are well separated, so richer label text could help the overlapping query and comment classes.","The emotion signal is tied to NRCLex's fixed lexicon, so a contextual emotion model trained on social media might strengthen the emotional-divergence vector on sarcastic or informal replies."],"forward_implications":["On the reported numbers, one architecture covers imbalanced (RumourEval), moderately imbalanced (SemEval), and balanced (P-Stance) datasets without dataset-specific changes.","The ablations show every component helps: removing dual cross-attention, hierarchical attention, label fusion, emotion synthesis, or feature closeness lowers accuracy and macro-F1 on all three datasets.","The dual cross-attention and hierarchical attention pair is the biggest driver, since removing both costs about 17 F1 points on RumourEval and more than 5 F1 points on SemEval.","The label-fusion component makes its largest contribution on SemEval, while emotion synthesis is especially important on P-Stance, indicating the two mechanisms address different failure modes.","With a reported 383 million parameters and 0.046-second inference per sample, the method is positioned for deployment-scale content moderation, provided the gains reproduce against the strongest published systems."],"supporting_citations":[{"why":"Provides the scaled dot-product attention operation that the dual cross-attention module adapts.","marker":"[18]"},{"why":"Serves as the shared RoBERTa encoder for source, reply, emotion, and label embeddings.","marker":"[20]"},{"why":"Supplies the emotion lexicon from which the top-K emotion features are selected.","marker":"[34]"},{"why":"Defines the highly imbalanced RumourEval dataset used for evaluation.","marker":"[49]"},{"why":"Defines the moderately imbalanced SemEval dataset used for evaluation.","marker":"[50]"},{"why":"Defines the balanced P-Stance dataset used for evaluation.","marker":"[51]"},{"why":"Attention-based baseline with topic-agnostic and topic-aware embeddings that SPLAENet is compared against.","marker":"[7]"},{"why":"Prior label-encoding framework that the label-fusion stage extends by measuring feature-label proximity.","marker":"[22]"},{"why":"Zero-shot contrastive baseline included in the comparison table.","marker":"[45]"},{"why":"Contrastive label-aware baseline included in the comparison table and qualitative analysis.","marker":"[60]"}],"fun_headline_variants":["SPLAENet uses emotion and dual attention to boost stance detection up to 8.9%","Dual cross-attention plus emotion lifts stance accuracy by up to 8.9%","Emotion-aware SPLAENet outperforms stance baselines by up to 17% F1","SPLAENet's emotion+attention fusion lifts stance detection by 8.9%","Emotion, dual attention, and label fusion lift stance detection by up to 8.9%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that SPLAENet improves on existing methods rests on the systems in Table 8 being a fair and representative set, trained and evaluated under the same conditions; if stronger published systems were included or the baselines were run under different protocols, the reported gains could narrow substantially or vanish.","fun_headline_variants_meta":{"raw":{"variants":["SPLAENet uses emotion and dual attention to boost stance detection up to 8.9%","Dual cross-attention plus emotion lifts stance accuracy by up to 8.9%","Emotion-aware SPLAENet outperforms stance baselines by up to 17% F1","SPLAENet's emotion+attention fusion lifts stance detection by 8.9%","Emotion, dual attention, and label fusion lift stance detection by up to 8.9%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001161,"raw_usage":{"total_tokens":4887,"prompt_tokens":1107,"completion_tokens":3780,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":723,"completion_tokens_details":{"reasoning_tokens":3657}},"tokens_in":723,"tokens_out":3780,"duration_ms":28643,"temperature":1.0,"reasoning_tokens":3657,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:29:47.271352+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SPLAENet and the top officially reported systems for RumourEval, SemEval, and P-Stance under identical train, validation, and test splits with the same preprocessing and hyperparameter search; if SPLAENet's macro-F1 advantage over the strongest published system for each dataset is close to zero or negative, the paper's central significance claim is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Serves as the shared RoBERTa encoder for source, reply, emotion, and label embeddings."},{"cited_title":"Mohammad, P","cited_arxiv_id":null,"evidence_quote":"Supplies the emotion lexicon from which the top-K emotion features are selected."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Attention-based baseline with topic-agnostic and topic-aware embeddings that SPLAENet is compared against."}],"review_version":1}