REVIEW 2 major objections 5 minor 29 references
Reasoner Outperforms: Generative Stance Detection with Rationalization for Social Media
T0 review · 2 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read A 780M FlanT5 model, trained with rationale multitask learning, beats GPT-3.5's zero-shot stance detection on SemEval-2016.
desk verdict Useful empirical result on rationale distillation for stance detection, weakened by test-set selection of the MTL weight and an unsupported faithfulness claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the label-conditioned rationale elicitation prompt (Prompt 2), which asks GPT-3.5 to begin its answer with "The comment is classified as [Stance] towards [Topic] because...", ensuring the generated rationale is tied to the true label. The distillation machinery is multitask learning on T5 and FlanT5, with task prefixes ("Stance:" and "Explain:") and a weighted loss $L = \alpha L_{\text{stance}} + (1-\alpha) L_{\text{rationale}}$; the coefficient $\alpha$ is tuned per model size. This flexible task grouping, rather than enforced sequential generation, is what carries the performance gain over single-task finetuning and single-task chain-of-thought.
What would settle it
A control experiment that replaces GPT-3.5 rationales with label-matched but content-free text, such as a template that only names the topic and stance, in the MTL training set: if F_avg does not drop, the claim that faithful rationales drive distillation is falsified.
Extended reading notes
Core claim
The discovery, in the authors' terms, is that reasoning is a transferable supervisory signal when decoupled from prediction. They reframe GPT-3.5's stance task as explanation generation with a prompt that conditions the rationale on the ground-truth label, so the explanation agrees with the known stance. Distilling those rationales into a small model via multitask learning, with stance and rationale as separate tasks sharing an encoder-decoder, consistently outperforms both ordinary finetuning and enforced chain-of-thought generation. The paper also reports that instruction tuning narrows the benefit: T5 gains more from rationale distillation than FlanT5, and that MTL is especially robust in low-data settings. Its strongest assertion is that faithful rationales are the reason distillation works, making the rationale content, not just the label, the mechanism behind the gain.
Load-bearing premise
The paper's central mechanism depends on GPT-3.5's label-conditioned rationales being faithful enough to serve as a trustworthy training signal, but no faithfulness metric, human evaluation, or comparison to gold rationales is reported.
Editorial extensions
If this is right
- A 780M-parameter FlanT5 model trained under MTL beats GPT-3.5's zero-shot performance on SemEval-2016, so small models are a viable deployment choice when large API access is costly.
- MTL reaches near-full-data performance at 10-20% of the training set for T5-Base and T5-Large, suggesting the rationale task supplies useful signal in low-resource settings.
- Enforcing sequential reasoning before prediction (ST-CoT) hurts small models, so rationale as an auxiliary task rather than a required reasoning step is the better distillation design.
- FlanT5 models need less help from rationale distillation than vanilla T5, indicating instruction-tuned models already carry implicit reasoning that reduces the marginal benefit.
- Outputs include human-readable justifications for each stance prediction, which is what the paper means by interpretable stance detection.
Reading between the lines
- If the faithfulness claim is right, rationale quality metrics should be added to the evaluation: automated plausibility checks or human annotations would tell whether the 9.57-point gain tracks rationale content or merely the presence of an auxiliary task.
- The same label-conditioned elicitation could be applied to other classification tasks where gold rationales are expensive, treating LLM rationales as a scalable training signal.
- A control with label-matched but content-free rationales would isolate whether the gain comes from semantic reasoning or from the regularization effect of a second task.
- Comparing against few-shot GPT-3.5 or a fine-tuned GPT-3.5 would clarify how much of the reported gap is zero-shot weakness rather than genuinely strong small-model reasoning.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a generative framework for stance detection in which GPT-3.5 produces rationales conditioned on ground-truth stance labels, and these rationales are distilled into smaller T5 and FlanT5 models under three paradigms: single-task fine-tuning (ST-FT), single-task chain-of-thought (ST-CoT), and multitask learning (MTL) with loss L = αL_stance + (1-α)L_rationale. Experiments on SemEval-2016 Task 6 report that MTL outperforms ST-FT and ST-CoT across model sizes, with the best FlanT5-Large MTL reaching 79.72 F_avg versus 70.15 for GPT-3.5 zero-shot. The paper claims that reasoning improves multitask learning performance, that MTL is especially effective in low-data settings, and that faithful rationales improve rationale distillation.
Significance. The paper addresses an important problem—interpretable stance detection for social media—and the idea of using LLM-generated rationales as supervisory signals for smaller models is timely. The systematic comparison of three distillation paradigms, the inclusion of standard deviations over three runs, and the low-data analysis are useful strengths. If the empirical claims hold, the work could inform practical deployment of explainable stance detectors. However, the central claims currently rest on an inadequately specified hyperparameter selection procedure and on an unmeasured notion of rationale faithfulness, so the significance is conditional on resolving these issues.
major comments (2)
- [§4, Table 2, Figure 2] The selection of the MTL weighting α is not described as validation-based. Table 2's caption states that MTL performance is reported at the 'optimal α,' and Figure 2 sweeps α from 0.1 to 0.9 without stating that the 10% validation split described in Section 4 was used to choose α. If α is selected using the test set, MTL receives a selection advantage that ST-FT and ST-CoT do not have, because they have no equivalent hyperparameter. This is load-bearing because the abstract's headline improvement of up to 9.57% relies on the FlanT5-Large MTL result (79.72), and the gap between MTL and ST-FT for that model is only 0.96 F_avg. Please clarify where α was selected and, preferably, report results with α chosen on the validation split or demonstrate that the conclusions are robust to α across validation folds.
- [§6 and Abstract] The claim that 'faithful rationales improve rationale distillation' is not supported by any direct evidence in the manuscript. The paper observes that Prompt 1 produces unfaithful rationales and conditions rationales on ground-truth labels in Prompt 2, but it never measures faithfulness of the generated rationales, either through human evaluation, automatic metrics, or comparison to gold rationales. The MTL-versus-ST-FT comparison shows only that adding a separate rationale-generation task helps, not that the rationales are faithful or that faithfulness drives the gain. Since the paper's central contribution is rationale distillation, this missing link is load-bearing. Please add a faithfulness evaluation or soften the claim to refer to 'label-conditioned rationales' without asserting faithfulness.
minor comments (5)
- [Title page and footer] The paper uses the ACM template with 'Under review, 2018' in the reference footer and the copyright line, which is inconsistent with the 2024/2025 submission date; please update the template fields.
- [Prompt 3 in Section 3.2] There is a typo in Prompt 3: 'Your task if to classify' should be 'Your task is to classify.'
- [Section 3.2 and Section 4] The task prefix is introduced as 'Explain:' in Section 3.2 but appears as 'Explanation:' in the experimental details of Section 4; please make the prefixes consistent.
- [§3.1 and Table 2] The GPT-3.5 zero-shot baseline is reported as 70.15 with a standard deviation, but the exact model version, temperature, and decoding parameters are not specified; please provide these details for reproducibility.
- [Figures 2 and 3] The axis labels and legend text in Figures 2 and 3 appear small and may be hard to read in print; please enlarge them or increase the figure resolution.
Circularity Check
MTL's reported optimal-α results are selected on the test set, partially constructing the claimed gains; no equation-level circularity or load-bearing self-citation found.
-
fitted input called prediction
[Table 2 caption; Section 5 (Weighting rationale generation varies performance) and Figure 2]
"MTL performance reported at the optimal α ... We examine the impact of the prediction and rationale generation tasks in MTL by varying the parameter α from 0.1 to 0.9, as shown in Figure 2. Optimal α values are generally low: 0.5 for T5-Small, 0.2 for T5-Base and T5-Large, 0.3 for FlanT5-Small, and 0.1 for FlanT5-Large, with FlanT5-Base as an exception at 0.9."
The MTL results are reported at the α that maximizes F_avg on the test set: Table 2 states 'MTL performance reported at the optimal α' and Figure 2 sweeps α from 0.1 to 0.9 with no statement that the validation split was used. ST-FT and ST-CoT appear at a single hyperparameter setting, so the comparison is not symmetric: MTL's reported number is, by construction, the maximum over the α grid, while the baselines are arbitrary fixed points. This selection effect is large enough to manufacture the FlanT5-Large margin (79.72 vs 78.76 for ST-FT) that underlies the abstract's 'up to 9.57%' claim. The T5-Large margin (76.79 vs 68.46) gives some independent support, so the circularity is partial rather than total.
full rationale
The paper's derivation is largely empirical and self-contained: the MTL loss is a convex combination of two cross-entropy losses (L = αL_stance + (1−α)L_rationale), and stance and rationale are separate outputs, so the stance prediction is not definitionally derived from the rationale. There are no load-bearing self-citations: references to prior work are standard (T5, CoT, SemEval-2016) and none are by the present authors. The main circularity concern is the α selection: Table 2 reports MTL 'at the optimal α' and Figure 2 sweeps α without stating that the 10% validation split was used, so the reported MTL scores are consistent with being the maximum over the test set. This gives MTL a selection advantage over ST-FT and ST-CoT, which are reported at a single configuration, and directly inflates the headline 'up to 9.57%' comparison (FlanT5-Large MTL 79.72 vs GPT-3.5 70.15), where the MTL-vs-ST-FT margin is only 0.96. The T5-Large result (76.79 vs 68.46) provides independent evidence for MTL's benefit, so the circularity is partial, not total. The faithfulness claim is unsupported (no faithfulness metric or human evaluation) but is an unverified assumption rather than a definitional circle.
Assumptions & free parameters
free parameters (1)
- alpha (MTL task weighting) =
0.5 (T5-Small), 0.2 (T5-Base, T5-Large), 0.3 (FlanT5-Small), 0.9 (FlanT5-Base), 0.1 (FlanT5-Large)
assumptions (2)
- domain assumption GPT-3.5 rationales conditioned on ground-truth labels are faithful and beneficial for distillation.
- domain assumption SemEval-2016 Task 6 is a representative benchmark for real-world social-media stance detection.
Cite this review
Pith. "Pith review of Reasoner Outperforms: Generative Stance Detection with Rationalization for Social Media." pith.science (2026). https://pith.science/paper/6OFZTBUO
@misc{pith2026241210266,
author = {Pith},
title = {Pith review of: Reasoner Outperforms: Generative Stance Detection with Rationalization for Social Media},
year = {2026},
howpublished = {\url{https://pith.science/paper/6OFZTBUO}},
note = {Machine review of arXiv:2412.10266}
}
read the original abstract
Stance detection is crucial for fostering a human-centric Web by analyzing user-generated content to identify biases and harmful narratives that undermine trust. With the development of Large Language Models (LLMs), existing approaches treat stance detection as a classification problem, providing robust methodologies for modeling complex group interactions and advancing capabilities in natural language tasks. However, these methods often lack interpretability, limiting their ability to offer transparent and understandable justifications for predictions. This study adopts a generative approach, where stance predictions include explicit, interpretable rationales, and integrates them into smaller language models through single-task and multitask learning. We find that incorporating reasoning into stance detection enables the smaller model (FlanT5) to outperform GPT-3.5's zero-shot performance, achieving an improvement of up to 9.57%. Moreover, our results show that reasoning capabilities enhance multitask learning performance but may reduce effectiveness in single-task settings. Crucially, we demonstrate that faithful rationales improve rationale distillation into SLMs, advancing efforts to build interpretable, trustworthy systems for addressing discrimination, fostering trust, and promoting equitable engagement on social media.
Figures
Reference graph
Works this paper leans on
-
[1]
Abeer Aldayel and Walid Magdy. 2019. Your Stance is Exposed! Analysing Possible Factors for Stance Detection on Social Media. Proceedings of the ACM on Human-Computer Interaction 3 (2019), 1 – 20. https://api.semanticscholar.org/ CorpusID:199501836
work page 2019
-
[2]
Abeer AlDayel and Walid Magdy. 2021. Stance detection on social media: State of the art and trends. Information Processing & Management 58, 4 (2021), 102597
work page 2021
-
[3]
Emily Allaway, Malavika Srikanth, and Kathleen McKeown. 2021. Adversarial Learning for Zero-Shot Stance Detection on Social Media. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Bel...
work page 2021
-
[4]
Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom
-
[5]
Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixi- ang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Wei Yu, Vincent Zhao, Yan- ping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, ...
arXiv 2022
-
[6]
Daniele De Vinco, Alessia Antelmi, Carmine Spagnuolo, and Luca Maria Aiello
-
[7]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. ArXiv abs/1810.04805 (2019)
arXiv 2019
-
[8]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computation...
Show all 29 references
-
[9]
Kuntal Dey, Ritvik Shrivastava, and Saroj Kaushik. 2017. Twitter Stance Detection - A Subjectivity and Sentiment Polarity Inspired Two-Phase Approach. 2017 IEEE International Conference on Data Mining Workshops (ICDMW) (2017), 365–372. https://api.semanticscholar.org/CorpusID:5741922
2017
-
[10]
Eduardo Graells-Garrido and Ricardo Baeza-Yates. 2022. Bots don”’t Vote, but They Surely Bother! A Study of Anomalous Accounts in a National Referendum. In Companion Publication of the ACM Web Science Conference (Barcelona, Spain). Association for Computing Machinery, NY, USA,...
2022
-
[11]
Eduardo Graells-Garrido, Ricardo Baeza-Yates, and Mounia Lalmas. 2020. Every Colour You Are: Stance Prediction and Turnaround in Controversial Issues. In Proceedings of the ACM Conference on Web Science (Southampton, United King- dom) (WebSci ’20). Association for Computing Ma...
2020
-
[12]
Elena Kochkina, Maria Liakata, and Isabelle Augenstein. 2017. Turing at SemEval- 2017 Task 8: Sequential Approach to Rumour Stance Classification with Branch- LSTM. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), Steven Bethard, Marine ...
2017 doi
-
[13]
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large Language Models are Zero-Shot Reasoners. ArXiv abs/2205.11916 (2022). https://api.semanticscholar.org/CorpusID:249017743
2022 arXiv
-
[14]
Yingjie Li and Yue Zhang. 2024. Pro-Woman, Anti-Man? Identifying Gen- der Bias in Stance Detection. In Findings of the Association for Computational Linguistics, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Associ- ation for Computational Linguistics, Bangkok, Thailan...
2024 doi
-
[15]
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering. In Advances in Neural Information Processing Systems , ...
2022
-
[16]
Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016. SemEval-2016 Task 6: Detecting Stance in Tweets. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016) , Steven Bethard, Marine Carpuat, Daniel Cer, Dav...
2016 doi
-
[17]
Marius Mosbach, Tiago Pimentel, Shauli Ravfogel, Dietrich Klakow, and Yanai Elazar. 2023. Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation. In Findings of the Association for Computational Linguistics: ACL 2023, Anna Rogers, Jordan Boyd-Graber, an...
2023 doi
-
[18]
Lynnette Hui Xian Ng and Kathleen M Carley. 2022. Is my stance the same as your stance? A cross validation study of stance detection datasets. Information Processing & Management 59, 6 (2022), 103070
2022
-
[19]
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. Instruction Tuning with GPT-4. arXiv preprint arXiv:2304.03277 (2023)
2023 arXiv
-
[20]
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.J. Mach. Learn. Res. 21, 1, Article 140 (jan 2020), 67 pages
2020
-
[21]
Alipoormolabashi
Yizhong Wang, Swaroop Mishra, and et al. Alipoormolabashi. 2022. Super- NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, Yoav Goldberg, Zornitsa Kozareva, ...
2022 doi
-
[22]
Xia, Quoc Le, and Denny Zhou
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoc Le, and Denny Zhou. 2022. Chain of Thought Prompting Elicits Reasoning in Large Language Models. ArXiv abs/2201.11903 (2022). https: //api.semanticscholar.org/CorpusID:246411621
2022 arXiv
-
[23]
Hanzi Xu, Slobodan Vucetic, and Wenpeng Yin. 2022. OpenStance: Real-world Zero-shot Stance Detection. In Proceedings of the 26th Conference on Computa- tional Natural Language Learning (CoNLL) , Antske Fokkens and Vivek Srikumar Under review, 978-1-4503-XXXX-X/18/06 Jiaqing Yu...
2022 doi
-
[24]
Song Yang and Jacopo Urbani. 2021. Tribrid: Stance Classification with Neu- ral Inconsistency Detection. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing , Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds...
2021 doi
-
[25]
Bowen Zhang, Daijun Ding, Liwen Jing, Genan Dai, and Nan Yin. 2022. How would stance detection techniques evolve after the launch of chatgpt? arXiv preprint arXiv:2212.14548 (2022)
2022 arXiv
-
[26]
Hong Zhang, Haewoon Kwak, Wei Gao, and Jisun An. 2023. Wearing Masks Implies Refuting Trump?: Towards Target-specific User Stance Prediction across Events in COVID-19 and US Election 2020 (WebSci ’23). Association for Comput- ing Machinery, NY, USA, 23–32. https://doi.org/10.1...
2023
-
[27]
Lixing Zhu, Yulan He, and Deyu Zhou. 2020. Neural Opinion Dynamics Model for the Prediction of User-Level Stance Dynamics. Inf. Process. Manage. 57, 2 (mar 2020), 13 pages. https://doi.org/10.1016/j.ipm.2019.03.010
2020 doi
-
[2018]
In Proceedings of the 32nd International Conference on Neural Information Process- ing Systems (Montréal, Canada) (NIPS’18)
E-SNLI: Natural Language Inference with Natural Language Explanations. In Proceedings of the 32nd International Conference on Neural Information Process- ing Systems (Montréal, Canada) (NIPS’18). Curran Associates Inc., Red Hook, NY, USA, 9560–9572
-
[2024]
InCompanion Publication of the ACM Web Science Conference(Stuttgart, Germany)
Deciphering Conversational Networks: Stance Detection via Hypergraphs and LLMs. InCompanion Publication of the ACM Web Science Conference(Stuttgart, Germany). Association for Computing Machinery, NY, USA, 3–4. https://doi. org/10.1145/3630744.3658418
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.