{"id":"658a2143-3e48-455a-850d-e1f07c536e9e","arxiv_id":"2412.10266","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Multitask training with GPT-generated rationales gives FlanT5 a small accuracy boost over plain fine-tuning on SemEval-2016 stance detection, while chain-of-thought fine-tuning hurts performance.","lead":"This paper tests whether training smaller language models to produce stance labels together with written rationales improves stance detection on social media posts. On the SemEval-2016 dataset, the best FlanT5 model reaches 79.72 mean F1, about 9.57 points above a zero-shot GPT-3.5 baseline, but most of that gain also appears without any rationale training.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MTL's reported superiority rests on α selected via the test set; without validation-based α selection, the reasoner-over-plain-finetuning claim is not established.","rationale":"The reader's weakest_assumption is that GPT-3.5 rationales are not shown to be faithful; that is a real limitation of the interpretability narrative. However, the most load-bearing issue for the paper's central MTL claim is test-set selection of α. Table 2 reports 'optimal α' and Figure 2 sweeps α without specifying validation, so the MTL row is a selected maximum over hyperparameters. Since FlanT5-Large ST-FT is already 78.76, the unique contribution of reasoning (MTL) is only +0.96, and this narrow margin is exactly the one most vulnerable to selection bias. A validation-based α search could preserve or overturn the result; until then, conditional acceptance with a required re-analysis is appropriate. This is consistent with the reader's conditional verdict, though the stress-test's focus differs from the reader's stated weakest assumption.","tokens_in":8308,"tokens_out":5562,"duration_ms":42288,"concrete_test":"Select α for each model and size on the 10% validation split created in Section 4 (or via nested cross-validation) instead of using the test-set F_avg curve in Figure 2; then recompute Table 2's MTL rows. If the validation-selected α differs from the reported 'optimal' α, or if MTL no longer beats ST-FT at the corresponding sizes, the MTL-superiority claim and the 9.57% headline are not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that MTL is the best rationale-distillation paradigm and that reasoning yields the reported gains—depends on an α selection procedure that is not described as validation-based. Table 2's caption says 'MTL performance reported at the optimal α,' and Figure 2 sweeps α from 0.1 to 0.9 with no statement that the 10% validation split from Section 4 was used to choose α. If the displayed MTL numbers are the best over the test set, MTL receives a selection advantage that ST-FT and ST-CoT do not. The reported optimal values (0.1–0.5 for most models) are consistent with test-set tuning. This matters because the abstract's 'up to 9.57%' comes from FlanT5-Large MTL (79.72 vs 70.15), while FlanT5-Large ST-FT already reaches 78.76 without any rationale; the reasoning-specific margin is at most 0.96 F_avg for that model and could shrink or vanish under proper α selection. The faithfulness of rationales is also unmeasured, but the immediate load-bearing flaw is the uncontrolled α.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a generative framework for stance detection in which GPT-3.5 produces rationales conditioned on ground-truth stance labels, and these rationales are distilled into smaller T5 and FlanT5 models under three paradigms: single-task fine-tuning (ST-FT), single-task chain-of-thought (ST-CoT), and multitask learning (MTL) with loss L = αL_stance + (1-α)L_rationale. Experiments on SemEval-2016 Task 6 report that MTL outperforms ST-FT and ST-CoT across model sizes, with the best FlanT5-Large MTL reaching 79.72 F_avg versus 70.15 for GPT-3.5 zero-shot. The paper claims that reasoning improves multitask learning performance, that MTL is especially effective in low-data settings, and that faithful rationales improve rationale distillation.","tokens_in":8473,"tokens_out":3410,"duration_ms":31906,"significance":"The paper addresses an important problem—interpretable stance detection for social media—and the idea of using LLM-generated rationales as supervisory signals for smaller models is timely. The systematic comparison of three distillation paradigms, the inclusion of standard deviations over three runs, and the low-data analysis are useful strengths. If the empirical claims hold, the work could inform practical deployment of explainable stance detectors. However, the central claims currently rest on an inadequately specified hyperparameter selection procedure and on an unmeasured notion of rationale faithfulness, so the significance is conditional on resolving these issues.","major_comments":[{"comment":"The selection of the MTL weighting α is not described as validation-based. Table 2's caption states that MTL performance is reported at the 'optimal α,' and Figure 2 sweeps α from 0.1 to 0.9 without stating that the 10% validation split described in Section 4 was used to choose α. If α is selected using the test set, MTL receives a selection advantage that ST-FT and ST-CoT do not have, because they have no equivalent hyperparameter. This is load-bearing because the abstract's headline improvement of up to 9.57% relies on the FlanT5-Large MTL result (79.72), and the gap between MTL and ST-FT for that model is only 0.96 F_avg. Please clarify where α was selected and, preferably, report results with α chosen on the validation split or demonstrate that the conclusions are robust to α across validation folds.","section":"§4, Table 2, Figure 2"},{"comment":"The claim that 'faithful rationales improve rationale distillation' is not supported by any direct evidence in the manuscript. The paper observes that Prompt 1 produces unfaithful rationales and conditions rationales on ground-truth labels in Prompt 2, but it never measures faithfulness of the generated rationales, either through human evaluation, automatic metrics, or comparison to gold rationales. The MTL-versus-ST-FT comparison shows only that adding a separate rationale-generation task helps, not that the rationales are faithful or that faithfulness drives the gain. Since the paper's central contribution is rationale distillation, this missing link is load-bearing. Please add a faithfulness evaluation or soften the claim to refer to 'label-conditioned rationales' without asserting faithfulness.","section":"§6 and Abstract"}],"minor_comments":[{"comment":"The paper uses the ACM template with 'Under review, 2018' in the reference footer and the copyright line, which is inconsistent with the 2024/2025 submission date; please update the template fields.","section":"Title page and footer"},{"comment":"There is a typo in Prompt 3: 'Your task if to classify' should be 'Your task is to classify.'","section":"Prompt 3 in Section 3.2"},{"comment":"The task prefix is introduced as 'Explain:' in Section 3.2 but appears as 'Explanation:' in the experimental details of Section 4; please make the prefixes consistent.","section":"Section 3.2 and Section 4"},{"comment":"The GPT-3.5 zero-shot baseline is reported as 70.15 with a standard deviation, but the exact model version, temperature, and decoding parameters are not specified; please provide these details for reproducibility.","section":"§3.1 and Table 2"},{"comment":"The axis labels and legend text in Figures 2 and 3 appear small and may be hard to read in print; please enlarge them or increase the figure resolution.","section":"Figures 2 and 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is potentially appropriate for the venue, but the two major issues above need to be addressed before I can recommend acceptance. The α-selection ambiguity is especially important because it directly affects the headline claim; if the authors can show that validation-based selection yields the same pattern, the empirical contribution would be much stronger. The faithfulness claim, as written, overstates what the experiments show; either adding a faithfulness evaluation or tempering the language would be acceptable. I would also encourage the authors to ensure that the comparison to GPT-3.5 is fair, perhaps by including a few-shot or CoT GPT-3.5 baseline, though this is not strictly required for the main claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"New here: distilling GPT-generated rationales into small models for stance detection, with a head-to-head of single-task CoT vs multitask learning. That specific comparison is not in the prior work cited, and the result — MTL beats ST-CoT and plain fine-tuning on SemEval-2016, especially in low-data regimes — is a practically useful finding. The experimental setup is clean: standard dataset, three model sizes, two architectures, standard deviations over three runs.\n\nThe main soft spot is the MTL weight α. Table 2 reports MTL \"at the optimal α,\" and Figure 2 sweeps α from 0.1 to 0.9, but the text never says α was chosen on the 10% validation split. It reads like the best test-set value was picked. That gives MTL a selection advantage that ST-FT and ST-CoT don't get. The effect is concrete: FlanT5-Large MTL hits 79.72 versus ST-FT's 78.76, so the reasoning-specific margin is under one F1 point; the abstract's 9.57% is against a zero-shot GPT-3.5 baseline, which is weak. A secondary issue is the faithfulness claim. The paper says faithful rationales improve distillation, but no faithfulness metric, human evaluation, or comparison to any ground-truth rationale appears anywhere. That causal claim is asserted, not supported.\n\nNone of this is fatal. The MTL advantage over ST-CoT is consistent across sizes and the low-data robustness is worth knowing. The authors need to state α selection explicitly (and re-run with validation-based selection if that's not what they did), add a strong supervised baseline, and either measure faithfulness or soften that statement.\n\nWho this is for: people working on stance detection or rationale distillation with small transformers. Worth sending to review, but expect requests for those revisions.","headline":"Useful empirical result on rationale distillation for stance detection, weakened by test-set selection of the MTL weight and an unsupported faithfulness claim.","tokens_in":9045,"tokens_out":2862,"would_cite":true,"duration_ms":607013,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 780M FlanT5 model, trained with rationale multitask learning, beats GPT-3.5's zero-shot stance detection on SemEval-2016.","keywords":["stance detection","rationalization","chain-of-thought reasoning","multitask learning","rationale distillation","small language models","generative classification"],"falsifier":"A control experiment that replaces GPT-3.5 rationales with label-matched but content-free text, such as a template that only names the topic and stance, in the MTL training set: if F_avg does not drop, the claim that faithful rationales drive distillation is falsified.","tokens_in":8069,"feed_emoji":"🧠","tokens_out":7066,"duration_ms":55885,"temperature":0.7,"pith_summary":"Stance detection on social media usually outputs a label with no explanation; this paper treats it as a generative task in which a model must produce the stance label and a rationale together. The authors use GPT-3.5 with a label-conditioned prompt to create rationales, then distill them into small T5 and FlanT5 models through single-task finetuning, single-task chain-of-thought, and multitask learning. Their central finding is that multitask learning works best: a FlanT5-Large trained this way reaches 79.72 macro-F1 on SemEval-2016, beating GPT-3.5's zero-shot 70.15 by 9.57 points, while forcing the small model to generate rationale before prediction hurts accuracy. The paper argues this shows faithful rationales improve rationale distillation and that small models can surpass a much larger zero-shot LLM at this task.","feed_headline":"A 780M FlanT5 beats GPT-3.5's zero-shot stance detection","feed_subtitle":"Multitask rationale distillation lifts a 780M model to 79.72 F1, above GPT-3.5's 70.15, with readable justifications.","key_machinery":"The central object is the label-conditioned rationale elicitation prompt (Prompt 2), which asks GPT-3.5 to begin its answer with \"The comment is classified as [Stance] towards [Topic] because...\", ensuring the generated rationale is tied to the true label. The distillation machinery is multitask learning on T5 and FlanT5, with task prefixes (\"Stance:\" and \"Explain:\") and a weighted loss $L = \\alpha L_{\\text{stance}} + (1-\\alpha) L_{\\text{rationale}}$; the coefficient $\\alpha$ is tuned per model size. This flexible task grouping, rather than enforced sequential generation, is what carries the performance gain over single-task finetuning and single-task chain-of-thought.","core_discovery":"The discovery, in the authors' terms, is that reasoning is a transferable supervisory signal when decoupled from prediction. They reframe GPT-3.5's stance task as explanation generation with a prompt that conditions the rationale on the ground-truth label, so the explanation agrees with the known stance. Distilling those rationales into a small model via multitask learning, with stance and rationale as separate tasks sharing an encoder-decoder, consistently outperforms both ordinary finetuning and enforced chain-of-thought generation. The paper also reports that instruction tuning narrows the benefit: T5 gains more from rationale distillation than FlanT5, and that MTL is especially robust in low-data settings. Its strongest assertion is that faithful rationales are the reason distillation works, making the rationale content, not just the label, the mechanism behind the gain.","pith_inferences":["If the faithfulness claim is right, rationale quality metrics should be added to the evaluation: automated plausibility checks or human annotations would tell whether the 9.57-point gain tracks rationale content or merely the presence of an auxiliary task.","The same label-conditioned elicitation could be applied to other classification tasks where gold rationales are expensive, treating LLM rationales as a scalable training signal.","A control with label-matched but content-free rationales would isolate whether the gain comes from semantic reasoning or from the regularization effect of a second task.","Comparing against few-shot GPT-3.5 or a fine-tuned GPT-3.5 would clarify how much of the reported gap is zero-shot weakness rather than genuinely strong small-model reasoning."],"forward_implications":["A 780M-parameter FlanT5 model trained under MTL beats GPT-3.5's zero-shot performance on SemEval-2016, so small models are a viable deployment choice when large API access is costly.","MTL reaches near-full-data performance at 10-20% of the training set for T5-Base and T5-Large, suggesting the rationale task supplies useful signal in low-resource settings.","Enforcing sequential reasoning before prediction (ST-CoT) hurts small models, so rationale as an auxiliary task rather than a required reasoning step is the better distillation design.","FlanT5 models need less help from rationale distillation than vanilla T5, indicating instruction-tuned models already carry implicit reasoning that reduces the marginal benefit.","Outputs include human-readable justifications for each stance prediction, which is what the paper means by interpretable stance detection."],"supporting_citations":[{"why":"Supplies the SemEval-2016 Task 6 dataset and the $F_{avg}$ evaluation metric used for all experiments.","marker":"[16]"},{"why":"Introduces chain-of-thought prompting, the reasoning mechanism the paper adapts for rationale generation.","marker":"[22]"},{"why":"Shows GPT can generate stance explanations and is cited as the basis for using GPT-3.5's explanations.","marker":"[25]"},{"why":"Provides the T5 architecture and text-to-text multitask learning paradigm on which ST-FT, ST-CoT, and MTL are built.","marker":"[20]"},{"why":"Supplies FlanT5, the instruction-tuned model family whose performance is compared against vanilla T5.","marker":"[5]"},{"why":"Introduces zero-shot-CoT prompting, used as background for eliciting rationales without human exemplars.","marker":"[13]"},{"why":"Motivates using machine-generated rationales by noting that human rationale annotation is costly and domain-specific.","marker":"[4]"},{"why":"Defines the verbalizer mechanism that ST-FT uses to map labels to words.","marker":"[17]"}],"fun_headline_variants":["Small model beats GPT-3.5 stance detection via rationale distillation","FlanT5 multitask rationale distillation beats GPT-3.5 zero-shot","Reasoning as transferable signal: 780M FlanT5 tops GPT-3.5","Multitask rationale distillation: 9.5% boost over GPT-3.5","780M FlanT5 with reasoning outperforms GPT-3.5 zero-shot"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's central mechanism depends on GPT-3.5's label-conditioned rationales being faithful enough to serve as a trustworthy training signal, but no faithfulness metric, human evaluation, or comparison to gold rationales is reported.","fun_headline_variants_meta":{"raw":{"variants":["Small model beats GPT-3.5 stance detection via rationale distillation","FlanT5 multitask rationale distillation beats GPT-3.5 zero-shot","Reasoning as transferable signal: 780M FlanT5 tops GPT-3.5","Multitask rationale distillation: 9.5% boost over GPT-3.5","780M FlanT5 with reasoning outperforms GPT-3.5 zero-shot"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001035,"raw_usage":{"total_tokens":4348,"prompt_tokens":927,"completion_tokens":3421,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":543,"completion_tokens_details":{"reasoning_tokens":3311}},"tokens_in":543,"tokens_out":3421,"duration_ms":22146,"temperature":1.0,"reasoning_tokens":3311,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:59:53.527434+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A control experiment that replaces GPT-3.5 rationales with label-matched but content-free text, such as a template that only names the topic and stance, in the MTL training set: if F_avg does not drop, the claim that faithful rationales drive distillation is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the T5 architecture and text-to-text multitask learning paradigm on which ST-FT, ST-CoT, and MTL are built."}],"review_version":1}