{"id":"7ad1a2b0-12f3-42f4-9445-d1db0187703d","arxiv_id":"2506.21053","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"The paper presents a large new English conversational stance detection dataset and a model that fuses LLM-generated relation and act knowledge, reporting state-of-the-art F1.","lead":"This paper introduces MT2-CSD, a 24,457-instance Reddit dataset for multi-target, multi-turn conversational stance detection, and an LLM-enhanced model called LLM-CRAN that uses GPT-4 to extract discourse relations and conversation acts. The authors report that LLM-CRAN beats baselines on this benchmark, but the split procedure and dataset tables contain ambiguities that need scrutiny.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Whether the 65/15/20 split is by conversation thread is the most load-bearing unknown; if it is by comment, LLM-CRAN sees near-duplicate context from the same threads in training and test, which can manufacture the Table IX margin.","rationale":"I agree with the reader's weakest assumption. The strongest claim—LLM-CRAN's 54.19 vs GLAN's 50.47—rests on all models seeing the same, non-overlapping conversations. Section III.D only gives the ratio; neither split unit nor seed is stated. A comment-level split is the classic failure mode for conversational datasets: because context is part of the input, test comments inherit thread history from training, and a model with explicit relational layers is the most likely beneficiary. I checked for other candidates: LLM-generated act labels are correlated with stance, but they are a legitimate discourse signal and the paper's ablation shows the expected direction; the Table V/VII inconsistency is serious for the dataset-construction claim but secondary for the model comparison; missing error bars and significance details are reporting gaps, not structural threats. If the released data show thread-level splitting, the central model claim is much safer; if not, the correct verdict is at best conditional pending rerun. Hence I leave the reader's CONDITIONAL unchanged.","tokens_in":19645,"tokens_out":6093,"duration_ms":75060,"concrete_test":"Inspect the released dataset or request the split metadata: group all instances by thread ID (the root post ID), then check whether any thread appears in both train and test. If overlap is nonzero, rerun LLM-CRAN and the strongest baselines (GLAN, GPT-4, KPT) on a thread-level 65/15/20 split with the same hyperparameters and report the Table IX average. If LLM-CRAN no longer beats GLAN by at least its stated margin with p<0.05, the headline result is an artifact of split leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is experimental isolation. Table IX claims LLM-CRAN beats GLAN by 3.72 F1 and GPT-4 by 6.01, but the comparison is only valid if no Reddit thread crosses the train/test boundary. Section III.D states only that the dataset was divided into training, validation, and test sets for all targets in a 65/15/20 ratio, without specifying whether the unit is a comment or a thread. Because every instance is defined with the full conversation as context, a comment-level random split places test comments inside threads whose parent or neighbor comments are in training; LLM-CRAN's GCN/RGCN layers and attention then have access to near-duplicate contextual structure at inference, exactly the signal the method is designed to exploit. This would inflate its advantage over weaker context models and could erase the reported gain. The same ambiguity affects the depth analysis in Table XI and the ablation study in Fig. 3. Separately, the dataset bookkeeping is not internally consistent: Table V lists 86,948 comments while Table VII sums to 24,457 annotated instances, and the stated minimum of 200 comments per post is hard to reconcile with the per-target comment/post ratios; this does not by itself refute the model comparison but strengthens the case that data processing and split construction need a precise public statement.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MT2-CSD, a Reddit-sourced dataset for multi-target, multi-turn conversational stance detection comprising 24,457 annotated instances across six targets (Bitcoin, Tesla, SpaceX, Biden, Trump, and a post-as-target setting called Post-T). The authors claim this is the largest human-labeled English conversational stance dataset and that it has greater conversational depth than existing CSD datasets. They also propose LLM-CRAN, which uses zero-shot LLM prompting to extract logical relations and conversation acts, and then fuses these with local CNN, GCN, and RGCN representations through a multi-hop attention mechanism. The main empirical claim is that LLM-CRAN outperforms strong baselines, with an average in-target F1 of 54.19 versus 50.47 for GLAN and 48.18 for GPT-4. The paper also reports cross-target experiments, depth analysis, ablations, and a case study.","tokens_in":19887,"tokens_out":6685,"duration_ms":72026,"significance":"If the dataset and experimental setup are sound, MT2-CSD would be a useful benchmark for conversational stance detection, and the LLM-knowledge-fusion architecture is a reasonable and interesting approach. The paper has several strengths: it evaluates a wide range of baselines, includes cross-target and depth analyses, reports annotation agreement, and the LLM relation extraction is zero-shot and not fitted to the stance labels, so there is no obvious definitional circularity. The ablations and the case study also provide useful qualitative evidence. However, the central empirical comparison is currently undermined by an unspecified data-split unit and by internal inconsistencies in the reported data counts, both of which are fixable but must be resolved before the results can be trusted.","major_comments":[{"comment":"The split unit is not specified: the text says only that the dataset was divided into training, validation, and test sets in a 65/15/20 ratio. Because every instance is defined with the full conversation as context (Section IV.A, Eqs. (1)-(2)), a comment-level random split would place test comments inside threads whose parent, child, or sibling comments appear in training. The GCN/RGCN layers and the multi-hop attention would then have access to near-duplicate conversational context at inference, and the zero-shot LLM in KAM would also see training-set messages from the same thread when labeling test comments. This could inflate the reported gains in Table IX and would also affect the depth analysis in Table XI and the ablation study in Fig. 3. Please state explicitly whether the split unit is a comment or a full Reddit thread, release thread IDs or split files, and if the current split is comment-level, re-run all experiments with a thread-level split.","section":"Section III.D"},{"comment":"The data bookkeeping is internally inconsistent. Table V lists 704 posts and 86,948 comments, but Section III.B requires a minimum of 200 comments per post, which would imply at least 140,800 comments. Moreover, Table VII reports only 24,457 annotated instances, and the relationship among raw collected comments, preprocessed comments, and annotated instances is never stated. Please clarify these counts per target, correct the filtering description, and make the data-preprocessing pipeline fully transparent.","section":"Section III.B and Tables V/VII"},{"comment":"The claim that LLM-CRAN 'significantly outperforms' baselines is supported by significance tests only against Branch-BERT, JoinCL, and TTS, not against the strongest competitors GLAN and GPT-4. Since the central empirical claim is the 3.72-point gain over GLAN and the 6.01-point gain over GPT-4, please report significance tests and confidence intervals for all relevant comparisons, or soften the significance claim accordingly.","section":"Section VI.A and Table IX"},{"comment":"The conversation-act taxonomy includes Agreement and Disagreement, which are semantically close to stance labels, and Fig. 4 shows a strong concentration of Favor and Against stances in these two acts. Because the LLM extracts these acts zero-shot and only LLM-CRAN receives them, the comparison may partly reflect access to a near-stance signal rather than general conversational understanding. Please discuss this explicitly, and ideally report an ablation in which Agreement and Disagreement are removed from the CA taxonomy, or in which the LLM is prompted to identify the act without being able to infer the target stance from the surrounding thread.","section":"Section IV.C, Table I, and Fig. 4"}],"minor_comments":[{"comment":"There are several typos: 'LLM-GRAN' should be 'LLM-CRAN' in Section V.B, 'SapceX' should be 'SpaceX' in Table IV, and 'Sepcifically' should be 'Specifically' in Section VI.A.","section":"Section V.B and Table IV"},{"comment":"The sentence 'Following [4], we used the Favor and Against classes to compute the kappa values' is ambiguous; Cohen's kappa is computed between annotators, not between classes. Please describe exactly which annotations entered the kappa calculation and how disagreement was resolved.","section":"Section III.C and Table VI"},{"comment":"The prompt template contains the placeholder '[Defined Logical Relations (or Conversation Acts)]' but the actual categories from Table I are not inserted in the template shown; please include the full prompt exactly as used for reproducibility.","section":"Section IV.C"},{"comment":"The repository link is given as https://github.com/nfq729/MT-CSD while the dataset is called MT2-CSD; please ensure the released files and README match the paper's target names, the Post-T schema, and the claimed annotation counts.","section":"Section III.A and GitHub link"},{"comment":"The tables report averages over three runs but no standard deviations or per-run values; adding these would help assess whether the reported gaps are stable.","section":"Tables IX and X"},{"comment":"For Post-T the depth bins are 2, 3-4, and 5-6, while for other targets they are 1-2, 3-5, and 6-8; please state why depth 1 is excluded for Post-T and define the depth convention consistently.","section":"Table XI"}],"recommendation":"major_revision","confidential_remarks":"The split-unit ambiguity is the decisive issue: if the actual split is comment-level, the headline comparison in Table IX is not trustworthy, and the same problem affects the ablation and depth analyses. The dataset count inconsistency in Section III.B versus Tables V and VII needs a clear correction. Both issues are fixable in a revision, and the methodological idea is worth pursuing, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my read of MT2-CSD. The dataset is the real contribution: 24,457 human-labeled instances across six targets, with deeper threads than SRQ, CANT-CSD, or CTSDT, plus the new Post-T target. That is a genuinely useful resource for the conversational stance detection subfield. The model side, LLM-CRAN, is an incremental extension of the authors' own GLAN with LLM-extracted logical relations and conversation acts; the gains over GLAN (3.72 F1 average) are plausible but not yet proven.\n\nThe load-bearing unknown is the train/test split. Section III.D says only that the data was divided 65/15/20 for all targets, with no statement about whether the unit is a thread or a comment. If it is comment-level, then contextual models like LLM-CRAN likely see near-duplicate context from the same thread in both training and test, which can inflate the reported margin. The depth analysis and ablation are subject to the same ambiguity. This has to be fixed before the headline numbers can be trusted.\n\nThere is also bookkeeping that does not add up. Table V lists 86,948 comments total, but Table VII sums to 24,457 annotated instances; the relation between collected and annotated comments is not explained. And the stated minimum of 200 comments per post does not square with Post-T's 42,536 comments across 374 posts, which is roughly 114 per post. These are not fatal to the dataset, but they suggest the data processing pipeline needs a precise, public description.\n\nOne more concern: the conversation-act taxonomy includes Agreement and Disagreement, which sit close to stance. The ablation shows removing the CA layer costs 8.84 F1 on average, the largest drop. Since the LLM extraction is zero-shot and not fitted to the labels, it is not definitional circularity, but it does mean the model may be using stance-like signals from the LLM. The heat map in Fig. 4 confirms the correlation. The authors should either validate the LLM-generated labels or release them so reviewers can check how much stance information they carry.\n\nThe paper is honestly written, the claims are proportionate to what they show conditional on the split issue, and the dataset is a real resource. I would send it to peer review, with explicit requests to clarify the split unit, reconcile the data bookkeeping, and report significance test details. For a reader in stance detection, the dataset is worth citing once those details are settled.","headline":"Useful new dataset, conditional model claims—the unspecified train/test split unit is the thing to check first.","tokens_in":20479,"tokens_out":2295,"would_cite":true,"duration_ms":24639,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MT2-CSD is the largest human-labeled conversational stance dataset, and the proposed LLM-CRAN model outperforms all strong baselines on it.","keywords":["conversational stance detection","multi-turn","Reddit","dataset","LLM","logical relations","conversation acts","attention network"],"falsifier":"Download the released MT2-CSD splits and check whether any Reddit thread contributes comments to both training and test sets; if leakage exists, re-run LLM-CRAN and GLAN with thread-level splits and see whether the 3.72-point F1 gap narrows or disappears.","tokens_in":19421,"feed_emoji":"🗣️","tokens_out":5252,"duration_ms":46719,"temperature":0.7,"pith_summary":"This paper tries to establish two things: that MT2-CSD is the largest English conversational stance dataset labeled by humans, and that a model which injects discourse knowledge extracted by large language models can substantially beat existing methods on it. The dataset contains 24,457 annotated Reddit comments across six targets, with 67.47% of instances deeper than three turns, a regime where prior conversational stance datasets are thin. The proposed model, LLM-CRAN, uses zero-shot prompts to label each comment with a logical relation and a conversation act, then fuses these signals with local, contextual, and graph-based representations through a multi-hop attention network. On the dataset's average F1, LLM-CRAN reaches 54.19, compared with 50.47 for the strongest baseline and 48.18 for GPT-4 prompted in context.","feed_headline":"LLM-extracted discourse cues lift stance F1 to 54.19","feed_subtitle":"A 24,457-comment human-labeled dataset with deep threads backs a model that beats GPT-4 and prior baselines.","key_machinery":"The load-bearing mechanism is LLM-CRAN's two-stage pipeline: a Knowledge Acquisition Module (KAM) uses zero-shot prompting with GPT-4 to label every comment's logical relation to its predecessor (one of contrastive, succession, causal, summary) and its conversation act (one of eight categories), and a Multi-Knowledge Integrated Attention Network (MKIAN) encodes these into a graph; four parallel layers (local CNN with masking, GCN for reply structure, and two RGCNs for logical and act relations) produce sentence vectors that a p-hop attention block repeatedly refines and fuses before the stance classifier reads the final vector.","core_discovery":"The paper's central claim is that conversational stance detection benefits from explicit multi-semantic knowledge — specifically logical discourse relations (contrastive, succession, causal, summary) and conversation acts (agreement, disagreement, suggestion, and others) — when that knowledge is extracted automatically by an LLM and fused into a purpose-built attention network. On the new MT2-CSD benchmark, this design yields an average F1 of 54.19 across six targets, beating the previous GLAN model (50.47), prompt-tuned BERT variants (around 45), and in-context LLMs including GPT-4 (48.18). The paper also claims that MT2-CSD, with 24,457 instances and the deepest reply chains among existing English CSD datasets, provides the first benchmark where such multi-turn context can be studied at scale.","pith_inferences":["If the dataset is released with thread-level metadata, an immediate test is to re-run LLM-CRAN against GLAN under splits that isolate whole threads; if the 3.72-point margin persists, the discourse-knowledge claim is robust to leakage concerns, and if it shrinks, part of the reported gain is memorization of adjacent comments.","The same LLM-as-knowledge-annotator recipe could be applied to other conversational tasks — argument mining, rumor detection, or dialogue act prediction — where logical-relation and act taxonomies are already defined, making MKIAN a reusable fusion block.","The paper reports LLM bias on political targets (GPT-3.5 at 27.47 on Trump, DEEM at 32.46 on Biden); this suggests that using LLM-generated knowledge for political sub-discussions may inject systematic stance bias, a risk worth quantifying by comparing KAM outputs against human annotation on the Trump and Biden subsets.","Because 67.47% of instances exceed depth 3, the dataset could serve as a stress test for long-context transformers and retrieval-based conversational models, not just CSD-specific architectures."],"forward_implications":["If the reported numbers hold under thread-level splitting, LLM-CRAN establishes that LLM-extracted discourse knowledge transfers across targets, since it also leads in all four in-domain cross-target experiments.","The dataset's depth distribution — 67.47% of comments beyond depth 3 — makes it possible to test how model performance degrades (or holds) as conversations lengthen; LLM-CRAN is the only method that maintains or improves F1 on the deepest bins in Table XI.","Removing the conversation-act layer costs an average 8.84 F1 and removing the logical-relation layer costs 6.10, so the paper's claim that explicit discourse knowledge is the main driver of the gain is directly testable by the provided ablations.","Since the model beats prompt-based GPT-4 by about 6 F1 while using GPT-4 only as a knowledge annotator, the architecture shows a division of labor: LLM reasoning is used offline to produce training signals, and a small trainable network makes the final decision."],"supporting_citations":[{"why":"the single-turn baseline dataset whose limitations MT2-CSD is designed to overcome.","marker":"[8]"},{"why":"the only prior multi-turn CSD dataset; supplies the depth comparison (6.3% beyond depth 3) and a baseline model.","marker":"[9]"},{"why":"the automated-annotation dataset contrasted with MT2-CSD's human labeling; also a comparison scale point.","marker":"[10]"},{"why":"the authors' prior model, extended here into LLM-CRAN, and the strongest baseline (50.47 F1) that LLM-CRAN beats.","marker":"[14]"},{"why":"source of the four logical-relation categories used in KAM prompts.","marker":"[43]"},{"why":"source of the conversation-act taxonomy (seven high-frequency acts plus 'other').","marker":"[44]"},{"why":"basis for using ChatGPT/GPT-4 as the LLM knowledge extractor and as an in-context baseline.","marker":"[36]"},{"why":"defines the Favg evaluation protocol and the kappa-computation convention adopted for annotation quality.","marker":"[4]"}],"fun_headline_variants":["LLM discourse cues boost stance F1 to 54.19 on new 24K-comment dataset","MT2-CSD: LLM-extracted semantic cues lift stance detection to 54.19","New largest CSD dataset plus LLM knowledge fusion hits 54.19 F1","Conversational stance detection: LLM fusion outperforms GPT-4 on new benchmark"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported performance gains assume the data split is made at the level of whole conversation threads, so that no thread appears in both training and test sets — but the paper only says the split ratio (65/15/20) and not the unit it splits on.","fun_headline_variants_meta":{"raw":{"variants":["LLM discourse cues boost stance F1 to 54.19 on new 24K-comment dataset","MT2-CSD: LLM-extracted semantic cues lift stance detection to 54.19","New largest CSD dataset plus LLM knowledge fusion hits 54.19 F1","Conversational stance detection: LLM fusion outperforms GPT-4 on new benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000947,"raw_usage":{"total_tokens":4040,"prompt_tokens":937,"completion_tokens":3103,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":553,"completion_tokens_details":{"reasoning_tokens":3007}},"tokens_in":553,"tokens_out":3103,"duration_ms":24180,"temperature":1.0,"reasoning_tokens":3007,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:34:49.960832+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Download the released MT2-CSD splits and check whether any Reddit thread contributes comments to both training and test sets; if leakage exists, re-run LLM-CRAN and GLAN with thread-level splits and see whether the 3.72-point F1 gap narrows or disappears.","supporting_citations":[{"cited_title":"Stance in Replies and Quotes (SRQ): A New Dataset For Learning Stance in Twitter Conversations","cited_arxiv_id":"2006.00691","evidence_quote":"the single-turn baseline dataset whose limitations MT2-CSD is designed to overcome."},{"cited_title":"Improved target-specific stance detection on social media platforms by delving into conversation threads,","cited_arxiv_id":null,"evidence_quote":"the only prior multi-turn CSD dataset; supplies the depth comparison (6.3% beyond depth 3) and a baseline model."},{"cited_title":"Contextual target-specific stance detection on twitter: Dataset and method,","cited_arxiv_id":null,"evidence_quote":"the automated-annotation dataset contrasted with MT2-CSD's human labeling; also a comparison scale point."},{"cited_title":"A challenge dataset and effective models for conversational stance detection,","cited_arxiv_id":null,"evidence_quote":"the authors' prior model, extended here into LLM-CRAN, and the strongest baseline (50.47 F1) that LLM-CRAN beats."},{"cited_title":"A survey of discourse parsing,","cited_arxiv_id":null,"evidence_quote":"source of the conversation-act taxonomy (seven high-frequency acts plus 'other')."},{"cited_title":"P-stance: A large dataset for stance detection in political domain,","cited_arxiv_id":null,"evidence_quote":"defines the Favg evaluation protocol and the kappa-computation convention adopted for annotation quality."}],"review_version":1}