{"id":"e06cacb7-1d5d-42b3-80a6-36815285dbf9","arxiv_id":"2412.04859","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"S2MAD, a multi-agent LLM debate pipeline with stance-separated comments and subjectivity-aware prompts, improves zero-shot rumor detection accuracy on two COVID-19 datasets by up to 12 percentage points.","lead":"This paper proposes a stance-separated multi-agent debate framework (S2MAD) that uses large language models to detect rumors in breaking events by splitting comments into supporting and opposing camps, then having two AI agents debate and a judge decide. It reports accuracy gains of 4-12% over vanilla LLM prompting on two COVID-19 rumor datasets, but the results lack statistical confidence intervals.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2's 'w/o Debate' ablation (0.645/0.737) is numerically identical to Table 1's Qwen1.5+S2MAD row, despite being labeled GPT-3.5; the 12%/7.3% debate-effect claim rests on this inconsistent row.","rationale":"The paper's central claim is that S2MAD, including the multi-agent debate, improves zero-shot rumor detection. The only direct ablation of the debate component is Table 2's 'w/o Debate' row, which reports accuracy drops of 12% and 7.3% relative to the full GPT-3.5-based S2MAD. However, those exact numbers—0.645 and 0.737—are identical to Table 1's Qwen1.5+S2MAD row, even though Table 2 states the experiments use GPT-3.5 turbo. This is not a small rounding coincidence; it is an exact match across two different backbone models. The most plausible explanations are either that the ablation was accidentally run with Qwen1.5, making the comparison invalid, or that the row was copied from Table 1, making the reported ablation unreliable. Either way, the paper's headline contribution—multi-agent debate—lacks credible supporting evidence as currently presented. The reader's identified weakest assumption (unvalidated stance scorer) is also legitimate, and it interacts with this issue: if the stance scorer is noisy, both the debate and the ablations are affected. But the Table 2 inconsistency is more concrete and checkable, and it directly undermines the named contribution. I do not claim the method is wrong; the headline results in Table 1 could still be reproducible. But the current published evidence does not permit a fair assessment of the debate's effect, so the appropriate verdict is UNVERDICTED pending correction or raw-log verification.","tokens_in":12509,"tokens_out":4986,"duration_ms":50245,"concrete_test":"Request the authors' raw logs or rerun the w/o Debate condition with GPT-3.5 turbo, using the same stance-separated top-k comment sets and initial-opinion prompts but no debate or judge, and compare the resulting ACC/Mac-F1 on both datasets to 0.645/0.737. If the true values differ, the debate-effect claim in §4.4 is unsupported; if they match exactly, then Table 1's Qwen row or Table 2's heading is misreported, and the paper should be corrected.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"Section 4.4's central ablation for the debate component is internally inconsistent. Table 2 reports 'w/o Debate' as 0.645 ACC on Twitter-COVID19 and 0.737 on Weibo-COVID19, explicitly 'using GPT-3.5 turbo'. These are exactly the same values as the 'Qwen1.5 + S2MAD' row in Table 1 (0.645, 0.737). Since GPT-3.5+S2MAD Full Model achieves 0.765/0.810, the claimed 12%/7.3% drop when removing debate is computed from a row that matches a different backbone's full S2MAD result. Either the w/o Debate experiment was run with Qwen1.5 rather than GPT-3.5, or the row was copied; in both cases the only direct evidence that 'multi-agent debate' contributes to S2MAD's gains is invalid. The paper's contribution list and title center on debate, so this corrupts the central claim's support. A secondary concern remains the unvalidated glm-3-turbo stance scorer, but the Table 2 inconsistency is more immediate.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes S2MAD, a zero-shot pipeline for rumor detection on breaking-event social media. It separates comments into supporting and opposing stance sets using an LLM scorer (glm-3-turbo), classifies each claim as subjective or non-subjective to select a prompt strategy, generates initial opinions for two debater agents, runs up to M debate rounds, and invokes a judge agent if no consensus is reached. The authors evaluate on Twitter-COVID19 and Weibo-COVID19, reporting that S2MAD improves Qwen1.5-7b-chat and GPT-3.5 turbo accuracy and outperforms trained baselines, with additional early-detection experiments.","tokens_in":12748,"tokens_out":4367,"duration_ms":42329,"significance":"If the empirical claims held, S2MAD would be a useful contribution to zero-shot rumor detection in breaking-event scenarios, where labeled data is unavailable. The component design is intuitive and the two-dataset evaluation against several baselines is a reasonable starting point. However, the current evidence does not yet support the central debate-specific claim: the key debate ablation row is internally inconsistent, and the reported gains are single-run point estimates without variance or statistical testing. The paper does not provide code, prompts in a reusable form, or scorer validation, which limits reproducibility.","major_comments":[{"comment":"The 'w/o Debate' row in Table 2 reports accuracy values of 0.645 on Twitter-COVID19 and 0.737 on Weibo-COVID19, which are exactly the same as the 'Qwen1.5 + S2MAD' row in Table 1, even though Table 2 is explicitly described as using GPT-3.5 turbo. If this is a copy error, the stated 12% and 7.3% drops attributed to removing the debate are not supported; if the experiment was actually run with Qwen1.5, it is not a valid ablation for the GPT-3.5-based full model. The authors must supply the correct GPT-3.5 no-debate numbers, with variance, before the central debate contribution can be assessed.","section":"Table 2 vs. Table 1"},{"comment":"The maximum number of debate rounds M is set to two after inspecting the accuracy curve on the test sets in Figure 2. This is a post-hoc selection on evaluation data, and because no validation split or confidence intervals are reported, the subsequent results may reflect selection artifacts rather than genuine improvement. The authors should fix M on a held-out validation set or report results across a range of M with error bars and a significance test.","section":"Section 4.4, Figure 2"},{"comment":"The stance separation stage depends entirely on glm-3-turbo as an LLM scorer, but the paper provides no validation of this scorer, no agreement statistics against human annotations, and no analysis of how scoring errors propagate to the selected comment sets P and N. The behavior of the 0.0 'not conforming to common sense' category is also not evaluated. Since both debaters receive only this filtered evidence, the reliability of the entire pipeline is not established without such an analysis.","section":"Section 3.2, Section 4.2"},{"comment":"All results are single-run point estimates without standard deviations, confidence intervals, or significance tests. The headline improvements (8.0% and 12.2% on Twitter-COVID19; 3.9% and 4.8% on Weibo-COVID19) are differences between point estimates, and the lack of variance reporting is especially problematic because LLM decoding is stochastic even at temperature 0.2. At least a small number of repeated runs with reported variance is needed to support the claimed improvements.","section":"Section 4.3, Table 1"}],"minor_comments":[{"comment":"The text uses '/o Debate' while Table 2 uses 'w/o Debate'; please correct the typo.","section":"Section 4.4"},{"comment":"The value of k, the number of comments selected per stance set, is never reported, although it is a free parameter in Eq. (2) and the 'w/o Stance' ablation uses a different sampling rule. Please report k and, ideally, include a sensitivity analysis over k.","section":"Section 4.2, Eq. (2)"},{"comment":"The caption for 'w/o Non-Sub/Sub' is ambiguous: it is not clear whether 'w/o Non-Sub' means removing the non-subjective prompt or applying the non-subjective prompt to the entire dataset. Please clarify each ablation row explicitly.","section":"Table 2 caption"},{"comment":"Figure 2 would benefit from error bars, a clear definition of 'round 0', and a statement of whether the reported values are averaged over multiple runs.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The Table 2 inconsistency is the most serious issue and must be resolved before the paper can be considered further. If the 'w/o Debate' row is indeed a copy-paste error, the authors need to rerun the experiment with GPT-3.5 turbo and report the correct number with error bars. The selection of M on the test set is also a clear evaluation leak. The methodological idea is potentially salvageable, but the current empirical support for the debate component is not trustworthy."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper has a sensible, clearly described pipeline for zero-shot rumor detection in breaking events — separate comments by stance, prompt two debaters with stance-specific evidence, add subjectivity-conditioned initial prompts, run a two-agent debate, and let a judge settle disagreements. That combination is new for this task. But the ablation that is supposed to isolate the debate component is internally inconsistent, and the empirical support otherwise is single-run point estimates with no variance. As written, the claim that multi-agent debate drives the gains is not supported.\n\nWhat the paper does well: the method is described in enough detail to reimplement (modulo the unreported top-k), the zero-shot framing fits the breaking-event setup, and the comparison against trained graph baselines like T3RD is appropriate. The early-detection curves are a genuine plus. Stance separation is a natural fit for rumor threads, and the subjective/objective split is a cheap, reasonable way to condition prompts.\n\nWhere it falls short, in order. First, Table 2's 'w/o Debate' row reports 0.645 accuracy on Twitter-COVID19 and 0.737 on Weibo-COVID19, explicitly under GPT-3.5 turbo. These are exactly the same numbers as the 'Qwen1.5 + S2MAD' row in Table 1. Since the full GPT-3.5 + S2MAD model gets 0.765/0.810, the claimed 12% and 7.3% drops when removing debate are computed from a row that matches a different backbone's full S2MAD result. Either the ablation ran with the wrong backbone or the row was copied; in both cases, the paper's only direct evidence for the debate component's contribution is invalid. This is a load-bearing flaw for the central contribution. Second, the glm-3-turbo stance scorer is unvalidated; if it mislabels comments, both debaters start from distorted evidence, and there is no error-propagation analysis. Third, M is selected by inspecting test-set accuracy in Figure 2, a form of fitting to the evaluation data. Fourth, no error bars or significance tests, so we cannot rule out stochastic noise. The value of k is not reported, and no code is released.\n\nI would send this to peer review, but with a clear ask: correct the w/o Debate numbers, add variance, report k and scorer validation, and re-examine the M selection. The idea is worth building on; the current evidence is not.","headline":"S2MAD is a promising zero-shot pipeline for rumor detection, but the debate ablation is invalidated by a table inconsistency and the empirical support is single-run point estimates.","tokens_in":13285,"tokens_out":4201,"would_cite":false,"duration_ms":37620,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Generic LLMs become competitive breaking-event rumor detectors when their reasoning is forced through a stance-separated, pro-con debate with a judge, improving accuracy by up to 12.2 percentage points on the two COVID-19 datasets.","keywords":["breaking event rumor detection","zero-shot rumor detection","multi-agent debate","stance separation","large language models","social media misinformation","COVID-19 rumor datasets"],"falsifier":"Take a sample of comments from the two datasets, have human annotators mark each as supporting, opposing, or off-topic, and compare their labels with the LLM stance scorer: if agreement is low, or if rerunning S2MAD with the scorer's labels replaced by human labels changes accuracy, the stance-separation stage is the fragile link. A second check is to replace stance separation with random comment assignment and measure whether the reported ablation drop of roughly 12 points on Twitter-COVID19 reproduces.","tokens_in":12296,"feed_emoji":"🗣️","tokens_out":10548,"duration_ms":101788,"temperature":0.7,"pith_summary":"The paper claims that generic large language models can detect rumors during breaking events if their reasoning is structured as a debate between two agents fed opposite sides of the comment thread. The proposed S2MAD pipeline separates comments into supporting and opposing sets, tailors each agent's initial prompt to whether the claim is subjective or objective, lets the two agents argue for a fixed number of rounds, and sends unresolved cases to a judge. On the Twitter-COVID19 and Weibo-COVID19 datasets, this training-free structure raises zero-shot accuracy over a single LLM by 8.0 and 3.9 percentage points on one backbone and by 12.2 and 4.8 on another, and it beats trained propagation-graph baselines tested out of domain. The practical upshot is that a sudden event, which by definition has no ready-made training labels, can still be handled by reasoning structure alone.","feed_headline":"Two AI agents that argue lift breaking-news rumor detection by 12%","feed_subtitle":"Zero-shot pro-con debate raises COVID-19 rumor accuracy by 12.2 percentage points and beats trained baselines.","key_machinery":"The object that carries the argument is the stance-separated debate loop. A commentary scorer labels each comment with a score in [-1,1]: positive for support of the claim, negative for opposition, and zero for comments that violate common sense; the top-k positives and top-k negatives become the two debaters' private evidence. The claim itself is separately classified as subjective or objective, which selects different initial-opinion prompts, with reasonability and trust-impact checks for subjective posts and support/rebuttal evaluation plus common-sense verification for objective posts. Two debater agents update their verdicts by critically reading each other's previous answer, and a judge agent resolves the cases where the fixed number of rounds ends in disagreement.","core_discovery":"On the paper's own terms, the central discovery is that the veracity of an unfolding claim can be decided more accurately by forcing two LLM agents to argue from deliberately separated evidence sets than by asking one model to reason over the whole thread. Stance separation converts an undifferentiated comment list into a support set and an oppose set; the support-agent and oppose-agent then generate initial judgments under prompts matched to the claim's subjectivity, exchange rebuttals for a fixed number of rounds, and only invoke a judge when they cannot agree. The reported result is that this structure turns weak zero-shot backbones into systems that outperform not only their vanilla prompts but also trained propagation-graph baselines applied out of domain: on Twitter-COVID19, S2MAD with GPT-3.5 turbo reaches 76.5 percent accuracy versus 73.5 for the strongest prior baseline, and on Weibo-COVID19 it reaches 81.0 percent versus 79.7. Ablations attribute the gain to stance separation, to subjectivity-matched prompts, and to the debate itself, with the debate component alone accounting for a drop of roughly 12 percentage points when removed from the Twitter-COVID19 setting.","pith_inferences":["Beyond the paper, the same pro-and-con evidence split could transfer to other low-resource verification tasks, such as fast-breaking health claims, disaster misinformation, or election rumors, whenever an unlabeled comment thread exists.","Beyond the paper, the judge's written synthesis doubles as a natural explanation artifact; whether that explanation is faithful to the evidence that drove the verdict is a testable question the paper does not explore.","Beyond the paper, the sensitivity of the reported gains to the scoring model, the number of selected comments k, and the number of debate rounds is still open, since the paper fixes these choices.","Beyond the paper, the small accuracy change after two debate rounds suggests that an adaptive stopping rule based on debater agreement could cut inference cost; this is a speculation, not a result of the paper."],"forward_implications":["A sudden event with no labeled data can still be handled at inference time: S2MAD outperforms propagation-graph models trained on public datasets and applied directly to the COVID-19 test sets.","The wrapper transfers across backbones: both tested LLMs improve when the stance-separated debate is added, so the gain is attributed to the structure rather than to one model.","Every component is load-bearing in the experiments: removing stance separation, dropping either subjectivity-matched prompt, or removing the debate reduces accuracy in the ablation study.","Early detection is not sacrificed: S2MAD stays competitive with baselines at early checkpoints with few comments, not only with full threads.","Unresolved debates are the reason for the judge: when the fixed two rounds do not produce consensus, the judge's synthesis supplies the final verdict."],"supporting_citations":[{"why":"Provides the competing-wisdom idea that motivates splitting comments into defending and attacking stances.","marker":"Wang et al. 2024a"},{"why":"Supplies the Twitter-COVID19 and Weibo-COVID19 datasets and the ACLR-BiGCN baseline.","marker":"Lin et al. 2022"},{"why":"Supplies the vanilla-prompt LLM baseline and the prompt-guidance result used to design initial-opinion prompts.","marker":"Liu et al. 2024"},{"why":"Defines the BiGCN propagation-graph baseline that S2MAD compares against in zero-shot transfer.","marker":"Bian et al. 2020"},{"why":"Defines the T3RD test-time-training baseline, the strongest trained system S2MAD must beat.","marker":"Zhang et al. 2024a"},{"why":"Provides the multi-agent debate formulation with divergent viewpoints that S2MAD adapts to rumor detection.","marker":"Liang et al. 2023"},{"why":"Shows preset stances with skeptical counter-debate, which motivates giving each debater a fixed stance.","marker":"Fang et al. 2024"}],"fun_headline_variants":["AI debate boosts rumor detection on breaking news by 12%","Stance-split debate strengthens LLMs for breaking-news rumor detection","Pro-con AI debate outperforms trained rumor detectors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the LLM that scores each comment as supporting, opposing, or nonsensical is reliable enough that the two debaters receive genuinely contrasting evidence; if that scorer is noisy or biased, the debate and judge inherit the distortion.","fun_headline_variants_meta":{"raw":{"variants":["AI debate boosts rumor detection on breaking news by 12%","Stance-split debate strengthens LLMs for breaking-news rumor detection","Pro-con AI debate outperforms trained rumor detectors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000478,"raw_usage":{"total_tokens":2396,"prompt_tokens":999,"completion_tokens":1397,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":1342}},"tokens_in":615,"tokens_out":1397,"duration_ms":11918,"temperature":1.0,"reasoning_tokens":1342,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:11:49.122391+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a sample of comments from the two datasets, have human annotators mark each as supporting, opposing, or off-topic, and compare their labels with the LLM stance scorer: if agreement is low, or if rerunning S2MAD with the scorer's labels replaced by human labels changes accuracy, the stance-separation stage is the fragile link. A second check is to replace stance separation with random comment assignment and measure whether the reported ablation drop of roughly 12 points on Twitter-COVID19 reproduces.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Twitter-COVID19 and Weibo-COVID19 datasets and the ACLR-BiGCN baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the BiGCN propagation-graph baseline that S2MAD compares against in zero-shot transfer."}],"review_version":1}