{"id":"19d2e073-290c-4d6c-a251-a9db770c32e8","arxiv_id":"2505.08532","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"TED uses structured pro/con debates between LLM agents plus a graph-based analysis model to detect fake news more accurately than prior methods on two benchmark datasets.","lead":"This paper presents TruEDebate (TED), a fake-news detector in which teams of large language model agents argue for and against a news item and a graph-based judge model makes the final call. A generalist might read it because it tests whether structured adversarial debate, not just a single LLM prompt, improves both accuracy and the transparency of automated fact-checking.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline gain over ARG is not controlled for LLM backbone: ARG was run with a pre-GPT-4o-mini LLM while TED uses GPT-4o-mini, so 'outperforms all' is not yet established.","rationale":"The reader's CONDITIONAL verdict is appropriate, but for a different reason than the stated weakest assumption. The paper releases code and provides ablations and backbone variations, which are strengths. However, the strongest_claim is an empirical comparison, and the comparison against the strongest baseline is not controlled for the LLM backbone. Since the numeric gap over ARG is small and TED with a different backbone ties ARG on ARG-EN, the missing control is the single most load-bearing issue. The proposed test is inexpensive and directly settles whether the headline gain exists under a fair comparison. If the test shows ARG+GPT-4o-mini closes the gap, the paper's central claim would need to be softened; if not, the current conditional acceptance stands.","tokens_in":16830,"tokens_out":10527,"duration_ms":105952,"concrete_test":"Re-run the ARG baseline from Hu et al. [15] on ARG-EN and ARG-CN with GPT-4o-mini as the LLM, keeping all other ARG components and hyperparameters fixed. If ARG+GPT-4o-mini reaches or exceeds TED's macF1/accuracy, the 'outperforms all' claim is not attributable to the debate framework; if ARG+GPT-4o-mini stays below TED by roughly the original gap, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1.3 specifies that TED's DebateFlow and Synthesis Agents use GPT-4o-mini. The closest LLM+SLM competitor, ARG, is listed in Table 1 with macF1 0.790 on ARG-EN and 0.784 on ARG-CN, but Section 4.1.2 says all baseline numbers except GPT-4o-mini and ChatEval are taken from Hu et al. [15]. The only LLM-only baseline in that source is GPT-3.5-turbo, and Hu et al. predates GPT-4o-mini, so ARG's rationales were almost certainly generated by an older, weaker backbone. TED's advantage over ARG is small (+0.013 and +0.011 macF1). Consistent with a backbone effect, Table 4 shows TED with Llama 3.1-70B reaches macF1 0.790 on ARG-EN, exactly matching the reported ARG score. The paper never re-runs ARG or ARG-D with GPT-4o-mini, nor any condition that holds the LLM backbone fixed while removing only the debate structure. Therefore the central claim 'TED outperforms all other methods' conflates the contribution of the debate with the contribution of a newer, stronger LLM.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TruEDebate (TED), a multi-agent LLM framework for fake news detection. In TED, DebateFlow Agents simulate a Lincoln-Douglas-style debate between Proponent and Opponent teams; a Synthesis Agent summarizes the debate log; and an Analysis Agent, built from a role-aware encoder, a debate graph processed by GATs, and an interactive attention mechanism over the debate and news embeddings, produces the final veracity label. Experiments on ARG-EN and ARG-CN compare TED with LLM-only, SLM-only, and LLM+SLM baselines, and include ablations and backbone-replacement experiments.","tokens_in":1478,"tokens_out":1378,"duration_ms":64113,"significance":"If the headline comparison were made fair, TED would be a useful contribution to interpretable fake news detection: it combines structured multi-agent debate with a trainable graph-based analysis component, attacks an important problem, and ships code. The paper's concrete strengths are the two-dataset evaluation, the ablation evidence that each component contributes, the multiple-backbone robustness check, and the interpretable case study. The classifier is trained and evaluated on held-out splits, so there is no circularity in the technical sense. However, the central claim that TED 'outperforms all other methods' is not yet established because the strongest LLM+SLM competitor, ARG, is run with an older and likely weaker LLM backbone than TED. The significance of the contribution therefore remains conditional on a controlled comparison.","major_comments":[{"comment":"The comparison against ARG/ARG-D is confounded by the LLM backbone. Table 1 states that all baselines except GPT-4o-mini and ChatEval are taken from Hu et al. [15], whose ARG system predates GPT-4o-mini, while TED uses GPT-4o-mini for its DebateFlow and Synthesis Agents (Section 4.1.3). The macF1 margin over ARG is only +0.013 on ARG-EN and +0.011 on ARG-CN, and Table 4 shows that TED with Llama 3.1-70B reaches exactly the ARG-reported macF1 of 0.790 on ARG-EN. The paper never re-runs ARG or ARG-D with GPT-4o-mini, nor any condition that holds the backbone fixed while removing only the debate structure. Consequently, the Section 4.2 conclusion that TED 'outperforms all other methods' conflates the debate contribution with the contribution of a stronger LLM. Please add a controlled comparison, e.g., ARG with GPT-4o-mini, or TED with the same backbone used for ARG in Hu et al.","section":"Section 4.1.2, 4.1.3, Table 1, Table 4"},{"comment":"Statistical significance is claimed for bold and underlined entries (t-test, p<0.01), but no run count, random seed, or standard deviation is reported anywhere in the paper. A single point estimate per condition cannot support pairwise t-test claims. Please report means and standard deviations over multiple independent runs (at least 3-5 seeds) and state the exact test, sample size, and pairing used.","section":"Tables 1, 3, 4; Section 4.1.3"},{"comment":"The debate graph and the ablation variants are under-specified. Equation (7) defines GAT updates on G=(V,E), but the paper does not say how many nodes a debate produces, what edge types exist, whether edges are directed, how many GAT layers and heads are used, or how the 'sequential order or explicit references' mentioned in Section 3.4.2 is encoded. Likewise, Table 3 does not define what replaces DebateFlow Agents in 'w/o DebateFlow Agents' or what replaces the Analysis Agent in 'w/o Analysis Agent'. Without these definitions, the ablation drops cannot be interpreted and the method cannot be reproduced from the text alone.","section":"Section 3.4.2 and Section 4.1.3"},{"comment":"The interpretability claim is not validated against ground truth. The Synthesis Agent's prompt hard-codes five heuristics (verifiable details, reliable sources, neutral tone, emotional language, corroboration), and the case study merely illustrates a generated summary. The paper does not measure whether the debate winner or the Synthesis summary actually agrees with the gold label more often than chance, nor whether the Analysis Agent's performance is driven by these stylistic cues rather than by debate structure. Please add an analysis of debate-outcome/gold-label agreement, or at least explicitly frame these heuristics as assumptions and test their correlation on the validation set.","section":"Section 3.4.1 and Section 5.3"}],"minor_comments":[{"comment":"Line 18 of Algorithm 1 reads 'Build Dabate graph'; this should be 'Build Debate graph'.","section":"Algorithm 1"},{"comment":"The baseline is listed as 'ENDEFA' in Section 4.1.2 but as 'ENDEF' in Table 1; please reconcile the name.","section":"Section 4.1.2 and Table 1"},{"comment":"The GitHub URL spells the repository as 'TED_fake-news-detction'; this appears to be a typo for 'detection' and should be corrected.","section":"Abstract footnote and Appendix A"},{"comment":"The ChatEval adaptation is mentioned only as 'applying these strategies to our fake news detection task'; please specify the prompts or provide a code pointer. Also, 'Deepseek' appears with inconsistent capitalization in Figure 3 and the surrounding text.","section":"Section 4.2 and Figure 3"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The first thing to know: this is a credible systems paper, but its central claim—'TED outperforms all other methods'—is not yet established. The strongest LLM+SLM baseline, ARG, was evaluated with GPT-3.5-turbo while TED runs on GPT-4o-mini. The gains over ARG are small (+0.013 and +0.011 macF1), and TED with Llama 3.1-70B lands exactly on ARG's score. The debate structure may be contributing, but the comparison conflates architecture with a stronger backbone.\n\nWhat's actually new is the assembly: a two-team Lincoln-Douglas debate, a synthesis agent, and a role-aware graph attention judge. The pieces exist, but their combination for fake news detection is a legitimate contribution. The paper also does a few things well: consistent results across English and Chinese, ablations that isolate each component, and a check across open- and closed-source LLMs. Code is released.\n\nSoft spots, in proportion: missing variance reporting, run counts, and seeds; the graph edge construction is described only as 'predefined' and not formally specified; and interpretability is claimed but only shown in one case study, not measured. There's also no direct check that the debate winner corresponds to the true label, so the judge might be exploiting debate style rather than veracity. All of this is fixable with a revision.\n\nThe backbone issue is the one that matters for the headline. If the authors re-run ARG and ARG-D with GPT-4o-mini and TED still wins, the claim holds. If not, the honest conclusion is that TED is a useful debate wrapper, not a clear improvement.\n\nI'd send this to a serious reviewer rather than desk reject. The architecture is coherent, the experiments are broadly reasonable, and the flaws are addressable. The revision must fix the baseline comparison.","headline":"Worth a careful look, but the 'outperforms all baselines' claim rests on an unfair backbone comparison: ARG was run on GPT-3.5, TED on GPT-4o-mini.","tokens_in":17644,"tokens_out":2987,"would_cite":false,"duration_ms":29741,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Structured LLM debate improves fake news detection over single-model and multi-agent baselines.","keywords":["fake news detection","multi-agent system","large language models","debate","interpretability","graph attention network","social media"],"falsifier":"Swap the stances assigned to the two teams for the same news items and retrain or re-run the pipeline: if the predictions flip or degrade while performance on the original stance assignments stays high, the system is following assigned roles and rhetorical cues rather than the truth content of the news.","tokens_in":16629,"feed_emoji":"⚖️","tokens_out":8080,"duration_ms":75110,"temperature":0.7,"pith_summary":"The paper argues that fake news detection improves when multiple large language model agents debate the news in a formal structure instead of classifying it in one pass. It introduces TruEDebate (TED), in which one team argues the news is true, another argues it is fake, and a synthesis-and-analysis stage converts the debate transcript into a label. Across the ARG-EN and ARG-CN datasets, TED reports the highest macro-F1 and accuracy, including gains over the strongest prior LLM-plus-small-model method, ARG. The point of the design is not only accuracy: the debate transcript and synthesis report are meant to make the detection decision explainable to a reader.","feed_headline":"Debating AI agents beat single models at fake news detection","feed_subtitle":"A pro-and-con debate plus a learned synthesis step tops prior models on English and Chinese news benchmarks.","key_machinery":"The central mechanism is the two-stage TED pipeline. DebateFlow Agents organize LLM-powered agents into Proponents and Opponents, who argue in Lincoln-Douglas style stages, producing a debate log. InsightFlow Agents then process that log: the Synthesis Agent condenses it into a report, and the Analysis Agent embeds each debate utterance with its role, builds a debate graph, applies Graph Attention Networks to model argument relationships, and uses interactive attention between the debate representation and the news text to classify. The role-aware encoding and graph structure are what let the model use who said what, not just what was said.","core_discovery":"TED's central claim is that a structured adversarial exchange surfaces signals that single-pass classifiers miss. In TED, DebateFlow Agents produce opening statements, cross-examinations, rebuttals, and closing statements from fixed pro and con stances; the Synthesis Agent then summarizes the debate around five veracity cues (specific verifiable details, credible sourcing, neutral tone, absence of emotional language, and external corroboration); and the Analysis Agent, a role-aware encoder over a debate graph, produces the final verdict. On ARG-EN, TED reaches a macro-F1 of 0.803 and accuracy of 0.892; on ARG-CN, 0.795 and 0.798, topping all reported baselines. Ablations attribute the largest drop to removing the Analysis Agent, with smaller but consistent drops from removing the DebateFlow or Synthesis Agents. The authors take these results as evidence that formal debate plus learned synthesis, rather than prompt-only LLM reasoning, is what drives detection.","pith_inferences":["A stance-swap test, in which Proponents are told to defend 'fake' and Opponents 'true' for the same news, would show whether the model tracks assigned rhetoric rather than veracity; the paper does not run this test.","Because the Synthesis Agent's five cues closely match standard journalistic verification heuristics, the same pipeline could plausibly generalize to claim verification beyond news; the paper only demonstrates news classification.","The Analysis Agent requires fine-tuning per dataset, so a practical open question, untouched here, is whether debate logs transfer across languages or domains without retraining.","If adversarial argumentation is the active ingredient, pairing TED with external evidence retrieval could push accuracy further, although the paper deliberately avoids tool use."],"forward_implications":["Structured debate improves on single-pass LLM classification: on both datasets, TED beats the same backbone used alone by clear margins.","Structured debate also beats a generic multi-agent debate scheme: TED outperforms ChatEval in both the one-by-one and simultaneous-talk configurations.","Each pipeline component contributes: removing the DebateFlow Agents, the Synthesis Agent, or the Analysis Agent lowers macro-F1, with the Analysis Agent's removal causing the largest drop.","The benefit is backbone-independent: TED improves over the LLM-only baseline whether the debate runs on GPT-4o-mini, Deepseek, Qwen 2.5, or Llama 3.1.","The debate transcript and synthesis report double as an explanation of the prediction, which is a stated goal and a practical advantage over black-box classifiers."],"supporting_citations":[{"why":"Supplies the ARG-EN and ARG-CN datasets, the ARG and ARG-D baselines, and the temporal split that prevents data leakage.","marker":"[15]"},{"why":"Defines the Lincoln-Douglas debate format whose stages (opening, cross-examination, rebuttal, closing) structure the DebateFlow Agents.","marker":"[8]"},{"why":"Provides the ChatEval multi-agent debate baseline that TED is compared against on the fake news task.","marker":"[3]"},{"why":"Provides the BERT encoder used by the Analysis Agent to represent debate interactions and news content.","marker":"[7]"},{"why":"Supplies the Graph Attention Network layers that propagate information across the debate graph.","marker":"[43]"},{"why":"Source of the English GossipCop data from which the ARG-EN evaluation set is derived.","marker":"[38]"},{"why":"Source of the Chinese Weibo21 data from which the ARG-CN evaluation set is derived.","marker":"[32]"}],"fun_headline_variants":["Multi-agent debate with synthesis beats single LLM classifiers","TruEDebate: Adversarial exchange improves fake news accuracy","Structured debate plus insight agents top fake news benchmarks","DebateFlow agents outperform solo LLMs on fake news"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The five cues the Synthesis Agent is told to weigh—verifiable specifics, reliable sourcing, neutral tone, absence of emotional language, and external corroboration—are assumed to track whether a news item is actually fake; if they do not, the classifier may be fitting debate-style artifacts and the reported gains would not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Multi-agent debate with synthesis beats single LLM classifiers","TruEDebate: Adversarial exchange improves fake news accuracy","Structured debate plus insight agents top fake news benchmarks","DebateFlow agents outperform solo LLMs on fake news"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000941,"raw_usage":{"total_tokens":4051,"prompt_tokens":1003,"completion_tokens":3048,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":619,"completion_tokens_details":{"reasoning_tokens":2980}},"tokens_in":619,"tokens_out":3048,"duration_ms":21922,"temperature":1.0,"reasoning_tokens":2980,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:52:22.086031+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Swap the stances assigned to the two teams for the same news items and retrain or re-run the pipeline: if the predictions flip or degrade while performance on the original stance assignments stays high, the system is following assigned roles and rhetorical cues rather than the truth content of the news.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ARG-EN and ARG-CN datasets, the ARG and ARG-D baselines, and the temporal split that prevents data leakage."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Lincoln-Douglas debate format whose stages (opening, cross-examination, rebuttal, closing) structure the DebateFlow Agents."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ChatEval multi-agent debate baseline that TED is compared against on the fake news task."}],"review_version":1}