{"id":"7dc5fdc1-5803-408c-ab37-c21ba145110c","arxiv_id":"1908.10818","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The authors release MCG-FNeWS, a large multimodal Weibo false-news dataset, and provide baseline scores for text-only, image-only, and multimodal detection subtasks.","lead":"This paper describes a new benchmark dataset and competition for detecting false news on Weibo using text, images, or both. It reports baseline results from standard models, with early text-image fusion reaching the highest accuracy at 87.6 percent.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own stated selection procedure for real-news examples is an unrecognized confound: the baselines reported in Table 1 may largely be measuring the annotated topic/event distribution rather than the linguistic or visual cues of false news.","rationale":"The reader's weakest_assumption already identifies the representativeness of real-news posts as the main risk, and the reader's verdict is already CONDITIONAL; my analysis lands on the same spot. Section 3 of the manuscript explicitly describes a matched negative-sampling procedure and the removal of 'meaningless statistical clues', which is a textbook setup for shortcut learning or dataset bias. Because no topic-stratified evaluation and no leakage analysis are reported, the concern is concrete and testable rather than speculative. I do not see a stronger load-bearing problem: the central contribution is a dataset, the baselines are reasonable, and the paper explicitly disclaims tuning as an objective. The dataset-construction confound is the single place where the central claim could fail in a way that the manuscript does not address. The correct verdict is therefore CONDITIONAL (retain the reader's verdict), with the condition being that the organizers either provide a matched-out or unmatched real-news evaluation or a topic-stratified breakdown.","tokens_in":7115,"tokens_out":1522,"duration_ms":14168,"concrete_test":"Re-run the two most competitive baselines (BERT for Subtask A, early fusion for Subtask C) on a small held-out set of ordinary real-news posts not drawn from the matched-pair pipeline, or, if no such set is available, recompute the Subtask A results separately for each of the nine topic tags and for each event-aligned pair. If accuracy drops by more than a few points on unmatched real news, or if performance varies by topic, the reported reference scores should be treated as specific to the dataset construction, not as an estimate of real-world performance.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that MCG-FNeWS supports a usable false-news benchmark and that the reported baselines provide reference performances. The load-bearing assumption is that the negative examples (real-news posts) are representative of the real-news distribution, or at least that the selection rule does not introduce an identifiable shortcut. Sections 3 and 5 state that false-news posts were collected from the Weibo community management center, while real-news posts were selected to match false-news posts, with real posts chosen from 'the same or similar events' as false posts, and that 'some meaningless statistical clues' were manually removed. This selection creates a matched-sample design. In such a design, the baseline classifiers are trained and evaluated on a mixture of (a) the actual false-vs-real difference and (b) whatever residual within-event/event-similarity structure remains after matching. If the matching is imperfect, the classifiers can exploit leftover event identity, topic, or time-period signals. The paper never reports topic-conditioned results, never reports performance on a held-out distribution of naturally occurring real news, and never reports any leakage analysis. The reported F1s are high but not extreme, and no error analysis is given to show that the classifiers use the advertised cues. The reader's concern about 'real news representativeness' is therefore not a hypothetical: it is a concrete, unmeasured confound inherent in the stated negative-sampling procedure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript, which reads as a competition overview rather than a full research paper, introduces a three-task false-news detection benchmark on Sina Weibo called MCG-FNeWS. The dataset contains 38,471 training posts (19,285 false, 19,186 real), a 4,000-post validation set, a 3,902-post test set, corresponding images, user profiles, and a 37,877-item debunking repository. The paper defines text-only (subtask A), image-only (subtask B), and multimodal (subtask C) classification tasks, and reports baseline results from LSTM, GRU, TextCNN, BERT, VGG19, MVNN, and attRNN. The central claim is that MCG-FNeWS is the largest publicly available multi-modal false news detection dataset and that the reported baselines provide reference performances for future work.","tokens_in":7409,"tokens_out":4765,"duration_ms":51309,"significance":"If the dataset is representative, it fills a genuine gap: most false-news datasets are text-only or small, and a public multi-modal resource with labels from Weibo's official debunking center is potentially valuable. The paper's strengths are its public release, the inclusion of multiple modalities, a moderately large labeled corpus, and a supplementary debunking repository. However, the scientific value depends on the negative-example construction: the real-news posts are deliberately matched to false-news linguistic patterns and events, so the benchmark may reward classifiers that detect residual event or topic identity rather than properties of false news in a natural stream. Because the baselines are also single-run and lightly specified, the reference numbers should not be treated as established until the authors supply additional representativeness and reproducibility analysis.","major_comments":[{"comment":"The real-news sampling procedure is a matched-sample design rather than a representative sample. The text states that real-news posts were selected to resemble the linguistic style of false-news posts, to belong to the same or similar events as false-news posts, and that some meaningless statistical clues were manually removed. Under this design, a classifier can exploit residual event identity, topic, time period, or stylistic markers that survived matching, so the high F1 values in Table 1 cannot be attributed to truthful linguistic or visual cues without a leakage or topic-conditioned analysis. I ask the authors to report per-topic performance, evaluate on naturally sampled real news from the same period, and provide an analysis of which features drive the decisions.","section":"Section 3 (Data & Resources) and Table 1"},{"comment":"The baseline table reports accuracy, precision, recall, and F1 to three decimal places for ten models, but gives no error bars, number of runs, random seeds, or hyperparameter settings. Section 5 explicitly states that the authors did not search for the best hyperparameters, so the reported numbers are not reproducible reference points and cannot support comparative claims such as early fusion outperforming other baselines for subtask C. Please provide model configurations, training details, and variance over multiple runs, or clearly reframe the table as indicative single-run results.","section":"Table 1 and Section 5"},{"comment":"The manual removal of meaningless statistical clues is underspecified. Without a concrete definition or examples of what was removed, this step is unrepeatable and may encode dataset-specific shortcuts that inflate baseline performance, and it also makes the dataset less clean for future users. The authors should either specify the removal rule precisely, provide the removed pattern list, or remove this step from the pipeline.","section":"Section 3 (preprocessing)"}],"minor_comments":[{"comment":"The paper lists nine topic categories but gives no distribution, annotation reliability, or examples. A small table of topic counts would help users and would enable the topic-conditioned checks requested in the major comments.","section":"Section 2.3"},{"comment":"There are typographical and grammatical errors, including 'detetcion' in the abstract and 'we doesn't focus' in Section 5; the manuscript should be proofread.","section":"Abstract and Section 5"},{"comment":"The claim that MCG-FNeWS is the largest multi-modal false news detection dataset should be supported by a comparison table of existing multi-modal datasets, including their sizes, modalities, and sources.","section":"Section 1"},{"comment":"It is not stated which split (training, validation, or testing) was used for the baseline results in Table 1; this should be clarified.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is short and competition-oriented, and its main contribution is the dataset rather than new methods. The dataset itself is plausible and potentially useful, but the evaluation component is too thin for a journal-level acceptance without the representativeness and reproducibility analyses described in the major comments. If the venue is a workshop or competition description, the bar can be relaxed, but I would hold the line for a formal journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a competition overview whose main contribution is a new multimodal false-news dataset for Weibo, MCG-FNeWS, probably the largest public resource of its kind. If you work on multimodal fake news detection, it's worth knowing about and likely worth citing. The modeling is just baselines; no new method is proposed.\n\nWhat is genuinely new and good: three subtasks (text, image, multimodal), 38,471 training posts with 34,096 images, separate validation and test splits, and a debunking repository of 37,877 refutations. The real-news collection goes beyond simple random sampling: the authors deliberately match real posts to false posts on linguistic style and event, and they manually remove \"meaningless statistical clues.\" That is a serious attempt to make the task non-trivial, and I credit them for it. The baselines are standard (LSTM, GRU, TextCNN, BERT, VGG19, MVNN, early/late fusion, attRNN), and the authors explicitly say these are not tuned, which is fine for reference numbers.\n\nThe soft spot is exactly what you flagged: the matched-sample design is a confound that is not measured. If matching leaves residual event or topic structure, the reported F1s (e.g., 0.875 for early fusion) may reflect that structure rather than the intended linguistic or visual cues. The paper gives no topic-conditioned results, no held-out evaluation on naturally occurring real news, and no error analysis to show what features the classifiers are actually using. This is a common problem in benchmark construction, and it does not make the dataset useless for relative comparisons among methods on the same split, but it does limit claims about absolute performance or generalization.\n\nMinor points: no error bars or hyperparameter details for the baselines, and the external data link is not verifiable from the manuscript alone. The citation pattern is heavy on the authors' own prior work, but those papers actually built this dataset and its precursors, so that is not a flaw.\n\nVerdict: the paper deserves a serious referee and likely publication after revision. The revision should document the sampling procedure in enough detail for readers to assess the confound, and ideally include a topic-conditioned breakdown or a small \"natural\" real-news sample for sanity-checking. I would use this dataset in my own work, with appropriate caveats.\n\nBest.","headline":"A useful new multimodal fake-news dataset for Weibo, with a real but unmeasured confound in its negative sampling; worth citing and refereeing.","tokens_in":7852,"tokens_out":3236,"would_cite":true,"duration_ms":34076,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper presents MCG-FNeWS, a dataset of 38,471 Weibo news posts with images plus a 37,877-item debunking repository, and reports baseline accuracy for text, image, and multi-modal false news detection.","keywords":["false news detection","multi-modal classification","social media","Weibo","dataset construction","benchmark","rumor detection","image-text fusion"],"falsifier":"Train the best reported fusion model on MCG-FNeWS, then run it on a freshly crawled, independently verified sample of Weibo news posts drawn without the style-matching procedure; if accuracy falls well below the reported 0.876, the matched real-news selection inflated the benchmark.","tokens_in":6973,"feed_emoji":"📰","tokens_out":8133,"duration_ms":70903,"temperature":0.7,"pith_summary":"The paper establishes a shared benchmark for real-time false news detection on Weibo: it defines three subtasks—classifying a post as false or real from its text alone, its image alone, or its text, image, and publisher profile together—and releases a dataset to train and test them. That dataset, MCG-FNeWS, contains 38,471 training posts balanced between 19,285 false and 19,186 real news items, with 34,096 corresponding images, a validation set of 4,000 posts, a test set of 3,902 posts, and a companion repository of 37,877 refutations of previously debunked false news. The paper reports baseline accuracies for all three subtasks, with the best results coming from BERT for text (0.867), MVNN for images (0.805), and early fusion for multi-modal input (0.876). A curious reader should care because this is a resource designed to let the community build and compare systems that judge truthfulness from raw content before a post goes viral, without waiting for retweets or comments.","feed_headline":"New dataset benchmarks fake-news detection on 38,471 Weibo posts","feed_subtitle":"The largest image-bearing false-news corpus gives text, image, and fused detectors one shared test.","key_machinery":"The load-bearing object is the MCG-FNeWS corpus and its construction pipeline. False-news posts are crawled from Weibo from May 2012 to November 2018 and verified by the official Weibo Community Management Center; real-news posts are collected over the same period by first text-mining false-news linguistic patterns (following the method of Jin et al.), crawling stylistically matched posts from the Weibo live stream, manually cross-checking them against online sources, and sampling to balance the two classes while preferring posts about the same or similar events as false posts. Statistical clues that would trivially separate the classes are removed in preprocessing. The corpus is organized around three subtask definitions—a text classifier $f(x_t)$, an image classifier $f(i_t)$, and a multi-modal classifier $f(x_t, i_t, u_t)$ using text, image list, and publisher profile—so that the same raw posts support ablation across modalities. The companion debunking repository of 37,877 refutations supplies the external knowledge the competition encourages participants to use.","core_discovery":"The central discovery on the paper's own terms is that a multi-modal false news detection corpus can be built at the scale of tens of thousands of posts, and that content-only classifiers reach usable accuracy on it. Concretely, the paper claims MCG-FNeWS is the largest such dataset released so far, with 38,471 training posts (19,285 false, 19,186 real), 34,096 images, a 4,000-post validation split, a 3,902-post test split, and a debunking repository of 37,877 refutations. The false-news posts come from the Weibo Community Management Center's verified records, and the real-news posts are crawled to match them in linguistic style and event, then manually cross-checked. The experimental contribution is a reference set of results: text classifiers reach 0.854 F1 (BERT), image classifiers reach 0.772 F1 (MVNN), and multi-modal classifiers reach 0.875 F1 (early fusion) when false news is the target class.","pith_inferences":["The deliberate matching of real posts to false posts by linguistic style likely makes MCG-FNeWS harder than a random news sample, so a system trained on it should generalize better to adversarial cases but may underperform on ordinary, easy-to-detect posts; that trade-off is worth testing directly.","The image-subtask numbers imply that images alone are a much weaker signal than text (best image F1 0.77 vs text 0.85); a natural next experiment, not run in the paper, is whether adding images to text changes post-level accuracy at all once early fusion reaches 0.876.","Because the authors do not promise that every false post has a matching refutation, a useful follow-up is computing the coverage rate of the 37,877-item repository against the 19,285 false training posts; retrieval-based detection would only be dependable for the covered fraction."],"forward_implications":["With text, image, and multi-modal tasks defined on the same posts, any submitted system can be directly ablated to measure how much each modality contributes to false-news detection.","Because real posts are matched to false posts by event, methods that use event context or adversarial event invariance can be tested without confounding on topic.","The separately released 37,877-item debunking repository opens a third, retrieval-based route: a detector can answer \"was this already refuted?\" before or alongside content classification.","Reported baselines give new teams a concrete bar to beat: roughly 0.85 F1 in the text task, 0.77 in the image task, and 0.875 in fused detection when the target class is false news."],"supporting_citations":[{"why":"Ma et al. 2016 establish the use of the Weibo Community Management Center as the ground-truth source for false-news posts, which this dataset relies on.","marker":"[4]"},{"why":"Vosoughi et al. 2018 document that false news spreads faster than true news, the stated motivation for demanding real-time, content-only detection.","marker":"[8]"},{"why":"Qi et al. 2019 supply MVNN, the state-of-the-art visual detector used as the subtask B baseline.","marker":"[12]"},{"why":"Jin et al. 2017 provide the attRNN fusion model used as a subtask C baseline and an earlier, smaller multi-modal dataset that the new corpus is compared against.","marker":"[13]"},{"why":"Jin et al. 2016 contribute the text-mining method used to discover false-news linguistic patterns and to crawl stylistically matched real-news posts as negative samples.","marker":"[27]"},{"why":"Devlin et al. 2019 supply BERT, the strongest text-only baseline in subtask A.","marker":"[31]"},{"why":"Simonyan and Zisserman 2014 supply VGG19, the CNN used as the image feature extractor in subtask B and the fusion baselines.","marker":"[33]"}],"fun_headline_variants":["38,471 Weibo posts set the bar for multi-modal fake-news detection","Largest fake-news detection benchmark on Weibo: 38,471 posts","Multi-modal fake-news detection gets a shared testbed: 38K Weibo posts","Content-only detectors hit usable F1 on 38,471 Weibo posts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's real-news posts are chosen to imitate false-news posts in language and event, so the results assume this matched set represents real news in general; if matched posts hide distinctive artifacts, reported accuracies may not hold on an uncurated stream of Weibo content.","fun_headline_variants_meta":{"raw":{"variants":["38,471 Weibo posts set the bar for multi-modal fake-news detection","Largest fake-news detection benchmark on Weibo: 38,471 posts","Multi-modal fake-news detection gets a shared testbed: 38K Weibo posts","Content-only detectors hit usable F1 on 38,471 Weibo posts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001107,"raw_usage":{"total_tokens":4593,"prompt_tokens":906,"completion_tokens":3687,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":522,"completion_tokens_details":{"reasoning_tokens":3599}},"tokens_in":522,"tokens_out":3687,"duration_ms":27522,"temperature":1.0,"reasoning_tokens":3599,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:32:05.992754+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the best reported fusion model on MCG-FNeWS, then run it on a freshly crawled, independently verified sample of Weibo news posts drawn without the style-matching procedure; if accuracy falls well below the reported 0.876, the matched real-news selection inflated the benchmark.","supporting_citations":[{"cited_title":"Detecting rumors from microblogs wi th recurrent neural networks","cited_arxiv_id":null,"evidence_quote":"Ma et al. 2016 establish the use of the Weibo Community Management Center as the ground-truth source for false-news posts, which this dataset relies on."},{"cited_title":"/T_he spread of tr ue and false news online","cited_arxiv_id":null,"evidence_quote":"Vosoughi et al. 2018 document that false news spreads faster than true news, the stated motivation for demanding real-time, content-only detection."},{"cited_title":"Exploiting multi- domain visual information for fake news detection","cited_arxiv_id":null,"evidence_quote":"Qi et al. 2019 supply MVNN, the state-of-the-art visual detector used as the subtask B baseline."},{"cited_title":"Multimodal fusion with recurrent neural networks for rumor detection o n microblogs","cited_arxiv_id":null,"evidence_quote":"Jin et al. 2017 provide the attRNN fusion model used as a subtask C baseline and an earlier, smaller multi-modal dataset that the new corpus is compared against."},{"cited_title":"Image Credibility Analysis with Effective Domain Transferred Deep Networks","cited_arxiv_id":"1611.05328","evidence_quote":"Jin et al. 2016 contribute the text-mining method used to discover false-news linguistic patterns and to crawl stylistically matched real-news posts as negative samples."}],"review_version":1}