{"id":"6bdfbd97-8900-48aa-ad58-4cf9aa2a2f70","arxiv_id":"2412.14686","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"The authors introduce AMG, an attribution-labeled multimodal fake news benchmark with five fake patterns, plus MGCA, a clue-alignment model that outperforms baselines on it.","lead":"This paper builds AMG, a new dataset of multimodal social media posts labeled both as real or fake and by five fake-news attribution types such as image fabrication and time inconsistency. It also proposes MGCA, a model that uses these labels, and reports state-of-the-art accuracy on the new benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Attribution ground truth is undermined by an internally contradictory coverage claim: the five category counts sum to all 2,004 fake posts while §3.3 also reports ~60 out-of-taxonomy cases, and acknowledged multi-cause cases cannot be represented by the single-label annotation.","rationale":"The paper's contribution (2) is the AMG benchmark and its attribution labels; without valid labels, the 'first attribution benchmark' claim and all reported attribution accuracies are uninterpretable. I searched for the weakest spot in that argument and found it in the dataset construction section, not in the model or experiments. The category counts adding to exactly 2,004 while a later sentence reports ~60 outside cases is an internal contradiction, not a matter of taste; it means the paper's own evidence for exhaustiveness is unusable. The Appendix's Figure 7 explicitly shows a post with both EntityInc and TimeInc, which the single-label protocol cannot represent, so even if coverage were fixed the taxonomy may not be mutually exclusive. The reliability evidence (100 pre-discussed cases with >95% agreement) does not establish annotator agreement on the full corpus. This does not mean the dataset is worthless; a multi-label revision, released raw annotations, and corrected statistics could support the claim. But as written, the central claim is conditional on a label layer that the paper has not shown to be sound. My proposed check is minimal and could be run on data the authors already have. The reader's weakest assumption identified the same label-validity concern, and my analysis sharpens it with concrete numeric and multi-label evidence, so no verdict change is needed: the paper remains conditional on resolving this issue.","tokens_in":17143,"tokens_out":6409,"duration_ms":55206,"concrete_test":"Obtain the pre-filter crawled fake-news pool (before removing out-of-taxonomy and multi-cause items). Take a stratified random sample of 200 posts (or all pre-filter items if fewer), and have three trained annotators independently assign every applicable attribution category (multi-label) plus 'other/none', without a forced single choice. Report per-category Fleiss kappa, the proportion of items with more than one applicable label, and the proportion labeled 'other/none'. If multi-label proportion is non-negligible (e.g. >10%) or kappa <0.6, the single-label AMG attribution labels and the 3% coverage claim are not supported; the benchmark needs a multi-label revision or a re-working of the taxonomy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that AMG is a first benchmark for multimodal fake-news attribution, so the validity of the five-way attribution labels is the load-bearing assumption. Two features of the construction make that assumption insecure. (1) The taxonomy is not shown to be mutually exclusive or jointly exhaustive. The five category counts reported in 'Data Collation and Analysis' (434, 295, 133, 667, 475) sum exactly to the reported 2,004 fake-news items. The same section then claims that ~60 instances (about 3%) fall outside the attribution categories. If those 60 were outside, the final fake count would be 2,064, or the category counts must exclude them; the paper asserts both, so the coverage statistic cannot be interpreted. The 60 is 3% of the 2,004 fake items, not 3% of the 5,022 total dataset, so the statement is also numerically mislabeled. Appendix Figure 7 shows a case with both EntityInc and TimeInc, while the annotation protocol forces each item to one of six classes by majority vote; multi-cause posts therefore receive a non-faithful label. (2) No full-corpus inter-annotator agreement is reported; the only reliability check is a 100-case pilot whose labels were chosen by group discussion before experts labeled them, so the >95% agreement on that pilot does not transfer to the full corpus. Since all detection/attribution scores in Tables 2-4 are computed against these labels, the headline 'first attribution benchmark' and all MGCA results are only as strong as this unvalidated label layer.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AMG, a multimodal fake news dataset with binary real/fake labels plus fine-grained attribution labels (ImageFab, ImageNoE, EntityInc, EventInc, TimeInc, and a 'None of the Above' category) for fake posts collected via fact-checking rulings from Snopes and CheckYourFact. It also proposes MGCA, a model that extracts multi-view clues (textual/visual entities, events, temporal information, and image-manipulation features) and aligns them for joint fake-news detection and attribution. Experiments on AMG and transfer experiments on Twitter, Weibo, and Weibo21 show that MGCA outperforms several baselines, and the authors argue that AMG is more challenging than prior datasets. The central claim is that AMG is the first benchmark for multimodal fake news attribution, with a multi-granularity label scheme that reveals the causes of falsity.","tokens_in":17516,"tokens_out":4357,"duration_ms":28434,"significance":"If the attribution labels are valid, AMG addresses a genuine gap: existing multimodal fake news datasets provide only binary labels, while AMG introduces a five-way causal attribution scheme, temporal metadata, and multi-platform coverage (Instagram, Facebook, Twitter). The dataset and code are promised to be released, which is a positive step for reproducibility. The MGCA model is a reasonable baseline with a sensible multi-clue design, and the transfer experiments provide external grounding of its effectiveness. However, the significance of the whole contribution hinges on the reliability of the attribution ground truth, and several unresolved issues in the dataset construction directly affect that reliability.","major_comments":[{"comment":"The coverage statistic is internally inconsistent. The category counts reported as '434, 295, 133, 667, 475' sum to exactly 2,004, which equals the stated number of fake news items, yet the same section claims that 'the samples that fall outside our attribution categories account for only around 3% of the total dataset, comprising approximately 60 instances.' If those 60 out-of-taxonomy instances were filtered out, they cannot simultaneously be used to argue that the taxonomy covers 97% of the data; if they were not filtered out, the category counts are incorrect. Moreover, 60 is about 3% of the 2,004 fake items, not 3% of the 5,022 total dataset, so the numerical framing is also wrong. This undermines the 'Rationality of our attribution rules' conclusion.","section":"Data Collation and Analysis / Rationality of our attribution rules"},{"comment":"The annotation protocol assigns each fake post to exactly one of six classes by majority vote, but the paper acknowledges multi-cause cases: Appendix Figure 7(a) shows a post with both EntityInc and TimeInc, and the Limitation section explicitly refers to 'multiple overlapping attribution anomalies.' The single-label six-class scheme cannot faithfully represent such posts; majority voting over mutually exclusive choices discards the multi-cause structure. As a result, the attribution ground truth used in Tables 2 and 4 is not faithful for at least some of the data, and the benchmark's claimed multi-granularity is undermined.","section":"Data Processing and Annotation / Cross Validation and Discussion; Preliminary, Task 2"},{"comment":"The only reliability check reported is a 100-case pilot in which cases were pre-selected through group discussion, and expert labels were compared against the pre-selected labels, with a threshold of >95% accuracy and F1 before the labels were allowed to proceed to voting. This measures agreement with a consensus-derived gold set on an artificially easy, pre-discussed sample; it does not establish inter-annotator agreement on the full corpus. The paper should report full-corpus inter-annotator agreement (e.g., Cohen's kappa or Fleiss' kappa) on a random sample of the data to support the reliability of the attribution labels.","section":"Appendix, Additions to Annotation Process"},{"comment":"The real-news set mixes posts from social platforms (Instagram, Facebook, Twitter) with articles from Reuters and NewsNation official accounts, plus a random supplement from archives. These sources differ systematically in format, length, and visual style, which can introduce confounds unrelated to authenticity. The later claim (Section 'Discussion on Dataset Difficulty') that AMG avoids entity bias is not supported by any distributional comparison between real and fake posts, and this weakens the conclusion that the dataset is challenging for content-based reasons rather than source artifacts.","section":"Data Collection, Real News Collection"}],"minor_comments":[{"comment":"The title contains unintended spacing: 'Each F ake News is F ake in its Own Way' should be 'Each Fake News is Fake in its Own Way.'","section":"Title"},{"comment":"In the Twitter row, '13.924' should likely be '13,924', and the MR2 row appears to duplicate the '6,976' entry; please correct the formatting.","section":"Table 1"},{"comment":"The text reads 'we utilize Baiduan APIs'; this should be 'Baidu APIs.'","section":"Visual Entity"},{"comment":"The phrase 'we exploit utilize BERT' is redundant; it should read 'we utilize BERT.'","section":"Multimodal Feature Learning"},{"comment":"The symbol T is used both for the temporal feature in the comparison function and for the temporal gap Tg; please use distinct notation to avoid confusion.","section":"Eq. (3) and surrounding text"},{"comment":"The citation '(Liu et al. 2023; ?)' contains a missing reference placeholder; please complete the citation.","section":"Appendix, Settings of LLaVA"},{"comment":"The Vicuna model is attributed to Zheng et al. 2023, but that reference is the MT-Bench paper; please cite the original Vicuna technical report or clarify the connection.","section":"References"},{"comment":"The 'Extended version' link is a placeholder URL (https://aaai.org/example/extended-version); either provide the actual link or remove it.","section":"Abstract"},{"comment":"The heatmap would be more interpretable with a colorbar and explicit axis labels indicating the classes; currently the reader cannot map colors to similarity values.","section":"Figure 6"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of a major NLP/CS conference, but the dataset release appears incomplete: the GitHub link is provided, yet no direct dataset download or annotation guideline is visible in the manuscript. The 'Extended version' placeholder URL suggests the submission is not final. Given that the central contribution is the benchmark, I strongly recommend that the authors release the full data, annotation guidelines, and a full-corpus agreement study for review. The positioning against Dong et al. 2024 should also be sharpened, since that work already explores deception patterns in multimodal fake news; the 'first' claim needs precise delimitation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The authors have built something the field actually lacks: a multimodal fake news benchmark with cause-based attribution labels (image fabrication, non-evidential image, entity/event/time inconsistency) derived from fact-checking rulings. That's a real gain over Fakeddit's subreddit-derived categories and over binary datasets. The novelty claim against Dong et al. 2024 looks fair—that paper studies deception patterns but releases no attribution benchmark. Time-inconsistency is a genuinely useful dimension, and collecting from Instagram, Facebook, and Twitter with timestamps adds practical value. MGCA is a credible baseline; it beats BMR and others on both detection and attribution, though the margin is small.\n\nThe load-bearing part is the annotation validity, and that has two soft spots. First, the coverage statistic is contradictory. The five category counts sum to 2,004—exactly the reported number of fake items—while the 'Rationality' section also says ~60 instances fall outside the categories, 'around 3% of the total dataset.' Both can't be true. If the 60 were filtered out, the category counts shouldn't sum to the final fake count; if they were included, the total would be 2,064. And 60 is 3% of 2,004, not of the 5,022 dataset. The paper may have meant something coherent, but as written it undermines the 'covers almost all cases' claim. Second, the only reliability check is a 100-case pilot where experts had to match consensus labels above 95% before voting. That's a qualification gate, not full-corpus inter-annotator agreement. We never see how often the three annotators disagreed on the remaining ~1,900 items or how many were settled by discussion.\n\nThe experimental reporting is thin in the same way: no error bars, no repeated runs, and the 2.5% detection gain could easily be noise. The real-news collection mixes social-media posts with Reuters/NewsNation articles, which risks a style shortcut. And the annotation forces one label per item while the paper itself shows a multi-cause case in Figure 7; the prevalence of multi-cause fakes is unquantified.\n\nThese are fixable, and they don't sink the contribution. The dataset idea is sound and the taxonomy is mostly defensible; the text just needs to reconcile the numbers, report IAA on the full corpus, and soften the statistical claims. This is for anyone working on explainable or fine-grained fake news detection, especially temporal misalignment. I'd send it to reviewers, not desk-reject. The reviewers should press on the annotation process and the coverage arithmetic. I wouldn't build quantitative conclusions on it yet, but I'd cite it as the first attribution benchmark once the caveats are cleaned up.","headline":"A genuinely new attribution benchmark for multimodal fake news, but the coverage claim is internally inconsistent and the annotation reliability is under-reported.","tokens_in":18025,"tokens_out":7401,"would_cite":false,"duration_ms":54649,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Multimodal fake news can be attributed to five recurrent causes, and the AMG benchmark plus the MGCA model make that attribution learnable.","keywords":["multimodal fake news detection","fake news attribution","benchmark dataset","image fabrication","entity inconsistency","event inconsistency","time inconsistency","multi-granularity clue alignment"],"falsifier":"Re-annotate a random sample of the 2,004 fake posts with a fresh set of annotators who are shown the same guidelines but not the original labels. If agreement falls well below the level reported in the pilot, or if a substantial share of posts are judged to fit multiple categories or none, then the attribution ground truth, and any attribution score computed on it, is not stable.","tokens_in":16940,"feed_emoji":"📰","tokens_out":5326,"duration_ms":36279,"temperature":0.7,"pith_summary":"The paper introduces AMG, a multimodal fake news dataset that goes beyond real/fake labels to say why a post is fake, assigning each false post to one of five causes: image fabrication, a non-evidential image, entity inconsistency, event inconsistency, or time inconsistency. It argues that existing benchmarks treat fake news as a single category, even though the error patterns behind false posts differ, and that attributing the cause improves the credibility and usefulness of detection. The paper also proposes MGCA, a model that extracts entity, event, time, and image-manipulation clues from text and images, aligns them, and predicts both authenticity and attribution. On AMG, MGCA outperforms the compared baselines on both tasks, and the authors show that AMG is harder than earlier datasets.","feed_headline":"New benchmark labels fake news by how it deceives","feed_subtitle":"AMG tags posts as fabricated images, mismatched entities, events, or times — and a new model learns to detect and attribute them.","key_machinery":"The load-bearing mechanism is multi-granularity clue alignment: the paper extracts textual entities with a pretrained language model, visual entities with an image-recognition API, image events with a multimodal language model, and temporal references from the post and from reverse image search. These clues are fed in pairs into a Compare-Net, a comparison network that scores feature pairs using the vector $[C_1, C_2, C_1-C_2, C_1*C_2]$, producing consistency features for entity, event, and time. Image manipulation features come from a frozen PSCC-NET. Each clue gets its own binary real/fake classifier, and the resulting probabilities gate the features before the final detection and attribution heads.","core_discovery":"The central claim is that multimodal fake news can be meaningfully attributed to five recurrent causes, and that a benchmark built on those causes supports a harder and more informative detection task. AMG contains 5,022 multimodal posts from Instagram, Twitter, and Facebook, with 2,004 fake posts labeled by expert annotators as either image fabrication, non-evidential image, entity inconsistency, event inconsistency, or time inconsistency, plus a None-of-the-Above category. The companion MGCA model treats detection and attribution jointly, aligning multi-granular clues extracted from text and images and using per-clue binary classifiers to weight each clue's contribution. The paper reports that MGCA reaches 83.23% accuracy and 0.8310 F1 on detection and 73.85% accuracy and 0.5666 F1 on attribution, outperforming the four baselines tested, and that removing the event-coherence or temporal modules causes the largest drops.","pith_inferences":["An attribution taxonomy of this kind could let platforms route suspected fake posts to different review queues depending on the predicted cause, since image fabrication needs a visual forensics check while time inconsistency needs a reverse-image lookup.","As synthetic images become more common, the ImageFab category may stop being a single coherent class; future work might need to split it into AIGC-generated, spliced, and repurposed images.","The model's reliance on reverse image search for temporal clues suggests that fake news using brand-new imagery no search engine has indexed yet would be a blind spot not covered by AMG."],"forward_implications":["Detection systems trained on AMG can output an attribution alongside the real/fake verdict, so a flag of 'fake' comes with a reason and a suggested check.","The ablation results show temporal coherence and event coherence are the strongest individual clues, so future detectors should model those rather than only image-text similarity.","Because MGCA also improves F1 on Twitter, Weibo, and Weibo21, attribution-aware training appears to transfer to older binary benchmarks.","AMG's inclusion of timestamps and its multi-platform collection make it possible to test whether models generalize across time and across social platforms."],"supporting_citations":[{"why":"Closest prior work on deception patterns in multimodal fake news, which AMG extends by providing a labeled attribution benchmark.","marker":"Dong et al. 2024"},{"why":"Source of the idea of attributing unanswerable questions, transferred here to attributing fake news.","marker":"Rajpurkar, Jia, and Liang 2018"},{"why":"Twitter verification dataset used as a generalization test bed for MGCA.","marker":"Boididou et al. 2015"},{"why":"Weibo rumor dataset used as a generalization test bed for MGCA.","marker":"Jin et al. 2017"},{"why":"Weibo21 dataset used as a generalization test bed for MGCA.","marker":"Nan et al. 2021"},{"why":"MR2 benchmark compared against AMG and cited as lacking attribution labels and timestamps.","marker":"Hu et al. 2023b"},{"why":"Fakeddit, the fine-grained multimodal dataset whose label scheme AMG differentiates from attribution labels.","marker":"Nakamura, Levy, and Wang 2020"},{"why":"FakeNewsNet datasets used to show AMG is harder than prior benchmarks.","marker":"Shu et al. 2020"},{"why":"PSCC-Net supplies the frozen image-manipulation features in MGCA.","marker":"Liu et al. 2022"},{"why":"CLIP provides the visual-textual feature backbone for MGCA and also serves as a baseline.","marker":"Radford et al. 2021"}],"fun_headline_variants":["Why each fake post is fake: a new taxonomy and detector","5 ways fake news deceives: new benchmark and AI","New dataset reveals five types of fake news trickery","Attributing fake news: model finds how each post misleads"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's ground truth rests on the five attribution categories being exhaustive and reliably assignable; the reported 3% coverage is hard to interpret because posts that did not fit the categories were filtered out before that figure was computed.","fun_headline_variants_meta":{"raw":{"variants":["Why each fake post is fake: a new taxonomy and detector","5 ways fake news deceives: new benchmark and AI","New dataset reveals five types of fake news trickery","Attributing fake news: model finds how each post misleads"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000347,"raw_usage":{"total_tokens":1870,"prompt_tokens":882,"completion_tokens":988,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":918}},"tokens_in":498,"tokens_out":988,"duration_ms":17332,"temperature":1.0,"reasoning_tokens":918,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:00:33.939933+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random sample of the 2,004 fake posts with a fresh set of annotators who are shown the same guidelines but not the original labels. If agreement falls well below the level reported in the pilot, or if a substantial share of posts are judged to fit multiple categories or none, then the attribution ground truth, and any attribution score computed on it, is not stable.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Closest prior work on deception patterns in multimodal fake news, which AMG extends by providing a labeled attribution benchmark."},{"cited_title":"T.; Boato, G.; Riegler, M.; Kompatsiaris, Y.; et al","cited_arxiv_id":null,"evidence_quote":"Twitter verification dataset used as a generalization test bed for MGCA."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Weibo rumor dataset used as a generalization test bed for MGCA."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Weibo21 dataset used as a generalization test bed for MGCA."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Fakeddit, the fine-grained multimodal dataset whose label scheme AMG differentiates from attribution labels."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"FakeNewsNet datasets used to show AMG is harder than prior benchmarks."}],"review_version":1}