{"id":"fd877e9d-f998-4e3b-b868-c0b4e63399d6","arxiv_id":"2505.18221","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A graph neural network that compares an evidence graph from reverse-image-search web pages against a claim graph detects out-of-context misinformation with 93.05% accuracy on a 461-sample overlap subset of Factify.","lead":"This paper builds a graph-based detector for out-of-context misinformation: it finds web pages that contain the image, turns those pages and the caption into entity-relation graphs, and uses a graph neural network to compare them. The authors report accuracy above 90% on a subset of a fact-checking benchmark, beating zero-shot LLMs that were given the same evidence as text.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 93.05% headline is measured on EVAL_COMMON (n=461), a subset defined by zero-shot LLMs agreeing the retrieved evidence is sufficient; on the full EVAL_ALL (n=1145) accuracy is 83.05%, so the central claim is not supported as stated.","rationale":"The reader's strongest_claim already notes that the 93.05% is on EVAL_COMMON and the full-set score is 83.05%; my analysis converges on that point and adds the mechanism: EVAL_COMMON is selected for evidence sufficiency via LLM abstention, so it is not a representative slice of the evaluation set. The reader's weakest_assumption focuses on evidence-retrieval relevance; while that is a real limitation (and is acknowledged in the paper), the most load-bearing issue for the advertised central claim is the choice of evaluation subset and the asymmetric LLM prompt that omits the image. For those reasons I partially agree with the reader rather than fully agreeing on the weakest assumption. The paper's contribution is still a plausible baseline with useful graph data, and the flaws are fixable by re-evaluating on EVAL_ALL with task-equivalent baselines and confidence intervals. I therefore keep the reader's CONDITIONAL verdict unchanged rather than moving to ACCEPT or REJECT.","tokens_in":12102,"tokens_out":6784,"duration_ms":54125,"concrete_test":"Re-run the full Table 2 evaluation on EVAL_ALL (n=1145) with the same trained EGMMG and the same LLM prompts augmented to include the claim image, and compute bootstrap 95% confidence intervals for the accuracy difference EGMMG minus best LLM. Also report EGMMG's accuracy on the 684-sample complement of EVAL_COMMON. If the EVAL_ALL margin is not statistically significant, or if the complement accuracy is far below 93.05%, the abstract's headline claim should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim ('93.05% accuracy on the evaluation set, outperforming the second-best LLM by 2.82%') is read off Table 2's EVAL_COMMON column. EVAL_COMMON is not a random holdout: it is the intersection of samples for which three zero-shot LLMs, prompted with the retrieved evidence text and the claim, chose not to abstain (Section 4.1). Abstention is triggered by insufficient or ambiguous evidence, so EVAL_COMMON is selected for cases where the reverse-image-search evidence pipeline already produced text that LLMs could use to decide. EGMMG never abstains and is supervised on Factify-style graphs, so reporting only this subset conditions the headline on evidence-retrieval success and on LLM-easy samples. On the full evaluation set (EVAL_ALL, n=1145), EGMMG scores 83.05% and GPT-4o 68.72%; the margin is large but the headline 93.05%/2.82% is not the evaluation-set result. A second confound is that the LLM prompt (Figure 5) supplies only EVIDENCE text and CLAIM text, not the image, while the evidence itself is obtained via image-based reverse search; the Section 6 statement that 'all methods have access to the same amount of data' is therefore not literally true. The published comparison thus does not isolate the graph-based architecture as the source of the gain.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EGMMG, a GNN-based detector for out-of-context multimodal misinformation. It first retrieves web evidence for an image via reverse image search, filters the retrieved documents by CLIP similarity, constructs entity-relation graphs from the evidence text and from the caption, and trains an attention-based GNN to classify image-caption pairs. Experiments on Factify, COSMOS, and MMFakeBench compare the method with zero-shot LLMs; the paper reports 93.05% accuracy on the EVAL_COMMON subset (n=461) and 83.05% on EVAL_ALL (n=1145), with ablations and generalization tests on additional datasets.","tokens_in":12423,"tokens_out":6069,"duration_ms":44732,"significance":"If the result holds, the paper is a useful demonstration that a small task-specific GNN fed with retrieved textual evidence can compete with frontier LLMs for out-of-context misinformation detection, and the evidence-hydration pipeline is a reusable resource. The strengths are the use of multiple public datasets, a clearly described retrieval-and-graph construction pipeline, and an honest limitations section. However, the headline accuracy is reported on a post hoc subset defined by LLM abstention, no confidence intervals or multiple runs are given for the main table, and the comparison lacks a trained non-graph supervised baseline, so the significance as stated is not yet fully established.","major_comments":[{"comment":"The headline claim of '93.05% detection accuracy on the evaluation set' is read from the EVAL_COMMON column (n=461), but EVAL_COMMON is not the evaluation set; it is the subset of EVAL_ALL (n=1145) on which the zero-shot LLMs did not abstain. Because abstention is triggered by insufficient or ambiguous evidence, EVAL_COMMON is selected toward samples that the LLMs already judged answerable, and EGMMG never abstains. The full-set accuracy is 83.05% on EVAL_ALL, which still beats GPT-4o by a large margin, so the central direction may survive, but the abstract's specific number and the 2.82% margin are not the evaluation-set result. Please report EVAL_ALL as the primary evaluation set and present EVAL_COMMON only with an explicit statement of the selection mechanism and its implications.","section":"Abstract, Section 4.1, Table 2"},{"comment":"The definition of EVAL_COMMON lists ESSonnet ∩ ESGpt4oMini ∩ ESHaiku, omitting GPT-4o, while the surrounding text says the set contains samples that all models considered answerable. This matters because GPT-4o is the strongest LLM baseline and the one closest to EGMMG on EVAL_COMMON. In addition, Table 2 reports no confidence intervals, bootstrap estimates, or multiple training seeds for the main comparison; with n=461, the 2.82% accuracy margin corresponds to roughly 13 samples, so a significance test or interval is needed before claiming a reliable margin over GPT-4o.","section":"Section 4.1, EVAL_COMMON definition"},{"comment":"The evidence-retrieval filter removes a large fraction of each dataset, e.g., Factify drops from 14,000 to 4,945 samples and the Factify validation set from 3,000 to 1,145. The paper does not check whether retained and discarded samples are exchangeable in label distribution, claim type, or image properties. If successful reverse-image search correlates with the nature of the claim or with veracity, then the evaluation is on a biased subpopulation and the generalization claims in Section 4.2 are not supported. Please report class balance and available content statistics before and after filtering, or justify why selection bias is not a concern.","section":"Section 3.1, Table 1"},{"comment":"The comparison is between EGMMG, which is trained on the dataset labels, and LLMs used in a zero-shot setting, so the experiment does not isolate the graph-based architecture as the source of the observed gain. A supervised non-graph baseline (e.g., a fine-tuned text classifier over the same retrieved evidence) is needed to support the claim that the graph structure itself is responsible for the improvement. Relatedly, Section 6's statement that 'all methods have access to the same amount of data (in text or graph format)' is not literally accurate: LLMs receive raw evidence text and the claim, EGMMG receives a lossy graph projection of the evidence text, and neither receives the image directly. Please add the missing supervised text baseline or temper the architecture-specific claim.","section":"Sections 4, 5, and 6"}],"minor_comments":[{"comment":"There are typos: 'Facitify' should be 'Factify', and 'postitive' should be 'positive'.","section":"Section 4.1"},{"comment":"The text introduces 'top-k evidences' and then fixes k=7; please make the notation consistent and state whether any sensitivity analysis was performed on the choice of k.","section":"Section 3.1"},{"comment":"The number of runs differs between the 384-dim setting (four runs) and the 768-dim setting (three runs); please state why and report mean ± standard deviation instead of raw run lists.","section":"Table 6"},{"comment":"Please clarify the relationship between the Factify split used to train the model for Table 2 (EVAL_ALL/EVAL_COMMON) and the 'Factify 85:15' split used for Tables 3 and 6, since both report accuracy on Factify-derived data.","section":"Tables 2 and 3"},{"comment":"For EVAL_ALL, Sonnet abstained on 64 samples even when prompted to answer strictly; the paper should state explicitly how abstained outputs are treated in the reported accuracy and F1 scores.","section":"Figure 5 and Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is promising and the underlying direction is worth publishing after revision, but the abstract oversells the EVAL_COMMON result and the statistical support is thin. Please verify whether the omission of GPT-4o from the EVAL_COMMON formula is an error, since that definition is load-bearing for the headline number."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my read on arXiv:2505.18221. The core contribution is a plausible evidence-grounded pipeline for OOC detection: reverse image search, evidence and claim graphs, attention-based GNN with cross-attention. The authors build new graph datasets from Factify, COSMOS, and MMFB, and report ablations showing component contributions. The 10M-parameter model is efficient, and the paper is honest in its limitations section.\n\nThe problem is the headline. The abstract claims 93.05% accuracy on 'the evaluation set' and outperforming the second-best LLM by 2.82%. Table 2 shows that number is on EVAL_COMMON (n=461), a subset where all zero-shot LLMs agreed they had enough evidence and answered. On the full EVAL_ALL (n=1145), accuracy is 83.05% — still 14 points above GPT-4o, but that's not what the abstract says. EVAL_COMMON is not a random holdout; it selects for cases where the evidence retrieval already produced text LLMs could use. EGMMG never abstains and is supervised, so the comparison there is not apples-to-apples.\n\nThere are also smaller issues: no confidence intervals or multiple seeds for the main table; the LLM baselines get only text, not the image, so the claim that 'all methods have access to the same data' is misleading; and the assumption that reverse image search returns relevant context is load-bearing, with a large chunk of images discarded (Factify drops from 14,000 to 4,945). The authors acknowledge these limitations, which is to their credit, but the evaluation design still overstates the result.\n\nWhat's genuinely useful: the graph construction rules and the resulting datasets could be reused, and the approach is a legitimate baseline for the 'contextualize first' line of work. The paper deserves a serious referee, but I'd ask for the headline to be corrected to the full-set number, plus variance estimates and a supervised baseline comparison before acceptance. Would I cite it? Not in my next 12 months, but I'd point students to it as a solid baseline example.","headline":"A useful baseline and new datasets, but the headline accuracy is measured on a cherry-picked LLM-abstention subset, not the full evaluation set.","tokens_in":12962,"tokens_out":1984,"would_cite":false,"duration_ms":18177,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A graph attention network that compares a caption's claim graph with an evidence graph built from reverse-image-search web pages outperforms frontier LLMs at out-of-context misinformation detection on the Factify benchmark.","keywords":["out-of-context misinformation","multimodal misinformation detection","graph neural networks","cross-graph attention","evidence graphs","reverse image search","Factify dataset","LLM comparison"],"falsifier":"Take the 1,855 Factify validation samples the pipeline discarded for lack of web matches, obtain ground-truth labels, build graphs for any that have partial matches, and run EGMMG: if accuracy on that group is near chance, the reported 83.05% to 93.05% results only characterize images with online evidence. Alternatively, have annotators judge whether the top-7 retrieved pages for a random sample of 100 pairs actually discuss the entities and event named in the caption; if most are irrelevant, the evidence graphs do not carry the context the method relies on.","tokens_in":11932,"feed_emoji":"🔎","tokens_out":10066,"duration_ms":90227,"temperature":0.7,"pith_summary":"This paper sets out to show that out-of-context misinformation—real images paired with false captions—can be detected by grounding the image in online textual evidence and comparing two graphs rather than by relying on an LLM's memory. The authors build an evidence graph from the top-ranked web pages that reuse the image and a claim graph from the caption, then train a small graph attention network to score whether the caption is consistent with the evidence. On the Factify validation subset the method reaches 93.05% accuracy on the 461 samples all LLMs could answer and 83.05% on all 1,145 samples, beating the best LLM by 2.82 points on the common set with about 10 million parameters. If this holds, task-specific graph detectors are a cheaper, less hallucination-prone alternative to LLM-based fact-checking.","feed_headline":"Small graph model beats LLMs at out-of-context fake news","feed_subtitle":"Grounding images in web evidence, the 10M-parameter model scores 93.05% on the shared test set.","key_machinery":"The load-bearing object is the pair of graphs produced by the EGMMG pipeline. A reverse image search collects web pages containing the image; after ranking by CLIP-based cosine similarity between page text and image, the top 7 pages are concatenated into the evidence. A dependency parser extracts entities, events, and locations as nodes, with rule-based edges such as PERFORMS, EXPERIENCES, TARGETS, LOCATED_IN, HAS_STATE, and SAME_AS derived from verb and dependency patterns; the caption is turned into the claim graph the same way. The classifier initializes node features as a learnable weighted sum of BERT label embeddings and neighborhood structural features, passes messages with transformer-style graph convolution layers, scores node importance, and computes cross-attention with claim nodes as queries and evidence nodes as keys and values. Global mean pooling concatenates evidence, claim, and attended representations into a sigmoid veracity score, so the model's decision depends directly on the consistency between the two graphs.","core_discovery":"The central claim is that the relative agreement between the claim graph and the evidence graph carries the veracity signal, and that a graph attention model reading these two graphs can outperform frontier LLMs given the same retrieved evidence. The paper reports 0.9305 accuracy and 0.9219 F1 on EVAL_COMMON and 0.8305 accuracy and 0.8455 F1 on EVAL_ALL, compared with GPT-4o's 0.9023 and 0.8936 on EVAL_COMMON and 0.6872 and 0.6209 on EVAL_ALL, plus cross-dataset results of 0.8248 on Factify, 0.7750 on COSMOS, and 0.7100/0.6823 on MMFakeBench under 85:15 splits. The evidence graph supplies the context that the caption must be checked against, while the claim graph encodes the caption's own entities and relations. On the paper's account, the advantage comes from doing this contextualization explicitly instead of relying on parametric knowledge.","pith_inferences":["The gap between 93.05% on the 461-sample common set and 83.05% on all 1,145 samples suggests the comparison with LLMs is most favorable on samples every LLM found answerable; deployment on arbitrary image-caption pairs may see the lower number.","Because images without any reverse-image-search match are dropped (Factify shrinks from 14,000 to 4,945 training samples and 3,000 to 1,145 validation samples), the reported accuracy may not hold for social-media posts whose images are not indexed online, and a selection-bias check on discarded samples would tell.","A direct testable extension is to feed the same top-7 evidence documents to an LLM in plain text and compare with the graph encoding, isolating whether the graph structure or the retrieval step is responsible for the gain.","If the evidence premise is right, adding image-derived entities to the evidence graph could recover context for the discarded no-match cases, since the paper notes it currently does not use the image content directly."],"forward_implications":["EGMMG's 93.05% accuracy on EVAL_COMMON and 83.05% on EVAL_ALL imply that explicit evidence grounding can beat zero-shot LLM prompting on the same evidence, at least on this benchmark.","At 10.7 million parameters running on a single T4, the approach is cheap enough to deploy at scale where API-based LLM fact-checking would be too costly.","The cross-dataset results indicate the pipeline transfers without per-dataset engineering, though with weaker performance on the more synthetic MMFakeBench sets than on Factify and COSMOS.","Ablation results show that weighted node features and 768-dimensional label embeddings matter most, while edge features reduce accuracy, pointing to node semantics rather than edge types as the informative part of the graph."],"supporting_citations":[{"why":"Supplies the Factify dataset and its labels, the primary benchmark for evaluating the method and comparing against LLMs.","marker":"Mishra et al. (2022)"},{"why":"Motivates gathering external online knowledge about both image and caption to detect out-of-context misinformation.","marker":"Abdelnabi et al. (2022)"},{"why":"Presents SNIFFER, an LLM-based out-of-context detector whose approach and performance frame the comparison.","marker":"Qi et al. (2024)"},{"why":"Provides the COSMOS out-of-context dataset used for generalization testing.","marker":"Aneja et al. (2023)"},{"why":"Provides MMFakeBench, the mixed-source benchmark used for generalization testing.","marker":"Liu et al. (2024)"},{"why":"Supplies the CLIP-based text and image embeddings used to rank evidence documents by cosine similarity.","marker":"Reimers and Gurevych (2019)"},{"why":"Provides the dependency parsing and named entity recognition that extract graph nodes and relations from evidence and claim text.","marker":"Honnibal et al. (2020)"},{"why":"Supplies BERT embeddings used as the language-model component of node features.","marker":"Devlin et al. (2019)"},{"why":"Introduces the graph attention mechanism on which the message-passing layers are built.","marker":"Veličković et al. (2018)"},{"why":"Introduces TransformerConv, the convolution variant selected for the final model.","marker":"Shi et al. (2021)"}],"fun_headline_variants":["Graph GNN beats GPT-4o on out-of-context misinformation","93% accuracy: graph attention stops repurposed-image lies","Explicit evidence graph beats LLM param memory on fake news","Small GNN uses web evidence to beat LLMs at fake image+text","Graph evidence check beats GPT-4o on out-of-context fakes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the web pages returned by reverse image search describe the true context of the image, and that ranking them by CLIP similarity and keeping the top 7 preserves the facts needed to judge the caption; it also assumes the images dropped for lack of matches are no different from the ones kept.","fun_headline_variants_meta":{"raw":{"variants":["Graph GNN beats GPT-4o on out-of-context misinformation","93% accuracy: graph attention stops repurposed-image lies","Explicit evidence graph beats LLM param memory on fake news","Small GNN uses web evidence to beat LLMs at fake image+text","Graph evidence check beats GPT-4o on out-of-context fakes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001058,"raw_usage":{"total_tokens":4450,"prompt_tokens":966,"completion_tokens":3484,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":3390}},"tokens_in":582,"tokens_out":3484,"duration_ms":19701,"temperature":1.0,"reasoning_tokens":3390,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:42:43.911595+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the 1,855 Factify validation samples the pipeline discarded for lack of web matches, obtain ground-truth labels, build graphs for any that have partial matches, and run EGMMG: if accuracy on that group is near chance, the reported 83.05% to 93.05% results only characterize images with online evidence. Alternatively, have annotators judge whether the top-7 retrieved pages for a random sample of 100 pairs actually discuss the entities and event named in the caption; if most are irrelevant, the evidence graphs do not carry the context the method relies on.","supporting_citations":[],"review_version":1}