Pith. sign in

REVIEW 5 major objections 7 minor 10 cited by

SIDA: Social Media Image Deepfake Detection, Localization and Explanation with Large Multimodal Model

T0 review · 5 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Pith's one-line read This paper claims that one large multimodal model can detect whether a social-media image is real, fully synthetic, or tampered, draw a pixel-level mask of the tampered region, and explain its verdict in text, and that the 300K-image…

desk verdict The dataset is the real contribution; the localization ground truth needs an alignment check before the headline claims are accepted. read the letter →

arxiv 2412.04292 v3 pith:QGWBW35F submitted 2024-12-05 cs.CV cs.AI

classification cs.CVcs.AI
keywords deepfakedetectiontamperlocalizationlargemultimodalmodelsvision-languagemodelsocialmediaimagessyntheticimagedatasetforensicsexplainableAI
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a single large multimodal model can do what previously required separate systems: decide whether a social-media image is real, fully synthetic, or tampered; draw a pixel mask over the tampered region; and say in plain language what cues led to that verdict. To test this, the authors build SID-Set, a 300,000-image benchmark of real, FLUX-generated, and locally tampered photos drawn from COCO and Flickr30k, with masks and textual explanations. They then introduce SIDA, a vision-language assistant built on LISA with two added tokens, and , whose hidden features drive a detection head and a mask decoder. The reported result is that SIDA outperforms or matches existing detectors and localization methods on SID-Set and generalizes to the external DMimage benchmark, at 93.5% overall accuracy and 87.3 localization AUC.

What carries the argument

The load-bearing mechanism is the token-head architecture borrowed from reasoning-segmentation vision-language models: two added tokens, <DET> and <SEG>, whose last-layer hidden states are extracted and fed to a lightweight detection head and, via a single multihead-attention residual module, to a mask decoder conditioned on the frozen image encoder's features. On the data side, the machinery is the four-stage tampering pipeline: GPT-4o extracts objects from captions, Language-SAM produces masks on the original images, replacement dictionaries define object swaps and attribute changes, and Latent Diffusion regenerates the image with the altered caption. The masks from stage two are the training ground truth for localization.

What would settle it

Take a random sample of SID-Set tampered images, have human annotators draw the actually changed pixels against the original photo, and recompute SIDA's localization AUC, F1, and IoU; if scores drop materially below the reported 87.3, 73.9, and 43.8, the masks used as ground truth did not match the true edits.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that detection, localization, and explanation are not competing tasks but can be fused in one vision-language model by letting the detection representation condition the segmentation features. The model adds <DET> and <SEG> tokens to the vocabulary; the <DET> hidden state classifies the image into real, fully synthetic, object-tampered, or part-tampered, and when tampering is found, a multihead-attention module uses detection features as queries over segmentation features, then a frozen-image-encoder decoder produces the mask. The same model generates a textual justification. Trained end-to-end on SID-Set with cross-entropy for detection, BCE+DICE for masks, and cross-entropy for text, SIDA-7B and SIDA-13B reach 93.5-93.6% overall accuracy and set the top localization scores among compared methods.

Load-bearing premise

The central load-bearing premise is that Language-SAM masks computed on the original images align with the regions Latent Diffusion actually altered, so the reported localization scores measure true tampered-region accuracy rather than agreement with approximate labels.

Editorial extensions

If this is right

  • A single SIDA-style model can replace separate detector and localization pipelines for social-media images, reducing the deployment cost of content moderation.
  • Because SIDA also outputs textual explanations, moderation systems gain a human-readable audit trail for every decision, not just a score.
  • SID-Set provides a training and evaluation ground for a wider variety of fakes than face-only datasets, covering fully synthetic and object- or part-tampered images.
  • Fine-tuning on SID-Set improves several prior detectors' performance, indicating the dataset is a useful transfer resource beyond SIDA itself.
  • The robustness experiments suggest SIDA tolerates JPEG compression, resizing, and Gaussian noise without explicit augmentation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If SID-Set's masks prove accurate, it could become the standard benchmark for non-facial image forensics, much as face-centric datasets were a decade ago.
  • The reliance on one generator for synthetic images and one diffusion model for tampering means SID-Set may bake in artifacts of those two generators; a testable next step is adding images from other generators and checking whether SIDA's advantage shrinks.
  • The explanation head is trained only on 3,000 GPT-4o descriptions, so SIDA's explanations likely reflect GPT-4o's judgment criteria rather than SIDA's own internal cues; a direct test would compare model explanations with its attention or occlusion maps.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The manuscript introduces SID-Set, a 300K-image benchmark for social-media deepfake detection, localization, and explanation, and proposes SIDA, a LISA-based large multimodal model that adds <DET> and <SEG> tokens to perform three-way detection (real / fully synthetic / tampered), pixel-level tamper localization, and textual explanation. The authors report state-of-the-art detection results against eight baselines on SID-Set (Table 2), the best localization numbers among four comparison methods (Table 3), robustness to JPEG compression, resizing, and Gaussian noise (Table 4), and strong performance on the DMimage benchmark (Table 5). The dataset, model, and code are promised for release.

Significance. If the benchmark's localization labels are valid, SID-Set would be the largest social-media-focused deepfake dataset with masks and explanations, and SIDA would be the first single VLM to unify three-way detection, pixel-level localization, and textual explanation. The paper has notable strengths: extensive retraining of detection baselines on SID-Set, a perturbation robustness study, qualitative failure-case analysis, and a commitment to release the dataset, code, and model. The significance is moderated by unresolved questions about the alignment between Language-SAM masks and the regions actually altered by Latent Diffusion, by an ambiguous evaluation protocol in Table 2, and by an unfair cross-benchmark comparison in Table 5.

major comments (5)
  1. [Section 3.2, Stages 2-4 and Table 3] The construction pipeline does not state whether the Language-SAM mask from Stage 2 is used as an inpainting or conditioning mask in the Stage 4 Latent Diffusion regeneration, nor does it verify that the pixels actually changed by the diffusion edit coincide with the Stage-2 mask. Because the Stage-2 masks are used both as training targets for Lmask in Eq. (5) and as inputs to the GPT-4o explanation prompts (Appendix Figure 14), any drift between the edited region and the mask makes the reported localization AUC/F1/IoU and the learned explanations measures of agreement with noisy labels rather than true tampering accuracy. Please specify the conditioning in Stage 4 and provide a quantitative alignment check, such as mean IoU between Stage-2 masks and pixel-wise differences of original versus regenerated images, or a human-verified sample.
  2. [Table 2 caption] The caption says parenthesized values are "performance changes after fine-tuning the models using the SID-Set on the validation set." If the models were fine-tuned on the validation set and evaluated on the same validation set, the comparison is invalid because of train/test leakage; if they were fine-tuned on the training split and evaluated on the validation split, the wording should be corrected. Please report results on the held-out test split and specify the exact protocol for each baseline.
  3. [Section 5.4, Table 5] SIDA is compared with baselines using their original pre-trained weights after SIDA has been trained on 300K SID-Set images, so the comparison is not apples-to-apples. The proper generalization test would fine-tune all baselines on the SID-Set training split and report test-split numbers, or evaluate all methods in a zero-shot setting without SID-Set training. As written, the "superior performance" on DMimage may reflect a training-data advantage rather than a genuine generalization advantage.
  4. [Section 4.2 / Section 5 Implementation Details / Section 5.5] The loss-weight specification is internally inconsistent. Implementation Details says the localization loss weights lambda_bce and lambda_dice are both set to 1.0, while Section 5.5 states "for our experiments, we set lambda_det to 1, lambda_bce to 2.0, and lambda_dice to 0.5," and Table 7 lists yet another configuration. Please state unambiguously which configuration produced Tables 2-4 and report the corresponding validation or test numbers; otherwise the experimental results are not reproducible.
  5. [Tables 2-4 and Section 5.5] The detection, localization, and ablation results are reported as single-run point estimates without standard deviations or the number of seeds. Given that the stated margins in Table 2 are large but the ablation differences in Table 6 are small (e.g., ACC 93.5 vs 91.1 for removing the attention module), please report mean and standard deviation over at least three runs, or otherwise justify that the reported differences are statistically meaningful.
minor comments (7)
  1. [Table 2 caption] The "Overall Acc and F1" are described as the average of the values from the three classification categories; this is a macro average, not an overall accuracy, and the caption should say so explicitly to avoid misleading readers.
  2. [Section 4.2 / Appendix Figures 7 and 14] The 3,000 explanation ground-truth texts are generated by GPT-4o and reviewed by five experts, but the paper reports no quantitative evaluation of explanation quality, only qualitative examples in Section 5.6. Moreover, the prompt for tampered images in Appendix Figure 14 explicitly provides the mask as input, so the explanation stage is conditioned on the same localization labels whose validity is in question; this limitation should be acknowledged.
  3. [Abstract and Section 3] The paper describes SID-Set as having "comprehensive annotations," but explanations are available for only 3,000 of the 300K images; the wording should be tempered to distinguish the mask annotations from the explanation annotations.
  4. [Section 5.5 and Table 7] The ablation study for training weights reports only two configurations and does not compare against the Implementation Details configuration of lambda_bce=lambda_dice=1.0; please include that configuration or justify its omission.
  5. [Throughout] There are several minor typos and naming inconsistencies, including "CnnSpott" in Table 2 versus "CnnSpot" in the text, "HIFI-Net" versus "HiFi-IFDL," and "textural descriptions" versus "textual descriptions." Please proofread for consistency.
  6. [Section 3.2 and Appendix D] The paper says MagicBrush images were "integrated" and "filtered," but provides no details on the filtering criteria or the proportion of MagicBrush images in the final dataset; please specify this, and also cite the OpenImages V7 dataset rather than only a URL.
  7. [Appendix E] The expert evaluation is described only qualitatively; please report at least the number of images flagged and excluded, and ideally inter-annotator agreement, so that the quality-control claims can be assessed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SIDA's detection and localization results follow from a standard supervised train/test split, and the self-citations are not load-bearing.

full rationale

The central detection and localization claims are derived from training SIDA on SID-Set with a held-out test split (Section 5, 7:1:2 ratio) and from comparison against external baselines, including the DMimage benchmark (Table 5), so they are not equivalent to any fitted input by construction. The localization ground truth is created by Language-SAM masks on original COCO/Flickr30k images (Section 3.2, Stage 2), and the paper does not state that the Stage-4 Latent Diffusion regeneration is guaranteed to alter exactly those pixels; this is a potential label-noise and validity threat rather than a circular derivation. The explanation stage is trained to match GPT-4o-generated descriptions (Section 4.2, Eq. 6), so the generated text is an imitation of an external label source and does not independently evidence SIDA's internal criteria; this is an evaluation limitation, not a circular reduction of the detection result. The only self-citations ([33], [70], [84]) appear in related-work enumerations of LMM/LISA-style models and are not load-bearing for the paper's claims. No fitted parameter is relabeled as a prediction, and no uniqueness theorem or prior-work ansatz is used to force the architecture choice.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central empirical claims rest on two constructed resources: the SID-Set labels and the GPT-4o explanations. No new physical entities are postulated, and the only numerical knobs are standard training hyperparameters and loss weights. The main risk is that the benchmark's ground-truth masks and explanations are generated by automated tools whose errors become the target the model learns, so 'superior performance' may partly mean better imitation of the annotation pipeline rather than better forensic truth.

free parameters (2)
  • Training loss weights = lambda_det=1.0, lambda_bce=2.0, lambda_dice=0.5 (second stage adds lambda_txt=1.0)
    Set by hand and validated via ablation in Table 7; the reported detection accuracy depends on these choices.
  • LoRA and optimization settings = LoRA alpha=16, dropout=0.05, LR=1e-4, batch size=2, grad accumulation=10
    Standard hyperparameters chosen without derivation; they affect the trained model but are not the scientific claim. Included for completeness.
assumptions (4)
  • domain assumption Language-SAM masks on the original image coincide with the pixels changed by Latent Diffusion in Stage 4.
    Used as ground truth for localization training and evaluation (Section 3.2); if false, localization metrics are measured against misaligned labels.
  • domain assumption OpenImages V7, COCO, and Flickr30k captions adequately represent the social media image distribution.
    The benchmark's 'social media' framing rests on this selection (Section 3.2); no social-media provenance metadata is provided.
  • domain assumption GPT-4o-generated descriptions, after expert review, are valid ground truth for explaining the model's judgment criteria.
    Used to fine-tune the text branch (Section 4.2, Appendix D); the paper does not verify that these explanations correspond to SIDA's internal detection cues.
  • domain assumption FLUX-generated images are representative of the deepfakes that appear on social media.
    All 100K synthetic images come from a single generator; the authors acknowledge possible data skew in Section 6.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SIDA: Social Media Image Deepfake Detection, Localization and Explanation with Large Multimodal Model." pith.science (2026). https://pith.science/paper/QGWBW35F

@misc{pith2026241204292,
  author       = {Pith},
  title        = {Pith review of: SIDA: Social Media Image Deepfake Detection, Localization and Explanation with Large Multimodal Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QGWBW35F}},
  note         = {Machine review of arXiv:2412.04292}
}
read the original abstract

The rapid advancement of generative models in creating highly realistic images poses substantial risks for misinformation dissemination. For instance, a synthetic image, when shared on social media, can mislead extensive audiences and erode trust in digital content, resulting in severe repercussions. Despite some progress, academia has not yet created a large and diversified deepfake detection dataset for social media, nor has it devised an effective solution to address this issue. In this paper, we introduce the Social media Image Detection dataSet (SID-Set), which offers three key advantages: (1) extensive volume, featuring 300K AI-generated/tampered and authentic images with comprehensive annotations, (2) broad diversity, encompassing fully synthetic and tampered images across various classes, and (3) elevated realism, with images that are predominantly indistinguishable from genuine ones through mere visual inspection. Furthermore, leveraging the exceptional capabilities of large multimodal models, we propose a new image deepfake detection, localization, and explanation framework, named SIDA (Social media Image Detection, localization, and explanation Assistant). SIDA not only discerns the authenticity of images, but also delineates tampered regions through mask prediction and provides textual explanations of the model's judgment criteria. Compared with state-of-the-art deepfake detection models on SID-Set and other benchmarks, extensive experiments demonstrate that SIDA achieves superior performance among diversified settings. The code, model, and dataset will be released.

Figures

Figures reproduced from arXiv: 2412.04292 by the authors.

Figure 1
Figure 1. The framework comparisons. Existing deepfake meth [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. SID-Set examples. The 1st row is the synthetic images, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Tampered image generation pipeline: It consists of four stages—extracting objects from captions using GPT-4o, obtaining object [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: Examples of tampered images. (Zoom in to view) [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: The pipeline of SIDA: Given an image xi and the corresponding text input xt, the last hidden layer for the <DET> token provides the detection result. If the detection result indicates a tampered image, SIDA extracts the <SEG> token to generate masks for the tampered re…
Figure 6
Figure 6. Figure 6: Visual results of SIDA on tampered images. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Generating ground truth descriptions for three different types of inputs. [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Visual comparison of SIDA with other localization methods. Both approaches were fine-tuned on the SID-Set for this evaluation. [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: Visual examples of SIDA. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: More Visual examples of SIDA. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Failure cases of SIDA. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Prompts for real images. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: Prompts for fully synthetic images. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 14
Figure 14. Figure 14: Prompts for tampered images. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]
Figure 15
Figure 15. Figure 15: Example output from SIDA when analyzing a real image. [PITH_FULL_IMAGE:figures/full_fig_p019_15.png]
Figure 16
Figure 16. Figure 16: Example output from SIDA when analyzing a fully synthetic image. [PITH_FULL_IMAGE:figures/full_fig_p020_16.png]
Figure 17
Figure 17. Figure 17: Example output from SIDA when analyzing a tampered image. [PITH_FULL_IMAGE:figures/full_fig_p021_17.png]
Figure 18
Figure 18. Figure 18: Object replacement directories for SID-Set. [PITH_FULL_IMAGE:figures/full_fig_p022_18.png]
Figure 19
Figure 19. Figure 19: Attribute modification directories for SID-Set. [PITH_FULL_IMAGE:figures/full_fig_p022_19.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DocShield: Towards AI Document Safety via Evidence-Grounded Agentic Reasoning

    cs.CV 2026-04 unverdicted novelty 7.0 of 10

    DocShield presents a new agentic reasoning framework using Cross-Cues-aware Chain of Thought to detect, localize, and explain text-centric forgeries in documents, with reported F1 gains of 41.4% over specialized metho...

  2. XPlainVerse: A Million-Scale Benchmark for Explainable Deepfake Detection

    cs.CV 2026-07 conditional novelty 6.5 of 10

    A million-scale deepfake benchmark with Edit-Check filtering, dual expert/lay explanations, and EntityScore/EvidenceScore shows fine-tuned detectors collapse under generator shift while surface fluency remains.

  3. Explaining AI-Image Detection: What the Heatmap Actually Shows

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Compression history, not synthesis, explains a near-perfect AI-image detector, and no tested heatmap causally explains its verdict.

  4. LaP-Forensics: Latent-Pixel Consistency Guided Multimodal Reasoning for Deepfake Detection

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A dual-stream deepfake forensic model that adds DDIM reconstruction residuals to RGB features improves artifact localization and cross-generator detection in evaluations, with honest caveats about text faithfulness.

  5. Detecting Text Manipulation in Images using Vision Language Models

    cs.CV 2025-09 conditional novelty 6.0 of 10

    In zero-shot benchmarks, GPT-4o outperforms open-source VLMs and specialized manipulation detectors on text tampering detection in scene images and fantasy ID documents.

  6. AIGI-Holmes: Towards Explainable and Generalizable AI-Generated Image Detection via Multimodal Large Language Models

    cs.CV 2025-07 conditional novelty 6.0 of 10

    AIGI-Holmes combines visual expert pretraining, SFT on explanation data, and direct preference optimization to deliver human-verifiable explanations and top detection accuracy on unseen AI generators.

  7. Omni-IML: Towards Unified Image Manipulation Localization

    cs.CV 2024-11 conditional novelty 6.0 of 10

    One jointly trained image-manipulation-localization model attains top average scores on natural, document, face, and scene-text forensics benchmarks and gains artifact-description ability from a new GPT-4o-annotated dataset.

  8. DFBench: Benchmarking Deepfake Image Detection Capability of Large Multimodal Models

    cs.CV 2025-06 conditional novelty 5.0 of 10

    DFBench adds a 540,000-image benchmark with 12 modern generators, partial edits, and distorted real images, and its three-model LMM ensemble, MoA-DF, reaches near-perfect recall on its own test split.

  9. Multiverse Through Deepfakes: The MultiFakeVerse Dataset of Person-Centric Visual and Conceptual Manipulations

    cs.MM 2025-06 conditional novelty 5.0 of 10

    MultiFakeVerse provides 845,286 person-centric images edited through VLM-generated instructions; state-of-the-art deepfake detectors and human observers misclassify a large fraction of them.

  10. Survey on AI-Generated Media Detection: From Non-MLLM to MLLM

    cs.CV 2025-02 unverdicted novelty 3.0 of 10

    A survey organizing AI-generated media detection into Non-MLLM and MLLM based methods, with task and benchmark taxonomies.

Reference graph

Works this paper leans on

90 extracted references · 75 canonical work pages · cited by 10 Pith papers

  1. [1]

    Kandinsky 3.0 technical report

    Vladimir Arkhipkin, Andrei Filatov, Viacheslav Vasilev, Anastasia Maltseva, Said Azizov, Igor Pavlov, Julia Aga- fonova, Andrey Kuznetsov, and Denis Dimitrov. Kandinsky 3.0 technical report. Arxiv, 2023. 4

  2. [2]

    Xiuli Bi, Bo Liu, Fan Yang, Bin Xiao, Weisheng Li, Gao Huang, and Pamela C. Cosman. Detecting generated images by real images only. Arxiv, 2023. 6, 7, 8, 9

  3. [3]

    Bird and Ahmad Lotfi

    Jordan J. Bird and Ahmad Lotfi. CIFAKE: image classifica- tion and explainable identification of ai-generated synthetic images. IEEE Access, 2024. 2

  4. [4]

    End-to-end reconstruction- classification learning for face forgery detection

    Junyi Cao, Chao Ma, Taiping Yao, Shen Chen, Shouhong Ding, and Xiaokang Yang. End-to-end reconstruction- classification learning for face forgery detection. In CVPR,

  5. [5]

    Antifakeprompt: Prompt-tuned vision-language models are fake image detectors

    You-Ming Chang, Chen Yeh, Wei-Chen Chiu, and Ning Yu. Antifakeprompt: Prompt-tuned vision-language models are fake image detectors. Arxiv, 2023. 3, 6, 7, 8, 9

  6. [6]

    Self-supervised learning of adversarial exam- ple: Towards good generalizations for deepfake detection

    Liang Chen, Yong Zhang, Yibing Song, Lingqiao Liu, and Jue Wang. Self-supervised learning of adversarial exam- ple: Towards good generalizations for deepfake detection. In CVPR, 2022. 3

  7. [7]

    Kankanhalli

    Harry Cheng, Yangyang Guo, Tianyi Wang, Liqiang Nie, and Mohan S. Kankanhalli. Diffusion facial forgery detection. In Proceedings of the 32nd ACM International Conference on Multimedia, 2024. 1, 2

  8. [8]

    On the de- tection of synthetic images generated by diffusion models

    Riccardo Corvi, Davide Cozzolino, Giada Zingarini, Gio- vanni Poggi, Koki Nagano, and Luisa Verdoliva. On the de- tection of synthetic images generated by diffusion models. In ICASSP, 2023. 7, 8

Show all 90 references
  1. [9]

    Diffusion models in vision: A survey

    Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. Diffusion models in vision: A survey. IEEE TPAMI, 2023. 1

  2. [10]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning. In NeurIPS, 2023. 3

  3. [11]

    Mvss-net: Multi-view multi-scale supervised networks for image manipulation detection

    Chengbo Dong, Xinru Chen, Ruohan Hu, Juan Cao, and Xirong Li. Mvss-net: Multi-view multi-scale supervised networks for image manipulation detection. IEEE TPAMI,

  4. [12]

    CASIA image tam- pering detection evaluation database

    Jing Dong, Wei Wang, and Tieniu Tan. CASIA image tam- pering detection evaluation database. In ISIP, 2013. 2

  5. [13]

    Test-time forgery de- tection with spatial-frequency prompt learning

    Junxian Duan, Yuang Ai, Jipeng Liu, Shenyuan Huang, Huaibo Huang, Jie Cao, and Ran He. Test-time forgery de- tection with spatial-frequency prompt learning. IJCV, 2024. 3

  6. [14]

    The llama 3 herd of models

    Abhimanyu Dubey et al. The llama 3 herd of models. Arxiv,

  7. [15]

    Llama: Open and efficient foundation language models

    Hugo Touvron et al. Llama: Open and efficient foundation language models. Arxiv, 2023

  8. [16]

    Llama 2: Open foundation and fine- tuned chat models

    Hugo Touvron et al. Llama 2: Open foundation and fine- tuned chat models. Arxiv, 2023. 3

  9. [17]

    Cnn-generated images are surprisingly easy to spot...for now

    Joel Frank and Thorsten Holz. Cnn-generated images are surprisingly easy to spot...for now. Arxiv, 2021. 6, 7, 8, 9

  10. [18]

    Leveraging fre- quency analysis for deep fake image recognition

    Joel Frank, Thorsten Eisenhofer, Lea Sch ¨onherr, Asja Fis- cher, Dorothea Kolossa, and Thorsten Holz. Leveraging fre- quency analysis for deep fake image recognition. In ICML,

  11. [19]

    Anomalygpt: Detecting in- dustrial anomalies using large vision-language models

    Zhaopeng Gu, Bingke Zhu, Guibo Zhu, Yingying Chen, Ming Tang, and Jinqiao Wang. Anomalygpt: Detecting in- dustrial anomalies using large vision-language models. In AAAI, 2024. 3

  12. [20]

    Trufor: Leveraging all-round clues for trustworthy image forgery detection and localiza- tion

    Fabrizio Guillaro, Davide Cozzolino, Avneesh Sud, Nicholas Dufour, and Luisa Verdoliva. Trufor: Leveraging all-round clues for trustworthy image forgery detection and localiza- tion. In CVPR, 2023. 3

  13. [21]

    Hierarchical fine-grained im- age forgery detection and localization

    Xiao Guo, Xiaohong Liu, Zhiyuan Ren, Steven Grosz, Ia- copo Masi, and Xiaoming Liu. Hierarchical fine-grained im- age forgery detection and localization. In CVPR, 2023. 1, 2, 7, 9

  14. [22]

    Language-guided hierarchical fine-grained image forgery detection and localization

    Xiao Guo, Xiaohong Liu, Iacopo Masi, and Xiaoming Liu. Language-guided hierarchical fine-grained image forgery detection and localization. Arxiv, 2024. 1, 2

  15. [23]

    Hdf-net: Capturing ho- mogeny difference features to localize the tampered image

    Ruidong Han, Xiaofeng Wang, Ningning Bai, Yihang Wang, Jianpeng Hou, and Jianru Xue. Hdf-net: Capturing ho- mogeny difference features to localize the tampered image. IEEE TPAMI, 2024. 2

  16. [24]

    Forgerynet: A versatile benchmark for comprehensive forgery analysis

    Yinan He, Bei Gan, Siyu Chen, Yichun Zhou, Guojun Yin, Luchuan Song, Lu Sheng, Jing Shao, and Ziwei Liu. Forgerynet: A versatile benchmark for comprehensive forgery analysis. In CVPR, 2021. 1, 2

  17. [25]

    FFAA: multimodal large language model based explainable open-world face forgery analysis assistant

    Zhengchao Huang, Bin Xia, Zicheng Lin, Zhun Mou, and Wenming Yang. FFAA: multimodal large language model based explainable open-world face forgery analysis assistant. Arxiv, abs/2408.10072, 2024. 9, 10

  18. [26]

    Frepgan: Robust deepfake detection using frequency- level perturbations

    Yonghyun Jeong, Doyeon Kim, Youngmin Ro, and Jongwon Choi. Frepgan: Robust deepfake detection using frequency- level perturbations. In AAAI, 2022. 3

  19. [27]

    Fusing global and local features for general- ized ai-synthesized image detection

    Yan Ju, Shan Jia, Lipeng Ke, Hongfei Xue, Koki Nagano, and Siwei Lyu. Fusing global and local features for general- ized ai-synthesized image detection. In ICIP, 2022. 6, 7, 8, 9

  20. [28]

    Jain, and Xiaoming Liu

    Minchul Kim, Feng Liu, Anil K. Jain, and Xiaoming Liu. Dcface: Synthetic face generation with dual condition diffu- sion model. In CVPR, 2023. 2

  21. [29]

    Vilt: Vision- and-language transformer without convolution or region su- pervision

    Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision- and-language transformer without convolution or region su- pervision. In ICML, 2021. 5

  22. [30]

    LISA: reasoning segmenta- tion via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. LISA: reasoning segmenta- tion via large language model. In CVPR, 2024. 3, 5, 7, 9, 10

  23. [31]

    Lang-segment-anything

    lang-sam team. Lang-segment-anything. https : / / github . com / luca - medeiros / lang - segment - anything, 2024. 4 23

  24. [32]

    Forgerygpt: Multimodal large language model for explainable image forgery detection and localization

    Jiawei Li, Fanrui Zhang, Jiaying Zhu, Esther Sun, Qiang Zhang, and Zheng-Jun Zha. Forgerygpt: Multimodal large language model for explainable image forgery detection and localization. Arxiv, 2024. 3, 9, 10

  25. [33]

    Transformer-based visual segmenta- tion: A survey

    Xiangtai Li, Henghui Ding, Wenwei Zhang, Haobo Yuan, Guangliang Cheng, Pang Jiangmiao, Kai Chen, Ziwei Liu, and Chen Change Loy. Transformer-based visual segmenta- tion: A survey. T-PAMI, 2024. 3

  26. [34]

    Detecting multimedia generated by large AI models: A survey

    Li Lin, Neeraj Gupta, Yue Zhang, Hainan Ren, Chun-Hao Liu, Feng Ding, Xin Wang, Xin Li, Luisa Verdoliva, and Shu Hu. Detecting multimedia generated by large AI models: A survey. Arxiv, 2024. 1

  27. [35]

    Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C

    Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In ECCV, 2014. 4, 10

  28. [36]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023. 3, 5

  29. [37]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR,

  30. [38]

    Pscc-net: Progressive spatio-channel correlation network for image manipulation detection and localization

    Xiaohong Liu, Yaojie Liu, Jun Chen, and Xiaoming Liu. Pscc-net: Progressive spatio-channel correlation network for image manipulation detection and localization. IEEE TCSVT, 2022. 7, 9, 10

  31. [39]

    Zhengzhe Liu, Xiaojuan Qi, and Philip H. S. Torr. Global texture enhancement for fake face detection in the wild. In CVPR, 2020. 6, 7, 8, 9

  32. [40]

    Deepfakes genera- tion and detection: state-of-the-art, open challenges, coun- termeasures, and way forward

    Momina Masood, Marriam Nawaz, Khalid Mahmood Malik, Ali Javed, Aun Irtaza, and Hafiz Malik. Deepfakes genera- tion and detection: state-of-the-art, open challenges, coun- termeasures, and way forward. Appl. Intell., 2023. 3

  33. [41]

    May, Kirill Trapeznikov, Shengbang Fang, and Matthew C

    Brandon B. May, Kirill Trapeznikov, Shengbang Fang, and Matthew C. Stamm. Comprehensive dataset of synthetic and manipulated overhead imagery for development and evalua- tion of forensic tools. In IH&MMSec, 2023. 2

  34. [42]

    Multi-spectral class center network for face manipulation de- tection and localization

    Changtao Miao, Qi Chu, Zhentao Tan, Zhenchao Jin, Wanyi Zhuang, Yue Wu, Bin Liu, Honggang Hu, and Nenghai Yu. Multi-spectral class center network for face manipulation de- tection and localization. Arxiv, 2023. 3

  35. [43]

    Large language models: A survey

    Shervin Minaee, Tom ´as Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large language models: A survey. Arxiv, 2024. 3

  36. [44]

    Absolutereality model repos- itory

    Absolutereality model team. Absolutereality model repos- itory. https://huggingface.co/jochemstoel/ absolutereality-model-repository , 2024. Ac- cessed: 2024-11-15. 4

  37. [45]

    Flux model

    Flux model team. Flux model. https : / / huggingface . co / black - forest - labs / FLUX . 1-dev, 2024. Accessed: 2024-11-15. 4

  38. [46]

    Laa-net: Localized artifact attention network for quality-agnostic and generalizable deepfake de- tection

    Dat Nguyen, Nesryne Mejri, Inder Pal Singh, Polina Kuleshova, Marcella Astrid, Anis Kacem, Enjie Ghorbel, and Djamila Aouada. Laa-net: Localized artifact attention network for quality-agnostic and generalizable deepfake de- tection. In CVPR, 2024. 3

  39. [47]

    IMD2020: A large-scale annotated dataset tailored for de- tecting manipulated images

    Adam Novoz ´amsk´y, Babak Mahdian, and Stanislav Saic. IMD2020: A large-scale annotated dataset tailored for de- tecting manipulated images. In WACVW, 2020. 2

  40. [48]

    Towards uni- versal fake image detectors that generalize across generative models

    Utkarsh Ojha, Yuheng Li, and Yong Jae Lee. Towards uni- versal fake image detectors that generalize across generative models. In CVPR, 2023. 6, 7, 8, 9

  41. [49]

    GPT-4 technical report

    OpenAI. GPT-4 technical report. Arxiv, 2023. 4

  42. [50]

    On the use of stable diffusion for creating realistic faces: from generation to detection

    Lorenzo Papa, Lorenzo Faiella, Luca Corvitto, Luca Maiano, and Irene Amerini. On the use of stable diffusion for creating realistic faces: from generation to detection. In IWBF, 2023. 2

  43. [51]

    Deepfake generation and detection: A benchmark and survey

    Gan Pei, Jiangning Zhang, Menghan Hu, Guangtao Zhai, Chengjie Wang, Zhenyu Zhang, Jian Yang, Chunhua Shen, and Dacheng Tao. Deepfake generation and detection: A benchmark and survey. Arxiv, 2024. 3

  44. [52]

    Kosmos-2: Grounding multimodal large language models to the world.Arxiv, 2023

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world.Arxiv, 2023. 3

  45. [53]

    Plummer, Liwei Wang, Chris M

    Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspon- dences for richer image-to-sentence models. IJCV, 2017. 4, 10

  46. [54]

    SDXL: improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. SDXL: improving latent diffusion models for high-resolution image synthesis. In ICLR, 2024. 4

  47. [55]

    Arti- fact: A large-scale dataset with artificial and factual images for generalizable and robust synthetic image detection

    Md Awsafur Rahman, Bishmoy Paul, Najibul Haque Sarker, Zaber Ibn Abdul Hakim, and Shaikh Anowarul Fattah. Arti- fact: A large-scale dataset with artificial and factual images for generalizable and robust synthetic image detection. In ICIP, 2023. 2

  48. [56]

    Shaker, Salman H

    Hanoona Abdul Rasheed, Muhammad Maaz, Sahal Shaji Mullappilly, Abdelrahman M. Shaker, Salman H. Khan, Hisham Cholakkal, Rao Muhammad Anwer, Eric P. Xing, Ming-Hsuan Yang, and Fahad Shahbaz Khan. Glamm: Pixel grounding large multimodal model. In CVPR, 2024. 3

  49. [57]

    Pixellm: Pixel reasoning with large multimodal model

    Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel reasoning with large multimodal model. In CVPR, 2024. 3

  50. [58]

    High-resolution image syn- thesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models. In CVPR, 2022. 4

  51. [59]

    A survey of multimodal-guided image editing with text-to-image diffu- sion models

    Xincheng Shuai, Henghui Ding, Xingjun Ma, Rongcheng Tu, Yu-Gang Jiang, and Dacheng Tao. A survey of multimodal-guided image editing with text-to-image diffu- sion models. Arxiv, 2024. 1

  52. [60]

    Robustness and generalizability of deepfake detection: A study with diffusion models

    Haixu Song, Shiyu Huang, Yinpeng Dong, and Wei-Wei Tu. Robustness and generalizability of deepfake detection: A study with diffusion models. Arxiv, 2023. 2

  53. [61]

    Learning on gradients: Generalized arti- facts representation for gan-generated images detection

    Chuangchuang Tan, Yao Zhao, Shikui Wei, Guanghua Gu, and Yunchao Wei. Learning on gradients: Generalized arti- facts representation for gan-generated images detection. In CVPR, 2023. 6, 7, 9

  54. [62]

    Frequency-aware deepfake de- tection: Improving generalizability through frequency space domain learning

    Chuangchuang Tan, Yao Zhao, Shikui Wei, Guanghua Gu, Ping Liu, and Yunchao Wei. Frequency-aware deepfake de- tection: Improving generalizability through frequency space domain learning. In AAAI, 2024. 3 24

  55. [63]

    Weakly-supervised deepfake localization in diffusion-generated images

    Dragos-Constantin T ˆantaru, Elisabeta Oneata, and Dan Oneata. Weakly-supervised deepfake localization in diffusion-generated images. In WACV, 2024. 3

  56. [64]

    Representative forgery mining for fake face detection

    Chengrui Wang and Weihong Deng. Representative forgery mining for fake face detection. In CVPR, 2021. 3

  57. [65]

    Fakespotter: A simple yet robust baseline for spotting ai-synthesized fake faces

    Run Wang, Felix Juefei-Xu, Lei Ma, Xiaofei Xie, Yihao Huang, Jian Wang, and Yang Liu. Fakespotter: A simple yet robust baseline for spotting ai-synthesized fake faces. In IJCAI, 2020. 2

  58. [66]

    Cnn-generated images are sur- prisingly easy to spot...for now

    Sheng-Yu Wang, Oliver Wang, Richard Zhang, Andrew Owens, and Alexei A Efros. Cnn-generated images are sur- prisingly easy to spot...for now. In CVPR, 2020. 2

  59. [67]

    Dynamic graph learning with content-guided spatial- frequency relation reasoning for deepfake detection

    Yuan Wang, Kun Yu, Chen Chen, Xiyuan Hu, and Silong Peng. Dynamic graph learning with content-guided spatial- frequency relation reasoning for deepfake detection. In CVPR, 2023. 3

  60. [68]

    DIRE for diffusion-generated image detection

    Zhendong Wang, Jianmin Bao, Wengang Zhou, Weilun Wang, Hezhen Hu, Hong Chen, and Houqiang Li. DIRE for diffusion-generated image detection. In ICCV, 2023. 2, 9

  61. [69]

    Lasagna: Language-based segmentation assistant for complex queries

    Cong Wei, Haoxian Tan, Yujie Zhong, Yujiu Yang, and Lin Ma. Lasagna: Language-based segmentation assistant for complex queries. Arxiv, 2024. 3

  62. [70]

    Towards se- mantic equivalence of tokenization in multimodal llm

    Shengqiong Wu, Hao Fei, Xiangtai Li, Jiayi Ji, Hanwang Zhang, Tat-Seng Chua, and Shuicheng Yan. Towards se- mantic equivalence of tokenization in multimodal llm. arXiv preprint arXiv:2406.05127, 2024. 3

  63. [71]

    GSV A: generalized segmentation via multimodal large language models

    Zhuofan Xia, Dongchen Han, Yizeng Han, Xuran Pan, Shiji Song, and Gao Huang. GSV A: generalized segmentation via multimodal large language models. In CVPR, 2024. 3

  64. [72]

    Fakeshield: Explainable image forgery detection and localization via multi-modal large lan- guage models

    Zhipei Xu, Xuanyu Zhang, Runyi Li, Zecheng Tang, Qing Huang, and Jian Zhang. Fakeshield: Explainable image forgery detection and localization via multi-modal large lan- guage models. Arxiv, 2024. 3, 9, 10

  65. [73]

    Deepfakebench: A comprehensive benchmark of deepfake detection

    Zhiyuan Yan, Yong Zhang, Xinhang Yuan, Siwei Lyu, and Baoyuan Wu. Deepfakebench: A comprehensive benchmark of deepfake detection. In NeurIPS, 2023. 1

  66. [74]

    An improved baseline for reasoning segmentation with large language model

    Senqiao Yang, Tianyuan Qu, Xin Lai, Zhuotao Tian, Bohao Peng, Shu Liu, and Jiaya Jia. An improved baseline for reasoning segmentation with large language model. Arxiv,

  67. [75]

    Ferret: Refer and ground anything anywhere at any granularity

    Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. In ICLR, 2024. 3

  68. [76]

    Diffforensics: Leveraging diffusion prior to image forgery detection and localization

    Zeqin Yu, Jiangqun Ni, Yuzhen Lin, Haoyi Deng, and Bin Li. Diffforensics: Leveraging diffusion prior to image forgery detection and localization. In CVPR, 2024. 2

  69. [77]

    Contextual object detection with mul- timodal large language models

    Yuhang Zang, Wei Li, Jun Han, Kaiyang Zhou, and Chen Change Loy. Contextual object detection with mul- timodal large language models. Arxiv, 2023. 3

  70. [78]

    Fangneng Zhan, Yingchen Yu, Rongliang Wu, Jiahui Zhang, Shijian Lu, Lingjie Liu, Adam Kortylewski, Christian Theobalt, and Eric P. Xing. Multimodal image synthesis and editing: The generative AI era. IEEE TPAMI, 2023. 1, 2

  71. [79]

    Text-to-image diffusion models in gen- erative AI: A survey

    Chenshuang Zhang, Chaoning Zhang, Mengchun Zhang, and In So Kweon. Text-to-image diffusion models in gen- erative AI: A survey. Arxiv, 2023. 2

  72. [80]

    Llava-grounding: Grounded visual chat with large multimodal models

    Hao Zhang, Hongyang Li, Feng Li, Tianhe Ren, Xueyan Zou, Shilong Liu, Shijia Huang, Jianfeng Gao, Leizhang, Chunyuan Li, and Jainwei Yang. Llava-grounding: Grounded visual chat with large multimodal models. In ECCV, 2024. 3

  73. [81]

    Vision-language models for vision tasks: A survey

    Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. IEEE TPAMI, 2024. 3

  74. [82]

    Magicbrush: A manually annotated dataset for instruction- guided image editing

    Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction- guided image editing. In NeurIPS, 2023. 6, 11

  75. [83]

    Perceptual artifacts localization for image synthesis tasks

    Lingzhi Zhang, Zhengjie Xu, Connelly Barnes, Yuqian Zhou, Qing Liu, He Zhang, Sohrab Amirghodsi, Zhe Lin, Eli Shechtman, and Jianbo Shi. Perceptual artifacts localization for image synthesis tasks. In ICCV, 2023. 3

  76. [84]

    Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding

    Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shunping Ji, Change Loy Chen, and Shuicheng Yan. Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding. In NeurIPS, 2024. 3

  77. [85]

    Common sense reasoning for deep fake detection

    Yue Zhang, Ben Colman, Ali Shahriyari, and Gaurav Bharaj. Common sense reasoning for deep fake detection. Arxiv,

  78. [86]

    MFMS: learning modality-fused and modality-specific features for deepfake detection and local- ization tasks

    Yi Zhang, Changtao Miao, Man Luo, Jianshu Li, Wenzhong Deng, Weibin Yao, Zhe Li, Bingyu Hu, Weiwei Feng, Tao Gong, and Qi Chu. MFMS: learning modality-fused and modality-specific features for deepfake detection and local- ization tasks. In ACM MM, 2024. 3

  79. [87]

    Rich and poor texture contrast: A simple yet effective ap- proach for ai-generated image detection

    Nan Zhong, Yiran Xu, Zhenxing Qian, and Xinpeng Zhang. Rich and poor texture contrast: A simple yet effective ap- proach for ai-generated image detection. Arxiv, 2023. 2, 3

  80. [88]

    Corso, and Jianfeng Gao

    Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Ja- son J. Corso, and Jianfeng Gao. Unified vision-language pre- training for image captioning and VQA. In AAAI, 2020. 5

  81. [89]

    Genimage: A million-scale benchmark for de- tecting ai-generated image

    Mingjian Zhu, Hanting Chen, Qiangyu Yan, Xudong Huang, Guanyu Lin, Wei Li, Zhijun Tu, Hailin Hu, Jie Hu, and Yunhe Wang. Genimage: A million-scale benchmark for de- tecting ai-generated image. In NeurIPS, 2023. 2, 3

  82. [90]

    M3DSYNTH: A dataset of medical 3d images with ai-generated local manipulations

    Giada Zingarini, Davide Cozzolino, Riccardo Corvi, Gio- vanni Poggi, and Luisa Verdoliva. M3DSYNTH: A dataset of medical 3d images with ai-generated local manipulations. In ICASSP, 2024. 2 25

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.