REVIEW 5 major objections 7 minor 10 cited by
SIDA: Social Media Image Deepfake Detection, Localization and Explanation with Large Multimodal Model
T0 review · 5 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read This paper claims that one large multimodal model can detect whether a social-media image is real, fully synthetic, or tampered, draw a pixel-level mask of the tampered region, and explain its verdict in text, and that the 300K-image…
desk verdict The dataset is the real contribution; the localization ground truth needs an alignment check before the headline claims are accepted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the token-head architecture borrowed from reasoning-segmentation vision-language models: two added tokens, <DET> and <SEG>, whose last-layer hidden states are extracted and fed to a lightweight detection head and, via a single multihead-attention residual module, to a mask decoder conditioned on the frozen image encoder's features. On the data side, the machinery is the four-stage tampering pipeline: GPT-4o extracts objects from captions, Language-SAM produces masks on the original images, replacement dictionaries define object swaps and attribute changes, and Latent Diffusion regenerates the image with the altered caption. The masks from stage two are the training ground truth for localization.
What would settle it
Take a random sample of SID-Set tampered images, have human annotators draw the actually changed pixels against the original photo, and recompute SIDA's localization AUC, F1, and IoU; if scores drop materially below the reported 87.3, 73.9, and 43.8, the masks used as ground truth did not match the true edits.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that detection, localization, and explanation are not competing tasks but can be fused in one vision-language model by letting the detection representation condition the segmentation features. The model adds <DET> and <SEG> tokens to the vocabulary; the <DET> hidden state classifies the image into real, fully synthetic, object-tampered, or part-tampered, and when tampering is found, a multihead-attention module uses detection features as queries over segmentation features, then a frozen-image-encoder decoder produces the mask. The same model generates a textual justification. Trained end-to-end on SID-Set with cross-entropy for detection, BCE+DICE for masks, and cross-entropy for text, SIDA-7B and SIDA-13B reach 93.5-93.6% overall accuracy and set the top localization scores among compared methods.
Load-bearing premise
The central load-bearing premise is that Language-SAM masks computed on the original images align with the regions Latent Diffusion actually altered, so the reported localization scores measure true tampered-region accuracy rather than agreement with approximate labels.
Editorial extensions
If this is right
- A single SIDA-style model can replace separate detector and localization pipelines for social-media images, reducing the deployment cost of content moderation.
- Because SIDA also outputs textual explanations, moderation systems gain a human-readable audit trail for every decision, not just a score.
- SID-Set provides a training and evaluation ground for a wider variety of fakes than face-only datasets, covering fully synthetic and object- or part-tampered images.
- Fine-tuning on SID-Set improves several prior detectors' performance, indicating the dataset is a useful transfer resource beyond SIDA itself.
- The robustness experiments suggest SIDA tolerates JPEG compression, resizing, and Gaussian noise without explicit augmentation.
Reading between the lines
- If SID-Set's masks prove accurate, it could become the standard benchmark for non-facial image forensics, much as face-centric datasets were a decade ago.
- The reliance on one generator for synthetic images and one diffusion model for tampering means SID-Set may bake in artifacts of those two generators; a testable next step is adding images from other generators and checking whether SIDA's advantage shrinks.
- The explanation head is trained only on 3,000 GPT-4o descriptions, so SIDA's explanations likely reflect GPT-4o's judgment criteria rather than SIDA's own internal cues; a direct test would compare model explanations with its attention or occlusion maps.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript introduces SID-Set, a 300K-image benchmark for social-media deepfake detection, localization, and explanation, and proposes SIDA, a LISA-based large multimodal model that adds <DET> and <SEG> tokens to perform three-way detection (real / fully synthetic / tampered), pixel-level tamper localization, and textual explanation. The authors report state-of-the-art detection results against eight baselines on SID-Set (Table 2), the best localization numbers among four comparison methods (Table 3), robustness to JPEG compression, resizing, and Gaussian noise (Table 4), and strong performance on the DMimage benchmark (Table 5). The dataset, model, and code are promised for release.
Significance. If the benchmark's localization labels are valid, SID-Set would be the largest social-media-focused deepfake dataset with masks and explanations, and SIDA would be the first single VLM to unify three-way detection, pixel-level localization, and textual explanation. The paper has notable strengths: extensive retraining of detection baselines on SID-Set, a perturbation robustness study, qualitative failure-case analysis, and a commitment to release the dataset, code, and model. The significance is moderated by unresolved questions about the alignment between Language-SAM masks and the regions actually altered by Latent Diffusion, by an ambiguous evaluation protocol in Table 2, and by an unfair cross-benchmark comparison in Table 5.
major comments (5)
- [Section 3.2, Stages 2-4 and Table 3] The construction pipeline does not state whether the Language-SAM mask from Stage 2 is used as an inpainting or conditioning mask in the Stage 4 Latent Diffusion regeneration, nor does it verify that the pixels actually changed by the diffusion edit coincide with the Stage-2 mask. Because the Stage-2 masks are used both as training targets for Lmask in Eq. (5) and as inputs to the GPT-4o explanation prompts (Appendix Figure 14), any drift between the edited region and the mask makes the reported localization AUC/F1/IoU and the learned explanations measures of agreement with noisy labels rather than true tampering accuracy. Please specify the conditioning in Stage 4 and provide a quantitative alignment check, such as mean IoU between Stage-2 masks and pixel-wise differences of original versus regenerated images, or a human-verified sample.
- [Table 2 caption] The caption says parenthesized values are "performance changes after fine-tuning the models using the SID-Set on the validation set." If the models were fine-tuned on the validation set and evaluated on the same validation set, the comparison is invalid because of train/test leakage; if they were fine-tuned on the training split and evaluated on the validation split, the wording should be corrected. Please report results on the held-out test split and specify the exact protocol for each baseline.
- [Section 5.4, Table 5] SIDA is compared with baselines using their original pre-trained weights after SIDA has been trained on 300K SID-Set images, so the comparison is not apples-to-apples. The proper generalization test would fine-tune all baselines on the SID-Set training split and report test-split numbers, or evaluate all methods in a zero-shot setting without SID-Set training. As written, the "superior performance" on DMimage may reflect a training-data advantage rather than a genuine generalization advantage.
- [Section 4.2 / Section 5 Implementation Details / Section 5.5] The loss-weight specification is internally inconsistent. Implementation Details says the localization loss weights lambda_bce and lambda_dice are both set to 1.0, while Section 5.5 states "for our experiments, we set lambda_det to 1, lambda_bce to 2.0, and lambda_dice to 0.5," and Table 7 lists yet another configuration. Please state unambiguously which configuration produced Tables 2-4 and report the corresponding validation or test numbers; otherwise the experimental results are not reproducible.
- [Tables 2-4 and Section 5.5] The detection, localization, and ablation results are reported as single-run point estimates without standard deviations or the number of seeds. Given that the stated margins in Table 2 are large but the ablation differences in Table 6 are small (e.g., ACC 93.5 vs 91.1 for removing the attention module), please report mean and standard deviation over at least three runs, or otherwise justify that the reported differences are statistically meaningful.
minor comments (7)
- [Table 2 caption] The "Overall Acc and F1" are described as the average of the values from the three classification categories; this is a macro average, not an overall accuracy, and the caption should say so explicitly to avoid misleading readers.
- [Section 4.2 / Appendix Figures 7 and 14] The 3,000 explanation ground-truth texts are generated by GPT-4o and reviewed by five experts, but the paper reports no quantitative evaluation of explanation quality, only qualitative examples in Section 5.6. Moreover, the prompt for tampered images in Appendix Figure 14 explicitly provides the mask as input, so the explanation stage is conditioned on the same localization labels whose validity is in question; this limitation should be acknowledged.
- [Abstract and Section 3] The paper describes SID-Set as having "comprehensive annotations," but explanations are available for only 3,000 of the 300K images; the wording should be tempered to distinguish the mask annotations from the explanation annotations.
- [Section 5.5 and Table 7] The ablation study for training weights reports only two configurations and does not compare against the Implementation Details configuration of lambda_bce=lambda_dice=1.0; please include that configuration or justify its omission.
- [Throughout] There are several minor typos and naming inconsistencies, including "CnnSpott" in Table 2 versus "CnnSpot" in the text, "HIFI-Net" versus "HiFi-IFDL," and "textural descriptions" versus "textual descriptions." Please proofread for consistency.
- [Section 3.2 and Appendix D] The paper says MagicBrush images were "integrated" and "filtered," but provides no details on the filtering criteria or the proportion of MagicBrush images in the final dataset; please specify this, and also cite the OpenImages V7 dataset rather than only a URL.
- [Appendix E] The expert evaluation is described only qualitatively; please report at least the number of images flagged and excluded, and ideally inter-annotator agreement, so that the quality-control claims can be assessed.
Circularity Check
No significant circularity: SIDA's detection and localization results follow from a standard supervised train/test split, and the self-citations are not load-bearing.
full rationale
The central detection and localization claims are derived from training SIDA on SID-Set with a held-out test split (Section 5, 7:1:2 ratio) and from comparison against external baselines, including the DMimage benchmark (Table 5), so they are not equivalent to any fitted input by construction. The localization ground truth is created by Language-SAM masks on original COCO/Flickr30k images (Section 3.2, Stage 2), and the paper does not state that the Stage-4 Latent Diffusion regeneration is guaranteed to alter exactly those pixels; this is a potential label-noise and validity threat rather than a circular derivation. The explanation stage is trained to match GPT-4o-generated descriptions (Section 4.2, Eq. 6), so the generated text is an imitation of an external label source and does not independently evidence SIDA's internal criteria; this is an evaluation limitation, not a circular reduction of the detection result. The only self-citations ([33], [70], [84]) appear in related-work enumerations of LMM/LISA-style models and are not load-bearing for the paper's claims. No fitted parameter is relabeled as a prediction, and no uniqueness theorem or prior-work ansatz is used to force the architecture choice.
Assumptions & free parameters
free parameters (2)
- Training loss weights =
lambda_det=1.0, lambda_bce=2.0, lambda_dice=0.5 (second stage adds lambda_txt=1.0)
- LoRA and optimization settings =
LoRA alpha=16, dropout=0.05, LR=1e-4, batch size=2, grad accumulation=10
assumptions (4)
- domain assumption Language-SAM masks on the original image coincide with the pixels changed by Latent Diffusion in Stage 4.
- domain assumption OpenImages V7, COCO, and Flickr30k captions adequately represent the social media image distribution.
- domain assumption GPT-4o-generated descriptions, after expert review, are valid ground truth for explaining the model's judgment criteria.
- domain assumption FLUX-generated images are representative of the deepfakes that appear on social media.
Cite this review
Pith. "Pith review of SIDA: Social Media Image Deepfake Detection, Localization and Explanation with Large Multimodal Model." pith.science (2026). https://pith.science/paper/QGWBW35F
@misc{pith2026241204292,
author = {Pith},
title = {Pith review of: SIDA: Social Media Image Deepfake Detection, Localization and Explanation with Large Multimodal Model},
year = {2026},
howpublished = {\url{https://pith.science/paper/QGWBW35F}},
note = {Machine review of arXiv:2412.04292}
}
read the original abstract
The rapid advancement of generative models in creating highly realistic images poses substantial risks for misinformation dissemination. For instance, a synthetic image, when shared on social media, can mislead extensive audiences and erode trust in digital content, resulting in severe repercussions. Despite some progress, academia has not yet created a large and diversified deepfake detection dataset for social media, nor has it devised an effective solution to address this issue. In this paper, we introduce the Social media Image Detection dataSet (SID-Set), which offers three key advantages: (1) extensive volume, featuring 300K AI-generated/tampered and authentic images with comprehensive annotations, (2) broad diversity, encompassing fully synthetic and tampered images across various classes, and (3) elevated realism, with images that are predominantly indistinguishable from genuine ones through mere visual inspection. Furthermore, leveraging the exceptional capabilities of large multimodal models, we propose a new image deepfake detection, localization, and explanation framework, named SIDA (Social media Image Detection, localization, and explanation Assistant). SIDA not only discerns the authenticity of images, but also delineates tampered regions through mask prediction and provides textual explanations of the model's judgment criteria. Compared with state-of-the-art deepfake detection models on SID-Set and other benchmarks, extensive experiments demonstrate that SIDA achieves superior performance among diversified settings. The code, model, and dataset will be released.
Figures
Figures from the paper (16 more)
Forward citations
Cited by 10 Pith papers
-
DocShield: Towards AI Document Safety via Evidence-Grounded Agentic Reasoning
DocShield presents a new agentic reasoning framework using Cross-Cues-aware Chain of Thought to detect, localize, and explain text-centric forgeries in documents, with reported F1 gains of 41.4% over specialized metho...
-
XPlainVerse: A Million-Scale Benchmark for Explainable Deepfake Detection
A million-scale deepfake benchmark with Edit-Check filtering, dual expert/lay explanations, and EntityScore/EvidenceScore shows fine-tuned detectors collapse under generator shift while surface fluency remains.
-
Explaining AI-Image Detection: What the Heatmap Actually Shows
Compression history, not synthesis, explains a near-perfect AI-image detector, and no tested heatmap causally explains its verdict.
-
LaP-Forensics: Latent-Pixel Consistency Guided Multimodal Reasoning for Deepfake Detection
A dual-stream deepfake forensic model that adds DDIM reconstruction residuals to RGB features improves artifact localization and cross-generator detection in evaluations, with honest caveats about text faithfulness.
-
Detecting Text Manipulation in Images using Vision Language Models
In zero-shot benchmarks, GPT-4o outperforms open-source VLMs and specialized manipulation detectors on text tampering detection in scene images and fantasy ID documents.
-
AIGI-Holmes: Towards Explainable and Generalizable AI-Generated Image Detection via Multimodal Large Language Models
AIGI-Holmes combines visual expert pretraining, SFT on explanation data, and direct preference optimization to deliver human-verifiable explanations and top detection accuracy on unseen AI generators.
-
Omni-IML: Towards Unified Image Manipulation Localization
One jointly trained image-manipulation-localization model attains top average scores on natural, document, face, and scene-text forensics benchmarks and gains artifact-description ability from a new GPT-4o-annotated dataset.
-
DFBench: Benchmarking Deepfake Image Detection Capability of Large Multimodal Models
DFBench adds a 540,000-image benchmark with 12 modern generators, partial edits, and distorted real images, and its three-model LMM ensemble, MoA-DF, reaches near-perfect recall on its own test split.
-
Multiverse Through Deepfakes: The MultiFakeVerse Dataset of Person-Centric Visual and Conceptual Manipulations
MultiFakeVerse provides 845,286 person-centric images edited through VLM-generated instructions; state-of-the-art deepfake detectors and human observers misclassify a large fraction of them.
-
Survey on AI-Generated Media Detection: From Non-MLLM to MLLM
A survey organizing AI-generated media detection into Non-MLLM and MLLM based methods, with task and benchmark taxonomies.
Reference graph
Works this paper leans on
-
[1]
Kandinsky 3.0 technical report
Vladimir Arkhipkin, Andrei Filatov, Viacheslav Vasilev, Anastasia Maltseva, Said Azizov, Igor Pavlov, Julia Aga- fonova, Andrey Kuznetsov, and Denis Dimitrov. Kandinsky 3.0 technical report. Arxiv, 2023. 4
2023
-
[2]
Xiuli Bi, Bo Liu, Fan Yang, Bin Xiao, Weisheng Li, Gao Huang, and Pamela C. Cosman. Detecting generated images by real images only. Arxiv, 2023. 6, 7, 8, 9
2023
-
[3]
Bird and Ahmad Lotfi
Jordan J. Bird and Ahmad Lotfi. CIFAKE: image classifica- tion and explainable identification of ai-generated synthetic images. IEEE Access, 2024. 2
2024
-
[4]
End-to-end reconstruction- classification learning for face forgery detection
Junyi Cao, Chao Ma, Taiping Yao, Shen Chen, Shouhong Ding, and Xiaokang Yang. End-to-end reconstruction- classification learning for face forgery detection. In CVPR,
-
[5]
Antifakeprompt: Prompt-tuned vision-language models are fake image detectors
You-Ming Chang, Chen Yeh, Wei-Chen Chiu, and Ning Yu. Antifakeprompt: Prompt-tuned vision-language models are fake image detectors. Arxiv, 2023. 3, 6, 7, 8, 9
2023
-
[6]
Self-supervised learning of adversarial exam- ple: Towards good generalizations for deepfake detection
Liang Chen, Yong Zhang, Yibing Song, Lingqiao Liu, and Jue Wang. Self-supervised learning of adversarial exam- ple: Towards good generalizations for deepfake detection. In CVPR, 2022. 3
2022
-
[7]
Kankanhalli
Harry Cheng, Yangyang Guo, Tianyi Wang, Liqiang Nie, and Mohan S. Kankanhalli. Diffusion facial forgery detection. In Proceedings of the 32nd ACM International Conference on Multimedia, 2024. 1, 2
2024
-
[8]
On the de- tection of synthetic images generated by diffusion models
Riccardo Corvi, Davide Cozzolino, Giada Zingarini, Gio- vanni Poggi, Koki Nagano, and Luisa Verdoliva. On the de- tection of synthetic images generated by diffusion models. In ICASSP, 2023. 7, 8
2023
Show all 90 references
-
[9]
Diffusion models in vision: A survey
Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. Diffusion models in vision: A survey. IEEE TPAMI, 2023. 1
2023
-
[10]
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning. In NeurIPS, 2023. 3
2023
-
[11]
Mvss-net: Multi-view multi-scale supervised networks for image manipulation detection
Chengbo Dong, Xinru Chen, Ruohan Hu, Juan Cao, and Xirong Li. Mvss-net: Multi-view multi-scale supervised networks for image manipulation detection. IEEE TPAMI,
-
[12]
CASIA image tam- pering detection evaluation database
Jing Dong, Wei Wang, and Tieniu Tan. CASIA image tam- pering detection evaluation database. In ISIP, 2013. 2
2013
-
[13]
Test-time forgery de- tection with spatial-frequency prompt learning
Junxian Duan, Yuang Ai, Jipeng Liu, Shenyuan Huang, Huaibo Huang, Jie Cao, and Ran He. Test-time forgery de- tection with spatial-frequency prompt learning. IJCV, 2024. 3
2024
-
[14]
The llama 3 herd of models
Abhimanyu Dubey et al. The llama 3 herd of models. Arxiv,
-
[15]
Llama: Open and efficient foundation language models
Hugo Touvron et al. Llama: Open and efficient foundation language models. Arxiv, 2023
2023
-
[16]
Llama 2: Open foundation and fine- tuned chat models
Hugo Touvron et al. Llama 2: Open foundation and fine- tuned chat models. Arxiv, 2023. 3
2023
-
[17]
Cnn-generated images are surprisingly easy to spot...for now
Joel Frank and Thorsten Holz. Cnn-generated images are surprisingly easy to spot...for now. Arxiv, 2021. 6, 7, 8, 9
2021
-
[18]
Leveraging fre- quency analysis for deep fake image recognition
Joel Frank, Thorsten Eisenhofer, Lea Sch ¨onherr, Asja Fis- cher, Dorothea Kolossa, and Thorsten Holz. Leveraging fre- quency analysis for deep fake image recognition. In ICML,
-
[19]
Anomalygpt: Detecting in- dustrial anomalies using large vision-language models
Zhaopeng Gu, Bingke Zhu, Guibo Zhu, Yingying Chen, Ming Tang, and Jinqiao Wang. Anomalygpt: Detecting in- dustrial anomalies using large vision-language models. In AAAI, 2024. 3
2024
-
[20]
Trufor: Leveraging all-round clues for trustworthy image forgery detection and localiza- tion
Fabrizio Guillaro, Davide Cozzolino, Avneesh Sud, Nicholas Dufour, and Luisa Verdoliva. Trufor: Leveraging all-round clues for trustworthy image forgery detection and localiza- tion. In CVPR, 2023. 3
2023
-
[21]
Hierarchical fine-grained im- age forgery detection and localization
Xiao Guo, Xiaohong Liu, Zhiyuan Ren, Steven Grosz, Ia- copo Masi, and Xiaoming Liu. Hierarchical fine-grained im- age forgery detection and localization. In CVPR, 2023. 1, 2, 7, 9
2023
-
[22]
Language-guided hierarchical fine-grained image forgery detection and localization
Xiao Guo, Xiaohong Liu, Iacopo Masi, and Xiaoming Liu. Language-guided hierarchical fine-grained image forgery detection and localization. Arxiv, 2024. 1, 2
2024
-
[23]
Hdf-net: Capturing ho- mogeny difference features to localize the tampered image
Ruidong Han, Xiaofeng Wang, Ningning Bai, Yihang Wang, Jianpeng Hou, and Jianru Xue. Hdf-net: Capturing ho- mogeny difference features to localize the tampered image. IEEE TPAMI, 2024. 2
2024
-
[24]
Forgerynet: A versatile benchmark for comprehensive forgery analysis
Yinan He, Bei Gan, Siyu Chen, Yichun Zhou, Guojun Yin, Luchuan Song, Lu Sheng, Jing Shao, and Ziwei Liu. Forgerynet: A versatile benchmark for comprehensive forgery analysis. In CVPR, 2021. 1, 2
2021
-
[25]
FFAA: multimodal large language model based explainable open-world face forgery analysis assistant
Zhengchao Huang, Bin Xia, Zicheng Lin, Zhun Mou, and Wenming Yang. FFAA: multimodal large language model based explainable open-world face forgery analysis assistant. Arxiv, abs/2408.10072, 2024. 9, 10
2024 arXiv
-
[26]
Frepgan: Robust deepfake detection using frequency- level perturbations
Yonghyun Jeong, Doyeon Kim, Youngmin Ro, and Jongwon Choi. Frepgan: Robust deepfake detection using frequency- level perturbations. In AAAI, 2022. 3
2022
-
[27]
Fusing global and local features for general- ized ai-synthesized image detection
Yan Ju, Shan Jia, Lipeng Ke, Hongfei Xue, Koki Nagano, and Siwei Lyu. Fusing global and local features for general- ized ai-synthesized image detection. In ICIP, 2022. 6, 7, 8, 9
2022
-
[28]
Jain, and Xiaoming Liu
Minchul Kim, Feng Liu, Anil K. Jain, and Xiaoming Liu. Dcface: Synthetic face generation with dual condition diffu- sion model. In CVPR, 2023. 2
2023
-
[29]
Vilt: Vision- and-language transformer without convolution or region su- pervision
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision- and-language transformer without convolution or region su- pervision. In ICML, 2021. 5
2021
-
[30]
LISA: reasoning segmenta- tion via large language model
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. LISA: reasoning segmenta- tion via large language model. In CVPR, 2024. 3, 5, 7, 9, 10
2024
-
[31]
Lang-segment-anything
lang-sam team. Lang-segment-anything. https : / / github . com / luca - medeiros / lang - segment - anything, 2024. 4 23
2024
-
[32]
Forgerygpt: Multimodal large language model for explainable image forgery detection and localization
Jiawei Li, Fanrui Zhang, Jiaying Zhu, Esther Sun, Qiang Zhang, and Zheng-Jun Zha. Forgerygpt: Multimodal large language model for explainable image forgery detection and localization. Arxiv, 2024. 3, 9, 10
2024
-
[33]
Transformer-based visual segmenta- tion: A survey
Xiangtai Li, Henghui Ding, Wenwei Zhang, Haobo Yuan, Guangliang Cheng, Pang Jiangmiao, Kai Chen, Ziwei Liu, and Chen Change Loy. Transformer-based visual segmenta- tion: A survey. T-PAMI, 2024. 3
2024
-
[34]
Detecting multimedia generated by large AI models: A survey
Li Lin, Neeraj Gupta, Yue Zhang, Hainan Ren, Chun-Hao Liu, Feng Ding, Xin Wang, Xin Li, Luisa Verdoliva, and Shu Hu. Detecting multimedia generated by large AI models: A survey. Arxiv, 2024. 1
2024
-
[35]
Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C
Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In ECCV, 2014. 4, 10
2014
-
[36]
Visual instruction tuning
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023. 3, 5
2023
-
[37]
Improved baselines with visual instruction tuning
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR,
-
[38]
Pscc-net: Progressive spatio-channel correlation network for image manipulation detection and localization
Xiaohong Liu, Yaojie Liu, Jun Chen, and Xiaoming Liu. Pscc-net: Progressive spatio-channel correlation network for image manipulation detection and localization. IEEE TCSVT, 2022. 7, 9, 10
2022
-
[39]
Zhengzhe Liu, Xiaojuan Qi, and Philip H. S. Torr. Global texture enhancement for fake face detection in the wild. In CVPR, 2020. 6, 7, 8, 9
2020
-
[40]
Deepfakes genera- tion and detection: state-of-the-art, open challenges, coun- termeasures, and way forward
Momina Masood, Marriam Nawaz, Khalid Mahmood Malik, Ali Javed, Aun Irtaza, and Hafiz Malik. Deepfakes genera- tion and detection: state-of-the-art, open challenges, coun- termeasures, and way forward. Appl. Intell., 2023. 3
2023
-
[41]
May, Kirill Trapeznikov, Shengbang Fang, and Matthew C
Brandon B. May, Kirill Trapeznikov, Shengbang Fang, and Matthew C. Stamm. Comprehensive dataset of synthetic and manipulated overhead imagery for development and evalua- tion of forensic tools. In IH&MMSec, 2023. 2
2023
-
[42]
Multi-spectral class center network for face manipulation de- tection and localization
Changtao Miao, Qi Chu, Zhentao Tan, Zhenchao Jin, Wanyi Zhuang, Yue Wu, Bin Liu, Honggang Hu, and Nenghai Yu. Multi-spectral class center network for face manipulation de- tection and localization. Arxiv, 2023. 3
2023
-
[43]
Large language models: A survey
Shervin Minaee, Tom ´as Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large language models: A survey. Arxiv, 2024. 3
2024
-
[44]
Absolutereality model repos- itory
Absolutereality model team. Absolutereality model repos- itory. https://huggingface.co/jochemstoel/ absolutereality-model-repository , 2024. Ac- cessed: 2024-11-15. 4
2024
-
[45]
Flux model
Flux model team. Flux model. https : / / huggingface . co / black - forest - labs / FLUX . 1-dev, 2024. Accessed: 2024-11-15. 4
2024
-
[46]
Laa-net: Localized artifact attention network for quality-agnostic and generalizable deepfake de- tection
Dat Nguyen, Nesryne Mejri, Inder Pal Singh, Polina Kuleshova, Marcella Astrid, Anis Kacem, Enjie Ghorbel, and Djamila Aouada. Laa-net: Localized artifact attention network for quality-agnostic and generalizable deepfake de- tection. In CVPR, 2024. 3
2024
-
[47]
IMD2020: A large-scale annotated dataset tailored for de- tecting manipulated images
Adam Novoz ´amsk´y, Babak Mahdian, and Stanislav Saic. IMD2020: A large-scale annotated dataset tailored for de- tecting manipulated images. In WACVW, 2020. 2
2020
-
[48]
Towards uni- versal fake image detectors that generalize across generative models
Utkarsh Ojha, Yuheng Li, and Yong Jae Lee. Towards uni- versal fake image detectors that generalize across generative models. In CVPR, 2023. 6, 7, 8, 9
2023
-
[49]
GPT-4 technical report
OpenAI. GPT-4 technical report. Arxiv, 2023. 4
2023
-
[50]
On the use of stable diffusion for creating realistic faces: from generation to detection
Lorenzo Papa, Lorenzo Faiella, Luca Corvitto, Luca Maiano, and Irene Amerini. On the use of stable diffusion for creating realistic faces: from generation to detection. In IWBF, 2023. 2
2023
-
[51]
Deepfake generation and detection: A benchmark and survey
Gan Pei, Jiangning Zhang, Menghan Hu, Guangtao Zhai, Chengjie Wang, Zhenyu Zhang, Jian Yang, Chunhua Shen, and Dacheng Tao. Deepfake generation and detection: A benchmark and survey. Arxiv, 2024. 3
2024
-
[52]
Kosmos-2: Grounding multimodal large language models to the world.Arxiv, 2023
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world.Arxiv, 2023. 3
2023
-
[53]
Plummer, Liwei Wang, Chris M
Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspon- dences for richer image-to-sentence models. IJCV, 2017. 4, 10
2017
-
[54]
SDXL: improving latent diffusion models for high-resolution image synthesis
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. SDXL: improving latent diffusion models for high-resolution image synthesis. In ICLR, 2024. 4
2024
-
[55]
Arti- fact: A large-scale dataset with artificial and factual images for generalizable and robust synthetic image detection
Md Awsafur Rahman, Bishmoy Paul, Najibul Haque Sarker, Zaber Ibn Abdul Hakim, and Shaikh Anowarul Fattah. Arti- fact: A large-scale dataset with artificial and factual images for generalizable and robust synthetic image detection. In ICIP, 2023. 2
2023
-
[56]
Shaker, Salman H
Hanoona Abdul Rasheed, Muhammad Maaz, Sahal Shaji Mullappilly, Abdelrahman M. Shaker, Salman H. Khan, Hisham Cholakkal, Rao Muhammad Anwer, Eric P. Xing, Ming-Hsuan Yang, and Fahad Shahbaz Khan. Glamm: Pixel grounding large multimodal model. In CVPR, 2024. 3
2024
-
[57]
Pixellm: Pixel reasoning with large multimodal model
Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel reasoning with large multimodal model. In CVPR, 2024. 3
2024
-
[58]
High-resolution image syn- thesis with latent diffusion models
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models. In CVPR, 2022. 4
2022
-
[59]
A survey of multimodal-guided image editing with text-to-image diffu- sion models
Xincheng Shuai, Henghui Ding, Xingjun Ma, Rongcheng Tu, Yu-Gang Jiang, and Dacheng Tao. A survey of multimodal-guided image editing with text-to-image diffu- sion models. Arxiv, 2024. 1
2024
-
[60]
Robustness and generalizability of deepfake detection: A study with diffusion models
Haixu Song, Shiyu Huang, Yinpeng Dong, and Wei-Wei Tu. Robustness and generalizability of deepfake detection: A study with diffusion models. Arxiv, 2023. 2
2023
-
[61]
Learning on gradients: Generalized arti- facts representation for gan-generated images detection
Chuangchuang Tan, Yao Zhao, Shikui Wei, Guanghua Gu, and Yunchao Wei. Learning on gradients: Generalized arti- facts representation for gan-generated images detection. In CVPR, 2023. 6, 7, 9
2023
-
[62]
Frequency-aware deepfake de- tection: Improving generalizability through frequency space domain learning
Chuangchuang Tan, Yao Zhao, Shikui Wei, Guanghua Gu, Ping Liu, and Yunchao Wei. Frequency-aware deepfake de- tection: Improving generalizability through frequency space domain learning. In AAAI, 2024. 3 24
2024
-
[63]
Weakly-supervised deepfake localization in diffusion-generated images
Dragos-Constantin T ˆantaru, Elisabeta Oneata, and Dan Oneata. Weakly-supervised deepfake localization in diffusion-generated images. In WACV, 2024. 3
2024
-
[64]
Representative forgery mining for fake face detection
Chengrui Wang and Weihong Deng. Representative forgery mining for fake face detection. In CVPR, 2021. 3
2021
-
[65]
Fakespotter: A simple yet robust baseline for spotting ai-synthesized fake faces
Run Wang, Felix Juefei-Xu, Lei Ma, Xiaofei Xie, Yihao Huang, Jian Wang, and Yang Liu. Fakespotter: A simple yet robust baseline for spotting ai-synthesized fake faces. In IJCAI, 2020. 2
2020
-
[66]
Cnn-generated images are sur- prisingly easy to spot...for now
Sheng-Yu Wang, Oliver Wang, Richard Zhang, Andrew Owens, and Alexei A Efros. Cnn-generated images are sur- prisingly easy to spot...for now. In CVPR, 2020. 2
2020
-
[67]
Dynamic graph learning with content-guided spatial- frequency relation reasoning for deepfake detection
Yuan Wang, Kun Yu, Chen Chen, Xiyuan Hu, and Silong Peng. Dynamic graph learning with content-guided spatial- frequency relation reasoning for deepfake detection. In CVPR, 2023. 3
2023
-
[68]
DIRE for diffusion-generated image detection
Zhendong Wang, Jianmin Bao, Wengang Zhou, Weilun Wang, Hezhen Hu, Hong Chen, and Houqiang Li. DIRE for diffusion-generated image detection. In ICCV, 2023. 2, 9
2023
-
[69]
Lasagna: Language-based segmentation assistant for complex queries
Cong Wei, Haoxian Tan, Yujie Zhong, Yujiu Yang, and Lin Ma. Lasagna: Language-based segmentation assistant for complex queries. Arxiv, 2024. 3
2024
-
[70]
Towards se- mantic equivalence of tokenization in multimodal llm
Shengqiong Wu, Hao Fei, Xiangtai Li, Jiayi Ji, Hanwang Zhang, Tat-Seng Chua, and Shuicheng Yan. Towards se- mantic equivalence of tokenization in multimodal llm. arXiv preprint arXiv:2406.05127, 2024. 3
2024 arXiv
-
[71]
GSV A: generalized segmentation via multimodal large language models
Zhuofan Xia, Dongchen Han, Yizeng Han, Xuran Pan, Shiji Song, and Gao Huang. GSV A: generalized segmentation via multimodal large language models. In CVPR, 2024. 3
2024
-
[72]
Fakeshield: Explainable image forgery detection and localization via multi-modal large lan- guage models
Zhipei Xu, Xuanyu Zhang, Runyi Li, Zecheng Tang, Qing Huang, and Jian Zhang. Fakeshield: Explainable image forgery detection and localization via multi-modal large lan- guage models. Arxiv, 2024. 3, 9, 10
2024
-
[73]
Deepfakebench: A comprehensive benchmark of deepfake detection
Zhiyuan Yan, Yong Zhang, Xinhang Yuan, Siwei Lyu, and Baoyuan Wu. Deepfakebench: A comprehensive benchmark of deepfake detection. In NeurIPS, 2023. 1
2023
-
[74]
An improved baseline for reasoning segmentation with large language model
Senqiao Yang, Tianyuan Qu, Xin Lai, Zhuotao Tian, Bohao Peng, Shu Liu, and Jiaya Jia. An improved baseline for reasoning segmentation with large language model. Arxiv,
-
[75]
Ferret: Refer and ground anything anywhere at any granularity
Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. In ICLR, 2024. 3
2024
-
[76]
Diffforensics: Leveraging diffusion prior to image forgery detection and localization
Zeqin Yu, Jiangqun Ni, Yuzhen Lin, Haoyi Deng, and Bin Li. Diffforensics: Leveraging diffusion prior to image forgery detection and localization. In CVPR, 2024. 2
2024
-
[77]
Contextual object detection with mul- timodal large language models
Yuhang Zang, Wei Li, Jun Han, Kaiyang Zhou, and Chen Change Loy. Contextual object detection with mul- timodal large language models. Arxiv, 2023. 3
2023
-
[78]
Fangneng Zhan, Yingchen Yu, Rongliang Wu, Jiahui Zhang, Shijian Lu, Lingjie Liu, Adam Kortylewski, Christian Theobalt, and Eric P. Xing. Multimodal image synthesis and editing: The generative AI era. IEEE TPAMI, 2023. 1, 2
2023
-
[79]
Text-to-image diffusion models in gen- erative AI: A survey
Chenshuang Zhang, Chaoning Zhang, Mengchun Zhang, and In So Kweon. Text-to-image diffusion models in gen- erative AI: A survey. Arxiv, 2023. 2
2023
-
[80]
Llava-grounding: Grounded visual chat with large multimodal models
Hao Zhang, Hongyang Li, Feng Li, Tianhe Ren, Xueyan Zou, Shilong Liu, Shijia Huang, Jianfeng Gao, Leizhang, Chunyuan Li, and Jainwei Yang. Llava-grounding: Grounded visual chat with large multimodal models. In ECCV, 2024. 3
2024
-
[81]
Vision-language models for vision tasks: A survey
Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. IEEE TPAMI, 2024. 3
2024
-
[82]
Magicbrush: A manually annotated dataset for instruction- guided image editing
Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction- guided image editing. In NeurIPS, 2023. 6, 11
2023
-
[83]
Perceptual artifacts localization for image synthesis tasks
Lingzhi Zhang, Zhengjie Xu, Connelly Barnes, Yuqian Zhou, Qing Liu, He Zhang, Sohrab Amirghodsi, Zhe Lin, Eli Shechtman, and Jianbo Shi. Perceptual artifacts localization for image synthesis tasks. In ICCV, 2023. 3
2023
-
[84]
Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding
Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shunping Ji, Change Loy Chen, and Shuicheng Yan. Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding. In NeurIPS, 2024. 3
2024
-
[85]
Common sense reasoning for deep fake detection
Yue Zhang, Ben Colman, Ali Shahriyari, and Gaurav Bharaj. Common sense reasoning for deep fake detection. Arxiv,
-
[86]
MFMS: learning modality-fused and modality-specific features for deepfake detection and local- ization tasks
Yi Zhang, Changtao Miao, Man Luo, Jianshu Li, Wenzhong Deng, Weibin Yao, Zhe Li, Bingyu Hu, Weiwei Feng, Tao Gong, and Qi Chu. MFMS: learning modality-fused and modality-specific features for deepfake detection and local- ization tasks. In ACM MM, 2024. 3
2024
-
[87]
Rich and poor texture contrast: A simple yet effective ap- proach for ai-generated image detection
Nan Zhong, Yiran Xu, Zhenxing Qian, and Xinpeng Zhang. Rich and poor texture contrast: A simple yet effective ap- proach for ai-generated image detection. Arxiv, 2023. 2, 3
2023
-
[88]
Corso, and Jianfeng Gao
Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Ja- son J. Corso, and Jianfeng Gao. Unified vision-language pre- training for image captioning and VQA. In AAAI, 2020. 5
2020
-
[89]
Genimage: A million-scale benchmark for de- tecting ai-generated image
Mingjian Zhu, Hanting Chen, Qiangyu Yan, Xudong Huang, Guanyu Lin, Wei Li, Zhijun Tu, Hailin Hu, Jie Hu, and Yunhe Wang. Genimage: A million-scale benchmark for de- tecting ai-generated image. In NeurIPS, 2023. 2, 3
2023
-
[90]
M3DSYNTH: A dataset of medical 3d images with ai-generated local manipulations
Giada Zingarini, Davide Cozzolino, Riccardo Corvi, Gio- vanni Poggi, and Luisa Verdoliva. M3DSYNTH: A dataset of medical 3d images with ai-generated local manipulations. In ICASSP, 2024. 2 25
2024
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.