REVIEW 4 major objections 5 minor 36 references
RAISE: Realness Assessment for Image Synthesis and Evaluation
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper introduces RAISE, a 600-image dataset with human realness scores, and shows that pretrained vision features can predict those scores.
desk verdict Useful small dataset, but the modeling claims are confounded by real-vs-generated separability and a sloppy ablation table. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The key machinery is the pairing of subjective mean opinion scores (MOS) obtained from a single-stimulus human study with transfer-learned deep features. The MOS values, averaged over at least 23 raters per image after outlier removal, provide a continuous ground-truth scale of realness from 0 to 100. The learned features come from a pretrained ResNet-18 backbone whose global-average-pooled 512-dimensional output is fed into a four-layer fully connected regressor, with the backbone kept fixed; this compact setup is what demonstrates that realness information is already present in generic foundation-model representations.
What would settle it
Re-evaluate all models under repeated random train/test splits or leave-one-out cross-validation on RAISE, and check whether the ranking of models and the reported SROCC values remain stable; alternatively, test the best model on an independent set of freshly generated images and see if its SROCC against human realness ratings stays well above zero.
Extended reading notes
Core claim
The discovery is that perceptual realness, a subjective quality distinct from technical image quality, is captured to a useful degree by generic features from deep vision backbones. Using the RAISE dataset's mean opinion scores as ground truth, the paper compares five approaches: handcrafted color, edge, contrast, texture, and frequency features with a decision tree (SROCC 0.5033); a lightweight CNN trained from scratch (0.6028); a regressor on frozen ImageNet-pretrained ResNet-18 features (0.6684); the existing JOINT naturalness model used as is (0.2998); and the JOINT rationality branch with partial fine-tuning (0.6798). The steady improvement from handcrafted features to foundation-model features is the evidence for the paper's claim that deep foundation vision models can effectively capture subjective realness.
Load-bearing premise
The comparisons between models rely on a single division of the 600 images into 510 training and 90 test images, so the small differences in correlation (for instance 0.6798 versus 0.6684) could vanish or swap under a different split.
Editorial extensions
If this is right
- Realness scores can be assigned to any single image without access to the text prompt that generated it, unlike prompt-dependent quality metrics.
- A small regressor on frozen pretrained features outperforms both handcrafted features and a CNN trained from scratch on 600 images, suggesting foundation-model representations are a strong default for perceptual tasks with limited data.
- The near-zero transfer of JOINT to RAISE indicates that technical naturalness and perceived realness are not the same thing, and that realness needs its own benchmark.
- The dataset provides a training and evaluation resource for realness models, and the authors propose using such models to guide generative models toward fewer unrealistic artifacts.
Reading between the lines
- Because real and AI-generated images overlap in rated realness, a model trained on RAISE is probably not a reliable forgery detector; it ranks plausibility, not provenance.
- The same kind of frozen-feature regressor could be adapted to rank or filter diffusion outputs during sampling, offering a parameter-free way to steer generation toward higher perceived realness.
- The reported differences between the top models rely on one fixed split, so the specific ordering of the transfer-learning variants should be treated as provisional until tested across multiple splits or an independent set.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces RAISE, a dataset of 600 images (480 AI-generated and 120 real photographs) annotated with subjective perceptual-realness scores from a 47-subject human study, and reports baseline models for predicting those scores. The baselines include a decision-tree regressor on handcrafted color, edge, contrast, texture/structure, and frequency features; a lightweight CNN trained from scratch; a neural regressor on frozen ResNet-18 features; and transfer-learning variants of the JOINT model. The central claim is that features from deep foundation vision models can effectively capture subjective realness.
Significance. If the central claim is properly established, RAISE would be a useful community resource: it focuses on perceptual realness rather than technical quality, it is prompt-agnostic, and it includes real images as reference anchors. The subjective study follows ITU recommendation procedures, reports inter-subject consistency, and releases the data and models publicly. However, the current evidence for the central claim is weakened by an uncontrolled source-label confound, a single small test split without significance assessment, and an internal inconsistency in the reported decision-tree results. These issues are fixable but load-bearing, because the paper's headline conclusion is about capturing subjective realness, not about separating real from generated images.
major comments (4)
- [Section V-A, Tables II and IV] There is an internal inconsistency in the reported decision-tree performance. Table II reports SROCC = 0.5033 for the model using all 55 handcrafted features, but Table IV reports SROCC = 0.5106 for the same decision-tree model. Moreover, the text states that the ablation study shows 'consistent drop in performance' when any feature category is excluded, yet Table II shows that excluding the Color category improves SROCC from 0.5033 to 0.5106. This directly contradicts the claimed consistency and needs to be corrected, not merely re-worded, because the ablation is used to support the contribution of each feature group.
- [Section V, Tables III and IV] All conclusions about model ranking rely on a single partition into 510 training and 90 test images, with no confidence intervals, bootstrap estimates, cross-validation, or significance tests. For example, the reported difference between the ResNet-18 transfer model (SROCC 0.6684) and the fine-tuned JOINT rationality model (SROCC 0.6798) may be within sampling noise on a 90-image test set. The paper should report uncertainty around the SROCC/PLCC values (e.g., bootstrap intervals or repeated random splits) before claiming that one model 'conclusively outperformed' another.
- [Section IV and Section V-D] The mixed real/AIGI test set confounds subjective realness with source-label separability. The test partition mixes AIGIs and real images, where real images have a mean MOS of 76.56 and AIGIs average 60.98, with a large FID of 177.07 between the subsets. A model that mostly learns to separate real images from AIGIs can therefore achieve a high SROCC on the mixed test set without capturing fine-grained realness within either category. To support the abstract's claim, the paper needs at least one of the following: (i) report SROCC/PLCC separately on the real-only and AIGI-only test subsets; (ii) compare against a trivial binary source classifier as a baseline; or (iii) report partial correlations controlling for the real/generated source label. Without such a control, the reported correlations (e.g., 0.6798) do not establish that the models assess perceived realness rather than synthetic-vs-real category.
- [Section V-C and V-D] The comparison between the frozen ResNet-18 feature extractor and the fine-tuned JOINT rationality branch is not matched in a way that isolates the contribution of the foundation-model features. The two models use different backbones (ResNet-18 vs ResNet-50), different numbers of trainable parameters, and different fine-tuning protocols. Since the paper's central contribution is the dataset rather than a new metric, this is not fatal, but the conclusion that 'features derived from deep foundation vision models can effectively capture the subjective realness' would be stronger if the same backbone were evaluated both frozen and fine-tuned, and if the fine-tuned JOINT variant were compared with a simple ResNet-50 baseline under the same protocol.
minor comments (5)
- [Introduction] The phrase 'the then RAISE dataset' in the first paragraph of Section I appears to be a typo; it should likely read 'the RAISE dataset.'
- [Section V-A, Table II] The column header 'Performance (excluding col. 1 for rows 1–5)' is confusing. It would be clearer to state explicitly in the caption that rows 1–5 report performance after omitting the indicated feature category.
- [Section V-D] The weights for the overall JOINT model are reported as 0.769 for the rationality branch and 0.145 for the technical quality branch, but these do not sum to 1. Please clarify whether these are the exact weights from [10] or whether a normalization step is implied.
- [Section IV] The statement that 'even real images can be perceived as synthetic by human raters' is inferred from the overlapping MOS distributions in Fig. 5. This inference would be more direct if the paper reported the fraction of ratings below a threshold separately for real and AIGI subsets.
- [References] Reference [5] is typeset as 'A V A' with unusual spacing; please verify the rendering and use the standard 'AVA' formatting.
Circularity Check
No significant circularity; the paper trains and evaluates supervised realness predictors on a held-out split of human MOS labels.
full rationale
The paper's derivation chain is empirical rather than definitional. The RAISE dataset is constructed from human subjective realness ratings (MOS), and each baseline model is trained on a 510-image partition and evaluated on a disjoint 90-image test partition (Section V). The reported correlations are computed against held-out human labels, so no 'prediction' is equivalent by construction to its training input. The JOINT model is an external baseline from Chen et al. with official weights, not a claim resting on the present authors' own prior work, and its adaptation involves training a new regressor on RAISE MOS labels. The skeptic's concern that mixed real/AIGI content may allow a model to succeed by source discrimination is a validity or interpretation risk, not a circularity, because the ground truth is perceived realness as rated by humans and the evaluation protocol does not substitute the source label for the human score. The dataset is self-referential only in the ordinary sense that 'realness' is operationalized through the collected ratings, which is the standard supervised-learning setup and does not make the evaluation circular. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors, and no known result is repackaged. Therefore the appropriate finding is no significant circularity.
Assumptions & free parameters
assumptions (3)
- domain assumption Perceptual realness is a distinct concept from technical quality and can be measured independently.
- domain assumption MOS computed from at least 23 ratings after kurtosis-based outlier removal is a reliable ground truth.
- domain assumption AIGIs sourced from Fake2M (Stable Diffusion v1.5) are representative of AI-generated images.
Cite this review
Pith. "Pith review of RAISE: Realness Assessment for Image Synthesis and Evaluation." pith.science (2026). https://pith.science/paper/4LTHMA6Q
@misc{pith2026250519233,
author = {Pith},
title = {Pith review of: RAISE: Realness Assessment for Image Synthesis and Evaluation},
year = {2026},
howpublished = {\url{https://pith.science/paper/4LTHMA6Q}},
note = {Machine review of arXiv:2505.19233}
}
read the original abstract
The rapid advancement of generative AI has enabled the creation of highly photorealistic visual content, offering practical substitutes for real images and videos in scenarios where acquiring real data is difficult or expensive. However, reliably substituting real visual content with AI-generated counterparts requires robust assessment of the perceived realness of AI-generated visual content, a challenging task due to its inherent subjective nature. To address this, we conducted a comprehensive human study evaluating the perceptual realness of both real and AI-generated images, resulting in a new dataset, containing images paired with subjective realness scores, introduced as RAISE in this paper. Further, we develop and train multiple models on RAISE to establish baselines for realness prediction. Our experimental results demonstrate that features derived from deep foundation vision models can effectively capture the subjective realness. RAISE thus provides a valuable resource for developing robust, objective models of perceptual realness assessment.
Figures
Reference graph
Works this paper leans on
-
[1]
High- resolution image synthesis with latent diffusion models,
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High- resolution image synthesis with latent diffusion models,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 10 684–10 695
2022
-
[2]
Hierarchical text-conditional image generation with clip latents,
A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, “Hierarchical text-conditional image generation with clip latents,” arXiv preprint arXiv:2204.06125, 2022
arXiv 2022
-
[3]
Making a “completely blind
A. Mittal, R. Soundararajan, and A. C. Bovik, “Making a “completely blind” image quality analyzer,” IEEE Signal Processing Letters, vol. 20, no. 3, pp. 209–212, 2013
2013
-
[4]
NIMA: Neural image assessment,
H. Talebi and P. Milanfar, “NIMA: Neural image assessment,” IEEE transactions on image processing , vol. 27, no. 8, pp. 3998–4011, 2018
work page 2018
-
[5]
A V A: A large-scale database for aesthetic visual analysis,
N. Murray, L. Marchesotti, and F. Perronnin, “A V A: A large-scale database for aesthetic visual analysis,” in IEEE conference on computer vision and pattern recognition , 2012, pp. 2408–2415
work page 2012
-
[6]
KonIQ-10k: An ecologi- cally valid database for deep learning of blind image quality assessment,
V . Hosu, H. Lin, T. Sziranyi, and D. Saupe, “KonIQ-10k: An ecologi- cally valid database for deep learning of blind image quality assessment,” IEEE Transactions on Image Processing, vol. 29, pp. 4041–4056, 2020
2020
-
[7]
A perceptual quality assessment exploration for AIGC images,
Z. Zhang, C. Li, W. Sun, X. Liu, X. Min, and G. Zhai, “A perceptual quality assessment exploration for AIGC images,” in 2023 IEEE In- ternational Conference on Multimedia and Expo Workshops (ICMEW) , 2023, pp. 440–445
work page 2023
-
[8]
AGIQA-3k: An open database for AI-generated image quality assessment,
C. Li, Z. Zhang, H. Wu, W. Sun, X. Min, X. Liu, G. Zhai, and W. Lin, “AGIQA-3k: An open database for AI-generated image quality assessment,” IEEE Transactions on Circuits and Systems for Video Technology, 2023
work page 2023
Show all 36 references
-
[9]
AIG- CIQA2023: A large-scale image quality assessment database for AI generated images: from the perspectives of quality, authenticity and correspondence,
J. Wang, H. Duan, J. Liu, S. Chen, X. Min, and G. Zhai, “AIG- CIQA2023: A large-scale image quality assessment database for AI generated images: from the perspectives of quality, authenticity and correspondence,” in CAAI International Conference on Artificial Intel- ligence, 2...
2023
-
[10]
Exploring the naturalness of AI-generated images,
Z. Chen, W. Sun, H. Wu, Z. Zhang, J. Jia, Z. Ji, F. Sun, S. Jui, X. Min, G. Zhai et al., “Exploring the naturalness of AI-generated images,” arXiv preprint arXiv:2312.05476, 2023
2023 arXiv
-
[11]
AIGIQA-20k: A large database for AI- generated image quality assessment,
C. Li, T. Kou, Y . Gao, Y . Cao, W. Sun, Z. Zhang, Y . Zhou, Z. Zhang, W. Zhang, H. Wu et al. , “AIGIQA-20k: A large database for AI- generated image quality assessment,” arXiv preprint arXiv:2404.03407 , vol. 2, no. 3, p. 5, 2024
2024 arXiv
-
[12]
Learning transferable visual models from natural language supervision,
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning , 2021, pp. 8748–8763
2021
-
[13]
Improved techniques for training GANs,
T. Salimans, I. Goodfellow, W. Zaremba, V . Cheung, A. Radford, and X. Chen, “Improved techniques for training GANs,” Advances in neural information processing systems , vol. 29, 2016
2016
-
[14]
GANs trained by a two time-scale update rule converge to a local nash equilibrium,
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “GANs trained by a two time-scale update rule converge to a local nash equilibrium,” Advances in neural information processing systems , vol. 30, 2017
2017
-
[15]
Demysti- fying MMD GANs,
M. Bi ´nkowski, D. J. Sutherland, M. Arbel, and A. Gretton, “Demysti- fying MMD GANs,” arXiv preprint arXiv:1801.01401 , 2018
2018 arXiv
-
[16]
NTIRE 2024 quality assessment of ai-generated content challenge,
X. Liu, X. Min, G. Zhai, C. Li, T. Kou, W. Sun, H. Wu, Y . Gao, Y . Cao, Z. Zhang et al., “NTIRE 2024 quality assessment of ai-generated content challenge,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 6337–6362
2024
-
[17]
Subjective-aligned dataset and metric for text-to-video quality assessment,
T. Kou, X. Liu, Z. Zhang, C. Li, H. Wu, X. Min, G. Zhai, and N. Liu, “Subjective-aligned dataset and metric for text-to-video quality assessment,” in Proceedings of the 32nd ACM International Conference on Multimedia, 2024, pp. 7793–7802
2024
-
[18]
AiGC image quality assessment via image-prompt correspondence,
F. Peng, H. Fu, A. Ming, C. Wang, H. Ma, S. He, Z. Dou, and S. Chen, “AiGC image quality assessment via image-prompt correspondence,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, vol. 6, 2024
2024
-
[19]
AIGCc-VQA: A holistic perception metric for aigc video quality assessment,
Y . Lu, X. Li, B. Li, Z. Yu, F. Guan, X. Wang, R. Liao, Y . Ye, and Z. Chen, “AIGCc-VQA: A holistic perception metric for aigc video quality assessment,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 6384–6394
2024
-
[20]
Global-local image perceptual score (GLIPS): Evaluating photorealistic quality of ai-generated images,
M. Aziz, U. Rehman, M. U. Danish, and K. Grolinger, “Global-local image perceptual score (GLIPS): Evaluating photorealistic quality of ai-generated images,” IEEE Transactions on Human-Machine Systems , 2025
2025
-
[21]
Seeing is not always believing: Benchmarking human and model perception of ai-generated images,
Z. Lu, D. Huang, L. Bai, J. Qu, C. Wu, X. Liu, and W. Ouyang, “Seeing is not always believing: Benchmarking human and model perception of ai-generated images,” Advances in Neural Information Processing Systems, vol. 36, pp. 25 435–25 447, 2023
2023
-
[22]
KADID-10k: A large-scale artificially distorted iqa database,
H. Lin, V . Hosu, and D. Saupe, “KADID-10k: A large-scale artificially distorted iqa database,” in 2019 Eleventh International Conference on Quality of Multimedia Experience (QoMEX) , 2019, pp. 1–3
2019
-
[23]
Most apparent distortion: full- reference image quality assessment and the role of strategy,
E. C. Larson and D. M. Chandler, “Most apparent distortion: full- reference image quality assessment and the role of strategy,” Journal of Electronic Imaging , vol. 19, no. 1, p. 011006, 2010. [Online]. Available: https://doi.org/10.1117/1.3267105
2010 doi
-
[24]
Subjective video quality assessment methods for multimedia applications,
ITU-T Recommendation P.910, “Subjective video quality assessment methods for multimedia applications,” 2008, https://www.itu.int/rec/ T-REC-P.910
2008
-
[25]
Recommendation ITU-R BT.500-15: Methodologies for the subjective assessment of the quality of television images,
International Telecommunication Union, Radiocommunication Sector (ITU-R), “Recommendation ITU-R BT.500-15: Methodologies for the subjective assessment of the quality of television images,” https://www. itu.int/rec/R-REC-BT.500-15-202305-I/en, 2023, accessed: 2025-05-04
2023
-
[26]
Exploring varying color spaces through representative forgery learning to improve deepfake detection,
M. A. Amin, Y . Hu, Y . Guan, and M. Z. Amin, “Exploring varying color spaces through representative forgery learning to improve deepfake detection,” Digital Signal Processing , vol. 147, p. 104426, 2024
2024
-
[27]
Leveraging edges and optical flow on faces for deepfake detection,
A. Chintha, A. Rao, S. Sohrawardi, K. Bhatt, M. Wright, and R. Ptucha, “Leveraging edges and optical flow on faces for deepfake detection,” in 2020 IEEE international joint conference on biometrics (IJCB) , 2020, pp. 1–10
2020
-
[28]
DeepFake videos detection based on texture features
B. Xu, J. Liu, J. Liang, W. Lu, and Y . Zhang, “DeepFake videos detection based on texture features.” Computers, Materials & Continua , vol. 68, no. 1, 2021
2021
-
[29]
Textural features for image classification,
R. M. Haralick, K. Shanmugam, and I. Dinstein, “Textural features for image classification,” IEEE Trans. Syst. Man Cybern., vol. SMC-3, no. 6, pp. 610–621, 1973
1973
-
[30]
Histograms of oriented gradients for human detection,
N. Dalal and B. Triggs, “Histograms of oriented gradients for human detection,” in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. (CVPR), vol. 1, 2005, pp. 886–893
2005
-
[31]
Image feature detectors for deepfake video detection,
F. F. Kharbat, T. Elamsy, A. Mahmoud, and R. Abdullah, “Image feature detectors for deepfake video detection,” in 2019 IEEE/ACS 16th International Conference on Computer Systems and Applications (AICCSA), 2019, pp. 1–4
2019
-
[32]
Leveraging frequency analysis for deep fake image recogni- tion,
J. Frank, T. Eisenhofer, L. Sch ¨onherr, A. Fischer, D. Kolossa, and T. Holz, “Leveraging frequency analysis for deep fake image recogni- tion,” in International conference on machine learning, 2020, pp. 3247– 3258
2020
-
[33]
Dropout: a simple way to prevent neural networks from over- fitting,
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhut- dinov, “Dropout: a simple way to prevent neural networks from over- fitting,” The journal of machine learning research , vol. 15, no. 1, pp. 1929–1958, 2014
1929
-
[34]
Batch normalization: Accelerating deep network training by reducing internal covariate shift,
S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in International conference on machine learning , 2015, pp. 448–456
2015
-
[35]
Deep residual learning for image recognition,
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2016, pp. 770–778
2016
-
[36]
ImageNet: A large-scale hierarchical image database,
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition . Ieee, 2009, pp. 248–255
2009
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.