Pith. sign in

REVIEW 2 major objections 5 minor 59 references

Towards Unified Benchmark and Models for Multi-Modal Perceptual Metrics

T0 review · 2 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A single multi-task perceptual metric, UniSim, beats task-specific specialists on average across a new 7-task, 25-dataset benchmark, while all existing metrics still fail to generalize to unseen task types.

desk verdict Useful benchmark paper with a genuine protocol-selection flaw in Appendix C that needs fixing before the ranking claims can be trusted. read the letter →

arxiv 2412.10594 v1 pith:WL6H63HW submitted 2024-12-13 cs.CV cs.LG

Sara Ghazanfari , Siddharth Garg , Nicolas Flammarion , Prashanth Krishnamurthy , Farshad Khorrami , Francesco Croce This is my paper · ORCID
classification cs.CVcs.LG
keywords perceptualsimilaritymetricsmulti-modalbenchmarktwo-alternativeforcedchoicevision-languagemodelsCLIPfine-tuningmulti-tasklearninggeneralizationhumanperceptionof
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that human perception of similarity should not be carved into separate narrow tasks, and that a single metric can be trained to handle several of them at once. To show this, it assembles UniSim-Bench, a benchmark of seven multi-modal similarity tasks drawn from 25 datasets, and measures how well general-purpose models (CLIP, LLaVA-NeXT, Mantis), task-specialized metrics (DreamSim, HPSv2, ImageReward, PAC-S, LIQE), and its own multi-task fine-tuned models (UniSim) align with human judgments. The central finding is that specialized metrics often lose to plain CLIP on tasks outside their training niche, while UniSim, fine-tuned simultaneously on four two-alternative forced-choice (2AFC) tasks, attains the highest average accuracy on the core tasks and generalizes to held-out datasets within those tasks. The paper also shows that none of these models generalizes well to truly unseen task types, which it takes as evidence that learning a unified perceptual metric that captures human similarity remains an open problem.

What carries the argument

The engine of the paper is the reduction of heterogeneous perceptual tasks to a common two-alternative forced choice (2AFC) format: every data point is a triplet (reference, alternative A, alternative B) plus a label saying which alternative humans find more similar to the reference, higher quality, or better aligned. This format lets UniSim-Bench compare all models on the same classification metric and lets the authors fine-tune a CLIP image encoder with a single hinge loss over all four core tasks simultaneously, keeping the text encoder frozen and using LoRA to limit overfitting. For the generative variant, the same 2AFC samples are converted into natural-language instructions and used to fine-tune LLaVA-NeXT, whose multi-image and image-text interleaved handling suits the triplet structure. The benchmark's train/held-out split is the third piece of machinery: by withholding specific datasets and whole task types from training, it defines three tiers of generalization—standard, intra-task, and inter-task—that structure all the conclusions.

What would settle it

Recompute the benchmark's task averages after normalizing each dataset by its chance-level difficulty or after calibrating each model separately per task; if plain CLIP then matches or beats UniSim on the average, the claim that multi-task training itself improves perceptual alignment would be disproved. Alternatively, add a new held-out 2AFC dataset whose human labels come from pairwise preference rankings rather than corruption or quality scores: if UniSim's advantage over its CLIP base vanishes there, its edge is task-specific rather than a general perceptual ability.

Watch

Extended reading notes

Core claim

The paper claims that a single multi-task fine-tuned perceptual metric, UniSim, achieves the highest average accuracy on the core 2AFC tasks of UniSim-Bench while also generalizing to held-out datasets within those tasks, and that specialized task-specific metrics frequently underperform general-purpose CLIP models on tasks outside their training domain. Concretely, UniSim models fine-tuned from CLIP (ViT-B/32 and ViT-L/14) and from LLaVA-NeXT-0.5B all rank first or second on nearly every core task and beat their own base models on every task, while DreamSim, ImageReward, HPSv2, PAC-S, and LIQE each show large drops when evaluated outside the task they were trained for. On out-of-distribution tasks (perceptual attribute assessment, odd-one-out, retrieval), however, both specialized and unified metrics fall below general-purpose baselines, except for near-OOD attribute assessment where multi-task training gives a small gain.

Load-bearing premise

The benchmark assumes the seven tasks, assembled from datasets with different annotation protocols, label meanings, and difficulty, all measure one underlying 'human perception of similarity,' so a simple average accuracy across tasks is a meaningful ranking of how well a model matches human perception.

Editorial extensions

If this is right

  • Specialized perceptual metrics should not be assumed to generalize: a metric trained for one task, such as text-to-image alignment, can fall below its own generic backbone on a closely related task such as caption-to-image alignment.
  • Multi-task fine-tuning on 2AFC data yields a single metric that beats both its base model and most specialists on the average of the core tasks, so unified training is a viable route rather than an obvious sacrifice in accuracy.
  • Intra-task generalization is achievable: UniSim improves on held-out datasets within a task it trained on, sometimes by large margins, as on CD-COCO and AGIQA-3K.
  • Inter-task generalization remains unsolved: on out-of-distribution tasks (PAA, OOO, IR), neither specialized nor unified perceptual metrics beat general-purpose models on average, indicating that current training fits task structure rather than a universal notion of similarity.
  • When the number of alternatives in IT-2AFC grows from 2 to 8, all models lose accuracy, but the CLIP-based UniSim retains roughly three times the accuracy of its base model, suggesting the unified training transfers some robustness to harder comparisons.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the benchmark averages tasks with equal weight, conclusions about which model wins could shift under a difficulty-weighted or application-weighted aggregate; this is an inference the paper does not explore.
  • The standardized 2AFC triplet structure suggests a natural extension: the same training recipe could be applied to other backbones (SigLIP, BLIP-2, larger LMMs) and to new datasets as they appear, effectively turning UniSim-Bench into a continuously growing evaluation instead of a fixed one.
  • If perceptual metrics are used as training rewards for generative models, the finding that specialists underperform general backbones out-of-domain implies that reward models should be evaluated on the target task's actual data distribution rather than only on the benchmark they were tuned on.
  • The near-OOD gain on perceptual attribute assessment hints that a metric trained on a broader set of tasks may transfer to simple photometric or geometric attributes; adding more attribute types beyond brightness, colorfulness, contrast, and sharpness would test this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper introduces UniSim-Bench, a benchmark of seven multi-modal perceptual tasks built from 25 existing datasets, and uses it to evaluate general-purpose vision-language models, task-specialized perceptual metrics, and the authors' proposed multi-task models UniSim (CLIP-based and LLaVA-NeXT-based). The main empirical claims are that specialized perceptual metrics often underperform general-purpose CLIP models on tasks outside their training domain, that a single multi-task fine-tuned model (UniSim) achieves the highest average accuracy on the core 2AFC tasks, and that all current models generalize poorly to the more distant OOD tasks. The paper also includes ablations on IT-2AFC training data and on scaling the number of alternatives in NAFC evaluation.

Significance. If the evaluation issues are resolved, this is a useful contribution: it provides a broad, openly available benchmark for perceptual similarity metrics and a clean multi-task training recipe that improves over single-task specialists while identifying their generalization limits. The paper is unusually thorough in its baseline coverage, and the authors release code and models, which strengthens reproducibility. The central claims are empirical rather than derived, so the main risk is not circularity but evaluation bias; the protocol-selection issue described below is the key concern that must be addressed before the headline rankings can be trusted.

major comments (2)
  1. [Appendix C, 'Evaluation setup'; Tables 1 and 2] The evaluation protocol for IQA and PAA is not fixed: for every encoder-based model, the authors test two scoring protocols (reference-prompt cosine similarity and the CLIP-IQA good/bad softmax) and report whichever 'performs best on average on the task.' This selection is made on the same test data used to compute Tables 1 and 7 and the PAA part of Tables 2 and 8, with no validation split and no correction for multiple comparisons. Because IQA contributes to the headline 'Avg' column and PAA contributes to the OOD average, the reported per-task numbers are maxima over two different metrics rather than the output of a single evaluation procedure. This is load-bearing for the central ranking claims: different models may benefit from the chosen protocol to different degrees, so the comparison of UniSim with CLIP and with specialized IQA models (e.g., LIQE, C2S) is not made under a single fixed rule. Please either (i) fix one protocol for all encoder models, (ii) select the protocol on a validation split, or (iii) report results separately under both protocols and demonstrate that the main conclusions are unchanged.
  2. [Section 3.2 and Tables 1-2 (average column)] The headline metric is a simple average of per-task average accuracies, but the tasks are heterogeneous in label semantics, difficulty, and chance level (2AFC, 3AFC, and retrieval). The paper states that the tasks are 'specific instances of a broader challenge' (Sec. 3.2), but it does not justify why equal weighting across tasks is the correct aggregation, nor does it show robustness to alternative aggregations (e.g., per-dataset averaging, chance-normalized accuracy, or leaving out a single task). Since the main conclusion that UniSim is the best unified metric and that specialized models underperform CLIP is stated in terms of this average, please add a robustness analysis of the aggregation choice; this would also help absorb the effect of the protocol-selection issue raised above.
minor comments (5)
  1. [Table 8, IR rows] The retrieval metric is not defined; please specify whether the reported numbers are mAP, recall@k, or another measure, and state how the per-dataset retrieval results are averaged into the task-level 'Avg'.
  2. [Appendix B.1 and Table 4] For datasets such as AGIQA-3K and KONIQ-10K, test samples are created by random pairing; please provide the exact pairing seed or release the generated pairs so that the benchmark results are exactly reproducible.
  3. [Table 1] The sentence 'UniSim ranks as the first or second best across nearly all tasks' is not accurate for IQA: UniSim ViT-L/14 (87.0) is behind LIQE (88.2) and C2S (90.2); please rephrase or qualify this claim.
  4. [Tables 1 and 7] Superscripts are used both to identify training datasets for specialized models and to label datasets in the table header, which makes the legend confusing; please use distinct notation for these two roles.
  5. [Equation (1)] Please state explicitly that y is the index of the more similar alternative (y=0 for z0, y=1 for z1), because the sign of the hinge loss depends on this convention and the current text is ambiguous.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the benchmark and model evaluation are empirical and held-out, with only a test-protocol selection caveat that is a validity concern rather than circular reasoning.

full rationale

The paper's central contributions are an empirical benchmark (UniSim-Bench) and fine-tuned models (UniSim) evaluated on held-out datasets. The derivation chain is not one in which inputs are transformed into outputs by construction: UniSim models are trained on a specified subset of datasets (Table 5) and then evaluated on left-out datasets within the core tasks (e.g., BAPPS, ImageReward, AGIQA-3K, CD-COCO, KONIQ-10K) as well as on OOD tasks (PAA, OOO, IR). No test labels enter the training objective, and the reported accuracies are measured on external human-annotated data, so the central ranking is an empirical finding rather than a fitted parameter renamed as a prediction. Self-citations in the related-work section (e.g., [7], [10], [11], [12]) are not load-bearing for the benchmark construction or the UniSim results. The main caveat is in App. C, where the authors state that for encoder-based models on IQA and PAA they 'test both approaches and report the results of the one which performs best on average on the task'; this is post-hoc protocol selection on the test data and could inflate the reported averages, but it is an evaluation-protocol concern, not a circular derivation. No reported quantity reduces to its own input by definition or by the paper's equations. Therefore, under the hard rules requiring an explicit reduction, no circular step qualifies.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The paper makes no new theoretical postulates. Its contributions are an empirical benchmark and fine-tuned models. The load-bearing assumptions are that the selected datasets and labels capture human perceptual similarity and that mean accuracy over tasks is a valid comparison metric. No free parameters are fitted to data in the sense of constants tuned to make a derivation work; training hyperparameters are standard and not part of the claim.

assumptions (2)
  • domain assumption The seven tasks and their 25 datasets jointly represent the relevant facets of human perceptual similarity, so that average accuracy across them is a meaningful measure of perceptual alignment.
    Invoked in Section 3.2 ('we consider these tasks as specific instances of a broader challenge') and in the averaging scheme of Tables 1 and 2. If the datasets are not comparable or representative, the comparative conclusions weaken.
  • domain assumption Human preference labels in the source datasets (NIGHTS, HPDv2, POLARIS, KADID-10K, etc.) are reliable ground truth for perceptual similarity and quality.
    All benchmark scores rest on these labels; the paper does not report annotator agreement or label noise (Section 3.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Unified Benchmark and Models for Multi-Modal Perceptual Metrics." pith.science (2026). https://pith.science/paper/WL6H63HW

@misc{pith2026241210594,
  author       = {Pith},
  title        = {Pith review of: Towards Unified Benchmark and Models for Multi-Modal Perceptual Metrics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WL6H63HW}},
  note         = {Machine review of arXiv:2412.10594}
}
read the original abstract

Human perception of similarity across uni- and multimodal inputs is highly complex, making it challenging to develop automated metrics that accurately mimic it. General purpose vision-language models, such as CLIP and large multi-modal models (LMMs), can be applied as zero-shot perceptual metrics, and several recent works have developed models specialized in narrow perceptual tasks. However, the extent to which existing perceptual metrics align with human perception remains unclear. To investigate this question, we introduce UniSim-Bench, a benchmark encompassing 7 multi-modal perceptual similarity tasks, with a total of 25 datasets. Our evaluation reveals that while general-purpose models perform reasonably well on average, they often lag behind specialized models on individual tasks. Conversely, metrics fine-tuned for specific tasks fail to generalize well to unseen, though related, tasks. As a first step towards a unified multi-task perceptual similarity metric, we fine-tune both encoder-based and generative vision-language models on a subset of the UniSim-Bench tasks. This approach yields the highest average performance, and in some cases, even surpasses taskspecific models. Nevertheless, these models still struggle with generalization to unseen tasks, highlighting the ongoing challenge of learning a robust, unified perceptual similarity metric capable of capturing the human notion of similarity. The code and models are available at https://github.com/SaraGhazanfari/UniSim.

Figures

Figures reproduced from arXiv: 2412.10594 by the authors.

Figure 1
Figure 1. Summary of our UniSim framework. First, we frame the existing multi-modal perceptual similarity tasks into our unified benchmark UniSim-Bench (from which the Core 2AFC Tasks are illustrated in the top row). Second, we show that models specialized in individual tasks (e.g. DreamSim [8], HPSv2 [48], PAC-S [40], LIQE [55]) do not generalize well to unseen perceptual tasks, with even worse accuracy than CLIP [39]. Final… view at source ↗
Figure 2
Figure 2. OOD Generalization Tasks in UniSim-Bench. We il￾lustrate samples from the three tasks not used for training, but to evaluate the model’s generalization capabilities. sufficient intra-class diversity, making the task non-trivial. A triplet consists of two images from the same macro-class and one from a different one, the ground-truth odd one. Image-to-Image Retrieval ( IR ). Perceptual metrics have also been employed… view at source ↗
Figure 3
Figure 3. Increasing the alternatives in Image-to-Text Align￾ment task. We report accuracy as the number of alternative im￾ages increases in the IT-2AFC (HPDV2 dataset). Both UniSim models preserve higher accuracy than the respective baselines (the gap is highlighted in the plot) as the number of alternatives grows. Notably, our encoder-based UniSim ViT-L/14 significantly outper￾forms the other metrics, including LMM-based Un… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 32 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  2. [2]

    Getting vit in shape: Scaling laws for compute-optimal model design

    Ibrahim M Alabdulmohsin, Xiaohua Zhai, Alexander Kolesnikov, and Lucas Beyer. Getting vit in shape: Scaling laws for compute-optimal model design. Advances in Neural Information Processing Systems, 36, 2024. 6

    work page 2024

  3. [3]

    Improving image captioning descriptive- ness by ranking and llm-based fusion

    Simone Bianco, Luigi Celona, Marco Donzella, and Paolo Napoletano. Improving image captioning descriptive- ness by ranking and llm-based fusion. arXiv preprint arXiv:2306.11593, 2023. 4, 13

    arXiv 2023

  4. [4]

    Learning a deep single image contrast enhancer from multi-exposure images

    Jianrui Cai, Shuhang Gu, and Lei Zhang. Learning a deep single image contrast enhancer from multi-exposure images. IEEE Transactions on Image Processing, 27(4):2049–2062,

  5. [5]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In ICCV, 2021. 2, 12, 17

    work page 2021

  6. [6]

    Reproducible scal- ing laws for contrastive language-image learning

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuh- mann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scal- ing laws for contrastive language-image learning. In CVPR,

  7. [7]

    Adversarially robust clip mod- els induce better (robust) perceptual metrics

    Francesco Croce, Christian Schlarmann, Naman Deep Singh, and Matthias Hein. Adversarially robust clip mod- els induce better (robust) perceptual metrics. In ICML 2024 Workshop on Foundation Models in the Wild, 2024. 3, 5, 12, 15

    work page 2024

  8. [8]

    Dream- sim: Learning new dimensions of human visual similarity using synthetic data

    Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dream- sim: Learning new dimensions of human visual similarity using synthetic data. In NeurIPS, 2023. 1, 2, 3, 6, 13, 14

    work page 2023

Show all 59 references
  1. [9]

    Blink: Multimodal large language models can see but not perceive

    Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. arXiv preprint arXiv:2404.12390,

    arXiv

  2. [10]

    R-LPIPS: An adversarially robust perceptual similarity metric

    Sara Ghazanfari, Siddharth Garg, Prashanth Krishnamurthy, Farshad Khorrami, and Alexandre Araujo. R-LPIPS: An adversarially robust perceptual similarity metric. In ICML Workshop on New Frontiers in Adversarial Machine Learn- ing, 2023. 12

    2023

  3. [11]

    Emma: Ef- ficient visual alignment in multi-modal llms

    Sara Ghazanfari, Alexandre Araujo, Prashanth Krishna- murthy, Siddharth Garg, and Farshad Khorrami. Emma: Ef- ficient visual alignment in multi-modal llms. arXiv preprint arXiv:2410.02080, 2024. 3

    2024 arXiv

  4. [12]

    Lipsim: A provably robust perceptual similarity metric

    Sara Ghazanfari, Alexandre Araujo, Prashanth Krishna- murthy, Farshad Khorrami, and Siddharth Garg. Lipsim: A provably robust perceptual similarity metric. In ICLR, 2024. 12

    2024

  5. [13]

    Generative adversarial networks

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Commu- nications of the ACM, 63(11):139–144, 2020. 13

    2020

  6. [14]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022. 12

    2022

  7. [15]

    Clipscore: A reference-free evaluation met- ric for image captioning

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation met- ric for image captioning. In EMNLP, 2021. 2, 3, 12

    2021

  8. [16]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 13

    2020

  9. [17]

    Koniq-10k: An ecologically valid database for deep learning of blind image quality assessment

    Vlad Hosu, Hanhe Lin, Tamas Sziranyi, and Dietmar Saupe. Koniq-10k: An ecologically valid database for deep learning of blind image quality assessment. IEEE Transactions on Image Processing, 29:4041–4056, 2020. 4, 13, 14

    2020

  10. [18]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In ICLR, 2022. 5, 15

    2022

  11. [19]

    Aesexpert: Towards multi-modality foun- dation model for image aesthetics perception

    Yipo Huang, Xiangfei Sheng, Zhichao Yang, Quan Yuan, Zhichao Duan, Pengfei Chen, Leida Li, Weisi Lin, and Guangming Shi. Aesexpert: Towards multi-modality foun- dation model for image aesthetics perception. InProceedings of the 32nd ACM International Conference on Multimedia ,...

    2024

  12. [20]

    Hq-edit: A high-quality dataset for instruction-based image editing

    Mude Hui, Siwei Yang, Bingchen Zhao, Yichun Shi, Heng Wang, Peng Wang, Yuyin Zhou, and Cihang Xie. Hq-edit: A high-quality dataset for instruction-based image editing. arXiv preprint arXiv:2404.09990, 2024. 4, 13, 14

    2024 arXiv

  13. [21]

    Mantis: Interleaved multi-image instruction tuning

    Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved multi-image instruction tuning. arXiv preprint arXiv:2405.01483, 2024. 2, 3, 6

    2024 arXiv

  14. [22]

    Pipal: a large-scale image quality assessment dataset for perceptual image restoration

    Gu Jinjin, Cai Haoming, Chen Haoyu, Ye Xiaoxing, Jimmy S Ren, and Dong Chao. Pipal: a large-scale image quality assessment dataset for perceptual image restoration. In ECCV, 2020. 4, 13, 14

    2020

  15. [23]

    Imagenhub: Standardizing the evaluation of conditional image generation models.arXiv preprint arXiv:2310.01596, 2023

    Max Ku, Tianle Li, Kai Zhang, Yujie Lu, Xingyu Fu, Wen- wen Zhuang, and Wenhu Chen. Imagenhub: Standardizing the evaluation of conditional image generation models.arXiv preprint arXiv:2310.01596, 2023. 3

    2023 arXiv

  16. [24]

    Umic: An unreferenced metric for image captioning via contrastive learning

    Hwanhee Lee, Seunghyun Yoon, Franck Dernoncourt, Trung Bui, and Kyomin Jung. Umic: An unreferenced metric for image captioning via contrastive learning. arXiv preprint arXiv:2106.14019, 2021. 2, 3

    2021 arXiv

  17. [25]

    Agiqa-3k: An open database for ai-generated image quality assessment

    Chunyi Li, Zicheng Zhang, Haoning Wu, Wei Sun, Xiongkuo Min, Xiaohong Liu, Guangtao Zhai, and Weisi Lin. Agiqa-3k: An open database for ai-generated image quality assessment. IEEE Transactions on Circuits and Sys- tems for Video Technology, 2023. 3, 4, 13

    2023

  18. [26]

    Aigiqa-20k: A large database for ai-generated image quality assessment

    Chunyi Li, Tengchuan Kou, Yixuan Gao, Yuqin Cao, Wei Sun, Zicheng Zhang, Yingjie Zhou, Zhichao Zhang, Weixia Zhang, Haoning Wu, et al. Aigiqa-20k: A large database for ai-generated image quality assessment. arXiv preprint arXiv:2404.03407, 2(3):5, 2024. 3

    2024 arXiv

  19. [27]

    Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895 , 2024. 2, 3, 5, 6, 15

    2024 arXiv

  20. [28]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv:2301.12597, 2023. 2, 3, 6, 12

    2023 arXiv

  21. [29]

    Kadid-10k: A large-scale artificially distorted iqa database

    Hanhe Lin, Vlad Hosu, and Dietmar Saupe. Kadid-10k: A large-scale artificially distorted iqa database. In 2019 Tenth International Conference on Quality of Multimedia Experi- ence (QoMEX), 2019. 4, 13, 14

    2019

  22. [30]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. 4, 13

    2014

  23. [31]

    S. W. Link and R. A. Heath. A sequential theory of psy- chological discrimination. Psychometrika, 40:77–105, 1975. 2

    1975

  24. [32]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 2, 3

    2024

  25. [33]

    Fusedream: Training-free text-to-image generation with improved clip+ gan space op- timization

    Xingchao Liu, Chengyue Gong, Lemeng Wu, Shujian Zhang, Hao Su, and Qiang Liu. Fusedream: Training-free text-to-image generation with improved clip+ gan space op- timization. arXiv preprint arXiv:2112.01573, 2021. 2, 3, 5, 12, 15

    2021 arXiv

  26. [34]

    Roberta: A robustly optimized bert pretraining approach

    Y Liu, M Ott, N Goyal, J Du, M Joshi, D Chen, O Levy, M Lewis, L Zettlemoyer, and V Stoyanov. Roberta: A robustly optimized bert pretraining approach. arxiv [preprint](2019). arXiv preprint arXiv:1907.11692, 1907. 12

    2019 arXiv

  27. [35]

    Vandermeulen, and Simon Kornblith

    Lukas Muttenthaler, Jonas Dippel, Lorenz Linhardt, Robert A. Vandermeulen, and Simon Kornblith. Human alignment of neural network representations. In ICLR, 2023. 4, 13, 14

    2023

  28. [36]

    Lost in quantization: Improving particu- lar object retrieval in large scale image databases

    James Philbin, Ondrej Chum, Michael Isard, Josef Sivic, and Andrew Zisserman. Lost in quantization: Improving particu- lar object retrieval in large scale image databases. In CVPR,

  29. [37]

    Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 2

    2023 arXiv

  30. [38]

    Pieapp: Perceptual image-error assessment through pairwise preference

    Ekta Prashnani, Hong Cai, Yasamin Mostofi, and Pradeep Sen. Pieapp: Perceptual image-error assessment through pairwise preference. In CVPR, 2018. 3, 4, 13, 14

    2018

  31. [39]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021. 1, 2,...

    2021

  32. [40]

    Positive-augmented contrastive learning for image and video captioning evaluation

    Sara Sarto, Manuele Barraco, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. Positive-augmented contrastive learning for image and video captioning evaluation. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6914–6924, 2023. ...

    2023

  33. [41]

    When does perceptual alignment benefit vision representations? arXiv preprint arXiv:2410.10817 ,

    Shobhita Sundaram, Stephanie Fu, Lukas Muttenthaler, Ne- tanel Y Tamir, Lucy Chai, Simon Kornblith, Trevor Darrell, and Phillip Isola. When does perceptual alignment benefit vision representations? arXiv preprint arXiv:2410.10817 ,

    arXiv

  34. [42]

    Chameleon: Mixed-modal early-fusion foundation models

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818, 2024. 2

    2024 arXiv

  35. [43]

    Polos: Multimodal metric learning from human feed- back for image captioning

    Yuiga Wada, Kanta Kaneda, Daichi Saito, and Komei Sug- iura. Polos: Multimodal metric learning from human feed- back for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13559–13568, 2024. 2, 3, 4, 12, 13, 14

    2024

  36. [44]

    Muirbench: A comprehensive bench- mark for robust multi-image understanding

    Fei Wang, Xingyu Fu, James Y Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, et al. Muirbench: A comprehensive bench- mark for robust multi-image understanding. arXiv preprint arXiv:2406.09411, 2024. 3

    2024 arXiv

  37. [45]

    Ex- ploring clip for assessing the look and feel of images

    Jianyi Wang, Kelvin CK Chan, and Chen Change Loy. Ex- ploring clip for assessing the look and feel of images. In AAAI, 2023. 2, 12, 16

    2023

  38. [46]

    Q-bench: A benchmark for general-purpose foundation models on low-level vision

    Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, et al. Q-bench: A benchmark for general-purpose foundation models on low-level vision. arXiv preprint arXiv:2309.14181, 2023. 2, 3

    2023 arXiv

  39. [47]

    Towards open-ended visual quality comparison

    Haoning Wu, Hanwei Zhu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Chunyi Li, Annan Wang, Wenxiu Sun, Qiong Yan, et al. Towards open-ended visual quality comparison. arXiv preprint arXiv:2402.16641, 2024. 2, 3, 6, 12

    2024 arXiv

  40. [48]

    Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis

    Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.09341 ,

    arXiv

  41. [49]

    Imagere- ward: Learning and evaluating human preferences for text- to-image generation

    Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagere- ward: Learning and evaluating human preferences for text- to-image generation. In NeurIPS, 2023. 2, 3, 4, 6, 12, 13

    2023

  42. [50]

    mplug- owl2: Revolutionizing multi-modal large language model with modality collaboration

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug- owl2: Revolutionizing multi-modal large language model with modality collaboration. In CVPR, 2024. 3, 6, 12

    2024

  43. [51]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. arXiv preprint arXiv:2303.15343, 2023. 17

    2023 arXiv

  44. [52]

    Text-to-image diffusion models in gener- ative ai: A survey

    Chenshuang Zhang, Chaoning Zhang, Mengchun Zhang, and In So Kweon. Text-to-image diffusion models in gener- ative ai: A survey. arXiv preprint arXiv:2303.07909, 2023. 2

    2023 arXiv

  45. [53]

    Magicbrush: A manually annotated dataset for instruction- guided image editing

    Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction- guided image editing. Advances in Neural Information Pro- cessing Systems, 36, 2024. 4, 13, 14

    2024

  46. [54]

    Efros, Eli Shecht- man, and Oliver Wang

    Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shecht- man, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018. 2, 3, 12, 13

    2018

  47. [55]

    Blind image quality assessment via vision- language correspondence: A multitask learning perspective

    Weixia Zhang, Guangtao Zhai, Ying Wei, Xiaokang Yang, and Kede Ma. Blind image quality assessment via vision- language correspondence: A multitask learning perspective. In CVPR, 2023. 1, 2, 3, 6, 12

    2023

  48. [56]

    A-bench: Are lmms masters at evaluat- ing ai-generated images? arXiv preprint arXiv:2406.03070,

    Zicheng Zhang, Haoning Wu, Chunyi Li, Yingjie Zhou, Wei Sun, Xiongkuo Min, Zijian Chen, Xiaohong Liu, Weisi Lin, and Guangtao Zhai. A-bench: Are lmms masters at evaluat- ing ai-generated images? arXiv preprint arXiv:2406.03070,

    arXiv

  49. [57]

    2afc prompting of large multimodal models for image quality assessment

    Hanwei Zhu, Xiangjie Sui, Baoliang Chen, Xuelin Liu, Peilin Chen, Yuming Fang, and Shiqi Wang. 2afc prompting of large multimodal models for image quality assessment. arXiv preprint arXiv:2402.01162, 2024. 3

    2024 arXiv

  50. [58]

    Adaptive image quality assess- ment via teaching large multimodal model to compare.arXiv preprint arXiv:2405.19298, 2024

    Hanwei Zhu, Haoning Wu, Yixuan Li, Zicheng Zhang, Bao- liang Chen, Lingyu Zhu, Yuming Fang, Guangtao Zhai, Weisi Lin, and Shiqi Wang. Adaptive image quality assess- ment via teaching large multimodal model to compare.arXiv preprint arXiv:2405.19298, 2024. 2, 3, 12 A. Extended ...

    2024 arXiv

  51. [2023]

    1, 2, 3, 4, 6, 12, 13, 14

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.