Pith. sign in

REVIEW 5 major objections 5 minor 65 references

Human Preference-Aligned Concept Customization Benchmark via Decomposed Evaluation

T0 review · 5 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that decomposing an image-generation evaluation into 18 aspect-wise GPT-4o scores yields a metric whose judgments track human preference better than existing concept-customization metrics.

desk verdict Useful benchmark and a plausible decomposed MLLM metric, but the 'surpasses all metrics' claim is tested only against weak baselines and the human ground truth is lab-internal. read the letter →

arxiv 2509.03385 v1 pith:QCBTXEIB submitted 2025-09-03 cs.CV

Reina Ishikawa , Ryo Fujii , Hideo Saito , Ryo Hachiuma This is my paper
classification cs.CV
keywords conceptcustomizationhuman-alignedevaluationMLLM-as-a-judgedecomposedmulti-conceptimagegenerationbenchmarkdatasetGPT-4o
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Concept customization—adapting a text-to-image model to generate a specific person or object from a few reference images—is hard to evaluate automatically. The paper's claim is that decomposing the evaluation into 18 named aspects, asking GPT-4o to score each aspect separately, and averaging those scores yields judgments that track human preference far better than existing metrics. On the paper's new benchmark, CC-AlignBench, D-GPTScore reaches an overall Pearson correlation of 0.78 and Spearman correlation of 0.69 with human preference scores, while the existing baselines stay well below that on both measures. The benchmark itself is designed to stage difficulty: single-person actions, two people acting independently, and two people in mutual interactions. A sympathetic reader would care because evaluation drives progress: if this holds, model developers can get interpretable, human-like feedback on what customization systems get right and wrong.

What carries the argument

D-GPTScore (Decomposed GPT Score): a two-phase metric. Phase 1 feeds each of 18 predefined evaluation aspects—one at a time—into an MLLM (GPT-4o) together with the generated image and, depending on the aspect, the text prompt and/or reference images, collecting a 1–5 score per aspect. Phase 2 aggregates via a simple average to a 1–10 overall score. The load-bearing idea is that a single MLLM pass produces a coarse impression, while aspect-wise prompting forces the model to attend to each fidelity and quality criterion separately. CC-AlignBench accompanies it with 980 prompts over three difficulty tiers, from a single person's action to two people in mutual interaction.

What would settle it

Take the same 720 generated images and collect preference scores from a large, independent crowd; if D-GPTScore's Pearson correlation drops well below 0.78 or Spearman below 0.69, the result is an artifact of the lab annotators rather than a general human-alignment property.

Watch

Extended reading notes

Core claim

The paper proposes that evaluation of concept customization should be neither a single holistic score nor one narrow similarity measure, but a decomposed judgment over a fixed aspect taxonomy. It defines 18 aspects—13 covering concept fidelity (subject type and quantity, layout, size, color, proportions, actions and expressions, facial similarity, clothing, surroundings, human/animal and object interactions) and 5 covering quality (deformation, artifacts, detail, style)—and prompts GPT-4o to rate each on a 1–5 scale while selectively receiving the text prompt and/or reference images. The 18 scores are averaged into a 1–10 overall score. On the 720-image human-annotated subset of CC-AlignBenc

Load-bearing premise

The ground-truth human preferences are the averaged ratings of 12 annotators from the authors' own laboratory; if those preferences differ from the wider population's, the claimed alignment is not general human alignment.

Editorial extensions

If this is right

  • If D-GPTScore's correlation holds, automated evaluation of concept customization no longer has to proxy human judgment through face-similarity or CLIP-text similarity alone; a single MLLM pipeline can serve as the primary metric.
  • The stage-wise benchmark makes model differences visible: OMG variants lead on easy prompts but lose ground on hard mutual interactions, while DreamBooth degrades least, so progress can be targeted at interaction fidelity.
  • Per-aspect scores give actionable feedback: all six models score low on Actions & Expressions and Human & Animal Interactions, pointing to the specific failure mode that next-generation customization systems need to solve.
  • Ablations imply that decomposition—not the choice of MLLM—is what buys human alignment, so cheaper models can be substituted when API cost matters.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the reported 0.78/0.69 alignment is measured against 12 annotators from a single lab; a fair test of the human-aligned claim would re-run the same correlation with a larger, more diverse annotator pool.
  • Beyond the paper: because the benchmark uses one synthetic male and one synthetic female identity, generalization to other identities, ages, and ethnicities is an open question; the metric's aspect prompts may need calibration if those subjects shift failure modes.
  • Beyond the paper: the same aspect-wise decomposition could be turned from an evaluation into a training signal—for example, as a reward or rejection filter at inference time—since each aspect score identifies which criterion a generated image fails.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes D-GPTScore, an MLLM-based evaluation metric for concept customization, and CC-AlignBench, a benchmark dataset focused on single- and multi-person actions and interactions. The metric decomposes evaluation into 18 aspects covering concept fidelity and image quality, scores each aspect with GPT-4o on a 1–5 scale, and aggregates the scores into an overall score. The benchmark contains 980 prompt variations across three difficulty levels. The central empirical claim is that D-GPTScore correlates strongly with human preference (Pearson 0.78, Spearman 0.69, Table 3) and significantly surpasses existing metrics, based on 720 images scored by 12 in-house annotators. The paper also reports benchmark scores for six customization models and ablations of decomposition, MLLM choice, and aggregation method.

Significance. If the results hold, the decomposed MLLM-judge approach is a plausible step toward more human-aligned evaluation of concept customization, and the benchmark's inclusion of multi-person interactions fills a real gap relative to existing datasets such as CustomConcept101. The paper's strengths include a clear decomposition of evaluation criteria, a new benchmark with staged difficulty, ablation studies isolating decomposition and aggregation choices, and a public release of code and data. However, the central claim of 'significantly surpassing all existing metrics' rests on a narrow set of baselines and a small, lab-only human annotation study without statistical inference. The scale inconsistency between Eq. (4) and Table 4 also needs to be resolved before the benchmark scores can be interpreted. The core idea is sound and likely useful, but the current evidence is not yet sufficient for the strong claims made.

major comments (5)
  1. [Section 2.3 vs. Section 5.2, Table 3] The claim that D-GPTScore 'significantly surpasses all existing metrics' is tested only against ArcFace, CLIP T2I/T2T, CLIP Aesthetic, and DINO. These are generic or partial-aspect metrics. The paper's own related-work section (Section 2.3) identifies VIEScore, CIGEval, DreamBench++, ImageReward, VQA Score, and GPT-4V Score as prior MLLM-based or decomposed metrics for conditional image synthesis and concept customization. None of these appear in Table 3. Without comparisons to the closest competitors, the empirical advantage cannot be attributed to decomposition, to the MLLM backbone, or simply to the choice of weak baselines. Please add at least the most relevant MLLM/decomposed baselines, or substantially qualify the 'all existing metrics' claim.
  2. [Section 5.1, Preference score annotation; Section 5.2, Table 3] The correlations in Table 3 are computed against human scores from 12 annotators, all members of the authors' laboratory, with no confidence intervals, p-values, or inter-annotator agreement reported. The word 'significantly' is used without any statistical test. Moreover, the 720 images are not independent: each prompt is evaluated across six generation models, so per-model correlations and the overall correlation are clustered. Please provide bootstrap confidence intervals or other uncertainty estimates for the Pearson/Spearman correlations, report inter-annotator agreement (e.g., Krippendorff's alpha or ICC), and account for clustering. This is necessary to support the central human-alignment claim.
  3. [Section 3.2, Eq. (4); Section 4; Table 4 and Table 5] There is a scale inconsistency. Aspect-wise scores s_n are collected on a 1–5 scale (Section 3.1 and Supplementary Table 12), and Eq. (4) defines the overall score as the simple average of these scores. Such an average cannot exceed 5, yet Table 4 reports benchmark scores up to 7.28, and Table 5 lists scores such as 8.250 and 3.375. Eq. (3) says the overall score is on a [1,10] scale, but the mapping is not given. This must be corrected—either by explaining a scaling factor (e.g., multiplication by 2) or by adjusting Eq. (4). The current text makes Table 4 uninterpretable and the benchmark results difficult to reproduce.
  4. [Section 5.1, Preference score annotation] The human ground truth comes exclusively from 12 members of the authors' laboratory. The paper does not discuss whether these annotators are representative of general human preference, nor does it provide any evidence of annotation reliability. The 'human-aligned' claim is only as strong as the ground truth. At minimum, the limitations section should acknowledge this, and the authors should report agreement statistics, perhaps on a held-out subset, and discuss potential demographic or expertise bias. Without this, the generalization of the reported correlations beyond the lab is uncertain.
  5. [Section 5.2, last paragraph] The statement that 'the strong correlation exceeding 0.7 suggests that the proposed decomposed aspects are necessary and sufficient' is a logical overreach. A high correlation with the human preference scores does not establish that the 18 aspects are necessary or sufficient; it only indicates that the resulting aggregate score tracks the human labels in this dataset. The decomposition ablation in Table 6 shows that decomposition helps, but it does not test individual aspect necessity. Please soften this claim or provide an aspect-removal ablation.
minor comments (5)
  1. [Figures 2 and 3] Typo: 'Sbject & Camera Positioning' should be 'Subject & Camera Positioning'. Also Figure 3 uses 'Target Deformation' while the text consistently uses 'Subject Deformation'; keep terminology unified.
  2. [Figure 1] The score example in the teaser is confusing: it shows aspect scores such as '2/5', '3/5', '5/5' and a final score of '5.625' or '6.75/10' without explaining the aggregation and scaling. This should be clarified or simplified.
  3. [Supplementary Section D] The claim that 'It has been empirically demonstrated that by eliminating unnecessary inputs, MLLM can provide more accurate evaluations' has no citation or supporting experiment in the paper. Please provide a reference or report the supporting ablation.
  4. [Table 4 caption] The caption mentions leave-one-out linear regression, but Table 4 shows only the average-aggregation results. Clarify whether the linear-regression numbers appear elsewhere, or remove this caption text from Table 4.
  5. [Section 5.5, MLLM ablation] Please report the temperature, maximum tokens, and other inference settings for the GPT-4o calls. API-based evaluation is stochastic, and this information is needed for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: D-GPTScore's default score is a fixed average of GPT-4o aspect ratings, not fitted to the benchmark's human labels.

full rationale

The derivation of D-GPTScore does not reduce to its own inputs. The default aggregation is a fixed average (Eq. 4) with no parameters fitted to the human preference labels; the 18 aspects are explicitly grounded in prior work [13,62] and author-defined taxonomies, not learned from the annotation data. The correlation reported in Table 3 is therefore an independent association between GPT-4o aspect scores and averaged human scores, not a fitted value. The linear-regression ablation (Table 6) is the only place where human labels enter the metric, and the paper uses leave-one-out per generative model and reports no improvement over averaging (0.75/0.62 vs 0.78/0.69), so it is not a 'prediction' smuggled from a fit. No self-citations are load-bearing: references [13,62] are by other research groups, and no uniqueness theorem or prior result by the present authors is invoked. The fact that all 12 annotators are from the authors' laboratory is a legitimate generalization limitation, and the absence of stronger baselines (VIEScore, DreamBench++, etc.) is a comparative-validity concern, but neither is an instance of the claimed derivation being equivalent to its assumptions. The central claim is therefore self-contained against the charge of circularity.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The metric and benchmark rest on three domain assumptions: MLLM scoring reliability, the sufficiency of the 18 aspect definitions, and the representativeness of the in-house human annotations. No nonstandard mathematical axioms or invented physical entities are introduced. The only fitted parameters appear in the optional linear-regression ablation.

free parameters (1)
  • Linear regression aggregation weights (ablation only) = not reported
    In Section 5.5, a linear regression over the 18 aspect scores is fit to human preference scores using leave-one-out per model. The learned weights are not listed and are not used in the main metric.
assumptions (3)
  • domain assumption GPT-4o can reliably judge each of the 18 visual aspects from images and text on a 1-5 scale.
    The entire metric relies on the MLLM's perceptual and reasoning abilities. The paper validates this only indirectly through correlation with in-house human scores (Section 5.2).
  • domain assumption The 18 predefined aspects are sufficient to capture what humans consider when evaluating concept customization.
    Aspects were chosen bottom-up by the authors (Supplementary C), and the paper infers sufficiency from the high correlation, but no direct validation of aspect coverage is provided.
  • domain assumption Human annotators' averaged scores are a stable, unbiased ground truth for image preference.
    Human preference scores come from 12 lab annotators with no reported inter-annotator agreement or external validation (Section 5.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Human Preference-Aligned Concept Customization Benchmark via Decomposed Evaluation." pith.science (2026). https://pith.science/paper/QCBTXEIB

@misc{pith2026250903385,
  author       = {Pith},
  title        = {Pith review of: Human Preference-Aligned Concept Customization Benchmark via Decomposed Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QCBTXEIB}},
  note         = {Machine review of arXiv:2509.03385}
}
read the original abstract

Evaluating concept customization is challenging, as it requires a comprehensive assessment of fidelity to generative prompts and concept images. Moreover, evaluating multiple concepts is considerably more difficult than evaluating a single concept, as it demands detailed assessment not only for each individual concept but also for the interactions among concepts. While humans can intuitively assess generated images, existing metrics often provide either overly narrow or overly generalized evaluations, resulting in misalignment with human preference. To address this, we propose Decomposed GPT Score (D-GPTScore), a novel human-aligned evaluation method that decomposes evaluation criteria into finer aspects and incorporates aspect-wise assessments using Multimodal Large Language Model (MLLM). Additionally, we release Human Preference-Aligned Concept Customization Benchmark (CC-AlignBench), a benchmark dataset containing both single- and multi-concept tasks, enabling stage-wise evaluation across a wide difficulty range -- from individual actions to multi-person interactions. Our method significantly outperforms existing approaches on this benchmark, exhibiting higher correlation with human preferences. This work establishes a new standard for evaluating concept customization and highlights key challenges for future research. The benchmark and associated materials are available at https://github.com/ReinaIshikawa/D-GPTScore.

Figures

Figures reproduced from arXiv: 2509.03385 by the authors.

Figure 1
Figure 1. D-GPTScore evaluates images generated by concept customization through a two-step process: aspect-wise evalua￾tion followed by aggregation. This approach achieves signifi￾cantly higher correlation with human preference scores than ex￾isting methods, establishing a more reliable and human-aligned metric. preferences remain underexplored in the research commu￾nity. In recent years, the rise of diffusion models [15, 38… view at source ↗
Figure 2
Figure 2. The pipeline of D-GPTScore. This metric comprises two phases: (1) [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Benchmark results of aspect-wise evaluation. Each aspect is scored on a scale from 1 to 5, with higher values indicat￾ing better performance. Spearman’s rank correlation of 0.69 with human prefer￾ences, significantly surpassing all existing metrics. These results demonstrate that the proposed method provides human-aligned evaluation for concept customization and outperforms existing metrics. Furthermore, since omiss… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Sample images in CC-AlignBench A. Existing Benchmark for Concept Cus￾tomization [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: Instruction of annotation FastComposer Mix-of-Show DreamBooth CustomDiffusion OMG+LoRA OMG+InstantID Overall [PITH_FULL_IMAGE:figures/full_fig_p021_5.png]
Figure 6
Figure 6. Figure 6: Scatter plots of the human preference scores versus our predicted scores. [PITH_FULL_IMAGE:figures/full_fig_p021_6.png]
Figure 7
Figure 7. Figure 7: Case study [PITH_FULL_IMAGE:figures/full_fig_p023_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

65 extracted references · 52 canonical work pages

  1. [1]

    HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Models

    Eslam Mohamed Bakr, Pengzhan Sun, Xiaoqian Shen, Faizan Farooq Khan, Li Erran Li, and Mohamed Elhoseiny. HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Models. In ICCV, 2023. 2, 3

    work page 2023

  2. [2]

    Videophy: Evaluating phys- ical commonsense for video generation

    Hritik Bansal, Zongyu Lin, Tianyi Xie, Zeshun Zong, Michal Yarom, Yonatan Bitton, Chenfanfu Jiang, Yizhou Sun, Kai- Wei Chang, and Aditya Grover. Videophy: Evaluating phys- ical commonsense for video generation. In ICRL, pages 102075–102121, 2025. 3

    work page 2025

  3. [3]

    VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evalua- tion in Video Generation

    Hritik Bansal, Clark Peng, Yonatan Bitton, Roman Golden- berg, Aditya Grover, and Kai-Wei Chang. VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evalua- tion in Video Generation. arXiv preprint arXiv:2503.06800,

    arXiv

  4. [4]

    Parkhi, and An- drew Zisserman

    Qiong Cao, Li Shen, Weidi Xie, Omkar M. Parkhi, and An- drew Zisserman. Vggface2: A dataset for recognising faces across pose and age. In FG, 2018. 2

    work page 2018

  5. [5]

    Emerg- ing Properties in Self-Supervised Vision Transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing Properties in Self-Supervised Vision Transformers. In ICCV, 2021. 3, 4

    work page 2021

  6. [6]

    AnyDoor: Zero-shot Object-level Image Customization

    Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. AnyDoor: Zero-shot Object-level Image Customization. In CVPR, 2024. 3

    work page 2024

  7. [7]

    DALL-EV AL: Probing the Reasoning Skills and Social Biases of Text-to- Image Generation Models

    Jaemin Cho, Abhay Zala, and Mohit Bansal. DALL-EV AL: Probing the Reasoning Skills and Social Biases of Text-to- Image Generation Models. In ICCV, 2023. 3, 2

    work page 2023

  8. [8]

    ArcFace: Additive Angu- lar Margin Loss for Deep Face Recognition

    Jiankang Deng, Jia Guo, Jing Yang, Niannan Xue, Irene Kotsia, and Stefanos Zafeiriou. ArcFace: Additive Angu- lar Margin Loss for Deep Face Recognition. IEEE TPAMI, 44(10 Part 1):5962–5979, 2022. 2, 4

    work page 2022

Show all 65 references
  1. [9]

    Benchmarking spatial relationships in text-to-image generation

    Tejas Gokhale, Hamid Palangi, Besmira Nushi, Vibhav Vi- neet, Eric Horvitz, Ece Kamar, Chitta Baral, and Yezhou Yang. Benchmarking spatial relationships in text-to-image generation. arXiv preprint arXiv:2212.10015, 2022. 2, 3

    2022 arXiv

  2. [10]

    GIQA: Generated Image Quality Assessment

    Shuyang Gu, Jianmin Bao, Dong Chen, and Fang Wen. GIQA: Generated Image Quality Assessment. In ECCV,

  3. [11]

    Mix-of-show: decentralized low-rank adaptation for multi-concept customization of diffusion models

    Yuchao Gu, Xintao Wang, Jay Zhangjie Wu, Yujun Shi, Yun- peng Chen, Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang, Weijia Wu, Yixiao Ge, Ying Shan, and Mike Zheng Shou. Mix-of-show: decentralized low-rank adaptation for multi-concept customization of diffusion models. In Neur...

    2023

  4. [12]

    UniPor- trait: A Unified Framework for Identity-Preserving Single- and Multi-Human Image Personalization

    Junjie He, Yifeng Geng, and Liefeng Bo. UniPor- trait: A Unified Framework for Identity-Preserving Single- and Multi-Human Image Personalization. arXiv preprint arXiv:2408.05939, 2024. 2

    2024 arXiv

  5. [13]

    VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feed- back for Video Generation

    Xuan He, Dongfu Jiang, Ge Zhang, Max Ku, Achint Soni, Sherman Siu, Haonan Chen, Abhranil Chandra, Ziyan Jiang, Aaran Arulraj, Kai Wang, Quy Duc Do, Yuansheng Ni, Bo- han Lyu, Yaswanth Narsupalli, Rongqi Fan, Zhiheng Lyu, Bill Yuchen Lin, and Wenhu Chen. VideoScore: Building Au...

    2024

  6. [14]

    Seman- tic Object Accuracy for Generative Text-to-Image Synthesis

    Tobias Hinz, Stefan Heinrich, and Stefan Wermter. Seman- tic Object Accuracy for Generative Text-to-Image Synthesis. IEEE TPAMI, 44(3):1552–1565, 2022. 2, 3

    2022

  7. [15]

    Denoising Dif- fusion Probabilistic Models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Dif- fusion Probabilistic Models. In NeurIPS, 2020. 1

    2020

  8. [16]

    Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Os- tendorf, Ranjay Krishna, and Noah A. Smith. TIFA: Accu- rate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering. In ICCV, 2023. 2, 3

    2023

  9. [17]

    A Style- Based Generator Architecture for Generative Adversarial Networks

    Tero Karras, Samuli Laine, and Timo Aila. A Style- Based Generator Architecture for Generative Adversarial Networks. In CVPR, 2019. 3, 2

    2019

  10. [18]

    Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick. Segment Anything. In ICCV, 2023. 3

    2023

  11. [19]

    OMG: Occlusion-Friendly Personalized Multi-concept Generation in Diffusion Models

    Zhe Kong, Yong Zhang, Tianyu Yang, Tao Wang, Kaihao Zhang, Bizhu Wu, Guanying Chen, Wei Liu, and Wenhan Luo. OMG: Occlusion-Friendly Personalized Multi-concept Generation in Diffusion Models. In ECCV, 2024. 5, 2, 3

    2024

  12. [20]

    VIEScore: Towards explainable metrics for condi- tional image synthesis evaluation

    Max Ku, Dongfu Jiang, Cong Wei, Xiang Yue, and Wenhu Chen. VIEScore: Towards explainable metrics for condi- tional image synthesis evaluation. In ACL, pages 12268– 12290. ACL, 2024. 3

    2024

  13. [21]

    Imagenhub: Standardizing the evaluation of conditional image generation models

    Max Ku, Tianle Li, Kai Zhang, Yujie Lu, Xingyu Fu, Wen- wen Zhuang, and Wenhu Chen. Imagenhub: Standardizing the evaluation of conditional image generation models. In ICLR, 2024. 3, 2

    2024

  14. [22]

    Multi-Concept Customization of Text-to-Image Diffusion

    Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-Concept Customization of Text-to-Image Diffusion. In CVPR, 2023. 2, 3, 5

    2023

  15. [23]

    Dongxu Li, Junnan Li, and Steven C.H. Hoi. BLIP-diffusion: pre-trained subject representation for controllable text-to- image generation and editing. In NeurIPS, 2023. 3

    2023

  16. [24]

    BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. In ICML, 2023. 3

    2023

  17. [25]

    Image Synthesis From Layout With Locality- Aware Mask Adaption

    Zejian Li, Jingyu Wu, Immanuel Koh, Yongchuan Tang, and Lingyun Sun. Image Synthesis From Layout With Locality- Aware Mask Adaption. In ICCV, 2021. 3

    2021

  18. [26]

    Lawrence Zitnick

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C. Lawrence Zitnick. Microsoft COCO: Common Objects in Context. In ECCV, 2014. 3, 2

    2014

  19. [27]

    Evaluating text-to-visual generation with image-to-text gen- eration

    Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text gen- eration. In ECCV, page 366–384, 2024. 3

    2024

  20. [28]

    Jun Liu, Amir Shahroudy, Mauricio Perez, Gang Wang, Ling-Yu Duan, and Alex C. Kot. NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understand- ing. IEEE TPAMI, 42(10):2684–2701, 2020. 1

    2020

  21. [29]

    Grounding dino: Marry- ing dino with grounded pre-training for open-set object de- tection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding dino: Marry- ing dino with grounded pre-training for open-set object de- tection. In ECCV, 2024. 3

    2024

  22. [30]

    Cones 2: customizable image synthesis with multiple subjects

    Zhiheng Liu, Yifei Zhang, Yujun Shen, Kecheng Zheng, Kai Zhu, Ruili Feng, Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao. Cones 2: customizable image synthesis with multiple subjects. In NeurIPS, 2023. 3

    2023

  23. [31]

    Subject- Diffusion: Open Domain Personalized Text-to-Image Gener- ation without Test-time Fine-tuning

    Jian Ma, Junhao Liang, Chen Chen, and Haonan Lu. Subject- Diffusion: Open Domain Personalized Text-to-Image Gener- ation without Test-time Fine-tuning. In SIGGRAPH, 2024. 3

    2024

  24. [32]

    Towards a Bet- ter Metric for Evaluating Question Generation Systems

    Nema, Preksha and Khapra, Mitesh M. Towards a Bet- ter Metric for Evaluating Question Generation Systems. In EMNLP, 2018. 3

    2018

  25. [33]

    Multi-modal, Multi-task, Multi-criteria Automatic Evaluation with Vision Language Models

    Masanari Ohi, Masahiro Kaneko, Naoaki Okazaki, and Nakamasa Inoue. Multi-modal, Multi-task, Multi-criteria Automatic Evaluation with Vision Language Models. arXiv preprint arXiv:2412.14613, 2025. 3

    2025

  26. [34]

    GPT-4 Technical Report

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, and Shya- mal Anadkat et al. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774, 2024. 2, 5

    2024 arXiv

  27. [35]

    Dreambench++: A human-aligned bench- mark for personalized image generation

    Yuang Peng, Yuxin Cui, Haomiao Tang, Zekun Qi, Runpei Dong, Jing Bai, Chunrui Han, Zheng Ge, Xiangyu Zhang, and Shu-Tao Xia. Dreambench++: A human-aligned bench- mark for personalized image generation. In ICLR, 2025. 3, 2

    2025

  28. [36]

    Grounded Text-to-Image Synthesis with Attention Refocusing

    Quynh Phung, Songwei Ge, and Jia-Bin Huang. Grounded Text-to-Image Synthesis with Attention Refocusing. In CVPR, 2024. 2, 3

    2024

  29. [37]

    Plummer, Liwei Wang, Chris M

    Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazeb- nik. Flickr30k Entities: Collecting Region-to-Phrase Corre- spondences for Richer Image-to-Sentence Models. In ICCV,

  30. [38]

    Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 1

    2023 arXiv

  31. [39]

    Learning Transferable Visual Models From Natural Language Supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In ICML,

  32. [40]

    High-Resolution Image Synthesis With Latent Diffusion Models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-Resolution Image Synthesis With Latent Diffusion Models. In CVPR, 2022. 1

    2022

  33. [41]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2023. 1, 3, 5, 2

    2023

  34. [42]

    Sara Mahdavi, Raphael Gontijo-Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Lit, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Raphael Gontijo-Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion ...

    2022

  35. [43]

    LAION- 400M: Open Dataset of CLIP-Filtered 400 Million Image- Text Pairs

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LAION- 400M: Open Dataset of CLIP-Filtered 400 Million Image- Text Pairs. arXiv preprint arXiv:2111.02114, 2021. 3, 2

    2021 arXiv

  36. [44]

    LAION-5B: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Je- nia Jitsev. L...

    2022 arXiv

  37. [45]

    NTU RGB+D: A Large Scale Dataset for 3D Human Activ- ity Analysis

    Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang. NTU RGB+D: A Large Scale Dataset for 3D Human Activ- ity Analysis. In CVPR, 2016. 1

    2016

  38. [46]

    Weiss, Niru Mah- eswaranathan, and Surya Ganguli

    Jascha Sohl-Dickstein, Eric A. Weiss, Niru Mah- eswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML,

  39. [47]

    Going Deeper with Convolutions

    Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going Deeper with Convolutions. arXiv preprint arXiv:1409.4842, 2014. 2

    2014 arXiv

  40. [48]

    Dai, Anja Hauth, Katie Millican, and David Sil- ver et al

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, and David Sil- ver et al. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2025. 4

    2025 arXiv

  41. [49]

    Diffusers: State-of-the-art diffusion models

    Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-the-art diffusion models. GitHub repository, 2022. 5

    2022

  42. [50]

    P+: Extended Textual Conditioning in Text-to- Image Generation

    Andrey V oynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman. P+: Extended Textual Conditioning in Text-to- Image Generation. arXiv preprint arXiv:2303.09522, 2023. 1

    2023 arXiv

  43. [51]

    AIGV-Assessor: Benchmarking and Evaluating the Perceptual Quality of Text-to-Video Gener- ation with LMM

    Jiarui Wang, Huiyu Duan, Guangtao Zhai, Juntong Wang, and Xiongkuo Min. AIGV-Assessor: Benchmarking and Evaluating the Perceptual Quality of Text-to-Video Gener- ation with LMM. arXiv preprint arXiv:2411.17221 , 2024. 3

    2024 arXiv

  44. [52]

    A unified agentic framework for evaluating conditional image generation

    Jifang Wang, Xue Yang, Longyue Wang, Zhenran Xu, Yiyu Wang, Yaowei Wang, Weihua Luo, Kaifu Zhang, Baotian Hu, and Min Zhang. A unified agentic framework for evaluating conditional image generation. arXiv preprint arXiv:2504.07046, 2025. 3

    2025 arXiv

  45. [53]

    MetaMetrics: Calibrat- ing Metrics For Generation Tasks Using Human Preferences

    Genta Indra Winata, David Anugraha, Lucky Susanto, Garry Kuwanto, and Derry Tanti Wijaya. MetaMetrics: Calibrat- ing Metrics For Generation Tasks Using Human Preferences. arXiv preprint arXiv:2410.02381, 2025. 3

    2025 arXiv

  46. [54]

    Q-Align: Teaching LMMs for Vi- sual Scoring via Discrete Text-Defined Levels

    Haoning Wu, Zicheng Zhang, Weixia Zhang, Chaofeng Chen, Liang Liao, Chunyi Li, Yixuan Gao, Annan Wang, Erli Zhang, Wenxiu Sun, Qiong Yan, Xiongkuo Min, Guang- tao Zhai, and Weisi Lin. Q-Align: Teaching LMMs for Vi- sual Scoring via Discrete Text-Defined Levels. In ICML,

  47. [55]

    Freeman, Fr ´edo Durand, and Song Han

    Guangxuan Xiao, Tianwei Yin, William T. Freeman, Fr ´edo Durand, and Song Han. FastComposer: Tuning-Free Multi- subject Image Generation with Localized Attention. IJCV, 133(3):1175–1194, 2024. 5, 2

    2024

  48. [56]

    Imagereward: learning and evaluating human preferences for text-to-image generation

    Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: learning and evaluating human preferences for text-to-image generation. In NeurIPS, pages 15903–15935, 2023. 3

    2023

  49. [57]

    From image descriptions to visual denotations: New similarity metrics for semantic inference over event descrip- tions

    Peter Young, Alice Lai, Micah Hodosh, and Julia Hocken- maier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descrip- tions. TACL, 2:67–78, 2014. 3, 2

    2014

  50. [58]

    Do llms really think step-by-step in implicit rea- soning? arXiv preprint arXiv:2411.15862, 2025

    Yijiong Yu. Do llms really think step-by-step in implicit rea- soning? arXiv preprint arXiv:2411.15862, 2025. 3

    2025 arXiv

  51. [59]

    Joint face detection and alignment using multitask cascaded convolutional networks

    Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. Joint face detection and alignment using multitask cascaded convolutional networks. SPL, 23:1499–1503, 2016. 2

    2016

  52. [60]

    Adding Conditional Control to Text-to-Image Diffusion Models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding Conditional Control to Text-to-Image Diffusion Models. In ICCV, 2023. 1

    2023

  53. [61]

    Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In NeurIPS, 2023. 2

    2023

  54. [62]

    Multi-LoRA Composition for Image Genera- tion

    Ming Zhong, Yelong Shen, Shuohang Wang, Yadong Lu, Yizhu Jiao, Siru Ouyang, Donghan Yu, Jiawei Han, and Weizhu Chen. Multi-LoRA Composition for Image Genera- tion. arXiv preprint arXiv:2402.16843, 2024. 3, 1

    2024 arXiv

  55. [63]

    Sim- ple multi-dataset detection

    Xingyi Zhou, Vladlen Koltun, and Philipp Kr¨ahenb¨uhl. Sim- ple multi-dataset detection. In CVPR, 2022. 3, 2 Human Preference-Aligned Concept Customization Benchmark via Decomposed Evaluation Supplementary Material Concept1Concept2 Figure 4. Sample images in CC-AlignBench A. E...

    2022

  56. [64]

    noisy, blurry, soft, deformed, ugly

    to localize faces and measures feature similarity within the embedding space [12, 19, 55]. In this study, faces de- tected with high confidence by MTCNN are used to extract embedding features for the specified number of concepts via Inception ResNet (V1) pretrained on VGGFace2...

  57. [65]

    <Text prompt>

    Pay attention to whether the characteristics of the individuals in the reference images (including clothing, etc.) are preserved and whether the generated image follows the text prompt. The text prompt “<Text prompt>” <Generated image, reference images> Score: A man A woman We...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.