Pith. sign in

REVIEW 5 major objections 5 minor 59 references

RA-ClipScore: Making Generative Model Evaluation More Interpretable

T0 review · 5 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read RA-CLIPScore claims to make generative-model evaluation interpretable by scoring each attribute independently through dual prompts and pooling local patch tokens; its spatial component, R-SaD, is reported to match human diversity…

desk verdict Strong idea for attribute- and spatial-level generative evaluation, but the missing projection in Eq. (6) is a load-bearing gap that undermines the quantitative claims until fixed. read the letter →

arxiv 2608.12088 v1 pith:4OXNJJMG submitted 2026-08-12 cs.CV

Yifan Lu , Taras Kucherenko , Hedvig Kjellström , Judith Bütepage This is my paper · ORCID
classification cs.CV
keywords generativemodelevaluationCLIPattribute-wisemetricsspatialdiversityinterpretabilitydualpromptsR-SaDexplainable
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to make generative-model evaluation explain why a model fails, not just how far its outputs sit from real images. It proposes RA-CLIPScore, a CLIP-based metric that scores each attribute independently through positive/negative prompt pairs and pools local patch tokens so the score is grounded in image regions. The authors claim this yields attribute-wise divergences, SaD and PaD, that stay stable when unrelated attributes are removed, and it extends CLIP-based evaluation to spatial distribution alignment through Regional Single-Attribute Divergence (R-SaD). If the claims hold, practitioners could diagnose biases such as objects always generated at fixed positions, which scalar scores like FID hide. The paper reports that R-SaD matched human judgments of sample diversity perfectly ($r=1$) in its user study, while standard diversity metrics scored between $0.29$ and $0.7$.

What carries the argument

The mechanism is a modified final block of CLIP's visual encoder combined with dual prompts. In the paper's Eq.~(6)-(7), the attention mask nulls attention among local patch tokens so they pass through the last residual block without the global mixing of the final attention layer, while the [CLS] token still aggregates global context; the output is the global token concatenated with the dense local tokens, $\tilde{E}_v(X)=[H^L[0],\,H^L_{\text{dense}}[1:]]$. Each attribute is turned into two fixed prompts, a positive and a negative, and the softmax of their cosine similarities with each patch token gives a binary presence score per region. Attention maps from earlier layers refine those patch scores, a softmax-weighted sum over patches pools them into one per-attribute score, and KL divergences (SaD, PaD) or per-patch mean differences (R-SaD) turn the scores into dataset-level diagnostics that can be drawn as heatmaps.

What would settle it

Implement the paper's Eq.~(6) on a released CLIP ViT and test whether patch-token--text-prompt cosine similarities can localize attributes where ground truth is known, such as object bounding boxes in COCO or attribute regions in CelebA; near-chance retrieval would falsify the joint-space assumption. A second check is to rerun the diversity user study with a larger participant pool and more attributes, since $r=1.0$ on 29 raters and 20 attributes is fragile.

Watch

Extended reading notes

Core claim

RA-CLIPScore replaces the single global embedding used by CLIPScore with a representation formed jointly by the final [CLS] token and dense local patch tokens taken from layer $L-1$ with the last attention operation bypassed. Each attribute $t_i$ is scored by a softmax over a positive prompt (``This is a photo of $t_i$'') and a negative prompt (``This is a photo without $t_i$''), and the regional scores are refined by attention and aggregated by patch-to-prompt similarity. The paper's central claim is that this decouples attributes that softmax contrastive training entangles, stays stable under attribute-set perturbation where HCS's mean-centering provably collapses into a two-attribute zero-sum relation, and reveals spatial biases such as BigGAN's tendency to draw electric guitars at a fixed $45^\circ$ diagonal. In the paper's user study, R-SaD achieves a Pearson correlation of $1.0$ with human preferences for sample diversity, against $0.29$-$0.7$ for FID, KID, LPIPS, precision/recall, density, and coverage.

Load-bearing premise

The load-bearing premise is that CLIP's layer $L-1$ patch tokens, after passing through the modified residual block (value projection plus MLP, no attention), lie in CLIP's shared vision-language embedding space, so cosine similarity with text prompts is semantically meaningful; in released CLIP models only the final [CLS] token is projected into that joint space.

Editorial extensions

If this is right

  • Attribute-wise evaluation becomes stable under attribute-set changes: RA-CLIPScore scores resist removal of a single attribute, where HCS shifts noticeably.
  • Spatial placement can be audited per class with no extra training or second forward pass; the paper reports distinct regional biases for StyleGAN-XL, BigGAN, and LDM.
  • Diversity judgments that ignore layout are incomplete: R-SaD's reported perfect human correlation implies where an object appears is part of perceived diversity.
  • SaD and PaD rise monotonically when biased images are injected into a dataset and stay flat for neutral injections, so they can serve as controlled sanity checks.
  • Because the aggregation runs in one forward pass and uses only CLIP features, the metric adds negligible cost over a plain CLIPScore evaluation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The $r=1.0$ result rests on 29 participants and 20 attributes; if it replicates with larger pools, spatial diversity should become a standard axis in generative evaluation, but the effect size needs re-measurement.
  • The paper's zero-sum argument against HCS applies generally: any attribute metric built on mean-centering CLIP embeddings in a two-attribute setting forces the scores to be exact opposites, so the fragility is not specific to HCS's particular implementation.
  • If the layer $L-1$ joint-space assumption fails for released CLIP models, the regional scores are cosine similarities in a space not semantically aligned with text; a projection head or a learned linear probe on patch tokens would be the minimal fix.
  • A cheap test of the spatial claim: retrain or fine-tune a generator on data with deliberately perturbed object positions and check whether R-SaD heatmaps track the perturbation magnitude per patch.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes RA-CLIPScore, a CLIP-based evaluation metric for generative image models that combines dual prompts (positive/negative attribute prompts) with local patch-token features to support attribute-wise and spatial distribution evaluation. The metric feeds refined patch-level scores into Single-Attribute Divergence (SaD), Pair-Attribute Divergence (PaD), and a new Regional Single-Attribute Divergence (R-SaD). The authors present experiments on CelebA, FFHQ, and ImageNet, an ablation study, injection controls, and a user study in which R-SaD reportedly achieves perfect correlation (r=1.0) with human judgments of sample diversity. The paper claims that RA-CLIPScore is more robust and interpretable than HCS and CLIPScore and reveals previously undocumented spatial biases in generative models without additional training or significant computational overhead.

Significance. If the load-bearing technical gaps are closed, RA-CLIPScore would be a genuinely useful contribution: it extends CLIP-based evaluation from global semantics to spatially localized attribute analysis, an underexplored direction, and it does so with a single forward pass and no learned parameters. The manuscript has clear strengths: a thoughtfully designed ablation (Table 3), well-controlled injection experiments (Fig. 3), an unusually detailed supplementary, and an attempted external check through human preferences. The claim of a new spatial-diversity dimension is interesting and, if substantiated, would be of value to the generative-model evaluation community. However, at present the method is not fully defined: Eq. (6) omits the projection that would place patch tokens in CLIP's joint embedding space, Eq. (8) contains a type error that makes the refinement step ambiguous, and the r=1.0 human-correlation figure is asserted on a small, possibly unrepresentative sample. These issues are correctable, but they currently block confident acceptance.

major comments (5)
  1. [Sec. 3.1, Eqs. (4) and (6)] The central mechanism is incomplete: Eq. (6) defines the dense features as the output of the final transformer block with attention bypassed for local tokens, leaving the patch features in the visual transformer's hidden space. No projection to CLIP's text-embedding dimension or to the joint vision-language space is specified. In released CLIP ViT models the visual hidden dimension and the text-embedding dimension differ, and only the final CLS token is projected into the shared space. Consequently, the cosine similarity in Eq. (4) between the adapted visual encoder output for patch j and the text prompt embedding is undefined as written, and all quantities derived from it (SaD, PaD, R-SaD, and the r=1.0 result) inherit this gap. The authors must either specify the exact projection used (learned, fixed, or the existing CLIP visual projection head) and validate that the projected patch features are semantically aligned with text embeddings, or reformulate the scoring so that it operates in a well-defined space.
  2. [Sec. 3.1, Eq. (8)] Equation (8) as printed is not implementable. The left side is the refined score for token j, but the right side multiplies the attention vector for token j (a row over all tokens) by the scalar coarse score for token j, and no summation over neighboring tokens is shown. The intended operation appears to be a weighted average of the coarse scores across tokens. As written, the definition of the refined scores that feed into RA-CLIPScore (Eq. 9) and R-SaD (Eq. 14) is ambiguous, and the reader cannot reproduce the experiments.
  3. [Sec. 4.5, Table 6] The claim that R-SaD achieves perfect correlation (r=1.0) with human-perceived diversity is disproportionate to the evidence. The user study uses only 20 attributes, selects classes with clear spatial biases (Tab. 5), and reports an inter-rater agreement of only 72.9%. A perfect Pearson correlation under such conditions strongly suggests a small or non-representative sample, a selection artifact, or an effective tie in the majority votes. The authors should report the exact number of paired comparisons, a confidence interval or significance test for r, a per-attribute breakdown, and an analysis of agreement (e.g., Cohen's kappa) rather than only the correlation coefficient. The wording 'perfect correlation (r=1)' should be revised unless the analysis is substantially expanded.
  4. [Sec. 3.2 and Fig. 4] The replacement of Gaussian KDE by a single Gaussian fit is asserted to be 'sufficient to capture distributional shifts' without evidence. The score distributions visualized in Fig. 4 appear asymmetric and in some cases bimodal, and a misspecified Gaussian model directly changes the KL-divergence values in Eqs. (12) and (13). The authors should justify the single-Gaussian assumption with a goodness-of-fit analysis or demonstrate that model rankings are unchanged when a more flexible estimator (e.g., KDE or a mixture model) is used. Without such validation, all SaD/PaD numbers in Tables 3, 4, and 8 are conditional on an unverified distributional assumption.
  5. [Sec. 4.4 and Fig. 5] The spatial-bias 'discoveries' (e.g., BigGAN's 45-degree guitar orientation, LDM's upward burrito shift) are outputs of the proposed metric alone; there is no external ground truth or independent localization check confirming that these heatmaps correspond to actual object-position differences between real and generated images. Since the spatial extension is a principal novelty of the paper, the authors should validate at least a few of the heatmaps against a detection-based or human-annotation-based position distribution, or explicitly frame Section 4.4 as a hypothesis-generating case study rather than a validated measurement.
minor comments (5)
  1. [General] There are several typos and inconsistencies: 'metri' in Section 4, 'explainations' in Section 4.1, and the terms 'R-SAD' and 'R-SaD' are used interchangeably; the abbreviations should be unified.
  2. [References] Reference [44] and [45] both point to the same paper (Rombach et al., CVPR 2022); one duplicate should be removed and the citation indices adjusted accordingly.
  3. [Eq. (7)] The description of the attention mask is not precise: the text says the first row contains all 0s and that the local submatrix becomes an identity, but the displayed matrix has -inf values on the diagonal of the local block, which would prevent tokens from attending to themselves and contradict the 'identity' statement. The mask should be specified explicitly.
  4. [Eq. (14)] In Eq. (14), the notation for the refined regional scores on the generated set is used, but Eq. (8) was introduced only for the training set; the authors should define the analogous refinement for generated samples consistently.
  5. [Fig. 5] The text 'dual prompts: first refine with attention, then softmax' appears in the heatmap figures and seems to be a leftover debugging annotation rather than a figure caption; it should be removed.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the load-bearing claims are validated against external human judgments; design choices tuned on the same data are selection effects, not definitional reductions.

full rationale

RA-CLIPScore and its derived divergences are defined from CLIP embeddings, dual prompts, and patch-token features; the central quantitative claim (r=1.0 with human preference, Tab. 6) is an empirical correlation computed against majority-voted human judgments, which are external to the metric's construction. There is no equation in the paper in which a target result is inserted as an input: Eq. (4) defines the attribute score, Eqs. (12)-(14) define SaD/PaD/R-SaD as divergences, and the user study then compares these outputs to human choices. The paper contains no self-citation chain or imported uniqueness theorem; the prior work cited ([16], [24]) is used as baseline, not as justification for this paper's choices. The empirical selections (psi = {1,...,L-1} in Eq. (8) described as 'Empirically, this configuration yields the most stable qualitative results' and the mean-difference R-SaD in Eq. (14) 'as it empirically yields the most stable results') are hyperparameter/design choices made on the same evaluation data, which is a mild self-confirmation loop and a legitimate concern for generalizability, but it does not make the reported correlation true by construction--human judgments could have disagreed. The skeptic's concern that Eq. (6) does not actually project patch tokens into CLIP's joint space is a correctness/internal-consistency issue with the released CLIP architecture, not a circularity of the derivation chain. No circular step can be exhibited, so the correct circularity finding is 0.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The metric itself has no fitted numeric constants, but several design choices are hand-selected on the evaluation data: the set of attention layers ψ, the mean-difference aggregation for R-SaD, and the single-Gaussian fit. The technical validity rests on unstated assumptions about CLIP's internal representation space and about the reliability of a small user study.

free parameters (2)
  • Attention refinement layer set ψ = {1,...,L-1}
    Chosen as 'empirically this configuration yields the most stable qualitative results' (Sec. 3.1); not derived or validated.
  • R-SaD aggregation choice = mean difference
    Chosen because 'empirically yields the most stable results and offers more interpretable insights' (Sec. 3.3); an alternative (KL) was available.
assumptions (5)
  • domain assumption CLIP's softmax contrastive training imposes mutual exclusivity among textual attributes, limiting multi-label representation.
    Relies on this from [41,51]; used to justify dual prompts.
  • domain assumption Spatial information in CLIP's local patch tokens is preserved in layer L-1 but lost in final layer.
    Cited from [13,31,59]; motivates bypassing final attention.
  • ad hoc to paper Local patch tokens after Eq. (6) lie in the same embedding space as text prompts, so cosine similarity is semantically valid.
    No projection to CLIP's shared space is specified; the paper asserts this without derivation.
  • ad hoc to paper A single Gaussian fit per score distribution is sufficient for divergence measurement.
    Stated 'we find that fitting a single Gaussian to the scores is sufficient' (Sec. 3.2), without empirical justification beyond their own experiments.
  • domain assumption User majority vote over 29 raters provides a reliable ground truth for diversity perception.
    72.9% inter-rater agreement; no significance or error analysis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RA-ClipScore: Making Generative Model Evaluation More Interpretable." pith.science (2026). https://pith.science/paper/4OXNJJMG

@misc{pith2026260812088,
  author       = {Pith},
  title        = {Pith review of: RA-ClipScore: Making Generative Model Evaluation More Interpretable},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4OXNJJMG}},
  note         = {Machine review of arXiv:2608.12088}
}
read the original abstract

Generative models can produce images nearly indistinguishable from real data, yet rigorous and interpretable evaluation remains challenging. Conventional metrics such as FID provide only scalar scores with limited diagnostic insight. Widely adopted CLIP-based metrics enable semantic evaluation beyond simple training class labels, but inherit limitations from CLIP's training paradigm that restrict attribute-wise analysis. We propose RA-CLIPScore, a novel metric that mitigates these issues and extends CLIP-based evaluation to spatial distribution alignment, measuring whether generated objects adhere to the positional priors found in the training data. RA-CLIPScore introduces dual prompts to decouple competing attributes and leverages local patch tokens to capture fine-grained regional semantics. We evaluate image generative models on their ability to match both attribute and spatial distributions of the training data. Extensive experiments show that RA-CLIPScore provides more robust and interpretable evaluations than prior methods, particularly under distribution misalignment or partially irrelevant textual attributes. We further demonstrate how it reveals spatial biases in generative models. User evaluations confirm that Regional Single Attribute Divergence based on our RA-CLIPScore aligns more closely with human perception of visual diversity than existing semantic metrics.

Figures

Figures reproduced from arXiv: 2608.12088 by the authors.

Figure 1
Figure 1. Our proposed RA-CLIPScore. We introduce dual prompts and regional feature extraction to improve interpretability of CLIP-based evaluation metrics while extend￾ing to spatial domain. Dual prompts decouple correlations among attributes, whereas regional feature extraction enhances fine-grained sensitivity through an adapted visual encoder, attention refinement, and region-aware feature aggregation. Grey blocks (left) … view at source ↗
Figure 2
Figure 2. Sensitivity to attribute removal. Removing a single attribute (long hair or elder) causes noticeable shifts in HCS, while RA-CLIPScore remains consistent under the same perturbations. Horizontal axes enumerate evaluated attributes. Three settings are compared: (1) using all 6 attributes (2) removing long hair (3) removing elder. the metric collapses into a forced zero-sum relationship, where HCS(x, tp) = −HCS(x, tq)… view at source ↗
Figure 3
Figure 3. Left: an example FFHQ image and its DiffuseIT translations under different prompts. Right: SaD and PaD responses as the fraction of injected images increases. outperforms others when irrelevant attributes are included, whereas HCS per￾formance degrades under this setting (see Sec.2 and Supp.A for explainations). This demonstrates the robustness of RA-CLIPScore under semantic mismatches, which commonly occur in pract… view at source ↗
Figures from the paper (21 more)
Figure 4
Figure 4. Figure 4: Ablation study comparing full method with w/o DP. Orange curves show score distributions for images with the attribute present, and blue curves for images without the attribute. Dual prompts improve distribution separation for both attributes [PITH_FULL_IMAGE:figures/…
Figure 6
Figure 6. Figure 6: Qualitative visualizations on sin￾gle images. Left: zebra (blue), sky (green), grass (orange). Right: pizza (blue), plate (green), table (orange). Model Top Classes StyleGAN-XL rotary dial telephone carousel iPod BigGAN balloon electric guitar microphone LDM (250) burr…
Figure 5
Figure 5. Figure 5: Spatial evaluation on ImageNet. Each row shows a real example (left), a generated sample (middle), and the corresponding R-SaD heatmap (right). Heatmaps visualize regional divergence normalized from 0 (blue, higher occur￾rence in generated images) to 1 (red, higher occ…
Figure 7
Figure 7. Figure 7: Experimental setup for human user study Metric Correlation r R-SaD 1.0 RA-CLIPScore 0.60 CLIPScore [16] 0.49 HCS [24] 0.21 FID [17] 0.50 KID [52] 0.60 LPIPS [58] 0.53 Precision [29, 46] 0.39 Recall [29, 46] 0.29 Density [37] 0.70 Coverage [37] 0.29 [PITH_FULL_IMAGE:fi…
Figure 8
Figure 8. Figure 8: Top: example COCO images containing chair, typically in complex scenes. Bottom: SaD and PaD responses as a function of the fraction of injected images. Bottom-left: injecting original COCO images (control group). Bottom-right: injecting chair images from COCO. and all …
Figure 9
Figure 9. Figure 9: Example images from CelebA (left to right): man without eyeglasses, man with eyeglasses, woman without eyeglasses, woman with eyeglasses. Method Metric A vs. B A vs. C RA-CLIPScore SaDEG 4.52 693.97 SaD¬EG 1.11 1.10 PaD 2.80 79.00 HCS SaDEG 3.89 93.86 SaD¬EG 3.43 5.94 …
Figure 10
Figure 10. Figure 10: Visualization of the top attributes and attribute pairs with the largest SaD and PaD scores. Eyeglasses and gender consistently rank among the top two attributes or attribute pairs, demonstrating successful detection of reversed attribute combinations. Hellinger Dista…
Figure 11
Figure 11. Figure 11: Ablation study on the male attribute. where R-SaDj (ti) is computed from Eq.(14), ti denotes the i-th ImageNet class, and j indexes local patches. For each class ti , mean R-SaD measures the average magnitude of regional bias over all patches. We sort all classes by t…
Figure 12
Figure 12. Figure 12: Ablation study on the eyeglasses attribute. Attributes Burrito, Carton, Shopping cart, School bus, Frying pan, Broccoli, Church, iPod, Ballon, Rotary dial telephone, Leather back sea turtle, Computer mouse, Flute, Spaghetti squash, Electric guitar, Paddle wheel, Clawn…
Figure 13
Figure 13. Figure 13: Ablation study on the smiling attribute [PITH_FULL_IMAGE:figures/full_fig_p027_13.png]
Figure 14
Figure 14. Figure 14: Visualization of the top attributes with the largest SaD scores. Top: LDM [45] with 50 denoising steps; Bottom: LDM [45] with 200 steps. Bald head consistently ranks as the top attribute, with larger denoising steps further increasing the divergence. Additionally, wea…
Figure 15
Figure 15. Figure 15: Rotary dial telephone. The first row shows samples from ImageNet, and the second row shows images generated by StyleGAN [49] [PITH_FULL_IMAGE:figures/full_fig_p028_15.png]
Figure 16
Figure 16. Figure 16: Carousel. The first row shows samples from ImageNet, and the second row shows images generated by StyleGAN [49]. 0 2 4 6 8 10 12 0 2 4 6 8 10 12 Heatmap of Mean Difference dual prompts: first refine with attention, then softmax 0.0 0.2 0.4 0.6 0.8 1.0 [PITH_FULL_IMAG…
Figure 17
Figure 17. Figure 17: iPod. The first row shows samples from ImageNet, and the second row shows images generated by StyleGAN [49]. 0 2 4 6 8 10 12 0 2 4 6 8 10 12 Heatmap of Mean Difference dual prompts: first refine with attention, then softmax 0.0 0.2 0.4 0.6 0.8 1.0 [PITH_FULL_IMAGE:fi…
Figure 18
Figure 18. Figure 18: Balloon. The first row shows samples from ImageNet, and the second row shows images generated by BigGAN [4] [PITH_FULL_IMAGE:figures/full_fig_p029_18.png]
Figure 19
Figure 19. Figure 19: Electric guitar. The first row shows samples from ImageNet, and the second row shows images generated by BigGAN [4]. 0 2 4 6 8 10 12 0 2 4 6 8 10 12 Heatmap of Mean Difference dual prompts: first refine with attention, then softmax 0.0 0.2 0.4 0.6 0.8 1.0 [PITH_FULL_…
Figure 20
Figure 20. Figure 20: Microphone. The first row shows samples from ImageNet, and the second row shows images generated by BigGAN [4]. 0 2 4 6 8 10 12 0 2 4 6 8 10 12 Heatmap of Mean Difference dual prompts: first refine with attention, then softmax 0.0 0.2 0.4 0.6 0.8 1.0 [PITH_FULL_IMAGE…
Figure 21
Figure 21. Figure 21: Burrito. The first row shows samples from ImageNet, and the second row shows images generated by LDM(250) [45] [PITH_FULL_IMAGE:figures/full_fig_p030_21.png]
Figure 22
Figure 22. Figure 22: Chain-link fence. The first row shows samples from ImageNet, and the second row shows images generated by LDM(250) [45]. 0 2 4 6 8 10 12 0 2 4 6 8 10 12 Heatmap of Mean Difference dual prompts: first refine with attention, then softmax 0.0 0.2 0.4 0.6 0.8 1.0 [PITH_F…
Figure 23
Figure 23. Figure 23: Carton. The first row shows samples from ImageNet, and the second row shows images generated by LDM(250) [45]. 23 24 25 26 27 28 29 31 32 33 34 35 36 41 42 0 2 4 6 1 1 1 4 2 6 2 3 2 1 1 1 2 1 1 Age Count 0 20 40 60 80 100 Male Female 64.5% 35.5% Percentage (%) [PITH_…
Figure 24
Figure 24. Figure 24: Age and gender distributions of participants in the user study [PITH_FULL_IMAGE:figures/full_fig_p031_24.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 31 canonical work pages

  1. [1]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Abdelfattah, R., Guo, Q., Li, X., Wang, X., Wang, S.: Cdul: Clip-driven un- supervised learning for multi-label image classification. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 1348–1357 (2023)

    work page 2023

  2. [2]

    In: International Conference on Machine Learning (2022)

    Alaa, A., Van Breugel, B., Saveliev, E.S., Van Der Schaar, M.: How faithful is your synthetic data? sample-level metrics for evaluating and auditing generative models. In: International Conference on Machine Learning (2022)

    work page 2022

  3. [3]

    arXiv preprint arXiv:2206.10935 (2022)

    Betzalel, E., Penso, C., Navon, A., Fetaya, E.: A study on the evaluation of gener- ative models. arXiv preprint arXiv:2206.10935 (2022)

    arXiv 2022

  4. [4]

    In: International Conference on Learning Representations (2019)

    Brock, A., Donahue, J., Simonyan, K.: Large scale GAN training for high fidelity natural image synthesis. In: International Conference on Learning Representations (2019)

    2019

  5. [5]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 9650–9660 (2021)

    2021

  6. [6]

    Lawrence Erl- baum Associates, Hillsdale, NJ, 2 edn

    Cohen, J.: Statistical Power Analysis for the Behavioral Sciences. Lawrence Erl- baum Associates, Hillsdale, NJ, 2 edn. (1988)

    work page 1988

  7. [7]

    In: 2009 IEEE conference on computer vision and pattern recognition

    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large- scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248–255. Ieee (2009)

    2009

  8. [8]

    Advances in neural information processing systems34, 8780–8794 (2021)

    Dhariwal, P., Nichol, A.: Diffusion models beat gans on image synthesis. Advances in neural information processing systems34, 8780–8794 (2021)

    2021

Show all 59 references
  1. [9]

    In: International Conference on Learning Representations (2017)

    Dinh, L., Sohl-Dickstein, J., Bengio, S.: Density estimation using real nvp. In: International Conference on Learning Representations (2017)

    2017

  2. [10]

    In: International Conference on Learning Representations (2021)

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16×16 words: Transformers for image recognition at scale. In: International Conference on L...

    2021

  3. [11]

    In: ICASSP 2023-2023 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP)

    Elizalde, B., Deshmukh, S., Al Ismail, M., Wang, H.: Clap learning audio con- cepts from natural language supervision. In: ICASSP 2023-2023 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 1–5. IEEE (2023)

    2023

  4. [12]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Gao, W., Wan, F., Pan, X., Peng, Z., Tian, Q., Han, Z., Zhou, B., Ye, Q.: Ts-cam: Token semantic coupled attention map for weakly supervised object localization. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 2886–2895 (2021) 16 Yifan Lu, Tara...

    2021

  5. [13]

    arXiv preprint arXiv:2212.06727 (2022)

    Ghiasi, A., Kazemi, H., Borgnia, E., Reich, S., Shu, M., Goldblum, M., Wilson, A.G., Goldstein, T.: What do vision transformers learn? a visual exploration. arXiv preprint arXiv:2212.06727 (2022)

    2022 arXiv

  6. [14]

    Advances in Neural Information Processing Systems36, 52132–52152 (2023)

    Ghosh, D., Hajishirzi, H., Schmidt, L.: Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems36, 52132–52152 (2023)

    2023

  7. [15]

    He,K.,Zhang,X.,Ren,S.,Sun,J.:Deepresiduallearningforimagerecognition.In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)

    2016

  8. [16]

    In: EMNLP (2021)

    Hessel, J., Holtzman, A., Forbes, M., Bras, R.L., Choi, Y.: CLIPScore: a reference- free evaluation metric for image captioning. In: EMNLP (2021)

    2021

  9. [17]

    In: Advances in Neural Information Processing Systems (2017)

    Heusel,M.,Ramsauer,H.,Unterthiner,T.,Nessler,B.,Hochreiter,S.:Ganstrained by a two time-scale update rule converge to a local nash equilibrium. In: Advances in Neural Information Processing Systems (2017)

    2017

  10. [18]

    In: Advances in Neural Information Processing Systems (2020)

    Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. In: Advances in Neural Information Processing Systems (2020)

    2020

  11. [19]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024)

    Jayasumana, S., Ramalingam, S., Veit, A., Glasner, D., Chakrabarti, A., Kumar, S.: Rethinking fid: Towards a better evaluation metric for image generation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024)

    2024

  12. [20]

    In: Advances in Neural Information Processing Systems (2023)

    Jiralerspong,M.,Bose,J.,Gemp,I.,Qin,C.,Bachrach,Y.,Gidel,G.:Featurelikeli- hood divergence: evaluating the generalization of generative models using samples. In: Advances in Neural Information Processing Systems (2023)

    2023

  13. [21]

    In: Advances in Neural Information Processing Systems (2021)

    Karras, T., Aittala, M., Laine, S., Härkönen, E., Hellsten, J., Lehtinen, J., Aila, T.: Alias-free generative adversarial networks. In: Advances in Neural Information Processing Systems (2021)

    2021

  14. [22]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 4401–4410 (2019)

    2019

  15. [23]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of stylegan. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 8110–8119 (2020)

    2020

  16. [24]

    In:Proceedingsofthe 41stInternational ConferenceonMachine Learning

    Kim, D., Kwon, M., Uh, Y.: Attribute based interpretable evaluation metrics for generative models. In:Proceedingsofthe 41stInternational ConferenceonMachine Learning. Proceedings of Machine Learning Research, vol. 235, pp. 24271–24293. PMLR (21–27 Jul 2024)

    2024

  17. [25]

    Kingma,D.P.,Welling,M.:Auto-encodingvariationalbayes.In:InternationalCon- ference on Learning Representations (2014)

    2014

  18. [26]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 4015–4026 (2023)

    2023

  19. [27]

    In: International Conference on Learning Representations (2023)

    Kwon, G., Ye, J.: Diffusion-based image translation using disentangled style and content representation. In: International Conference on Learning Representations (2023)

    2023

  20. [28]

    Kynkäänniemi, T., Karras, T., Aittala, M., Aila, T., Lehtinen, J.: The role of ima- genetclassesinfréchetinceptiondistance.In:InternationalConferenceonLearning Representations (2023)

    2023

  21. [29]

    In: Advances in Neural Information Processing Systems (2019) RA-ClipScore: Making Generative Model Evaluation More Interpretable 17

    Kynkäänniemi, T., Karras, T., Laine, S., Lehtinen, J., Aila, T.: Improved pre- cision and recall metric for assessing generative models. In: Advances in Neural Information Processing Systems (2019) RA-ClipScore: Making Generative Model Evaluation More Interpretable 17

    2019

  22. [30]

    In: European conference on computer vision

    Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: European conference on computer vision. pp. 740–755. Springer (2014)

    2014

  23. [31]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Lin, Y., Chen, M., Zhang, K., Li, H., Li, M., Yang, Z., Lv, D., Lin, B., Liu, H., Cai, D.: Tagclip: A local-to-global framework to enhance open-vocabulary multi-label classification of clip without training. In: Proceedings of the AAAI Conference on Artificial Intelligence. vo...

    2024

  24. [32]

    In: Proceedings of International Conference on Computer Vision (ICCV) (December 2015)

    Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Proceedings of International Conference on Computer Vision (ICCV) (December 2015)

    2015

  25. [33]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022)

    Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent dif- fusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022)

    2022

  26. [34]

    In: International conference on artificial intelligence and statistics (2020)

    Meehan, C., Chaudhuri, K., Dasgupta, S.: A non-parametric test to detect data- copying in generative models. In: International conference on artificial intelligence and statistics (2020)

    2020

  27. [35]

    In: International Conference on Learning Representations (2021)

    Morozov, S., Voynov, A., Babenko, A.: On self-supervised image representations for gan evaluation. In: International Conference on Learning Representations (2021)

    2021

  28. [36]

    Mukherjee, U.S.: Furniture image dataset.https://www.kaggle.com/datasets/ udaysankarmukherjee/furniture-image-dataset(2020)

    2020

  29. [37]

    In: International conference on machine learning

    Naeem, M.F., Oh, S.J., Uh, Y., Choi, Y., Yoo, J.: Reliable fidelity and diversity metrics for generative models. In: International conference on machine learning. pp. 7176–7185. PMLR (2020)

    2020

  30. [38]

    In: Proceedings of the 39th Interna- tional Conference on Machine Learning

    Nichol, A.Q., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., Mcgrew, B., Sutskever, I., Chen, M.: GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In: Proceedings of the 39th Interna- tional Conference on Machine Learning. Procee...

    2022

  31. [39]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Parmar, G., Zhang, R., Zhu, J.Y.: On aliased resizing and surprising subtleties in gan evaluation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11410–11420 (2022)

    2022

  32. [40]

    In: Advances in Neural Information Processing Systems (2019)

    Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.: Pytorch: An imperative style, high- performance deep learning library. In: Advances in Neural Information Processing Systems (2019)

    2019

  33. [41]

    In: International conference on machine learning

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021)

    2021

  34. [42]

    arXiv preprint arXiv:2204.06125 1(2), 3 (2022)

    Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 1(2), 3 (2022)

    2022 arXiv

  35. [43]

    In: Inter- national conference on machine learning

    Rezende, D., Mohamed, S.: Variational inference with normalizing flows. In: Inter- national conference on machine learning. pp. 1530–1538. PMLR (2015)

    2015

  36. [44]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022)

    2022

  37. [45]

    In: IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR)

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR). pp. 10684–10695 (2022) 18 Yifan Lu, Taras Kucherenko, Hedvig Kjellström, and...

    2022

  38. [46]

    In: Advances in Neural Information Processing Systems (2018)

    Sajjadi, M.S., Bachem, O., Lucic, M., Bousquet, O., Gelly, S.: Assessing generative models via precision and recall. In: Advances in Neural Information Processing Systems (2018)

    2018

  39. [47]

    In: Advances in Neural Information Processing Systems (2016)

    Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X.: Im- proved techniques for training gans. In: Advances in Neural Information Processing Systems (2016)

    2016

  40. [48]

    In: Advances in Neural Information Processing Systems (2021)

    Sauer, A., Chitta, K., Müller, J., Geiger, A.: Projected gans converge faster. In: Advances in Neural Information Processing Systems (2021)

    2021

  41. [49]

    In: ACM SIGGRAPH 2022 conference proceedings

    Sauer, A., Schwarz, K., Geiger, A.: Stylegan-xl: Scaling stylegan to large diverse datasets. In: ACM SIGGRAPH 2022 conference proceedings. pp. 1–10 (2022)

    2022

  42. [50]

    Song,J.,Meng,C.,Ermon,S.:Denoisingdiffusionimplicitmodels.In:International Conference on Learning Representations (2021)

    2021

  43. [51]

    In: Advances in Neural Information Processing Systems (2022)

    Sun, X., Hu, P., Saenko, K.: Dualcoop: Fast adaptation to multi-label recognition with limited annotations. In: Advances in Neural Information Processing Systems (2022)

    2022

  44. [52]

    In: International conference for learning representations

    Sutherland, J., Arbel, M., Gretton, A.: Demystifying mmd gans. In: International conference for learning representations. vol. 6 (2018)

    2018

  45. [53]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the incep- tion architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2818–2826 (2016)

    2016

  46. [54]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) 2025 Workshops (2025)

    Uchida, Y., Togo, R., Maeda, K., Ogawa, T., Haseyama, M.: Objectness similar- ity: Capturing object-level fidelity in 3d scene evaluation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) 2025 Workshops (2025)

    2025

  47. [55]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Xu, L., Ouyang, W., Bennamoun, M., Boussaid, F., Xu, D.: Multi-class token transformer for weakly supervised semantic segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 4310–4319 (2022)

    2022

  48. [56]

    Yuksekgonul, M., Bianchi, F., Kalluri, P., Jurafsky, D., Zou, J.Y.: When and why vision-language models behave like bags-of-words, and what to do about it? In: International Conference on Learning Representations (2023)

    2023

  49. [57]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Zhai, X., Mustafa, B., Kolesnikov, A., Beyer, L.: Sigmoid loss for language im- age pre-training. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 11975–11986 (2023)

    2023

  50. [58]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 586–595 (2018)

    2018

  51. [59]

    arXiv preprint arXiv:2103.11886 (2021) RA-ClipScore: Making Generative Model Evaluation More Interpretable 1 Supplementary Material A Rethinking HCS Kimet al

    Zhou, D., Kang, B., Jin, X., Yang, L., Lian, X., Hou, Q., Feng, J.: Deepvit: Towards deeper vision transformer. arXiv preprint arXiv:2103.11886 (2021) RA-ClipScore: Making Generative Model Evaluation More Interpretable 1 Supplementary Material A Rethinking HCS Kimet al. [24] i...

    2021 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.