Pith. sign in

REVIEW 4 major objections 6 minor 7 cited by

DiffSim: Taming Diffusion Models for Evaluating Visual Similarity

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Diffusion U-Net attention can judge image similarity.

desk verdict The attention-aligned diffusion feature trick is new and useful, but the per-benchmark test-set grid search makes the SOTA claims untrustworthy. read the letter →

arxiv 2412.14580 v1 pith:L4NIXX7U submitted 2024-12-19 cs.CV

Yiren Song , Xiaokang Liu , Mike Zheng Shou This is my paper · ORCID
classification cs.CV
keywords visualsimilaritydiffusionmodelsperceptualmetricsAlignedAttentionScorestyleinstancegenerativeimageevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the features inside a pretrained text-to-image diffusion model, specifically the attention layers of its denoising U-Net, can be repurposed as a perceptual similarity metric without any fine-tuning. The proposed DiffSim score aligns two images through attention before comparing them, which the authors argue captures both appearance and style while avoiding the pixel-alignment problem of low-level metrics and the information compression of CLIP and DINO. Across human-judgment, style, instance, and low-level benchmarks, DiffSim is reported to match or beat CLIP, DINO v2, and LPIPS, and an ensemble with CLIP and DINO improves further. The paper also introduces two new benchmarks, Sref for style and IP for instance consistency, to test similarity in custom generation settings. If correct, this gives generative-model practitioners a training-free way to check that generated images stay visually consistent with references.

What carries the argument

The load-bearing object is the Aligned Attention Score (AAS), computed inside an attention layer of a pretrained diffusion U-Net. For two images A and B, AAS takes A's query against B's keys and values and B's query against A's keys and values, then averages the cosine similarity of the two attended outputs; this symmetric cross-attention alignment is what replaces pixel-aligned feature comparison. DiffSim-S applies AAS in self-attention layers of Stable Diffusion 1.5, while DiffSim-C applies it in cross-attention layers using IP-Adapter Plus image tokens. The layer index and denoising timestep act as a control dial: shallow blocks and high timesteps correlate with style and low-level appearance, and deep blocks and low timesteps correlate with semantic and instance content.

What would settle it

Re-run the NIGHTS, Sref, and IP comparisons with DiffSim configured by a single layer and timestep chosen on a validation split before seeing test labels; if NIGHTS accuracy drops from 86.52% to at or below DINO v2's 85.24%, the headline state-of-the-art claim depends on test-set selection.

Watch

Extended reading notes

Core claim

DiffSim's central claim is that the self-attention and cross-attention layers of a Stable Diffusion U-Net carry features that can measure human-perceived visual similarity. The paper introduces the Aligned Attention Score (AAS), which feeds image A's query features to image B's key/value features in an attention layer and symmetrically does the reverse, then averages the cosine similarities of the two attention outputs. This implicit alignment lets the metric compare images that differ in layout, pose, or style without first registering pixels. By varying which U-Net block and which denoising timestep are used, the same method can emphasize low-level/style similarity (shallow layers, high timesteps) or semantic/instance similarity (deeper layers, lower timesteps). The paper reports that DiffSim outperforms CLIP, DINO v2, and LPIPS on the NIGHTS human-alignment benchmark and on its new Sref and IP style/instance benchmarks, and that applying AAS to CLIP and DINO v2 also improves them on some tasks.

Load-bearing premise

The reported state-of-the-art results assume that picking the best U-Net layer and denoising timestep by grid search on each benchmark's own test set is a fair way to estimate DiffSim's performance.

Editorial extensions

If this is right

  • A training-free perceptual metric for generated images becomes available directly from the U-Net already used to create those images, so no extra data labeling or model training is needed.
  • The same metric can be tuned by layer and timestep to measure style, instance identity, or low-level distortion, replacing several specialized metrics with one configurable score.
  • AAS is architecture-agnostic enough to improve CLIP and DINO v2 on style and low-level tasks, suggesting attention alignment is a generally useful add-on for similarity scoring.
  • An ensemble of CLIP, DINO v2, and DiffSim votes to higher accuracy on several benchmarks, meaning DiffSim adds signal not already captured by existing metrics.
  • For video, AAS-based scores have lower variance across frames of the same subject, which the paper interprets as better robustness to motion and layout change.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A practical version of DiffSim would need a fixed default layer and timestep chosen on validation data; the paper's reported numbers select the best setting on each test set, so a frozen configuration would likely change the exact margins.
  • Because AAS also helps CLIP and DINO, the active ingredient may be attention-based feature alignment rather than diffusion-specific knowledge; a control experiment using random or weakly trained attention features would isolate this.
  • The observed layer and timestep gradient suggests a multi-scale score that pools features across several blocks and timesteps could be more robust than any single setting, though the paper does not test this.
  • The approach could extend to video diffusion models directly, evaluating frame-to-frame consistency with the same attention alignment rather than a separate video metric.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. DiffSim proposes a training-free visual similarity metric that extracts features from the attention layers of a pretrained Stable Diffusion U-Net and computes an Aligned Attention Score (AAS) to handle spatial misalignment. The paper claims that this is the first use of pretrained diffusion models for visual similarity assessment, reports state-of-the-art accuracy on human-align, style, and instance benchmarks, and introduces two new benchmarks, Sref and IP bench, for style and instance consistency evaluation. The method is evaluated against CLIP, DINO v2, LPIPS, FFA, and Gram on seven benchmarks, with additional ablations, video consistency experiments, and image retrieval results.

Significance. If the empirical claims are correct, DiffSim would be a valuable addition to the toolkit for evaluating generated images: it is training-free, requires no fine-tuning, and leverages the rich appearance information in diffusion U-Net features. The AAS alignment mechanism is simple and is shown to also improve CLIP and DINO v2 when applied to their attention layers. The two new benchmarks, Sref and IP bench, address an important gap in style and instance consistency evaluation. However, the reported state-of-the-art status rests on a protocol that selects hyperparameters on the test set, and the margins over strong baselines are often very small. Without a corrected evaluation protocol and uncertainty quantification, the central contribution is not yet established.

major comments (4)
  1. [Section 4.1, Table 1, Supplementary Figures 7-13] The authors state that they "conducted grid searches across different layers and denoising time steps for each task, reporting the best results among these choices." This is test-set hyperparameter selection: the U-Net layer and denoising timestep are chosen on the same benchmark used for evaluation. Supplementary Figures 7-13 show that accuracy varies by tens of percentage points across timesteps and layers (for NIGHTS, the range is roughly 65% to 86%). The winning margins over the next-best baseline are small (e.g., 1.28 points on NIGHTS, 0.14 points on IP). A state-of-the-art claim requires a fixed configuration chosen without access to test labels, or a nested validation protocol with the selected configuration reported for each benchmark. As written, the comparison is optimistically biased and the central claim is not established.
  2. [Section 4.4, Table 1] No error bars, significance tests, or repeated-noise averaging are reported. DiffSim injects noise into the input images before feature extraction, yet each accuracy appears to come from a single run without multiple noise realizations or seeds. Several decisive margins are very narrow (IP: 91.84% vs 91.70% for CLIP; NIGHTS: 86.52% vs 85.24% for DINO v2). Without bootstrap confidence intervals or paired significance tests, it is unclear whether these differences are statistically meaningful. The authors should report repeated runs with different noise samples and provide uncertainty estimates.
  3. [Section 3.4, Section 4.4] The two newly introduced benchmarks, Sref bench and IP bench, are constructed with diffusion-based generators (Midjourney, Flux, and IP-Adapter). Because DiffSim extracts features from the denoising U-Net of Stable Diffusion, there is a risk that the benchmark favors the proposed method by construction, since the reference and candidate images share the same generative prior. The paper does not analyze this potential bias. A concrete test would be to evaluate on non-diffusion style and instance benchmarks, or to show that the performance advantage persists when candidate images are produced by GANs or other non-diffusion generators.
  4. [Abstract, Section 4.4, Table 1] The abstract claims "state-of-the-art performance" across benchmarks, but Table 1 shows that DiffSim is not the best method on several benchmarks: LPIPS outperforms on TID2013 (94.50% vs 94.17%), and DINO v2 outperforms on CUTE (77.27% vs 76.17%) and Dreambench++ (72.25% vs 71.50%). The text describes the Dreambench++ result as "comparable to" DINO v2, which is misleading given the numerical gap. The claims should be scoped to the benchmarks where DiffSim actually leads, or the method should be improved to be competitive across all evaluated settings.
minor comments (6)
  1. [Section 1] There is a typo in the Introduction: "video gneration" should be "video generation."
  2. [Section 3.3.1] The word "caculate" should be "calculate."
  3. [Equation (11)] Equation (11) appears to be missing the AAS operator in the second term: it reads "+zB_t,cross,n, IPA)" but should be "+AAS(zB_t,cross,n, IPA))."
  4. [Section 4.4] There are typos in the experimental section: "orginal" should be "original," and "stylish similarity" should be "style similarity."
  5. [Figure 4 caption] The caption contains a typo: "middel" should be "middle."
  6. [References] References [53] and [54] are duplicate entries for the same IP-Adapter paper; they should be consolidated into a single reference.

Circularity Check

1 steps flagged · score 4.0 of 10

The AAS derivation is not circular, but the central SOTA claim is weakened by per-benchmark grid-search selection of layer and timestep on the test set, which turns the reported accuracy into a fitted maximum rather than a fixed-protocol prediction.

  1. fitted input called prediction [Section 4.1, Experimental Setting]
    "We also conducted grid searches across different layers and denoising time steps for each task, reporting the best results among these choices; further details are provided in the supplementary materials."

    The layer index n and denoising timestep t in DiffSim-S(IA, IB, n, t) are free hyperparameters of the metric. The paper selects them by grid search on each benchmark's own test set and then reports the best accuracy in Table 1. The reported number is therefore the maximum over many configurations evaluated on the same triplets that define the task, not the accuracy of a single fixed DiffSim configuration. Supplementary Figures 7-13 show accuracy varies by tens of points across configurations (e.g., NIGHTS from roughly 65% to 86%), and the SOTA margins in Table 1 are narrow (NIGHTS 86.52% vs. DINO v2 85.24%; IP 91.84% vs. CLIP 91.70%).

full rationale

No formal circularity is present in the derivation of AAS: Eq. 7-10 define an attention-aligned cosine score without presupposing the benchmark labels, and the method is training-free rather than fitted to a target output. The paper does not lean on a load-bearing self-citation chain or a uniqueness theorem; its motivation cites external works such as ReferenceNet, Custom Diffusion, and IP-Adapter. The central claim about diffusion U-Net features measuring visual similarity has independent empirical content, and external benchmarks such as NIGHTS, TID2013, CUTE, and Dreambench++ provide some non-circular evidence. The main validity problem is the per-benchmark grid search over layers and timesteps on the test set, with the best result explicitly reported. That makes the headline state-of-the-art numbers selected maxima rather than fixed-configuration predictions, and the small margins in Table 1 mean this selection could determine the ranking. The newly introduced Sref and IP benchmarks are also diffusion-generated and share related attention mechanisms, which weakens the style/instance evidence but does not amount to an equation-level circularity. Overall, the derivation is not tautological, but the evaluation protocol partially manufactures the claimed advantage, so a moderate score is appropriate.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central comparison depends on per-benchmark choices of layer, timestep, and resolution, which are tuned on the test sets. The method also relies on the assumption that diffusion attention features carry perceptual content and that the two new synthetic benchmarks are valid ground truth.

free parameters (3)
  • U-Net layer index = per benchmark (D_0 through U_2)
    Grid-searched over Downsample 0-2, Middle, and Upsample 0-2 on each benchmark's test set; Section 4.1 and Figures 7-13.
  • Denoising timestep t = e.g., 900 for Sref, 750 for IP, 600 for NIGHTS
    Grid-searched over t=100 to t=900 per benchmark; Section 4.5 reports fixed t values for selected experimental curves.
  • Input resolution = 512x512
    The paper tested resolutions from 384 to 1024 and standardized to 512x512 to match SD1.5 training; Section 4.5.
assumptions (5)
  • domain assumption Self-attention features of a pretrained SD1.5 U-Net encode appearance, style, and semantic information sufficient for perceptual similarity.
    Central premise behind DiffSim-S, motivated by ReferenceNet, Custom Diffusion, and IP-Adapter observations in Section 3.3.1.
  • ad hoc to paper The Aligned Attention Score, comparing attn(Q_A,K_A,V_A) with attn(Q_A,K_B,V_B) via cosine, corrects for spatial misalignment without losing detail.
    Introduced in Equations 7-9; supported only by the paper's own ablations and not by a theoretical guarantee.
  • domain assumption IP-Adapter Plus tokens from the CLIP encoder faithfully represent image appearance for cross-attention similarity.
    Used in DiffSim-C in Section 3.3.2; the paper inherits this property from the IP-Adapter method without re-validation.
  • domain assumption The labels in NIGHTS, Dreambench++, CUTE, and TID2013 are reliable perceptual ground truth.
    Used as external benchmarks in Section 4.3 and treated as standard references.
  • domain assumption The newly introduced Sref and IP benchmarks correctly order images by style and instance similarity.
    Sref styles are artist-picked, but IP variants are generated by Flux with different IP-Adapter weights, and no human-rating validation is reported for either benchmark in Section 3.4.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DiffSim: Taming Diffusion Models for Evaluating Visual Similarity." pith.science (2026). https://pith.science/paper/L4NIXX7U

@misc{pith2026241214580,
  author       = {Pith},
  title        = {Pith review of: DiffSim: Taming Diffusion Models for Evaluating Visual Similarity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L4NIXX7U}},
  note         = {Machine review of arXiv:2412.14580}
}
read the original abstract

Diffusion models have fundamentally transformed the field of generative models, making the assessment of similarity between customized model outputs and reference inputs critically important. However, traditional perceptual similarity metrics operate primarily at the pixel and patch levels, comparing low-level colors and textures but failing to capture mid-level similarities and differences in image layout, object pose, and semantic content. Contrastive learning-based CLIP and self-supervised learning-based DINO are often used to measure semantic similarity, but they highly compress image features, inadequately assessing appearance details. This paper is the first to discover that pretrained diffusion models can be utilized for measuring visual similarity and introduces the DiffSim method, addressing the limitations of traditional metrics in capturing perceptual consistency in custom generation tasks. By aligning features in the attention layers of the denoising U-Net, DiffSim evaluates both appearance and style similarity, showing superior alignment with human visual preferences. Additionally, we introduce the Sref and IP benchmarks to evaluate visual similarity at the level of style and instance, respectively. Comprehensive evaluations across multiple benchmarks demonstrate that DiffSim achieves state-of-the-art performance, providing a robust tool for measuring visual coherence in generative models.

Figures

Figures reproduced from arXiv: 2412.14580 by the authors.

Figure 1
Figure 1. We propose DiffSim, a method that utilizes pre-trained diffusion models to extract image features for evaluating visual similarity. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The illustration shows two DiffSim implementations: DiffSim-S using self-attention, where U-Net extracts features from both [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. To evaluate style similarity and instance-level similarity, we introduced the Sref bench and IP bench. The Sref dataset contains 508 [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Some retrieval examples using DiffSim, CLIP, and DINO v2. The left, middel and right column displays retrieval results from [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Evaluation of different benchmarks across different [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Failure case. 4.8. Limitation and Failure Case DiffSim occasionally overemphasizes background features, neglecting key subject details, especially in images with smaller subjects. As shown in [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 8
Figure 8. Figure 8: Results on Dreambench++ dataset. 100 200 300 400 500 600 700 800 900 Time Step (t) 50 55 60 65 70 Accuracy (%) Performance of Different Time Steps and Layers on CUTE Dataset Layers D_0 D_1 D_2 M U_0 U_1 U_2 [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Results on CUTE dataset. 100 200 300 400 500 600 700 800 900 Time Step (t) 65 70 75 80 85 90 Accuracy (%) Performance of Different Time Steps and Layers on IP Bench Layers D_0 D_1 D_2 M U_0 U_1 U_2 [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Results on IP bench [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: Results on TID2013 dataset. 100 200 300 400 500 600 700 800 900 Time Step (t) 70 75 80 85 90 95 Accuracy (%) Performance of Different Time Steps and Layers on Sref Bench Layers D_0 D_1 D_2 M U_0 U_1 U_2 [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 12
Figure 12. Figure 12: Results on Sref bench. 100 200 300 400 500 600 700 800 900 Time Step (t) 80 85 90 95 100 Accuracy (%) Performance of Different Time Steps and Layers on InstantStyle Bench Layers D_0 D_1 D_2 M U_0 U_1 U_2 [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]
Figure 13
Figure 13. Figure 13: Results on InstantStyle bench [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]
Figure 14
Figure 14. Figure 14: Examples in Sref bench we proposed [PITH_FULL_IMAGE:figures/full_fig_p015_14.png]
Figure 15
Figure 15. Figure 15: Examples in IP bench we proposed [PITH_FULL_IMAGE:figures/full_fig_p016_15.png]
Figure 16
Figure 16. Figure 16: More image retrieval results [PITH_FULL_IMAGE:figures/full_fig_p017_16.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. IRIS: A Real-World Benchmark for Inverse Recovery and Identification of Physical Dynamic Systems from Monocular Video

    cs.CV 2026-03 accept novelty 6.0 of 10

    IRIS releases 220 real 4K videos of eight dynamical systems with ground-truth parameters plus a protocol that measures parameter recovery, equation selection, and multi-body failure modes of unsupervised video-to-phys...

  2. ReMoT: Reinforcement Learning with Motion Contrast Triplets

    cs.CV 2026-02 conditional novelty 6.0 of 10

    Training a 4B vision-language model on rule-generated motion-contrast triplets with GRPO lifts spatio-temporal QA accuracy by about 17 points on the authors' own benchmark and by smaller margins on standard benchmarks.

  3. CoCA: Step-level Reward for Free in RL-based T2I Diffusion Model Fine-tuning

    cs.CV 2025-05 conditional novelty 6.0 of 10

    CoCA redistributes a single final image reward across denoising steps using cosine similarity between intermediate and final latents, improving RL fine-tuning sample efficiency on four human preference rewards.

  4. MakeAnything: Harnessing Diffusion Transformers for Multi-Domain Procedural Sequence Generation

    cs.CV 2025-02 conditional novelty 6.0 of 10

    Fine-tuning a diffusion transformer with asymmetric LoRA plus a new 24,000-sequence dataset enables multi-domain, step-by-step procedural generation and image-to-process reconstruction.

  5. LayerTracer: Cognitive-Aligned Layered SVG Synthesis via Diffusion Transformer

    cs.CV 2025-02 conditional novelty 6.0 of 10

    A diffusion transformer trained on SVG construction sequences generates and vectorizes layered SVG graphics, breaking creation into editable steps.

  6. Any2AnyTryon: Leveraging Adaptive Position Embeddings for Versatile Virtual Clothing Tasks

    cs.CV 2025-01 conditional novelty 6.0 of 10

    A single DiT-based model with adaptive position embeddings performs virtual try-on, garment reconstruction, model-free try-on, and layered try-on from text and variable-size image inputs.

  7. RelationAdapter: Learning and Transferring Visual Relation with Diffusion Transformers

    cs.CV 2025-06 conditional novelty 5.0 of 10

    A decoupled-attention adapter transfers image-pair edits to new photos in diffusion transformers, trained with a new 218-task visual editing dataset.

Reference graph

Works this paper leans on

65 extracted references · 31 canonical work pages · cited by 7 Pith papers

  1. [1]

    https: //midjourneysref.com/discover?page=1

    Midjourney sref: Prompt library and examples. https: //midjourneysref.com/discover?page=1 . Ac- cessed: 2024-11-21. 5

    work page 2024

  2. [2]

    Flux.1 ai, 2024

    Flux.1 AI. Flux.1 ai, 2024. 5

    work page 2024

  3. [3]

    Cross-image attention for zero- shot appearance transfer

    Yuval Alaluf, Daniel Garibi, Or Patashnik, Hadar Averbuch- Elor, and Daniel Cohen-Or. Cross-image attention for zero- shot appearance transfer. In ACM SIGGRAPH 2024 Confer- ence Papers, pages 1–12, 2024. 5

    work page 2024

  4. [4]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 2

    arXiv 2023

  5. [5]

    In- structpix2pix: Learning to follow image editing instructions

    Tim Brooks, Aleksander Holynski, and Alexei A Efros. In- structpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392–18402, 2023. 2

    2023

  6. [6]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J’egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. 2021 ieee. In CVF International Conference on Computer Vision (ICCV), 2021. 1

    work page 2021

  7. [7]

    Robust clas- sification via a single diffusion model

    Huanran Chen, Yinpeng Dong, Zhengyi Wang, Xiao Yang, Chengqi Duan, Hang Su, and Jun Zhu. Robust clas- sification via a single diffusion model. arXiv preprint arXiv:2305.15241, 2023. 3

    arXiv 2023

  8. [8]

    Livephoto: Real image animation with text-guided motion control

    Xi Chen, Zhiheng Liu, Mengting Chen, Yutong Feng, Yu Liu, Yujun Shen, and Hengshuang Zhao. Livephoto: Real image animation with text-guided motion control. arXiv preprint arXiv:2312.02928, 2023. 5

    arXiv 2023

Show all 65 references
  1. [9]

    Echomimic: Lifelike audio-driven por- trait animations through editable landmark conditions

    Zhiyuan Chen, Jiajiong Cao, Zhiquan Chen, Yuming Li, and Chenguang Ma. Echomimic: Lifelike audio-driven por- trait animations through editable landmark conditions. arXiv preprint arXiv:2407.08136, 2024. 5

    2024 arXiv

  2. [10]

    Arcface: Additive angular margin loss for deep face recognition

    Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 4690–4699, 2019. 3

    2019

  3. [11]

    Prob- ing the 3d awareness of visual foundation models

    Mohamed El Banani, Amit Raj, Kevis-Kokitsi Maninis, Ab- hishek Kar, Yuanzhen Li, Michael Rubinstein, Deqing Sun, Leonidas Guibas, Justin Johnson, and Varun Jampani. Prob- ing the 3d awareness of visual foundation models. In Pro- ceedings of the IEEE/CVF Conference on Computer ...

    2024

  4. [12]

    Dream- sim: Learning new dimensions of human visual similar- ity using synthetic data

    Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dream- sim: Learning new dimensions of human visual similar- ity using synthetic data. arXiv preprint arXiv:2306.09344 ,

    arXiv

  5. [13]

    A neural algorithm of artistic style

    Leon A Gatys. A neural algorithm of artistic style. arXiv preprint arXiv:1508.06576, 2015. 6, 7

    2015 arXiv

  6. [14]

    Gatys, Alexander S

    Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2414–2423, 2016. 3

    2016

  7. [15]

    I2v-adapter: A general image-to-video adapter for diffusion models

    Xun Guo, Mingwu Zheng, Liang Hou, Yuan Gao, Yufan Deng, Pengfei Wan, Di Zhang, Yufan Liu, Weiming Hu, Zhengjun Zha, et al. I2v-adapter: A general image-to-video adapter for diffusion models. In ACM SIGGRAPH 2024 Conference Papers, pages 1–12, 2024. 3

    2024

  8. [16]

    Animatediff: Animate your personalized text-to-image diffusion models without specific tuning

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023. 2

    2023 arXiv

  9. [17]

    Id-animator: Zero-shot identity-preserving human video generation.arXiv preprint arXiv:2404.15275, 2024

    Xuanhua He, Quande Liu, Shengju Qian, Xin Wang, Tao Hu, Ke Cao, Keyu Yan, Man Zhou, and Jie Zhang. Id-animator: Zero-shot identity-preserving human video generation.arXiv preprint arXiv:2404.15275, 2024. 5

    2024 arXiv

  10. [18]

    Things: A database of 1,854 object concepts and more than 26,000 naturalistic object images

    Martin N Hebart, Adam H Dickter, Alexis Kidder, Wan Y Kwok, Anna Corriveau, Caitlin Van Wicklin, and Chris I Baker. Things: A database of 1,854 object concepts and more than 26,000 naturalistic object images. PloS one, 14 (10):e0223792, 2019. 3

    2019

  11. [19]

    Prompt-to-prompt im- age editing with cross attention control

    Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt im- age editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022. 2

    2022 arXiv

  12. [20]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 2

    2020

  13. [21]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. 3

    2021

  14. [22]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. InIn- ternational Conference on Learning Representations , 2022. 3

    2022

  15. [23]

    Animate anyone: Consistent and controllable image- to-video synthesis for character animation

    Li Hu. Animate anyone: Consistent and controllable image- to-video synthesis for character animation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8153–8163, 2024. 2, 3, 5

    2024

  16. [24]

    Hughes, Daniel J

    James M. Hughes, Daniel J. Graham, C. Robert Jacobsen, and Daniel N. Rockmore. Comparing higher-order spatial statistics and perceptual judgements in the stylometric anal- ysis of art. In 2011 19th European Signal Processing Con- ference, pages 1244–1248, 2011. 3

    2011

  17. [25]

    Self-supervised 3d representation learning of dressed humans from social media videos

    Yasamin Jafarian and Hyun Soo Park. Self-supervised 3d representation learning of dressed humans from social media videos. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(7):8969–8983, 2022. 6

    2022

  18. [26]

    Are these the same apple? comparing images based on object intrinsics

    Klemen Kotar, Stephen Tian, Hong-Xing Yu, Dan Yamins, and Jiajun Wu. Are these the same apple? comparing images based on object intrinsics. Advances in Neural Information Processing Systems, 36:40853–40871, 2023. 6, 7

    2023

  19. [27]

    Imagenet classification with deep convolutional neural net- works

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural net- works. Advances in neural information processing systems , 25, 2012. 3

    2012

  20. [28]

    Multi-concept customization of text-to-image diffusion

    Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1931–1941, 2023. 2, 3, 5

    1931

  21. [29]

    Deep photo style transfer, 2017

    Fujun Luan, Sylvain Paris, Eli Shechtman, and Kavita Bala. Deep photo style transfer, 2017. 3

    2017

  22. [30]

    T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models

    Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhon- gang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.08453, 2023. 2

    2023 arXiv

  23. [31]

    Human alignment of neural network representations

    Lukas Muttenthaler, Jonas Dippel, Lorenz Linhardt, Robert A Vandermeulen, and Simon Kornblith. Human alignment of neural network representations. arXiv preprint arXiv:2211.01201, 2022. 3

    2022 arXiv

  24. [32]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 3, 6, 7

    2023 arXiv

  25. [33]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 4195–4205,

  26. [34]

    Dreambench++: A human-aligned bench- mark for personalized image generation

    Yuang Peng, Yuxin Cui, Haomiao Tang, Zekun Qi, Runpei Dong, Jing Bai, Chunrui Han, Zheng Ge, Xiangyu Zhang, and Shu-Tao Xia. Dreambench++: A human-aligned bench- mark for personalized image generation. arXiv preprint arXiv:2406.16855, 2024. 1, 6

    2024 arXiv

  27. [35]

    A new color image database tid2013: Innovations and results

    Nikolay Ponomarenko, Oleg Ieremeiev, Vladimir Lukin, Lina Jin, Karen Egiazarian, Jaakko Astola, Benoit V ozel, Kacem Chehdi, Marco Carli, Federica Battisti, et al. A new color image database tid2013: Innovations and results. In Advanced Concepts for Intelligent Vision Systems:...

    2013

  28. [36]

    Pieapp: Perceptual image-error assessment through pairwise preference

    Ekta Prashnani, Hong Cai, Yasamin Mostofi, and Pradeep Sen. Pieapp: Perceptual image-error assessment through pairwise preference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1808– 1817, 2018. 3

    2018

  29. [37]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of th...

    2021

  30. [38]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 2

    2022

  31. [39]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2250...

    2023

  32. [40]

    Hi- erarchical classification of paintings using face- and brush stroke models

    Robert Sablatnig, Paul Kammerer, and Ernestine Zolda. Hi- erarchical classification of paintings using face- and brush stroke models. Proc. 14th Int. Conference on Pattern Recog- nition, 1, 2002. 3

    2002

  33. [41]

    Very deep convolutional networks for large-scale image recognition

    Karen Simonyan. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. 3

    2014 arXiv

  34. [42]

    Measuring style similarity in diffusion models

    Gowthami Somepalli, Anubhav Gupta, Kamal Gupta, Shra- may Palta, Micah Goldblum, Jonas Geiping, Abhinav Shri- vastava, and Tom Goldstein. Measuring style similarity in diffusion models. arXiv preprint arXiv:2404.01292, 2024. 3

    2024 arXiv

  35. [43]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 2

    2010 arXiv

  36. [44]

    Proces- spainter: Learn painting process from sequence data

    Yiren Song, Shijie Huang, Chen Yao, Xiaojun Ye, Hai Ci, Jiaming Liu, Yuxuan Zhang, and Mike Zheng Shou. Proces- spainter: Learn painting process from sequence data. arXiv preprint arXiv:2406.06062, 2024. 2

    2024 arXiv

  37. [45]

    Emergent correspondence from image diffusion

    Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. Advances in Neural Information Pro- cessing Systems, 36:1363–1389, 2023. 2, 3

    2023

  38. [46]

    Stable diffusion reference implementa- tion, 2023

    Hugging Face Team. Stable diffusion reference implementa- tion, 2023. Available online. 2, 5

    2023

  39. [47]

    Emo: Emote portrait alive-generating expressive portrait videos with audio2video diffusion model under weak conditions

    Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. Emo: Emote portrait alive-generating expressive portrait videos with audio2video diffusion model under weak conditions. arXiv preprint arXiv:2402.17485, 2024. 5

    2024 arXiv

  40. [48]

    Instantstyle: Free lunch towards style-preserving in text-to-image generation

    Haofan Wang, Matteo Spinelli, Qixun Wang, Xu Bai, Zekui Qin, and Anthony Chen. Instantstyle: Free lunch towards style-preserving in text-to-image generation. arXiv preprint arXiv:2404.02733, 2024. 3, 6

    2024 arXiv

  41. [49]

    Instantid: Zero-shot identity-preserving gener- ation in seconds

    Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, and An- thony Chen. Instantid: Zero-shot identity-preserving gener- ation in seconds. arXiv preprint arXiv:2401.07519 , 2024. 3

    2024 arXiv

  42. [50]

    Efros, Jun-Yan Zhu, and Richard Zhang

    Sheng-Yu Wang, Alexei A. Efros, Jun-Yan Zhu, and Richard Zhang. Evaluating data attribution for text-to-image models. In ICCV, 2023. 3

    2023

  43. [51]

    X-portrait: Expressive portrait anima- tion with hierarchical motion attention

    You Xie, Hongyi Xu, Guoxian Song, Chao Wang, Yichun Shi, and Linjie Luo. X-portrait: Expressive portrait anima- tion with hierarchical motion attention. In ACM SIGGRAPH 2024 Conference Papers, pages 1–11, 2024. 5

    2024

  44. [52]

    Magicanimate: Temporally consistent human im- age animation using diffusion model

    Zhongcong Xu, Jianfeng Zhang, Jun Hao Liew, Hanshu Yan, Jia-Wei Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. Magicanimate: Temporally consistent human im- age animation using diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco...

    2024

  45. [54]

    Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models

    Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models. arXiv preprint arXiv:2308.06721,

    arXiv

  46. [55]

    A general protocol to probe large vision models for 3d physical understanding

    Guanqi Zhan, Chuanxia Zheng, Weidi Xie, and Andrew Zis- serman. A general protocol to probe large vision models for 3d physical understanding. InThe Thirty-eighth Annual Con- ference on Neural Information Processing Systems, 2023. 2

    2023

  47. [56]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543, 2023. 2

    2023 arXiv

  48. [57]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shecht- man, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 586–595, 2018. 3, 6, 7

    2018

  49. [58]

    I2vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models

    Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin, Xiang Wang, Deli Zhao, and Jingren Zhou. I2vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models. arXiv preprint arXiv:2311.04145, 2023. 3, 5

    2023 arXiv

  50. [59]

    Ssr-encoder: Encoding selective subject representation for subject-driven generation

    Yuxuan Zhang, Yiren Song, Jiaming Liu, Rui Wang, Jinpeng Yu, Hao Tang, Huaxia Li, Xu Tang, Yao Hu, Han Pan, et al. Ssr-encoder: Encoding selective subject representation for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Rec...

    2024

  51. [60]

    Fast personalized text to image synthesis with attention injection

    Yuxuan Zhang, Yiren Song, Jinpeng Yu, Han Pan, and Zhongliang Jing. Fast personalized text to image synthesis with attention injection. In ICASSP 2024-2024 IEEE Inter- national Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP), pages 6195–6199. IEEE, 2024. 3

    2024

  52. [61]

    Stable- makeup: When real-world makeup transfer meets diffusion model

    Yuxuan Zhang, Lifu Wei, Qing Zhang, Yiren Song, Jiaming Liu, Huaxia Li, Xu Tang, Yao Hu, and Haibo Zhao. Stable- makeup: When real-world makeup transfer meets diffusion model. arXiv preprint arXiv:2403.07764, 2024. 2, 3, 5

    2024 arXiv

  53. [62]

    Stable-hair: Real-world hair transfer via diffusion model

    Yuxuan Zhang, Qing Zhang, Yiren Song, and Jiaming Liu. Stable-hair: Real-world hair transfer via diffusion model. arXiv preprint arXiv:2407.14078, 2024. 2, 3, 5 DiffSim: Taming Diffusion Models for Evaluating Visual Similarity Supplementary Material

    2024 arXiv

  54. [63]

    Image pair with the higher score is selected as the choice of current evaluated model

    Experimental Details in Different Bench On each benchmark, the similarity scores are computed be- tween a reference image and two candidate images, one of which is closer to the reference image. Image pair with the higher score is selected as the choice of current evaluated mo...

  55. [64]

    DiffSim-S SD1.5 leads in all benchmarks except for the CUTE dataset

    Exploring Different Model Architectures In Table 5, we present the performance differences of Diff- Sim using pre-trained models with different architectures. DiffSim-S SD1.5 leads in all benchmarks except for the CUTE dataset. DiffSim-C SD1.5 performs better on the CUTE datas...

  56. [65]

    Additional Experimental Results In Figures 7 to 13 , we present the default implementation of DiffSim, which is based on the self-attention layers of SD1.5, showing results across different layers and denois- ing time steps t

  57. [66]

    Additional Visual Examples Figure 14 and 15 show more examples of images from Sref bench and IP bench; Figure 16 presents more top-4 retrieval results of DiffSim, CLIP, DINO v2 on MS COCO, Sref bench and IP bench. 100 200 300 400 500 600 700 800 900 Time Step (t) 65 70 75 80 8...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.