Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Appearance Matching Adapter for Exemplar-based Semantic Image Synthesis in-the-Wild

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read AM-Adapter transfers multi-object appearance from a single exemplar scene by refining cross-image matching with semantic cost.

desk verdict A genuinely useful matching adapter for scene-level appearance transfer, but the 'in-the-wild' claim is stretched by training on cropped/flipped copies of the same image. read the letter →

arxiv 2412.03150 v2 pith:2I3Z76ZR submitted 2024-12-04 cs.CV

Siyoon Jin , Jisu Nam , Jiyoung Kim , Dahyun Chung , Yeong-Seok Kim , Joonhyung Park , Heonjeong Chu , Seungryong Kim This is my paper · ORCID
classification cs.CV
keywords exemplar-basedsemanticimagesynthesisappearancetransferaugmentedself-attentiondiffusionmodelssegmentation4Dconvolutiongenerationmulti-object
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that local appearance can be transferred from a single scene-level exemplar image to a target synthesis with many objects by refining the implicit cross-image matching inside augmented self-attention with semantic cues. Prior matching-only approaches misalign categories in complex scenes, so the paper introduces a categorical matching cost derived from segmentation maps and refines the combined cost with a 4D convolution network to resolve those mismatches. The refined cost is added residually to the original implicit cost, allowing the pre-trained diffusion model and structure-guidance network to stay frozen while only a small adapter learns matching. If correct, this makes exemplar-based semantic image synthesis practical in-the-wild: one scene-level exemplar can specify the appearance of many objects at once, with optional user control for one-to-one object mapping.

What carries the argument

The load-bearing object is the refined matching cost $O^{Y\to X}_t = \varphi(R^{Y\to X}_t) + A^{Y\to X}_t$, where $A^{Y\to X}_t$ is the implicit query-key matching cost from augmented self-attention and $R^{Y\to X}_t$ concatenates that cost with a binary categorical matching cost $C^{Y\to X}$ built from the exemplar and target segmentation maps. A lightweight 4D convolution network $\varphi$ aggregates the combined cost so that correspondences are locally consistent across the two images rather than isolated pixel matches. The residual addition to the original cost stabilizes training and keeps the pre-trained generation and structure networks frozen, so the framework disentangles generation from matching. Stage-wise training first fits the structure-guidance and diffusion models, then trains the adapter alone.

What would settle it

Build a test set of exemplar-target pairs with deliberately low structural similarity, such as different scene layouts or large camera shifts, and measure appearance preservation (CLIP and DINO similarity) and structural consistency (Self-Sim.) against pairs with high similarity; if the gains over the no-adapter baseline vanish or reverse when structure differs, the claim that the adapter generalizes in-the-wild fails.

Watch

Extended reading notes

Core claim

The central claim is that cross-image matching in augmented self-attention, rather than the generation network, is the bottleneck for scene-level exemplar transfer, and that this bottleneck can be fixed by a learned adapter. The adapter computes a binary categorical matching cost between exemplar and target segmentation maps, concatenates that cost along the head dimension with the implicit matching cost from self-attention, processes the combined 4D cost with a small 4D convolution network, and adds the output back to the original cost. This refined matching cost is what the target self-attention uses to select exemplar values, so appearance is warped along semantically consistent correspondences instead of arbitrary ones. The paper supports this with attention visualizations showing that query points on cars, buildings, and other objects attend to correct exemplar regions after the adapter is applied, and with quantitative gains in semantic alignment, appearance preservation, and image quality relative to previous attention-control baselines.

Load-bearing premise

The adapter is trained on exemplar-target pairs made by randomly cropping and flipping a single anchor image, so the in-the-wild claim assumes that matching learned under near-identical geometric transformations transfers to genuinely different scenes and large viewpoint changes.

Editorial extensions

If this is right

  • A single scene-level exemplar can set the appearance of multiple objects in the generated image at once, without per-object exemplars or manual assignment.
  • User-defined one-to-one matching becomes possible by restricting the categorical cost to chosen source and destination regions, enabling precise object-level control.
  • Because only the adapter is trained, the method adds few learnable parameters while preserving the structural consistency and image quality of the pre-trained models.
  • An automatic retrieval step can select a suitable exemplar from a pool, removing the labor-intensive manual selection required by prior tuning-free approaches.
  • The method generalizes across domains such as driving and indoor scenes, supporting image-to-image translation, segmentation-based editing, and consecutive video frame generation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The categorical cost makes the method directly sensitive to segmentation quality; coarse or missing labels would likely propagate into matching errors, so a segmentation-free variant would be needed for fully unconstrained use.
  • The same refined-cost mechanism could be lifted to other dense correspondence tasks, such as video object transfer or few-shot segmentation, where semantic labels are available.
  • Training pairs built by random crop and flip of the same anchor cap the structural variation the adapter sees, so fine-tuning on video or multi-view pairs with larger geometric changes is a natural test of the in-the-wild claim.
  • The retrieval step could be replaced by a learned structural-similarity embedding, which might improve both speed and matching quality on very large exemplar pools.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes AM-Adapter, a learnable adapter for exemplar-based semantic image synthesis. The method augments the self-attention of a pre-trained diffusion model with a categorical matching cost derived from segmentation maps (Eq. 6), concatenates it with the implicit attention matching cost, and refines the combined 4D cost volume with a lightweight 4D convolution network (Eqs. 7-8). Training is stage-wise: ControlNeXt and the diffusion model are trained first, then the adapter is trained on random crop/flip pairs from a single anchor image while the earlier stages are frozen. At inference, the paper proposes automatic retrieval of structurally similar exemplars and a matching-cost guidance (Eq. 9). The method is evaluated on BDD100K and NYUv2 with Self-Sim, ICLIP, IDINO, DINO[cls] loss, FID, and user studies, reporting state-of-the-art results.

Significance. If the central claim holds, the contribution is meaningful: it extends exemplar-based appearance transfer from single-object or foreground-centric settings to multi-object scene-level synthesis, and the 4D cost-aggregation design is a novel way to inject semantic segmentation priors into augmented self-attention. The paper ships a clear architectural description, extensive ablations, attention visualizations, and user studies, and the authors promise code and weights, which strengthens reproducibility. However, the significance is tempered by the fact that the headline in-the-wild claim is evaluated on a self-constructed benchmark with retrieval-selected exemplars, and the paper itself concedes that structurally dissimilar exemplars are ignored rather than transferred. The core idea is plausible, but the evidence for broad in-the-wild generality is not yet convincing.

major comments (3)
  1. [Sec. 3.5 and Sec. E] The training data premise is load-bearing for the in-the-wild claim. Sec. 3.5 states that exemplar-target pairs are generated by random cropping and flipping of the same anchor image, so the two views share near-identical content, pose, illumination, and object instances. Under this distribution, the adapter learns to refine matching only when the exemplar and target are already roughly aligned. The paper's own Sec. E and Fig. 8 concede that for non-homogeneous pairs the model 'robustly ignores' the exemplar rather than transferring its local appearance, and that temporal consistency degrades under large scene changes. This is consistent with the concern that the method does not actually transfer appearance across genuine structural differences. I recommend an explicit out-of-distribution test: evaluate on exemplar-target pairs from different scenes, different object instances, or large viewpoint changes where the exemplar is not retrieved by the proposed retrieval step, and report both qualitative and quantitative results separated by structural-similarity level. Without such a test, the central claim of automatic local appearance transfer in-the-wild is not supported.
  2. [Table 3 and Sec. 3.6] Table 3 shows that the proposed retrieval step alone raises ICLIP from 0.741 to 0.814, while matching guidance adds only 0.005 (0.814 to 0.819). This indicates that the majority of the appearance gain reported in the main comparison is attributable to retrieving exemplars that are structurally similar to the target, not to the adapter's matching refinement. Since retrieval is a component of the proposed inference pipeline, this is not circular, but it substantially narrows the in-the-wild claim: the headline numbers largely reflect favorable exemplar selection. The paper should report results with randomly chosen exemplars as a separate condition alongside the retrieved-exemplar condition, and should state clearly that the method's automatic transfer is demonstrated only for structurally similar pairs. The current presentation, with a single set of numbers in Table 1, overstates the adapter's ability to handle arbitrary user-provided exemplars.
  3. [Sec. 4.1 and Appendix B.2] The evaluation metrics do not directly measure the paper's central claim of local appearance transfer. The main appearance metrics ICLIP, IDINO, and DINO[cls] loss are global image-level similarities between the generated image and the exemplar; they do not verify that object-specific details (e.g., car color, building texture) are transferred to the correct spatial locations in the target. The object-wise local CLIP similarity in Appendix B.2 and Fig. 10 is a step in the right direction, but it is reported only as a figure on BDD100K and not included in the main table, and it still measures per-category global similarity rather than per-instance correspondence. Without a spatially resolved metric, such as per-segment or per-mask similarity aligned against the target segmentation, the evidence for 'local appearance fidelity' is indirect. I recommend adding such a metric, or at least reporting the object-wise results numerically for all tested datasets.
minor comments (5)
  1. [Fig. 24 caption] The caption contains a typo: 'Additon' should be 'Addition'.
  2. [References] References [12] and [13] appear to be the same paper (both are Animate Anyone); this duplication should be fixed.
  3. [Sec. 3.5 vs. Appendix A] Sec. 3.5 describes a three-stage training procedure (ControlNeXt, then diffusion model, then AM-Adapter), but Appendix A describes only two stages and does not mention fine-tuning the diffusion model. Please clarify whether the diffusion model was actually fine-tuned and, if so, on what data and for how many steps.
  4. [Table 1] The caption of Table 1 does not specify which dataset(s) the reported numbers are computed on. The main text says the dataset comprises 300 pairs from BDD100K and NYUv2, while Appendix B.1 mentions evalu ation on BDD100K, Cityscapes, and NYUv2 with 900 segmentation maps. This inconsistency should be resolved so the reader knows exactly which benchmark each table row refers to.
  5. [Sec. 1] The paper repeatedly claims that AM-Adapter uses 'minimal learnable parameters' but never reports the parameter count of the adapter. Adding a number (e.g., parameters and GPU memory) would make this claim concrete and comparable to future work.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central matching refinement is a learned empirical module evaluated on external data, not a quantity that reduces to its inputs by construction.

full rationale

This paper makes no analytical derivation claim whose output is equivalent to an input by definition. The core construction O^{Y→X}_t = φ(concat(A^{Y→X}_t, Downsample(C^{Y→X}))) + A^{Y→X}_t (Eqs. 7-8) is a learned residual correction applied to an implicit attention matching cost; the categorical cost C^{Y→X} is computed directly from the given segmentation maps SX and SY (Eq. 6), which are legitimate conditioning inputs rather than quantities the paper claims to predict. The Self-Sim metric is a standard external structural-consistency measure and is not the same function as the categorical matching cost, so evaluating semantic alignment with it does not reduce to the model's own input. The training-pair construction via random cropping and flipping of a single anchor image (Sec. 3.5) is a data-generating assumption, not a circular derivation; it is openly discussed, and the paper's own limitation section (Sec. E) concedes that temporal consistency degrades under large scene changes, which is a generalization risk rather than a definitional identification of output with input. The automated retrieval ablation (Table 3) separately shows the contributions of retrieval and matching guidance, and the quantitative comparison uses external datasets (BDD100K, Cityscapes, NYUv2) and a 45-participant user study, providing independent evidence beyond the authors' own fitted values. Self-citations appear only as a baseline (DreamMatcher [28], with overlapping authors) and as architectural inspiration (Neural Matching Fields [10], Hypercorrelation Squeeze [24]); neither is invoked as a uniqueness theorem or as the sole justification for the central claim. Consequently, no specific reduction of a claimed prediction to a fitted input or self-citation chain can be exhibited, so the correct finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The ledger for this paper is not about mathematical constants but about the design choices and external priors the method depends on. No new physical or conceptual entities are postulated; the 'adapter' is a trained module, not an invented entity. The main unpaid-for inputs are the pretrained diffusion and ControlNeXt priors, the aligned segmentation taxonomies, and the hand-selected hyperparameters (guidance scale, layer range, training budget, retrieval top-1).

free parameters (4)
  • guidance scale s = 7.5 for both text and matching guidance
    Hand-tuned in Eq. 9; controls the strength of the matching-cost guidance during inference and directly affects output; no sensitivity analysis is reported.
  • adapter layer range L in [1,9] = all self-attention layers except the first encoder block
    Applied to layers where ControlNeXt cross-normalized features are present; no ablation is shown on which layers benefit matching.
  • training data budget = 7K augmentation pairs, 25,000 steps
    Chosen without dataset-size or step-count ablation; the reported performance depends on this budget.
  • retrieval top-1 selection = top-1 grayscale structural similarity
    Inference-time hyperparameter; the appearance metrics in Table 1 are partly earned by choosing the most structurally similar exemplar, not by the adapter alone.
assumptions (4)
  • domain assumption A pre-trained text-to-image diffusion model (Stable Diffusion/LDM) provides a powerful generative prior that can be conditioned on exemplar appearance.
    Invoked throughout; the method builds on a pre-trained UNet and VAE (Sec. 3.1, 3.2).
  • domain assumption ControlNeXt, when pretrained on the target domain, reliably converts segmentation maps into structure guidance.
    Stage 1 of training assumes this; Sec. 3.5 states AM-Adapter is learned on frozen ControlNeXt; Appendix E notes performance degrades if ControlNeXt is weak.
  • domain assumption Semantic segmentation maps are available at inference with class labels that align between exemplar and target domains.
    Eq. (6) requires exact label equality S_Y(i,j) = S_X(k,l); mismatched label taxonomies or absent classes break the categorical cost.
  • domain assumption DDIM inversion of the exemplar into the pretrained latent space preserves enough appearance information for transfer.
    The Appearance Net reconstructs z0^X from zT^X via DDIM inversion (Sec. 3.2); failure of reconstruction would degrade key/value features.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Appearance Matching Adapter for Exemplar-based Semantic Image Synthesis in-the-Wild." pith.science (2026). https://pith.science/paper/2I3Z76ZR

@misc{pith2026241203150,
  author       = {Pith},
  title        = {Pith review of: Appearance Matching Adapter for Exemplar-based Semantic Image Synthesis in-the-Wild},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2I3Z76ZR}},
  note         = {Machine review of arXiv:2412.03150}
}
read the original abstract

Exemplar-based semantic image synthesis generates images aligned with semantic content while preserving the appearance of an exemplar. Conventional structure-guidance models like ControlNet, are limited as they rely solely on text prompts to control appearance and cannot utilize exemplar images as input. Recent tuning-free approaches address this by transferring local appearance via implicit cross-image matching in the augmented self-attention mechanism of pre-trained diffusion models. However, prior works are often restricted to single-object cases or foreground object appearance transfer, struggling with complex scenes involving multiple objects. To overcome this, we propose AM-Adapter (Appearance Matching Adapter) to address exemplar-based semantic image synthesis in-the-wild, enabling multi-object appearance transfer from a single scene-level image. AM-Adapter automatically transfers local appearances from the scene-level input. AM-Adapter alternatively provides controllability to map user-defined object details to specific locations in the synthesized images. Our learnable framework enhances cross-image matching within augmented self-attention by integrating semantic information from segmentation maps. To disentangle generation and matching, we adopt stage-wise training. We first train the structure-guidance and generation networks, followed by training the matching adapter while keeping the others frozen. During inference, we introduce an automated exemplar retrieval method for selecting exemplar image-segmentation pairs efficiently. Despite utilizing minimal learnable parameters, AM-Adapter achieves state-of-the-art performance, excelling in both semantic alignment and local appearance fidelity. Extensive ablations validate our design choices. Code and weights will be released.: https://cvlab-kaist.github.io/AM-Adapter/

Figures

Figures reproduced from arXiv: 2412.03150 by the authors.

Figure 1
Figure 1. AM-Adapter enables Exemplar-based Semantic Image Synthesis in-the-Wild. (a) Given an exemplar image and a target segmentation, AM-Adapter generates high-quality images that retain the local appearance of the exemplar and the accurate image structure defined by the segmentation map. We demonstrate the versatility of our method in various applications, including (b) controllable one-to￾one appearance transfer with use… view at source ↗
Figure 2
Figure 2. Controllability of our AM-Adapter: The M-to-N setting refers to a multiple-object many-to-many transfer, where M and N denote the number of instances of a specific category in the exemplar and target, respectively. By default, as shown in (a), AM-Adapter automatically matches appearance based on structural similarity. In addition, as shown in (b), it allows user-defined guidance to precisely transfer specific object… view at source ↗
Figure 3
Figure 3. Attention Visualization: (a) Exemplar image, (b) target segmentation, and (g) generated image. Green and orange markers in (b) indicate query points. (c) and (d) show the augmented self-attention map Q Y t (KX t ) T from the green marker, before and after applying AM-Adapter, respectively. (e) and (f) show the augmented self-attention map Q Y t (KX t ) T from the orange marker, before and after applying AM-Adapter, … view at source ↗
Figures from the paper (23 more)
Figure 4
Figure 4. Figure 4: Overall Architecture. 4D Conv 𝑂𝑡 Y→X 𝜙(𝑅𝑡 Y→X ) 𝐴𝑡 {Y,X} 𝑅𝑡 Y→X 𝐴𝑡 Y→X ( 𝑚 + 1 , ℎ, 𝑤, ℎ, 𝑤) 𝑄𝑡 Y𝐾𝑡 Y 𝑄𝑡 Y𝐾𝑡 X (ℎ, 𝑤, ℎ, 𝑤) (ℎ, 𝑤, ℎ, 𝑤) 𝑤X ℎ X 𝑤Y ℎ Y 𝐶 Y→X 𝑤Y ℎ Y 𝑤Y ℎ Y 𝑤X ℎ X 𝑤Y ℎ Y Channel Concatenation Residual Connection [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Architecture of Appearance Matching Adapter. Exemplar Segmentation Target Segmentation Categorical Matching Cost [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Visualization of Categorical Matching Cost. Green and pink markers denote query points in the exemplar segmen￾tation map. Our categorical matching cost effectively identifies binary categorical matches between exemplar and target segmen￾tation maps (best viewed in colo…
Figure 7
Figure 7. Figure 7: Qualitative Comparison on BDD100K [45] and NYUv2 [29]: Compared to previous methods[1, 19, 25, 44, 46] that fail to transfer local appearance or find accurate matching, AM-Adapter effectively achieves precise matching in content-rich, complex scenes. Furthermore, our a…
Figure 8
Figure 8. Figure 8: Robustness to Exemplar. If the exemplar lacks structural similarity to the target segmentation maps, AM-Adapter robustly ignores it and remains effective on non-homogeneous pairs. Methods Training Self-sim. (↓) IDINO(↑) DINO [cls] loss (↓) ICLIP(↑) FID (↓) ControlNet […
Figure 9
Figure 9. Figure 9: User Study. the CLIP image similarity denoted as ICLIP (↑) [35] be￾tween the generated image and the exemplar image. To assess image quality, we calculate FID (↓) [5]. Addition￾ally, to better capture local details, we calculated DINO￾based metrics, including IDINO(↑) …
Figure 10
Figure 10. Figure 10: Object-wise CLIP Image Similarity per Class. Augmented SA +Categorical matching cost Structure Consistency MasaCtrl DreamMatcher Augmented SA Augmented SA w/ finetuning Ours Appearance Preservation Image Quality 0.11 0.17 0.15 0.18 0.11 0.27 0.29 0.11 0.12 0.13 0.11 0…
Figure 11
Figure 11. Figure 11: User Study on Ablations. similarity. Our approach achieves the best performance in this evaluation [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]
Figure 12
Figure 12. Figure 12: Ablation Study on Individual Components: (a) ControlNeXt [32] + MasaCtrl [2], (b) ControlNeXt [32] + Dream￾Matcher [28], (c) ControlNeXt [32] + Augmented Self-Attention, (d) (c) + w/ Fine-tuning, (e) (c) + Categorical Matching Cost, (f) AM-Adapter (Ours). 13 [PITH_FU…
Figure 13
Figure 13. Figure 13: Ablation on Inference: (a) random exemplar image, (b) retrieved exemplar image, (c) target segmentation map with desired structure, results generated (d) without retrieval or matching guidance, (e) with retrieval but without matching guidance, and (f) with both retrie…
Figure 14
Figure 14. Figure 14: Additional Attention Visualization: (a) Exemplar image with desired appearance, (b) target segmentation with desired structure, and (g) generated image. Green and orange markers in (b) indicate query points. (c) and (d) show the augmented self-attention map Q Y t (KX …
Figure 15
Figure 15. Figure 15: Additional qualitative results on NYUv2 [29] dataset. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]
Figure 16
Figure 16. Figure 16: An Example of a User Study Comparing AM-Adapter with Previous Methods. For structure preservation, we provide the target segmentation map and generated images from different methods, ControlNet [46] + IP-Adapter [44], FreeControl [25], Cross￾Image Attention [1], Ctrl-…
Figure 17
Figure 17. Figure 17: An Example of a User Study Comparing AM-Adapter with Ablation Studies. For structure preservation, we provide the target segmentation map and generated images from different methods, key-value replacement (MasaCtrl [2]), DreamMatcher [28], augmented self-attention, au…
Figure 18
Figure 18. Figure 18: Additional Qualitative Results of AM-Adapter. Visualization of results generated by AM-Adapter (Ours) across various scenarios. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_18.png]
Figure 19
Figure 19. Figure 19: Additional Qualitative Comparison on BDD100K [45] Dataset. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_19.png]
Figure 20
Figure 20. Figure 20: Details of the Retrieval-based Inference. BDD100K Cityscapes Retrieved Image Target Segmentation [PITH_FULL_IMAGE:figures/full_fig_p022_20.png]
Figure 21
Figure 21. Figure 21: Retrieval Examples. During inference, our retrieval technique selects the exemplar image that exhibits the highest structural similarity to the target segmentation map. User-defined Target Seg Generated Image Generated Image Exemplar Image User-defined Exemplar Seg Ex…
Figure 22
Figure 22. Figure 22: Application: Controllable One-to-One Matching with User Guidance. The AM-Adapter enforces one-to-one mapping in a many-to-many setting, enabling controlled appearance transfer. The white regions in the exemplar and target segmentations indicate the source and destinat…
Figure 23
Figure 23. Figure 23: Application: Object Removal by Segmentation-Based Image Editing. (a) Original target segmentation with desired struc￾ture, (b) generated image given the target segmentation map (a), (c) edited target segmentation map, (d) generated image given the target segmentation …
Figure 24
Figure 24. Figure 24: Application: Object Additon by Segmentation-Based Image Editing. (a) Edited target segmentation with desired structure, (b) generated image given the target segmentation map (a), (c) edited target segmentation map, (d) generated image given the target segmentation map…
Figure 25
Figure 25. Figure 25: Application: Image-to-Image Translation. The first and third row represent exemplar images reflecting various weather conditions and times of day, including various categories such as cloudy days, sunset hours, sunny, night, and rainy conditions. The second and fourth…
Figure 26
Figure 26. Figure 26: Application: Appearance-Consistent Consecutive Video Frame Generation. The target segmentation maps in the first row are consecutive frames provided by the BDD100K [45] dataset. The second row displays the generated image results corresponding to each target segmentat…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Emergent Temporal Correspondences from Video Diffusion Transformers

    cs.CV 2025-06 conditional novelty 6.0 of 10

    Video diffusion transformers encode temporal correspondences primarily in query-key similarities of a few specific attention layers, which can be extracted for zero-shot point tracking and used for training-free motio...

Reference graph

Works this paper leans on

50 extracted references · 38 canonical work pages · cited by 1 Pith paper

  1. [1]

    Cross-image attention for zero- shot appearance transfer

    Yuval Alaluf, Daniel Garibi, Or Patashnik, Hadar Averbuch- Elor, and Daniel Cohen-Or. Cross-image attention for zero- shot appearance transfer. In ACM SIGGRAPH 2024 Confer- ence Papers, pages 1–12, 2024. 2, 3, 4, 6, 7, 8, 12, 19

    work page 2024

  2. [2]

    Masactrl: Tuning-free mu- tual self-attention control for consistent image synthesis and editing

    Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xi- aohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mu- tual self-attention control for consistent image synthesis and editing. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 22560–22570, 2023. 2, 4, 8, 12, 13, 14, 19

    work page 2023

  3. [3]

    Finding the subjective truth: Collecting 2 million votes for comprehensive gen-ai model evaluation

    Dimitrios Christodoulou and Mads Kuhlmann-Jørgensen. Finding the subjective truth: Collecting 2 million votes for comprehensive gen-ai model evaluation. arXiv preprint arXiv:2409.11904, 2024. 11

    arXiv 2024

  4. [4]

    The cityscapes dataset for semantic urban scene understanding

    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 3213–3223, 2016. 3, 11

    work page 2016

  5. [5]

    Diffusion models beat gans on image synthesis, 2021

    Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis, 2021. 8, 11

    work page 2021

  6. [6]

    Towards pragmatic semantic image synthesis for ur- ban scenes, 2023

    George Eskandar, Diandian Guo, Karim Guirguis, and Bin Yang. Towards pragmatic semantic image synthesis for ur- ban scenes, 2023. 2

    work page 2023

  7. [7]

    Encoder-based domain tuning for fast personalization of text-to-image models.ACM Transactions on Graphics (TOG), 42(4):1–13, 2023

    Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. Encoder-based domain tuning for fast personalization of text-to-image models.ACM Transactions on Graphics (TOG), 42(4):1–13, 2023. 2, 3, 4, 11

    work page 2023

  8. [8]

    Generative view synthesis: From single- view semantics to novel-view images, 2020

    Tewodros Habtegebrial, Varun Jampani, Orazio Gallo, and Didier Stricker. Generative view synthesis: From single- view semantics to novel-view images, 2020. 2

    work page 2020

Show all 50 references
  1. [9]

    Classifier-free diffusion guidance, 2022

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance, 2022. 7, 14

    2022

  2. [10]

    Neural matching fields: Implicit representation of matching fields for visual correspondence, 2022

    Sunghwan Hong, Jisu Nam, Seokju Cho, Susung Hong, San- gryul Jeon, Dongbo Min, and Seungryong Kim. Neural matching fields: Implicit representation of matching fields for visual correspondence, 2022. 5

    2022

  3. [11]

    Hexiang Hu, Kelvin C. K. Chan, Yu-Chuan Su, Wenhu Chen, Yandong Li, Kihyuk Sohn, Yang Zhao, Xue Ben, Bo- qing Gong, William Cohen, Ming-Wei Chang, and Xuhui Jia. Instruct-imagen: Image generation with multi-modal in- struction, 2024. 3

    2024

  4. [12]

    Animate anyone: Consistent and controllable image- to-video synthesis for character animation

    Li Hu. Animate anyone: Consistent and controllable image- to-video synthesis for character animation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8153–8163, 2024. 2

    2024

  5. [13]

    Animate anyone: Consistent and controllable image-to-video synthesis for character animation, 2024

    Li Hu, Xin Gao, Peng Zhang, Ke Sun, Bang Zhang, and Liefeng Bo. Animate anyone: Consistent and controllable image-to-video synthesis for character animation, 2024. 2

    2024

  6. [14]

    Genai arena: An open evaluation platform for generative models

    Dongfu Jiang, Max Ku, Tianle Li, Yuansheng Ni, Shizhuo Sun, Rongqi Fan, and Wenhu Chen. Genai arena: An open evaluation platform for generative models. arXiv preprint arXiv:2406.04485, 2024. 11

    2024 arXiv

  7. [15]

    Genai-bench: Evaluating and improv- ing compositional text-to-visual generation

    Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jiayao Li, Yixin Fei, Kewen Wu, Tiffany Ling, Xide Xia, Pengchuan Zhang, Gra- ham Neubig, et al. Genai-bench: Evaluating and improv- ing compositional text-to-visual generation. arXiv preprint arXiv:2406.13743, 2024. 11

    2024 arXiv

  8. [16]

    Controlnet++: Improving conditional controls with efficient consistency feedback

    Ming Li, Taojiannan Yang, Huafeng Kuang, Jie Wu, Zhaon- ing Wang, Xuefeng Xiao, and Chen Chen. Controlnet++: Improving conditional controls with efficient consistency feedback. In European Conference on Computer Vision , pages 129–147. Springer, 2025. 2, 3, 11

    2025

  9. [17]

    Gligen: Open-set grounded text-to-image generation

    Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jian- wei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22511–22521, 2023. 3, 11

    2023

  10. [18]

    Photomaker: Customizing re- alistic human photos via stacked id embedding

    Zhen Li, Mingdeng Cao, Xintao Wang, Zhongang Qi, Ming- Ming Cheng, and Ying Shan. Photomaker: Customizing re- alistic human photos via stacked id embedding. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8640–8650, 2024. 2, 3, 11

    2024

  11. [19]

    Ctrl-x: Controlling structure and appear- ance for text-to-image generation without guidance

    Kuan Heng Lin, Sicheng Mo, Ben Klingher, Fangzhou Mu, and Bolei Zhou. Ctrl-x: Controlling structure and appear- ance for text-to-image generation without guidance. arXiv preprint arXiv:2406.07540, 2024. 2, 3, 4, 6, 7, 8, 11, 12, 19

    2024 arXiv

  12. [20]

    Evaluating text-to-visual generation with image-to-text gen- eration

    Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text gen- eration. In European Conference on Computer Vision, pages 366–384. Springer, 2025. 11

    2025

  13. [21]

    Decoupled weight decay regularization, 2019

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. 11

    2019

  14. [22]

    Siedob: Semantic image editing by disentangling object and background, 2023

    Wuyang Luo, Su Yang, Xinjian Zhang, and Weishan Zhang. Siedob: Semantic image editing by disentangling object and background, 2023. 16

    2023

  15. [23]

    Zhengyao Lv, Yuxiang Wei, Wangmeng Zuo, and Kwan- Yee K. Wong. Place: Adaptive layout-semantic fusion for semantic image synthesis, 2024. 3

    2024

  16. [24]

    Hypercorrela- tion squeeze for few-shot segmentation, 2021

    Juhong Min, Dahyun Kang, and Minsu Cho. Hypercorrela- tion squeeze for few-shot segmentation, 2021. 5

    2021

  17. [25]

    Freecontrol: Training-free spatial control of any text-to-image diffusion model with any condition

    Sicheng Mo, Fangzhou Mu, Kuan Heng Lin, Yanli Liu, Bochen Guan, Yin Li, and Bolei Zhou. Freecontrol: Training-free spatial control of any text-to-image diffusion model with any condition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pa...

    2024

  18. [26]

    Null-text inversion for editing real im- ages using guided diffusion models

    Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real im- ages using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6038–6047, 2023. 4

    2023

  19. [27]

    T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models

    Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 4296–4304, 20...

    2024

  20. [28]

    Dreammatcher: Appearance matching self-attention for semantically-consistent text-to- image personalization

    Jisu Nam, Heesu Kim, DongJae Lee, Siyoon Jin, Seungry- ong Kim, and Seunggyu Chang. Dreammatcher: Appearance matching self-attention for semantically-consistent text-to- image personalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition...

    2024

  21. [29]

    Indoor segmentation and support inference from rgbd images

    Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, 2012. 6, 7, 11, 15, 18

    2012

  22. [30]

    Semantic image synthesis with spatially-adaptive nor- malization

    Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially-adaptive nor- malization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2337–2346,

  23. [31]

    One-step image translation with text-to-image models, 2024

    Gaurav Parmar, Taesung Park, Srinivasa Narasimhan, and Jun-Yan Zhu. One-step image translation with text-to-image models, 2024. 11

    2024

  24. [32]

    Controlnext: Powerful and effi- cient control for image and video generation

    Bohao Peng, Jian Wang, Yuechen Zhang, Wenbo Li, Ming- Chang Yang, and Jiaya Jia. Controlnext: Powerful and effi- cient control for image and video generation. arXiv preprint arXiv:2408.06070, 2024. 2, 3, 4, 6, 7, 8, 11, 13, 16

    2024 arXiv

  25. [33]

    Portraitbooth: A versatile portrait model for fast identity-preserved personalization

    Xu Peng, Junwei Zhu, Boyuan Jiang, Ying Tai, Donghao Luo, Jiangning Zhang, Wei Lin, Taisong Jin, Chengjie Wang, and Rongrong Ji. Portraitbooth: A versatile portrait model for fast identity-preserved personalization. InProceedings of the IEEE/CVF Conference on Computer Vision a...

    2024

  26. [34]

    Variational autoencoder for deep learning of images, labels and cap- tions

    Yunchen Pu, Zhe Gan, Ricardo Henao, Xin Yuan, Chun- yuan Li, Andrew Stevens, and Lawrence Carin. Variational autoencoder for deep learning of images, labels and cap- tions. Advances in neural information processing systems , 29, 2016. 4

    2016

  27. [35]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. 2, 3, 8, 11

    2021

  28. [36]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 3

    2022

  29. [37]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 11

    2010 arXiv

  30. [38]

    Data augmentation using random image cropping and patch- ing for deep cnns

    Ryo Takahashi, Takashi Matsubara, and Kuniaki Uehara. Data augmentation using random image cropping and patch- ing for deep cnns. IEEE Transactions on Circuits and Sys- tems for Video Technology, 30(9):2917–2931, 2020. 6

    2020

  31. [39]

    Evalalign: Evaluating text- to-image models through precision alignment of multimodal large models with supervised fine-tuning to human annota- tions

    Zhiyu Tan, Xiaomeng Yang, Luozheng Qin, Mengping Yang, Cheng Zhang, and Hao Li. Evalalign: Evaluating text- to-image models through precision alignment of multimodal large models with supervised fine-tuning to human annota- tions. arXiv preprint arXiv:2406.16562, 2024. 11

    2024 arXiv

  32. [40]

    Splicing vit features for semantic appearance transfer,

    Narek Tumanyan, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Splicing vit features for semantic appearance transfer,

  33. [41]

    Seman- tic image synthesis via diffusion models

    Weilun Wang, Jianmin Bao, Wengang Zhou, Dongdong Chen, Dong Chen, Lu Yuan, and Houqiang Li. Seman- tic image synthesis via diffusion models. arXiv preprint arXiv:2207.00050, 2022. 3, 7, 11

    2022

  34. [42]

    Fastcomposer: Tuning-free multi- subject image generation with localized attention

    Guangxuan Xiao, Tianwei Yin, William T Freeman, Fr ´edo Durand, and Song Han. Fastcomposer: Tuning-free multi- subject image generation with localized attention. Interna- tional Journal of Computer Vision, pages 1–20, 2024. 2, 3, 11

    2024

  35. [43]

    Freestyle layout-to-image synthesis, 2023

    Han Xue, Zhiwu Huang, Qianru Sun, Li Song, and Wenjun Zhang. Freestyle layout-to-image synthesis, 2023. 3

    2023

  36. [44]

    Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models

    Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models. arXiv preprint arXiv:2308.06721,

    arXiv

  37. [45]

    Bdd100k: A diverse driving dataset for heterogeneous multitask learning

    Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Dar- rell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition ...

    2020

  38. [46]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023. 3, 4, 6, 7, 8, 11, 12, 19

    2023

  39. [47]

    Uni-controlnet: All-in-one control to text-to-image diffusion models

    Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan-Yee K Wong. Uni-controlnet: All-in-one control to text-to-image diffusion models. Advances in Neural Information Processing Sys- tems, 36, 2024. 2, 3, 11

    2024

  40. [48]

    Flipda: Effective and robust data augmentation for few-shot learning, 2022

    Jing Zhou, Yanan Zheng, Jie Tang, Jian Li, and Zhilin Yang. Flipda: Effective and robust data augmentation for few-shot learning, 2022. 6

    2022

  41. [49]

    Yan Zhuang, Benjamin Hou, Tejas Sudharshan Mathai, Pri- tam Mukherjee, Boah Kim, and Ronald M. Summers. Se- mantic image synthesis for abdominal ct, 2023. 2 10 AM-Adapter: Appearance Matching Adapter for Exemplar-based Semantic Image Synthesis in-the-Wild Supplementary Materia...

    2023

  42. [2023]

    2, 3, 4, 6, 7, 8, 11, 12, 19

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.