{"id":"868b57ff-07a9-4849-ae93-e9ca76dbc296","arxiv_id":"2412.03150","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"AM-Adapter injects segmentation-derived matching costs into diffusion self-attention to transfer local object appearances from an exemplar to a target segmentation-driven scene.","lead":"This paper introduces a diffusion-model adapter that transfers the appearance of objects from a single reference image into a new image whose layout is set by a segmentation map. It targets complex multi-object scenes, where earlier exemplar-based methods could only handle one object or global style.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No significant objection identified; the central claim is plausible but its in-the-wild generality rests on training data that may not cover large structural changes, so a targeted out-of-distribution test is the main remaining check.","rationale":"The reader's verdict is CONDITIONAL, and the stress-test pass finds the same load-bearing concern: training on random-crop/flip pairs (§3.5) may not support the in-the-wild claim. The paper is internally consistent, and its own limitations section (Sec. E) concedes degraded consistency under large scene changes; Figure 8 concedes that structurally dissimilar exemplars are ignored. Table 3 shows retrieval accounts for the largest appearance gain, supporting the concern that the benchmark rewards near-homologous exemplars. An honest non-finding is not appropriate because the concern is real and concrete, but it does not invalidate the method's core contribution, which is the architecture and stage-wise training. The proposed cross-scene, no-retrieval test would settle whether the central claim of in-the-wild transfer holds. Since the reader already conditioned on exactly this issue, the verdict remains CONDITIONAL and no adjustment is needed.","tokens_in":18611,"tokens_out":1561,"duration_ms":12876,"concrete_test":"Build an out-of-distribution evaluation set where the target segmentation and the exemplar image come from different scenes with no retrieval-based preselection: e.g., take 300 BDD100K target segmentation maps and pair each with an exemplar image of the opposite weather/area composition or with an exemplar from Cityscapes, then measure ICLIP and DINO[cls] loss against the random-exemplar baseline in Table 3. If ICLIP drops to the random-exemplar level (about 0.741) while FID stays similar, the appearance gain is mostly retrieval-driven and the in-the-wild transfer claim is not supported; if the drop is modest, the adapter generalizes beyond its near-identical augmentation training.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that AM-Adapter's learned 4D refinement of the categorical plus implicit matching cost (Eqs. 6-8) enables automatic, multi-object local appearance transfer from a single scene-level exemplar in-the-wild. The weakest load-bearing assumption is not the architecture itself but the training-data premise in Sec. 3.5: exemplar-target pairs are generated by random cropping and flipping of one anchor image. Under this augmentation, the exemplar and target share near-identical content, pose, illumination, and object instances; the adapter therefore learns to refine matches when the two views are already roughly aligned. The in-the-wild claim requires that this refinement transfers to exemplar-target pairs with genuine structural differences, such as different scenes, different object instances, or large viewpoint changes. The paper's own evidence partially concedes this concern: Fig. 8 shows that when the exemplar lacks structural similarity the model 'robustly ignores it' rather than transferring appearance, and Sec. E states that temporal consistency degrades under large scene changes. Table 3 further shows that retrieval, which selects an exemplar structurally similar to the target, contributes most of the appearance gain (ICLIP 0.741 to 0.814 vs. 0.814 to 0.819 for matching guidance). Thus the headline quantitative result may largely reflect the retrieval step supplying near-homologous exemplars, not the adapter transferring appearance across structurally different scenes. This is a scope concern rather than an internal inconsistency: the method is coherent, and the paper acknowledges the behavior, but the 'in-the-wild' wording implies stronger generality than the training distribution supports.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AM-Adapter, a learnable adapter for exemplar-based semantic image synthesis. The method augments the self-attention of a pre-trained diffusion model with a categorical matching cost derived from segmentation maps (Eq. 6), concatenates it with the implicit attention matching cost, and refines the combined 4D cost volume with a lightweight 4D convolution network (Eqs. 7-8). Training is stage-wise: ControlNeXt and the diffusion model are trained first, then the adapter is trained on random crop/flip pairs from a single anchor image while the earlier stages are frozen. At inference, the paper proposes automatic retrieval of structurally similar exemplars and a matching-cost guidance (Eq. 9). The method is evaluated on BDD100K and NYUv2 with Self-Sim, ICLIP, IDINO, DINO[cls] loss, FID, and user studies, reporting state-of-the-art results.","tokens_in":18873,"tokens_out":4531,"duration_ms":41056,"significance":"If the central claim holds, the contribution is meaningful: it extends exemplar-based appearance transfer from single-object or foreground-centric settings to multi-object scene-level synthesis, and the 4D cost-aggregation design is a novel way to inject semantic segmentation priors into augmented self-attention. The paper ships a clear architectural description, extensive ablations, attention visualizations, and user studies, and the authors promise code and weights, which strengthens reproducibility. However, the significance is tempered by the fact that the headline in-the-wild claim is evaluated on a self-constructed benchmark with retrieval-selected exemplars, and the paper itself concedes that structurally dissimilar exemplars are ignored rather than transferred. The core idea is plausible, but the evidence for broad in-the-wild generality is not yet convincing.","major_comments":[{"comment":"The training data premise is load-bearing for the in-the-wild claim. Sec. 3.5 states that exemplar-target pairs are generated by random cropping and flipping of the same anchor image, so the two views share near-identical content, pose, illumination, and object instances. Under this distribution, the adapter learns to refine matching only when the exemplar and target are already roughly aligned. The paper's own Sec. E and Fig. 8 concede that for non-homogeneous pairs the model 'robustly ignores' the exemplar rather than transferring its local appearance, and that temporal consistency degrades under large scene changes. This is consistent with the concern that the method does not actually transfer appearance across genuine structural differences. I recommend an explicit out-of-distribution test: evaluate on exemplar-target pairs from different scenes, different object instances, or large viewpoint changes where the exemplar is not retrieved by the proposed retrieval step, and report both qualitative and quantitative results separated by structural-similarity level. Without such a test, the central claim of automatic local appearance transfer in-the-wild is not supported.","section":"Sec. 3.5 and Sec. E"},{"comment":"Table 3 shows that the proposed retrieval step alone raises ICLIP from 0.741 to 0.814, while matching guidance adds only 0.005 (0.814 to 0.819). This indicates that the majority of the appearance gain reported in the main comparison is attributable to retrieving exemplars that are structurally similar to the target, not to the adapter's matching refinement. Since retrieval is a component of the proposed inference pipeline, this is not circular, but it substantially narrows the in-the-wild claim: the headline numbers largely reflect favorable exemplar selection. The paper should report results with randomly chosen exemplars as a separate condition alongside the retrieved-exemplar condition, and should state clearly that the method's automatic transfer is demonstrated only for structurally similar pairs. The current presentation, with a single set of numbers in Table 1, overstates the adapter's ability to handle arbitrary user-provided exemplars.","section":"Table 3 and Sec. 3.6"},{"comment":"The evaluation metrics do not directly measure the paper's central claim of local appearance transfer. The main appearance metrics ICLIP, IDINO, and DINO[cls] loss are global image-level similarities between the generated image and the exemplar; they do not verify that object-specific details (e.g., car color, building texture) are transferred to the correct spatial locations in the target. The object-wise local CLIP similarity in Appendix B.2 and Fig. 10 is a step in the right direction, but it is reported only as a figure on BDD100K and not included in the main table, and it still measures per-category global similarity rather than per-instance correspondence. Without a spatially resolved metric, such as per-segment or per-mask similarity aligned against the target segmentation, the evidence for 'local appearance fidelity' is indirect. I recommend adding such a metric, or at least reporting the object-wise results numerically for all tested datasets.","section":"Sec. 4.1 and Appendix B.2"}],"minor_comments":[{"comment":"The caption contains a typo: 'Additon' should be 'Addition'.","section":"Fig. 24 caption"},{"comment":"References [12] and [13] appear to be the same paper (both are Animate Anyone); this duplication should be fixed.","section":"References"},{"comment":"Sec. 3.5 describes a three-stage training procedure (ControlNeXt, then diffusion model, then AM-Adapter), but Appendix A describes only two stages and does not mention fine-tuning the diffusion model. Please clarify whether the diffusion model was actually fine-tuned and, if so, on what data and for how many steps.","section":"Sec. 3.5 vs. Appendix A"},{"comment":"The caption of Table 1 does not specify which dataset(s) the reported numbers are computed on. The main text says the dataset comprises 300 pairs from BDD100K and NYUv2, while Appendix B.1 mentions evalu ation on BDD100K, Cityscapes, and NYUv2 with 900 segmentation maps. This inconsistency should be resolved so the reader knows exactly which benchmark each table row refers to.","section":"Table 1"},{"comment":"The paper repeatedly claims that AM-Adapter uses 'minimal learnable parameters' but never reports the parameter count of the adapter. Adding a number (e.g., parameters and GPU memory) would make this claim concrete and comparable to future work.","section":"Sec. 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is from a well-known group and the architectural idea is interesting, but the evaluation strategy is the main weakness. The self-built benchmark with retrieval-selected exemplars, combined with the training pairs derived from augmentation of a single image, makes the 'in-the-wild' claim substantially narrower than presented. The reported gains in Table 1 largely come from the retrieval component (Table 3), and the paper itself acknowledges that non-homogeneous exemplars are ignored. I would ask the authors to provide an out-of-distribution evaluation with random or deliberately dissimilar exemplars and to report per-dataset numbers. If those results confirm the retrieval-dependence, the claims should be softened accordingly. I do not see a fatal internal inconsistency, so major revision rather than rejection seems appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the 4D-conv refinement of a categorical plus implicit matching cost is a sensible and novel mechanism, and the stage-wise training is well motivated. The paper deserves a serious referee. But don't let the 'in-the-wild' framing go through unchanged: the adapter is trained on pairs made by random crop/flip of one anchor image, so exemplar and target are near-duplicates. The authors partly concede this in Fig. 8 and Sec. E, but the abstract and intro still imply more generality than the evidence supports.\n\nWhat is actually new: combining a segmentation-derived categorical matching cost with the implicit attention cost, then refining it with a lightweight 4D CNN inside augmented self-attention, and training it stage-wise with frozen diffusion and control nets. That combination is not in Cross-Image Attention, DreamMatcher, or Ctrl-X, as far as I know. The ablation table shows each component earning its place, and the user studies are consistent with the quantitative claims. On its own terms, the method works: on their BDD100K/NYU test set it beats the comparisons on structure, appearance, and FID.\n\nThe soft spots are real but not disqualifying. The biggest is the training-data premise: random crop/flip of the same image teaches the adapter to refine matches when the two views are already roughly aligned. The in-the-wild claim presumes this transfers to genuinely different scenes or large viewpoint changes. Their own Table 3 shows that retrieval, which picks structurally similar exemplars, accounts for most of the ICLIP gain (0.741 to 0.814) compared to the matching guidance (0.814 to 0.819). That suggests the headline gain is largely from the retrieval step supplying near-homologous exemplars, not from the adapter transferring appearance across structural differences. I'd want to see an out-of-distribution test: exemplar and target from different scenes or with large viewpoint changes, with and without retrieval.\n\nOther issues: the benchmark is self-constructed (900 segmentation maps, 300 each from BDD100K, Cityscapes, NYUv2), no error bars are reported, and code/weights are promised but not yet public. None of these are fatal; they are standard fixable issues. The paper is internally consistent and the limitation section is honest about temporal consistency under large scene changes.\n\nWho should read it: anyone working on exemplar-based appearance transfer, diffusion-based semantic synthesis, or attention control. For a referee, I'd ask for the OOD experiment, error bars, and code release. It deserves peer review; desk rejection would be wrong.","headline":"A genuinely useful matching adapter for scene-level appearance transfer, but the 'in-the-wild' claim is stretched by training on cropped/flipped copies of the same image.","tokens_in":19487,"tokens_out":2453,"would_cite":false,"duration_ms":20894,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AM-Adapter transfers multi-object appearance from a single exemplar scene by refining cross-image matching with semantic cost.","keywords":["exemplar-based semantic image synthesis","appearance transfer","augmented self-attention","diffusion models","semantic segmentation","4D convolution","image generation","multi-object transfer"],"falsifier":"Build a test set of exemplar-target pairs with deliberately low structural similarity, such as different scene layouts or large camera shifts, and measure appearance preservation (CLIP and DINO similarity) and structural consistency (Self-Sim.) against pairs with high similarity; if the gains over the no-adapter baseline vanish or reverse when structure differs, the claim that the adapter generalizes in-the-wild fails.","tokens_in":18401,"feed_emoji":"🎨","tokens_out":4524,"duration_ms":38686,"temperature":0.7,"pith_summary":"This paper claims that local appearance can be transferred from a single scene-level exemplar image to a target synthesis with many objects by refining the implicit cross-image matching inside augmented self-attention with semantic cues. Prior matching-only approaches misalign categories in complex scenes, so the paper introduces a categorical matching cost derived from segmentation maps and refines the combined cost with a 4D convolution network to resolve those mismatches. The refined cost is added residually to the original implicit cost, allowing the pre-trained diffusion model and structure-guidance network to stay frozen while only a small adapter learns matching. If correct, this makes exemplar-based semantic image synthesis practical in-the-wild: one scene-level exemplar can specify the appearance of many objects at once, with optional user control for one-to-one object mapping.","feed_headline":"Adapter transfers many objects' looks from a single exemplar image","feed_subtitle":"A 4D-conv adapter adds segmentation-aware cost to self-attention, fixing category mis-matching.","key_machinery":"The load-bearing object is the refined matching cost $O^{Y\\to X}_t = \\varphi(R^{Y\\to X}_t) + A^{Y\\to X}_t$, where $A^{Y\\to X}_t$ is the implicit query-key matching cost from augmented self-attention and $R^{Y\\to X}_t$ concatenates that cost with a binary categorical matching cost $C^{Y\\to X}$ built from the exemplar and target segmentation maps. A lightweight 4D convolution network $\\varphi$ aggregates the combined cost so that correspondences are locally consistent across the two images rather than isolated pixel matches. The residual addition to the original cost stabilizes training and keeps the pre-trained generation and structure networks frozen, so the framework disentangles generation from matching. Stage-wise training first fits the structure-guidance and diffusion models, then trains the adapter alone.","core_discovery":"The central claim is that cross-image matching in augmented self-attention, rather than the generation network, is the bottleneck for scene-level exemplar transfer, and that this bottleneck can be fixed by a learned adapter. The adapter computes a binary categorical matching cost between exemplar and target segmentation maps, concatenates that cost along the head dimension with the implicit matching cost from self-attention, processes the combined 4D cost with a small 4D convolution network, and adds the output back to the original cost. This refined matching cost is what the target self-attention uses to select exemplar values, so appearance is warped along semantically consistent correspondences instead of arbitrary ones. The paper supports this with attention visualizations showing that query points on cars, buildings, and other objects attend to correct exemplar regions after the adapter is applied, and with quantitative gains in semantic alignment, appearance preservation, and image quality relative to previous attention-control baselines.","pith_inferences":["The categorical cost makes the method directly sensitive to segmentation quality; coarse or missing labels would likely propagate into matching errors, so a segmentation-free variant would be needed for fully unconstrained use.","The same refined-cost mechanism could be lifted to other dense correspondence tasks, such as video object transfer or few-shot segmentation, where semantic labels are available.","Training pairs built by random crop and flip of the same anchor cap the structural variation the adapter sees, so fine-tuning on video or multi-view pairs with larger geometric changes is a natural test of the in-the-wild claim.","The retrieval step could be replaced by a learned structural-similarity embedding, which might improve both speed and matching quality on very large exemplar pools."],"forward_implications":["A single scene-level exemplar can set the appearance of multiple objects in the generated image at once, without per-object exemplars or manual assignment.","User-defined one-to-one matching becomes possible by restricting the categorical cost to chosen source and destination regions, enabling precise object-level control.","Because only the adapter is trained, the method adds few learnable parameters while preserving the structural consistency and image quality of the pre-trained models.","An automatic retrieval step can select a suitable exemplar from a pool, removing the labor-intensive manual selection required by prior tuning-free approaches.","The method generalizes across domains such as driving and indoor scenes, supporting image-to-image translation, segmentation-based editing, and consecutive video frame generation."],"supporting_citations":[{"why":"Supplies the structure-guidance network that anchors the generation branch and provides the segmentation-conditioned backbone.","marker":"[32]"},{"why":"Establishes the augmented self-attention baseline whose implicit matching the adapter refines.","marker":"[1]"},{"why":"Baseline tuning-free method for structure-and-appearance control used for comparison.","marker":"[19]"},{"why":"Baseline appearance matching self-attention method that the adapter extends.","marker":"[28]"},{"why":"Baseline key-value replacement method used in ablations.","marker":"[2]"},{"why":"Global appearance adapter baseline demonstrating the insufficiency of global embeddings for local transfer.","marker":"[44]"},{"why":"Structure-guidance baseline that relies on text prompts for appearance, showing the need for exemplar input.","marker":"[46]"},{"why":"Classifier-free guidance applied to the refined matching cost at inference.","marker":"[9]"},{"why":"DDIM inversion used to encode the exemplar into the appearance branch.","marker":"[26]"}],"fun_headline_variants":["Segmentation-aware 4D conv fixes cross-image matching for multi-object transfer","One exemplar image, many objects: a 4D conv adapter matches local appearance","AM-Adapter warps appearance along semantic boundaries, not arbitrary attention","Segmentation-aware matching cost steers exemplar values to correct objects in a scene","Scene-level exemplar transfer done right: adaptive 4D cost for self-attention"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The adapter is trained on exemplar-target pairs made by randomly cropping and flipping a single anchor image, so the in-the-wild claim assumes that matching learned under near-identical geometric transformations transfers to genuinely different scenes and large viewpoint changes.","fun_headline_variants_meta":{"raw":{"variants":["Segmentation-aware 4D conv fixes cross-image matching for multi-object transfer","One exemplar image, many objects: a 4D conv adapter matches local appearance","AM-Adapter warps appearance along semantic boundaries, not arbitrary attention","Segmentation-aware matching cost steers exemplar values to correct objects in a scene","Scene-level exemplar transfer done right: adaptive 4D cost for self-attention"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000922,"raw_usage":{"total_tokens":3993,"prompt_tokens":1021,"completion_tokens":2972,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":637,"completion_tokens_details":{"reasoning_tokens":2868}},"tokens_in":637,"tokens_out":2972,"duration_ms":19813,"temperature":1.0,"reasoning_tokens":2868,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:42:35.987657+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a test set of exemplar-target pairs with deliberately low structural similarity, such as different scene layouts or large camera shifts, and measure appearance preservation (CLIP and DINO similarity) and structural consistency (Self-Sim.) against pairs with high similarity; if the gains over the no-adapter baseline vanish or reverse when structure differs, the claim that the adapter generalizes in-the-wild fails.","supporting_citations":[{"cited_title":"Cross-image attention for zero- shot appearance transfer","cited_arxiv_id":null,"evidence_quote":"Establishes the augmented self-attention baseline whose implicit matching the adapter refines."},{"cited_title":"Dreammatcher: Appearance matching self-attention for semantically-consistent text-to- image personalization","cited_arxiv_id":null,"evidence_quote":"Baseline appearance matching self-attention method that the adapter extends."},{"cited_title":"Masactrl: Tuning-free mu- tual self-attention control for consistent image synthesis and editing","cited_arxiv_id":null,"evidence_quote":"Baseline key-value replacement method used in ablations."},{"cited_title":"Adding conditional control to text-to-image diffusion models","cited_arxiv_id":null,"evidence_quote":"Structure-guidance baseline that relies on text prompts for appearance, showing the need for exemplar input."},{"cited_title":"Classifier-free diffusion guidance, 2022","cited_arxiv_id":null,"evidence_quote":"Classifier-free guidance applied to the refined matching cost at inference."},{"cited_title":"Null-text inversion for editing real im- ages using guided diffusion models","cited_arxiv_id":null,"evidence_quote":"DDIM inversion used to encode the exemplar into the appearance branch."}],"review_version":1}