REVIEW 3 major objections 6 minor 36 references
InstancePin: Instance-Addressable Layout-to-Image Diffusion via Coordinate Pinning
T0 review · 3 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read InstancePin pins each object to a coordinate anchor so layout-to-image diffusion can tell apart crowded same-category objects.
desk verdict Sensible architecture and a real problem, but the main comparison is confounded by an input asymmetry: baselines lack the instance maps that InstancePin consumes, so the gains in Table 1 don't isolate coordinate pinning. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Coordinate Pinning Attention is the central mechanism: each instance center is converted into a coordinate token via Fourier features and a small mapping network, and latent image features cross-attend to these tokens so that every spatial location can be associated with a specific instance. The coordinate alignment loss forces each token's attention map to activate its own instance region, and the Instance-Mask Guided Fusion module combines the resulting adapter features with frozen backbone features using a binary instance mask. Together these components separate same-category instances while preserving the pretrained model's global semantic prior.
What would settle it
Run InstancePin and the FreestyleNet baseline over multiple training seeds and compute FID/mIoU confidence intervals, or evaluate with an instance-level metric such as per-instance segmentation of generated images matched to ground-truth instances. If the gains vanish within noise, or if per-instance matching accuracy is no better than the category-level baseline, the claim that coordinate pinning achieves instance disentanglement is not established.
Extended reading notes
Core claim
The central claim is that coordinate-pinned instance anchors, combined with mask-guided feature fusion, let a layout-to-image diffusion model treat each object instance as a separately addressable unit without degrading category-level semantic control. Rather than injecting instance masks into the pretrained denoising backbone, InstancePin adds an independent instance-aware adapter: instance centers are encoded with Fourier features and projected into coordinate tokens, latent image features query these tokens through coordinate pinning attention, and the resulting attention maps are explicitly supervised to align with instance regions. An instance-mask guided fusion module then routes backb
Load-bearing premise
The central claim rests on the assumption that FID and mIoU differences of roughly one to three points, measured on a single evaluation without error bars or an instance-level metric, reflect genuine per-instance separation rather than run-to-run noise or improved global fidelity.
Editorial extensions
If this is right
- Adjacent same-category objects in dense urban layouts can be generated with distinct boundaries and appearances instead of merged, averaged regions.
- Instance-level control can be added to a frozen pretrained diffusion backbone through an independent adapter, avoiding the need to retrain the category-level model.
- Coordinate attention maps become spatially interpretable, indicating which latent region corresponds to which instance anchor during denoising.
- The pedestrian-focused subset shows that gains concentrate exactly where instance entanglement is most severe, supporting the method's stated purpose.
Reading between the lines
- The coordinate-token design suggests a direct route to instance-level editing: by manipulating a single token's value or attention map, an editor could localize changes to one object without touching its neighbors or the background.
- Because the method only needs center coordinates and a binary instance mask, it should transfer to other instance-annotated layout domains such as indoor scenes or autonomous-vehicle datasets, though the paper only tests Cityscapes.
- The absence of an instance-level evaluation metric leaves open whether the FID/mIoU gains reflect true per-instance separation or simply better global fidelity; a per-instance matching metric would settle that question.
- For heavily occluded or overlapping instances, a single center coordinate may not uniquely identify an object, so the anchor representation may need to be extended to multiple keypoints per instance in such settings.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes InstancePin, a layout-to-image diffusion method that adds instance-level control through coordinate pinning. Each instance is represented by a coordinate token derived from its center via Fourier features and a mapping network. An instance-aware adapter runs in parallel with a frozen pretrained backbone, and coordinate pinning attention lets latent positions query these tokens, with the attention maps supervised by instance regions (Eq. 4). An instance-mask guided fusion module blends backbone features in non-instance regions with adapter features in instance regions (Eq. 5). Experiments on Cityscapes and a pedestrian-focused subset report improved FID and mIoU over OASIS, SDM, ControlNet, Uni-ControlNet, and FreestyleNet, along with qualitative evidence of clearer instance separation.
Significance. If the central claim is established, InstancePin addresses a real limitation of category-level layout-to-image diffusion: the lack of instance addressability in dense same-category regions. The design is practical: it avoids fine-tuning the pretrained backbone, uses an adapter for instance-specific control, and employs a mask-guided fusion to preserve global structure. The paper is clearly written and the proposed components are motivated. However, the experimental evidence currently does not separate the effect of the proposed coordinate-pinning mechanism from the effect of simply adding instance-map conditioning, and the reported metrics are category-level rather than instance-level. The method is a plausible contribution, but the quantitative support for the central 'instance disentanglement' claim needs substantial strengthening.
major comments (3)
- [§4.2, Table 1] The comparison against baselines is confounded by an input asymmetry. InstancePin conditions on the instance map M and its derivatives P and B, while none of the baselines (OASIS, SDM, ControlNet, Uni-ControlNet, FreestyleNet) receive instance-level annotations. The reported gains (FID 40.3→37.6, mIoU 73.3→75.1 on Cityscapes; FID 46.6→43.5, mIoU 74.1→75.8 on Cityscapes-pedestrian) could therefore stem from the extra instance information rather than from coordinate pinning itself. The ablations in Table 2 all hold M fixed, so they cannot separate the value of the additional input from the value of the proposed architecture. Please add an instance-conditioned baseline, such as FreestyleNet or ControlNet fed the same instance mask/centers (e.g., via simple concatenation or an adapter without coordinate pinning), and/or an ablation that removes CPA while retaining P,B conditioning. Without t
- [§4.1–§4.2, Tables 1–2] No error bars, repeated seeds, or significance tests are reported. The FID and mIoU differences over FreestyleNet are modest (0.9–2.7 FID points, 1.0–1.8 mIoU points), and the ablation differences are similarly small. On a single evaluation it is impossible to tell whether these gains are stable or within run-to-run noise. Please report means and standard deviations over at least three seeds (or equivalent bootstrapped confidence intervals) for all numbers in Tables 1 and 2, and ideally state whether the differences are statistically significant. This is essential given that the paper's quantitative case rests on small margins.
- [§4.2, §4.4] The paper's stated goal is to reduce instance entanglement, but no instance-level metric is reported. mIoU measures category-level semantic consistency, and FID measures global fidelity; neither directly quantifies whether adjacent same-category instances are generated as distinct objects. The qualitative zoomed regions in Fig. 3 are suggestive but anecdotal. Please add an instance-level evaluation, for example: predicted instance segmentation mAP (using a pretrained Mask R-CNN or similar), Panoptic Quality, or a metric that measures whether the number of generated instances matches the ground-truth instance count in the layout. Such a metric is necessary to support the central 'instance-addressable' claim quantitatively.
minor comments (6)
- [§4.3, Fig. 4] The attention maps in Fig. 4 are directly supervised by instance-region targets via Eq. (4), so their alignment with instance regions is a trained behavior, not an emergent property. The text in §4.3 should be tempered: it is a verification that the supervisory signal is learned, rather than evidence that coordinate pinning attention is inherently interpretable. This does not invalidate the method, but the framing should be adjusted.
- [§4.1] The 'Cityscapes-pedestrian' subset is not precisely defined. Is it the subset of validation images containing pedestrians, or a cropping of pedestrian regions? Its size and selection criteria should be stated, as the reported numbers depend on this definition.
- [§3.3, Eq. (3)] The projection matrices W_Q, W_K, W_V are written without the scale index l, which is confusing since the feature F^l is scale-dependent. Clarify that these are per-scale projections, or explicitly denote them as W^l_Q etc.
- [§4.1] Several hyperparameters are not ablated or justified: the coordinate loss weight λ_coord=10, the number of Fourier frequency bands F, and the coordinate token dimension d_c. At least λ_coord should be examined, given that it controls the strength of the instance-region supervision that drives the central mechanism.
- [§4.1] No code or trained models are released. Given the reliance on a specific pretrained backbone (FreestyleNet) and detailed adapter architecture, releasing code would substantially aid reproducibility.
- [References] Several references have formatting errors: Ref. [4] is missing author names after 'M., Heusel'; Ref. [17] contains a typo 'ZhoubTinghui, A'; Ref. [7] lists 'Ian, G.' rather than the full author name; Ref. [20] has inconsistent publisher capitalization ('Pmlr'). These should be corrected.
Circularity Check
No central circularity; the only by-construction evidence is the supervised coordinate-attention visualization, which is not load-bearing for the main FID/mIoU comparison.
-
self definitional
[Section 3.3 (Eq. 4) and Section 4.3]
"To make these anchors spatially meaningful, we further supervise the coordinate attention maps with instance regions... Lcoord = E[1/|Ω| Σ_{l∈Ω} ||A^l_c(z_t,t,P) − R^l||^2_2] ... As shown in Fig. 4, the attention response of each coordinate token is concentrated around the corresponding instance region... This observation suggests that coordinate pinning attention provides a spatially interpretable mechanism for associating latent image features with individual instances."
Equation 4 defines a training loss that directly minimizes the squared distance between the coordinate attention map A_c and the instance-region target R derived from the ground-truth instance map. Therefore the Fig. 4 observation that attention is concentrated on the corresponding instance region is enforced by construction: it is the objective being optimized, not an emergent or independent finding. Presenting it as evidence that the anchors are 'spatially interpretable' is circular, because the interpretability is imposed by supervision, not discovered. This step is not load-bearing for the main quantitative claim, which rests on external FID/mIoU comparisons.
full rationale
The paper's central quantitative claims are the FID/mIoU improvements over OASIS, SDM, ControlNet, Uni-ControlNet, and FreestyleNet in Table 1. Those comparisons are against external, independently trained baselines and do not reduce to the paper's own equations or fitted constants by construction. The ablations in Table 2 also hold the instance-map input fixed across variants, so they provide a meaningful internal comparison of the proposed components. The one circular element is the attention-map visualization in Section 4.3: because Eq. 4 explicitly supervises coordinate attention maps to match instance regions, showing that they match instance regions is a self-fulfilling observation rather than independent evidence. This is a minor, non-central circularity and does not invalidate the main comparison. The additional-input confound identified by the skeptic (baselines do not receive the instance map M, centers P, or mask B) is a real experimental-design/validity concern, but it is not circularity in the derivation-chain sense: the reported gains are not equal to the inputs by definition. No load-bearing self-citation chain or imported uniqueness theorem is present; references from the same group are used only as general prior-work support.
Assumptions & free parameters
free parameters (3)
- lambda_coord =
10
- Fourier frequency bands and count F =
not specified
- Coordinate token dimension d_c =
not specified
assumptions (4)
- domain assumption Frozen pretrained FreestyleNet preserves category-level generation prior while only the adapter is trained.
- domain assumption Ground-truth instance maps M are available at training and inference and are perfectly aligned with semantic layout S.
- domain assumption FID and DRN-D-105 mIoU at 512x512 are valid measures of the claimed instance-level improvement.
- standard math Standard Fourier features and softmax cross-attention behave as expected for coordinate conditioning.
invented entities (1)
-
Coordinate token u_k
Cite this review
Pith. "Pith review of InstancePin: Instance-Addressable Layout-to-Image Diffusion via Coordinate Pinning." pith.science (2026). https://pith.science/paper/KRNMTXLY
@misc{pith2026260800588,
author = {Pith},
title = {Pith review of: InstancePin: Instance-Addressable Layout-to-Image Diffusion via Coordinate Pinning},
year = {2026},
howpublished = {\url{https://pith.science/paper/KRNMTXLY}},
note = {Machine review of arXiv:2608.00588}
}
read the original abstract
Layout-to-image diffusion models have achieved impressive semantic controllability by conditioning generation on category-level segmentation maps. However, such category-aligned control is not necessarily instance-addressable: multiple nearby objects from the same category are often treated as a shared semantic region, leading to ambiguous boundaries, averaged appearances, and feature confusion among instances. This limitation is particularly evident in urban scene synthesis, where small and crowded pedestrians or vehicles require fine-grained instance separation while preserving global scene consistency. In this paper, we propose InstancePin, an instance-addressable layout-to-image diffusion framework that pins each object instance with an explicit coordinate anchor. Instead of directly injecting instance masks into the pretrained backbone, InstancePin introduces an independent instance-aware adapter to preserve the category-level generation prior while learning instance-specific spatial control. For each instance, its center coordinate is encoded with Fourier features and projected into a coordinate token, which serves as a spatial anchor queried by latent image features through coordinate pinning attention. To make these anchors spatially meaningful, we further supervise the coordinate attention maps with instance regions, encouraging each coordinate token to activate its corresponding object area. Finally, an instance-mask guided fusion module routes pretrained backbone features to non-instance regions and adapter features to instance regions, enabling local instance refinement without sacrificing global semantic fidelity. Extensive experiments on Cityscapes demonstrate that InstancePin mitigates instance entanglement in dense layouts and improves both image fidelity and semantic consistency.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Avrahami, O., Hayes, T., Gafni, O., Gupta, S., Taigman, Y., Parikh, D., Lischinski, D., Fried, O., Yin, X.: Spatext: Spatio-textual representation for controllable image generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18370–18380 (2023)
work page 2023
-
[2]
In: Proceedings of the AAAI Conference on Artificial Intelligence
Cai, Y., Wei, Y., Ji, Z., Bai, J., Han, H., Zuo, W.: Decoupled textual embeddings for customized image generation. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 38, pp. 909–917 (2024)
work page 2024
-
[3]
In: Pro- ceedings of the IEEE conference on computer vision and pattern recognition
Choi, Y., Choi, M., Kim, M., Ha, J.W., Kim, S., Choo, J.: Stargan: Unified gener- ative adversarial networks for multi-domain image-to-image translation. In: Pro- ceedings of the IEEE conference on computer vision and pattern recognition. pp. 8789–8797 (2018)
2018
- [4]
-
[5]
Advances in neural information processing systems33, 6840–6851 (2020)
Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in neural information processing systems33, 6840–6851 (2020)
2020
-
[6]
In: Proceedings of the European conference on computer vision (ECCV)
Huang, X., Liu, M.Y., Belongie, S., Kautz, J.: Multimodal unsupervised image-to- image translation. In: Proceedings of the European conference on computer vision (ECCV). pp. 172–189 (2018) InstancePin 13
work page 2018
-
[7]
Advances in neural information processing systems27(2014)
Ian, G., Jean, P.A., Mehdi, M., Bing, X., David, W.F., Sherjil, O., Aaron, C., Yoshua, B.: Generative adversarial nets. Advances in neural information processing systems27(2014)
work page 2014
-
[8]
In: Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition
Ifriqi, T.B., Verbeek, J., Couprie, C., Alahari, K.: Unlocking pre-trained image backbones for semantic image synthesis. In: Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition. pp. 7840–7849 (2024)
work page 2024
Show all 36 references
-
[9]
In: International conference on ma- chine learning
Kim, T., Cha, M., Kim, H., Lee, J.K., Kim, J.: Learning to discover cross-domain relations with generative adversarial networks. In: International conference on ma- chine learning. pp. 1857–1865. Pmlr (2017)
2017
-
[10]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Li, X., Hou, X., Loy, C.C.: When stylegan meets stable diffusion: a w+ adapter for personalized image generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2187–2196 (2024)
2024
-
[11]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Li, Y., Liu, H., Wu, Q., Mu, F., Yang, J., Gao, J., Li, C., Lee, Y.J.: Gligen: Open-set grounded text-to-image generation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 22511–22521 (2023)
2023
-
[12]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Lv, Z., Wei, Y., Zuo, W., Wong, K.Y.K.: Place: Adaptive layout-semantic fusion for semantic image synthesis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9264–9274 (2024)
2024
-
[13]
CoRR abs/1411.1784(2014)
Mirza, M., Osindero, S.: Conditional generative adversarial nets. CoRR abs/1411.1784(2014)
2014 arXiv
-
[14]
In: Proceedings of the AAAI conference on artificial intelligence
Mou, C., Wang, X., Xie, L., Wu, Y., Zhang, J., Qi, Z., Shan, Y.: T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In: Proceedings of the AAAI conference on artificial intelligence. vol. 38, pp. 4296–4304 (2024)
2024
-
[15]
arXiv preprint arXiv:2112.10741 (2021)
Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., Chen, M.: Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 (2021)
2021 arXiv
-
[16]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Park, T., Liu, M.Y., Wang, T.C., Zhu, J.Y.: Semantic image synthesis with spatially-adaptive normalization. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 2337–2346 (2019)
2019
-
[17]
In: Proceedings of the IEEE conference on com- puter vision and pattern recognition
Phillip, I., Jun-Yan, Z., ZhoubTinghui, A, E.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE conference on com- puter vision and pattern recognition. pp. 1125–1134 (2017)
2017
-
[18]
In: International conference on machine learning
Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021)
2021
-
[19]
arXiv preprint arXiv:2204.06125 1(2), 3 (2022)
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 1(2), 3 (2022)
2022 arXiv
-
[20]
In: International conference on machine learning
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., Sutskever, I.: Zero-shot text-to-image generation. In: International conference on machine learning. pp. 8821–8831. Pmlr (2021)
2021
-
[21]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022)
2022
-
[22]
arXiv preprint arXiv:2012.04781 (2020) 14 C
Sushko, V., Sch¨ onfeld, E., Zhang, D., Gall, J., Schiele, B., Khoreva, A.: You only need adversarial supervision for semantic image synthesis. arXiv preprint arXiv:2012.04781 (2020) 14 C. Wu et al
2012 arXiv
-
[23]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Tan, Z., Chai, M., Chen, D., Liao, J., Chu, Q., Liu, B., Hua, G., Yu, N.: Diverse semantic image synthesis via probability distribution modeling. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 7962– 7971 (2021)
2021
-
[24]
IEEE Trans- actions on Pattern Analysis and Machine Intelligence44(9), 4852–4866 (2021)
Tan, Z., Chen, D., Chu, Q., Chai, M., Liao, J., He, M., Yuan, L., Hua, G., Yu, N.: Efficient semantic image synthesis via class-adaptive normalization. IEEE Trans- actions on Pattern Analysis and Machine Intelligence44(9), 4852–4866 (2021)
2021
-
[25]
arXiv preprint arXiv:2205.12952 (2022)
Wang, T., Zhang, T., Zhang, B., Ouyang, H., Chen, D., Chen, Q., Wen, F.: Pretraining is all you need for image-to-image translation. arXiv preprint arXiv:2205.12952 (2022)
2022 arXiv
-
[26]
In: Proceedings of the IEEE conference on computer vision and pattern recognition
Wang, T.C., Liu, M.Y., Zhu, J.Y., Tao, A., Kautz, J., Catanzaro, B.: High- resolution image synthesis and semantic manipulation with conditional gans. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 8798–8807 (2018)
2018
-
[27]
arXiv preprint arXiv:2207.00050 (2022)
Wang, W., Bao, J., Zhou, W., Chen, D., Chen, D., Yuan, L., Li, H.: Semantic image synthesis via diffusion models. arXiv preprint arXiv:2207.00050 (2022)
2022
-
[28]
In: Proceedings of the IEEE/CVF International Conference on Computer Vision
Wang, Y., Qi, L., Chen, Y.C., Zhang, X., Jia, J.: Image synthesis via semantic com- position. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 13749–13758 (2021)
2021
-
[29]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Xue, H., Huang, Z., Sun, Q., Song, L., Zhang, W.: Freestyle layout-to-image synthe- sis. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 14256–14266 (2023)
2023
-
[30]
In: Proceedings of the IEEE conference on computer vision and pattern recognition
Yu, F., Koltun, V., Funkhouser, T.: Dilated residual networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 472–480 (2017)
2017
-
[31]
In: Proceedings of the IEEE/CVF international conference on computer vision
Zhang, L., Rao, A., Agrawala, M.: Adding conditional control to text-to-image diffusion models. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 3836–3847 (2023)
2023
-
[32]
Knowledge-Based Systems p
Zhang, Y., Huo, X., Chen, T., Wu, S., Wong, H.S.: Class-conditional image syn- thesis with intra-class relation preservation. Knowledge-Based Systems p. 114487 (2025)
2025
-
[33]
IEEE Transactions on Multimedia (2026)
Zhang, Y., Wang, J., Huang, Y., Chen, T., Wong, H.S., Wu, S.: Classbooth: Boost class semantics with bidirectional feature fusion in text-to-image diffusion models. IEEE Transactions on Multimedia (2026)
2026
-
[34]
Advances in Neural Information Processing Systems36, 11127–11150 (2023)
Zhao, S., Chen, D., Chen, Y.C., Bao, J., Hao, S., Yuan, L., Wong, K.Y.K.: Uni- controlnet: All-in-one control to text-to-image diffusion models. Advances in Neural Information Processing Systems36, 11127–11150 (2023)
2023
-
[35]
In: Proceedings of the IEEE interna- tional conference on computer vision
Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE interna- tional conference on computer vision. pp. 2223–2232 (2017)
2017
-
[36]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition
Zhu, Z., Xu, Z., You, A., Bai, X.: Semantically multi-modal image synthesis. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition. pp. 5467–5476 (2020)
2020
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.