Pith. sign in

REVIEW 4 major objections 6 minor 68 references

SPADE: Spatial-Aware Denoising Network for Open-vocabulary Panoptic Scene Graph Generation with Long- and Local-range Context Reasoning

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SPADE injects spatial awareness into diffusion-based scene graph generation and outperforms prior open-set models.

desk verdict A useful new combination for open-vocabulary PSG with a clear spatial-prior idea, but the central calibration loss is underspecified and needs a major revision. read the letter →

arxiv 2507.05798 v1 pith:VRI4S3IC submitted 2025-07-08 cs.CV

Xin Hu , Ke Qin , Guiduo Duan , Ming Li , Yuan-Fang Li , Tao He This is my paper · ORCID
classification cs.CV
keywords panopticscenegraphgenerationopen-vocabularyspatialrelationreasoningdiffusionmodelinversionDDIMcross-attentiondistillationLoRAfine-tuningtransformer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that open-vocabulary panoptic scene graph generation models built on vision-language models fail at spatial relation reasoning, especially when the subject and object are far apart in the image. SPADE is a two-stage remedy: first calibrate a pre-trained diffusion model into a PSG-specific denoising network by aligning its inversion-derived cross-attention maps with those of a LoRA-tuned student UNet; then run a relation graph transformer that mixes long-range and local context to build relation queries. On the PSG and Visual Genome benchmarks, the paper reports state-of-the-art closed-set and open-set results, with the largest gains on spatial predicates and distant relation pairs.

What carries the argument

The load-bearing object is the calibration loss of Eq. (6), an L1 alignment between two cross-attention maps: $A_i$ produced by the student UNet conditioned on CLIP image features through an MLP adapter, and $A'_i$ produced by a frozen teacher diffusion model conditioned on the prompt '[subject] is [predicate] [object]' during deterministic DDIM inversion, the process that maps an image to noise while preserving its spatial layout. This loss is the only conduit through which spatial priors enter the UNet; it is paired with LoRA updates restricted to cross-attention layers to avoid destroying the pretrained model. The second stage, a relation graph transformer with long-range neighbor/non-neighbor self-attention and a GCN local block, builds pairwise relation queries.

What would settle it

Run the identical SPADE pipeline but condition the teacher inversion on a non-spatial prompt such as 'a photo' instead of '[subject] is [predicate] [object]'. If open-set recall on PSG stays high, the spatial prompt is not carrying the result; if recall drops to baseline, the inversion geometry alone is insufficient and the calibration signal is doing the work.

Watch

Extended reading notes

Core claim

SPADE's central claim is that the spatial structure a DDIM inversion preserves can be transferred into a PSG model without sacrificing the open-world knowledge of the frozen diffusion and CLIP backbones. Using cross-attention maps from the teacher's inversion as spatial priors, the method calibrates a UNet via rank-limited updates, then fuses diffusion features with pooled CLIP features for open-vocabulary classification. A relation graph transformer captures both connected and non-connected object context. On PSG closed-set, R/mR@50 reaches 45.1/41.2 versus 42.8/38.9 for OpenPSG, and open-set OvR R/mR@50 reaches 26.7/23.3 versus 21.2/19.8; on VG, open-set OvR R/mR@50 reaches 24.1/11.2 versus 20.4/9.4.

Load-bearing premise

The calibration loss assumes that the cross-attention maps produced by the student UNet and the teacher diffusion model are comparable L1-alignable quantities with matching spatial and semantic structure; the paper does not establish that the two maps are compatible.

Editorial extensions

If this is right

  • Spatial predicate prediction in open-vocabulary PSG can be improved without retraining the VLM or diffusion backbone, by supervised alignment to inversion cross-attention maps.
  • Distant relation pairs, which current VLM models handle poorly, become a targetable weakness: the calibration plus graph transformer nearly closes the gap between distant and non-distant mR@50 (38.7 vs 42.8 on PSG).
  • The same two-stage recipe could transfer to other structured prediction tasks that need spatial awareness, such as human-object interaction detection, since no task-specific spatial head is assumed beyond the graph transformer.
  • The reported gains are consistent across two datasets (PSG and VG) and across closed-set, OvR, and OvD+R settings, suggesting the mechanism generalizes beyond a single benchmark.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The calibration loss treats cross-attention maps as a spatial prior, but the paper does not establish what those maps mean semantically; a natural test is whether the alignment transfers spatial layout rather than just matching global statistics.
  • The method's dependence on the prompt template '[subject] is [predicate] [object]' may inherit the teacher's own spatial reasoning limits, so relations not expressible in that form are a likely failure mode.
  • The graph transformer selects object pairs by cosine distance above a fixed threshold; making that threshold adaptive per image is a testable extension the paper leaves open.
  • If the gains come mostly from the calibration alignment, a lighter alternative that distills inversion maps directly into a non-diffusion encoder might capture similar spatial priors at lower inference cost.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. SPADE is a two-stage framework for open-vocabulary panoptic scene graph generation. Stage one calibrates a pre-trained text-to-image diffusion UNet into a PSG-specific denoising network: DDIM inversion of real images with relation prompts yields teacher cross-attention maps, and a calibration loss aligns student cross-attention maps to these under LoRA adaptation of the UNet. Stage two constructs a spatial-semantic graph from instance masks and applies a relation graph transformer with long-range neighbor/non-neighbor attention and graph-convolutional local correlation, followed by relation-query construction and prompt-based open-vocabulary classification that fuses diffusion and pooled CLIP features. On PSG and VG, the paper reports state-of-the-art closed-set and open-set results, with the largest margins on spatial predicate prediction and distant relation pairs.

Significance. The paper addresses a genuine weakness of VLM-based PSG, namely spatial reasoning at distance, with a novel and plausible mechanism. The empirical study is broad: two datasets, closed-set/OvR/OvD+R settings, component ablations, and a deliberate seen-only calibration split for open-set experiments. If the inversion-guided calibration is made well-defined, the method is a useful contribution. As written, however, the central loss in Eq. (6) is underspecified to the point of being undefined, so the significance currently rests on an unclear mechanism.

major comments (4)
  1. [§3.1, Eq. (6)] The calibration loss is central to the method, but the two cross-attention maps it aligns have different conditioning modalities and token dimensions. The student map A_i is computed with MLP∘CLIP_image(x_i) as the UNet conditioning signal (Eq. (4)), while the teacher map A'_i is computed from the text prompt p='[subject] is [predicate] [object]' (§3.1). CLIP image features and text-token embeddings do not share a sequence length, and no pooling, padding, or projection is specified. As written, ||A_i - A'_i||_1 is undefined. Please specify the exact alignment used and explain why the aligned maps are semantically comparable enough to serve as spatial priors.
  2. [§3.2, Eqs. (8)-(9)] The attention equations for long-range correlation learning are not well-formed. The expression contains 'p |P(r)+|' in the numerator without a division or a defined symbol p, and the scaling convention is missing. The notation φ[P(r)+] for averaging neighbor features is also not formally defined. Since RGT is the core context-reasoning module, these equations need to be rewritten with clear tensor shapes and scaling.
  3. [§3.3, Eqs. (16)-(17)] The prompt-based classification formulas are dimensionally unclear: dividing the feature-transpose product by ϵ_t(T_o), a text embedding, is not a valid operation as written. Presumably the intended form is a scaled cosine similarity or inner product between H and text embeddings with a temperature, but the notation prevents the reader from knowing. Please rewrite these equations and define the shapes of all tensors.
  4. [§4.3, Table 5] The 'w/o inversion' ablation, which replaces deterministic inversion with random Gaussian sampling, is central to attributing the gain to inversion guidance, but the description does not state whether the teacher cross-attention maps are then computed from random-noise sampling or whether the student UNet is trained with the same LoRA recipe. Please clarify the exact protocol; otherwise the 21.0→26.7 gain cannot be assigned to the inversion prior.
minor comments (6)
  1. [§4.1] The default diffusion timestep is t=0, but the inversion and calibration section implies features from the 'final timestep t'; please state the inversion step count and which timesteps produce the teacher maps.
  2. [Eq. (14)] The denominator |ˆQr||ˆQr^T| is not defined as a matrix norm; specify the norm and the exact construction of Ψ'.
  3. [Eq. (13)] It is unclear whether Dis is a distance (smaller=closer) or similarity (larger=closer) given Select> η; please clarify.
  4. [§4.1] The DR/NDR split is based on one-third image width; state whether this uses normalized coordinates and how border cases are treated.
  5. [General] No variance or significance tests are reported for the main tables; since the margins over OpenPSG are 2–5 points, multiple seeds or standard deviations would increase confidence.
  6. [General] Typographical issues: 'reversion process' in the contribution list, 'Lmaks' for L_mask, the project page link 'at here', and duplicate GLIP references [28]/[29] should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: SPADE's headline results are external benchmark numbers and its calibration target is an independent teacher model.

full rationale

The derivation chain is not circular. The central claims are empirical: Tables 1-4 report Recall/mRecall on held-out PSG and VG test splits, against external baselines, and the open-set protocol restricts calibration to seen-category images to avoid leakage. The inversion-guided calibration loss (Eq. 6) fits the student UNet's cross-attention maps to a frozen teacher diffusion model's inversion maps; the teacher (BELM/DDIM) is prior external work, and the fitted maps are not themselves the reported evaluation metric. The relation graph transformer and open-vocabulary losses are standard training objectives followed by held-out evaluation, not quantities that equal the output by construction. Coauthor Tao He's earlier papers are cited only in related work and are not load-bearing for the method or the comparison. The underspecified alignment in Eq. (6) between text-token-conditioned teacher maps and CLIP-image-conditioned student maps is a reproducibility/correctness concern, not a circular reduction; it does not make the benchmark gains equivalent to the training objective.

Assumptions & free parameters 8 free parameters · 4 assumptions · 0 invented entities

The central method rests on several domain assumptions: that DDIM inversion preserves spatial structure, that cross-attention maps from a text-conditioned teacher can be distilled into a CLIP-image-conditioned student, and that CLIP image features are an adequate text substitute. The empirical protocol also introduces a set of hyperparameters whose values are fixed without sensitivity analysis. No new physical or ontological entities are proposed.

free parameters (8)
  • calibration loss weight lambda = not reported
    Factor lambda in Eq. (6); no value or sensitivity study is given.
  • relation query threshold eta = 0.65
    Threshold in Eq. (13) for selecting object pairs; fixed without sensitivity analysis.
  • fusion exponent alpha = 0.34
    Geometric-mean fusion weight for diffusion and pooled CLIP scores; fixed without sensitivity analysis.
  • relation query construction loss weight lambda_rqc = 0.6
    Loss weight in Eq. (20); fixed without ablation across values.
  • mask loss weight lambda_mask = 1
    Loss weight in Eq. (20); fixed.
  • learnable temperature tau = learned
    Temperature in prompt-based classifier Eqs. (16)-(17); final value not reported.
  • semantic graph similarity threshold = not reported
    Threshold for binarizing semantic distances in graph construction in Section 3.2.
  • LoRA rank r = not reported
    Decomposition rank in Eq. (5); value not reported.
assumptions (4)
  • domain assumption DDIM inversion preserves the spatial structure of the input image, so cross-attention maps extracted during inversion encode usable spatial priors.
    Invoked in Section 3.1 based on references [45,50,64]; the paper does not validate this directly on PSG and VG distributions beyond the reported ablation.
  • ad hoc to paper The student UNet's cross-attention map A_i, computed with CLIP image features as the conditioning signal, is L1-comparable to the teacher's inversion cross-attention map A'_i, computed with the text prompt '[subject] is [predicate] [object]'.
    The entire calibration in Eq. (6) depends on this correspondence, which is never demonstrated; the two maps may have different token lengths and conditioning semantics.
  • domain assumption CLIP image features are a sufficient replacement for text embeddings when conditioning the denoising UNet in the panoptic scene graph task.
    Adopted in Section 3.1 following prior diffusion-perception works [56,58]; no dedicated experiment isolates the quality of this substitution.
  • domain assumption The 7:3 base/novel split with calibration restricted to images whose object and predicate categories are in the seen group prevents open-vocabulary data leakage.
    Stated in Section 4.1; it assumes the frozen CLIP and diffusion models do not leak unseen-category information through pretrained weights during calibration.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SPADE: Spatial-Aware Denoising Network for Open-vocabulary Panoptic Scene Graph Generation with Long- and Local-range Context Reasoning." pith.science (2026). https://pith.science/paper/VRI4S3IC

@misc{pith2026250705798,
  author       = {Pith},
  title        = {Pith review of: SPADE: Spatial-Aware Denoising Network for Open-vocabulary Panoptic Scene Graph Generation with Long- and Local-range Context Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VRI4S3IC}},
  note         = {Machine review of arXiv:2507.05798}
}
read the original abstract

Panoptic Scene Graph Generation (PSG) integrates instance segmentation with relation understanding to capture pixel-level structural relationships in complex scenes. Although recent approaches leveraging pre-trained vision-language models (VLMs) have significantly improved performance in the open-vocabulary setting, they commonly ignore the inherent limitations of VLMs in spatial relation reasoning, such as difficulty in distinguishing object relative positions, which results in suboptimal relation prediction. Motivated by the denoising diffusion model's inversion process in preserving the spatial structure of input images, we propose SPADE (SPatial-Aware Denoising-nEtwork) framework -- a novel approach for open-vocabulary PSG. SPADE consists of two key steps: (1) inversion-guided calibration for the UNet adaptation, and (2) spatial-aware context reasoning. In the first step, we calibrate a general pre-trained teacher diffusion model into a PSG-specific denoising network with cross-attention maps derived during inversion through a lightweight LoRA-based fine-tuning strategy. In the second step, we develop a spatial-aware relation graph transformer that captures both local and long-range contextual information, facilitating the generation of high-quality relation queries. Extensive experiments on benchmark PSG and Visual Genome datasets demonstrate that SPADE outperforms state-of-the-art methods in both closed- and open-set scenarios, particularly for spatial relationship prediction.

Figures

Figures reproduced from arXiv: 2507.05798 by the authors.

Figure 1
Figure 1. Mean Recall@50 of our proposed method and other [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The overview of SPADE, which comprises two key steps: (1) inversion-guided calibration for the UNet to adapt a general pre-trained teacher diffusion model into a PSG-specific denoising network and (2) spatial-aware context reasoning (SCR) over relativeness and non-relativeness context through a relation graph Transformer to generate high-quality relation queries. Similarly, we apply the same approach to learn the ot… view at source ↗
Figure 3
Figure 3. Overview of the proposed Relation Graph Transformer, [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: The visualization results of PSG by SPADE. The first column shows the input images, while the second column presents the ground truth. The other columns display the predictions by our model, where unseen predicted objects and relationships are highlighted on a yellow g…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

68 extracted references · 61 canonical work pages

  1. [1]

    ediff-i: Text-to-image diffusion models with an ensem- ble of expert denoisers

    Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming-Yu Liu. ediff-i: Text-to-image diffusion models with an ensem- ble of expert denoisers. CoRR, abs/2211.01324, 2022. 2

    arXiv 2022

  2. [2]

    Hico: A benchmark for recognizing human-object interactions in images

    Yu-Wei Chao, Zhan Wang, Yugeng He, Jiaxuan Wang, and Jia Deng. Hico: A benchmark for recognizing human-object interactions in images. In Proceedings of the IEEE inter- national conference on computer vision , pages 1017–1025,

    work page

  3. [3]

    Spatialvlm: Endow- ing vision-language models with spatial reasoning capabili- ties

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endow- ing vision-language models with spatial reasoning capabili- ties. In CVPR, pages 14455–14465, 2024. 1

    work page 2024

  4. [4]

    Expanding scene graph boundaries: fully open-vocabulary scene graph generation via visual-concept alignment and retention

    Zuyao Chen, Jinlin Wu, Zhen Lei, Zhaoxiang Zhang, and Chang Wen Chen. Expanding scene graph boundaries: fully open-vocabulary scene graph generation via visual-concept alignment and retention. In European Conference on Com- puter Vision, pages 108–124. Springer, 2024. 1, 4, 5, 6, 7

    work page 2024

  5. [5]

    Spatial- rgpt: Grounded spatial reasoning in vision-language mod- els

    An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Rui- han Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatial- rgpt: Grounded spatial reasoning in vision-language mod- els. Advances in Neural Information Processing Systems , 37:135062–135093, 2025. 1

    work page 2025

  6. [6]

    Masked-attention mask transformer for universal image segmentation

    Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexan- der Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In CVPR, pages 1290–1299, 2022. 6, 7

    work page 2022

  7. [7]

    Recovering the unbiased scene graphs from the biased ones

    Meng-Jiun Chiou, Henghui Ding, Hanshu Yan, Changhu Wang, Roger Zimmermann, and Jiashi Feng. Recovering the unbiased scene graphs from the biased ones. In Proceedings of the 29th ACM International Conference on Multimedia , pages 1581–1590, 2021. 2

    work page 2021

  8. [8]

    Reltr: Relation transformer for scene graph generation.IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9):11169–11183, 2023

    Yuren Cong, Michael Ying Yang, and Bodo Rosenhahn. Reltr: Relation transformer for scene graph generation.IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9):11169–11183, 2023. 1, 3

    work page 2023

Show all 68 references
  1. [9]

    De- coupling zero-shot semantic segmentation

    Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. De- coupling zero-shot semantic segmentation. In CVPR, pages 11583–11592, 2022. 6

    2022

  2. [10]

    Concept sliders: Lora adaptors for precise control in diffusion models

    Rohit Gandikota, Joanna Materzy ´nska, Tingrui Zhou, Anto- nio Torralba, and David Bau. Concept sliders: Lora adaptors for precise control in diffusion models. In European Con- ference on Computer Vision, pages 172–188. Springer, 2024. 3

    2024

  3. [11]

    ican: Instance- centric attention network for human-object interaction detec- tion

    Chen Gao, Yuliang Zou, and Jia-Bin Huang. ican: Instance- centric attention network for human-object interaction detec- tion. In British Machine Vision Conference 2018, BMVC 2018, Newcastle, UK, September 3-6, 2018, page 41. BMV A Press, 2018. 1

    2018

  4. [12]

    Open-vocabulary object detection via vision and language knowledge distillation

    Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. In ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. 6

    2022

  5. [13]

    Dsgg: Dense relation transformer for an end-to-end scene graph generation

    Zeeshan Hayder and Xuming He. Dsgg: Dense relation transformer for an end-to-end scene graph generation. In CVPR, pages 28317–28326, 2024. 1, 6

    2024

  6. [14]

    Learning from the scene and borrowing from the rich: tackling the long tail in scene graph generation

    Tao He, Lianli Gao, Jingkuan Song, Jianfei Cai, and Yuan- Fang Li. Learning from the scene and borrowing from the rich: tackling the long tail in scene graph generation. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificia...

    2021

  7. [15]

    Semantic compositional learning for low-shot scene graph generation

    Tao He, Lianli Gao, Jingkuan Song, Jianfei Cai, and Yuan- Fang Li. Semantic compositional learning for low-shot scene graph generation. arXiv preprint arXiv:2108.08600, 2021. 2

    2021 arXiv

  8. [16]

    Ex- ploiting scene graphs for human-object interaction detection

    Tao He, Lianli Gao, Jingkuan Song, and Yuan-Fang Li. Ex- ploiting scene graphs for human-object interaction detection. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 15964–15973. IEEE Computer Soci- ety, 2021. 1

    2021

  9. [17]

    To- wards open-vocabulary scene graph generation with prompt- based finetuning

    Tao He, Lianli Gao, Jingkuan Song, and Yuan-Fang Li. To- wards open-vocabulary scene graph generation with prompt- based finetuning. In European Conference on Computer Vi- sion, pages 56–73. Springer, 2022. 1, 4, 6

    2022

  10. [18]

    To- ward a unified transformer-based framework for scene graph generation and human-object interaction detection

    Tao He, Lianli Gao, Jingkuan Song, and Yuan-Fang Li. To- ward a unified transformer-based framework for scene graph generation and human-object interaction detection. IEEE Transactions on Image Processing, 32:6274–6288, 2023. 2

    2023

  11. [19]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR 2022, Virtual Event, April 25-29, 2022 . OpenRe- view.net, 2022. 2, 3

    2022

  12. [20]

    Egtr: Extracting graph from trans- former for scene graph generation

    Jinbae Im, JeongYeon Nam, Nokyung Park, Hyungmin Lee, and Seunghyun Park. Egtr: Extracting graph from trans- former for scene graph generation. In CVPR, pages 24229– 24238, 2024. 6

    2024

  13. [21]

    Zero-shot scene graph relation prediction through commonsense knowledge inte- gration

    Xuan Kan, Hejie Cui, and Carl Yang. Zero-shot scene graph relation prediction through commonsense knowledge inte- gration. In Machine Learning and Knowledge Discovery in Databases. Research Track: European Conference, ECML PKDD 2021, Bilbao, Spain, September 13–17, 2021, Pro- ...

    2021

  14. [22]

    Imagic: Text-based real image editing with diffusion models

    Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In CVPR, pages 6007–6017, 2023. 2, 3

    2023

  15. [23]

    Text-image align- ment for diffusion-based perception

    Neehar Kondapaneni, Markus Marks, Manuel Knott, Rog´erio Guimaraes, and Pietro Perona. Text-image align- ment for diffusion-based perception. InCVPR, pages 13883– 13893, 2024. 3

    2024

  16. [24]

    Visual genome: Connecting language and vision using crowdsourced dense image annotations

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalan- tidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of compute...

    2017

  17. [25]

    Topviewrs: Vision-language models as top-view spatial reasoners

    Chengzu Li, Caiqi Zhang, Han Zhou, Nigel Collier, Anna Korhonen, and Ivan Vuli ´c. Topviewrs: Vision-language models as top-view spatial reasoners. In Proceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1786–1807, 2024. 1

    2024

  18. [26]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 1, 2

    2023

  19. [27]

    Panoptic scene graph genera- tion with semantics-prototype learning

    Li Li, Wei Ji, Yiming Wu, Mengze Li, You Qin, Lina Wei, and Roger Zimmermann. Panoptic scene graph genera- tion with semantics-prototype learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 3145– 3153, 2024. 6

    2024

  20. [29]

    Grounded language-image pre-training

    Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jian- wei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In CVPR, pages 10965–10975,

  21. [30]

    Sgtr: End- to-end scene graph generation with transformer

    Rongjie Li, Songyang Zhang, and Xuming He. Sgtr: End- to-end scene graph generation with transformer. In CVPR, pages 19486–19496, 2022. 1, 6

    2022

  22. [31]

    From pixels to graphs: Open-vocabulary scene graph generation with vision-language models

    Rongjie Li, Songyang Zhang, Dahua Lin, Kai Chen, and Xuming He. From pixels to graphs: Open-vocabulary scene graph generation with vision-language models. In CVPR, pages 28076–28086, 2024. 1, 4, 6, 7

    2024

  23. [32]

    Open-vocabulary object segmentation with diffusion models

    Ziyi Li, Qinye Zhou, Xiaoyun Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Open-vocabulary object segmentation with diffusion models. In ICCV, pages 7667–7676, 2023. 3

    2023

  24. [33]

    Gps-net: Graph property sensing network for scene graph generation

    Xin Lin, Changxing Ding, Jinquan Zeng, and Dacheng Tao. Gps-net: Graph property sensing network for scene graph generation. In CVPR, pages 3746–3753, 2020. 6

    2020

  25. [34]

    Path aggregation network for instance segmentation

    Shu Liu, Lu Qi, Haifang Qin, Jianping Shi, and Jiaya Jia. Path aggregation network for instance segmentation. In CVPR (CVPR), 2018. 1

    2018

  26. [35]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 10012–10022, 2021. 7

    2021

  27. [36]

    Attending to graph transformers

    Luis M ¨uller, Mikhail Galkin, Christopher Morris, and Ladislav Ramp´asek. Attending to graph transformers. Trans. Mach. Learn. Res., 2024, 2024. 3

    2024

  28. [37]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  29. [38]

    Faster r-cnn: Towards real-time object detection with region proposal networks

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information process- ing systems, 28, 2015. 1

    2015

  30. [39]

    High-resolution image syn- thesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models. In CVPR, pages 10684– 10695, 2022. 2, 6

    2022

  31. [40]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information...

    2022

  32. [41]

    Self- attention with relative position representations

    Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self- attention with relative position representations. In NAACL- HLT, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 2 (Short Papers), pages 464–468. Association for Computa- tional Linguistics, 2018. 3

    2018

  33. [42]

    Graph trans- formers: A survey

    Ahsan Shehzad, Feng Xia, Shagufta Abid, Ciyuan Peng, Shuo Yu, Dongyu Zhang, and Karin Verspoor. Graph trans- formers: A survey. arXiv preprint arXiv:2407.09777, 2024. 3

    2024

  34. [43]

    An empirical analysis on spatial reason- ing capabilities of large multimodal models

    Fatemeh Shiri, Xiao-Yu Guo, Mona Far, Xin Yu, Reza Haf, and Yuan-Fang Li. An empirical analysis on spatial reason- ing capabilities of large multimodal models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21440–21455, 2024. 1

    2024

  35. [44]

    Continual dif- fusion: Continual customization of text-to-image diffusion with c-lora

    James Seale Smith, Yen-Chang Hsu, Lingyu Zhang, Ting Hua, Zsolt Kira, Yilin Shen, and Hongxia Jin. Continual dif- fusion: Continual customization of text-to-image diffusion with c-lora. Trans. Mach. Learn. Res., 2024, 2024. 3

    2024

  36. [45]

    Denois- ing diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denois- ing diffusion implicit models. In ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. 2, 3

    2021

  37. [46]

    Transformer-based image generation from scene graphs

    Renato Sortino, Simone Palazzo, Francesco Rundo, and Concetto Spampinato. Transformer-based image generation from scene graphs. Computer Vision and Image Understand- ing, 233:103721, 2023. 3

    2023

  38. [47]

    Reclip: A strong zero-shot baseline for referring expression compre- hension

    Sanjay Subramanian, William Merrill, Trevor Darrell, Matt Gardner, Sameer Singh, and Anna Rohrbach. Reclip: A strong zero-shot baseline for referring expression compre- hension. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume ...

    2022

  39. [48]

    Learning to compose dynamic tree structures for visual contexts

    Kaihua Tang, Hanwang Zhang, Baoyuan Wu, Wenhan Luo, and Wei Liu. Learning to compose dynamic tree structures for visual contexts. In CVPR, pages 6619–6628, 2019. 2, 6

    2019

  40. [49]

    Structured sparse r-cnn for di- rect scene graph generation

    Yao Teng and Limin Wang. Structured sparse r-cnn for di- rect scene graph generation. In CVPR, pages 19437–19446,

  41. [50]

    Belm: Bidirec- tional explicit linear multi-step sampler for exact inversion in diffusion models

    Fangyikang Wang, Hubery Yin, Yue-Jiang Dong, Huminhao Zhu, Hanbin Zhao, Hui Qian, Chen Li, et al. Belm: Bidirec- tional explicit linear multi-step sampler for exact inversion in diffusion models. Advances in Neural Information Process- ing Systems, 37:46118–46159, 2025. 2, 3

    2025

  42. [51]

    Pair then relation: Pair-net for panoptic scene graph generation

    Jinghao Wang, Zhengyu Wen, Xiangtai Li, Zujin Guo, Jingkang Yang, and Ziwei Liu. Pair then relation: Pair-net for panoptic scene graph generation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. 1, 6

    2024

  43. [52]

    Stylediffusion: Controllable disentangled style transfer via diffusion models

    Zhizhong Wang, Lei Zhao, and Wei Xing. Stylediffusion: Controllable disentangled style transfer via diffusion models. In ICCV, pages 7677–7689, 2023. 2, 3

    2023

  44. [53]

    Chief: Clustering with higher-order motifs in big networks

    Feng Xia, Shuo Yu, Chengfei Liu, Jianxin Li, and Ivan Lee. Chief: Clustering with higher-order motifs in big networks. IEEE Transactions on Network Science and Engineering , 9 (3):990–1005, 2021. 3

    2021

  45. [54]

    Scene graph generation by iterative message passing

    Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. In CVPR, pages 5410–5419, 2017. 6

    2017

  46. [55]

    Scene graph generation by iterative message passing

    Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. In CVPR, pages 5410–5419, 2017. 2

    2017

  47. [56]

    Open-vocabulary panop- tic segmentation with text-to-image diffusion models

    Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiao- long Wang, and Shalini De Mello. Open-vocabulary panop- tic segmentation with text-to-image diffusion models. In CVPR, pages 2955–2966, 2023. 1, 3, 6

    2023

  48. [57]

    Panoptic scene graph gen- eration

    Jingkang Yang, Yi Zhe Ang, Zujin Guo, Kaiyang Zhou, Wayne Zhang, and Ziwei Liu. Panoptic scene graph gen- eration. In European Conference on Computer Vision, pages 178–196. Springer, 2022. 1, 2, 4, 6

    2022

  49. [58]

    Open-world human-object interaction detection via multi-modal prompts

    Jie Yang, Bingliang Li, Ailing Zeng, Lei Zhang, and Ruimao Zhang. Open-world human-object interaction detection via multi-modal prompts. In IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024 , pages 16954–16964. IEEE,

    2024

  50. [59]

    Visually-prompted language model for fine-grained scene graph generation in an open world

    Qifan Yu, Juncheng Li, Yu Wu, Siliang Tang, Wei Ji, and Yueting Zhuang. Visually-prompted language model for fine-grained scene graph generation in an open world. In ICCV, pages 21560–21571, 2023. 1, 2, 7

    2023

  51. [60]

    Zero-shot scene graph generation with knowledge graph completion

    Xiang Yu, Ruoxin Chen, Jie Li, Jiawei Sun, Shijing Yuan, Huxiao Ji, Xinyu Lu, and Chentao Wu. Zero-shot scene graph generation with knowledge graph completion. In2022 IEEE International Conference on Multimedia and Expo (ICME), pages 1–6. IEEE, 2022. 2

    2022

  52. [61]

    Graph transformer networks.Advances in neural information processing systems, 32, 2019

    Seongjun Yun, Minbyul Jeong, Raehyun Kim, Jaewoo Kang, and Hyunwoo J Kim. Graph transformer networks.Advances in neural information processing systems, 32, 2019. 3

    2019

  53. [62]

    Open-vocabulary object detection using captions

    Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih- Fu Chang. Open-vocabulary object detection using captions. In CVPR, pages 14393–14402, 2021. 1, 2

    2021

  54. [63]

    Neural motifs: Scene graph parsing with global con- text

    Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph parsing with global con- text. In CVPR, pages 5831–5840, 2018. 2, 6

    2018

  55. [64]

    gddim: Generalized denoising diffusion implicit models

    Qinsheng Zhang, Molei Tao, and Yongxin Chen. gddim: Generalized denoising diffusion implicit models. In ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net,

    2023

  56. [65]

    Learning to generate language- supervised and open-vocabulary scene graph using pre- trained visual-semantic space

    Yong Zhang, Yingwei Pan, Ting Yao, Rui Huang, Tao Mei, and Chang-Wen Chen. Learning to generate language- supervised and open-vocabulary scene graph using pre- trained visual-semantic space. In CVPR, pages 2915–2924,

  57. [66]

    Unleashing text-to-image diffusion models for visual perception

    Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text-to-image diffusion models for visual perception. In ICCV, pages 5729–5739,

  58. [67]

    Prototype-based embedding network for scene graph generation

    Chaofan Zheng, Xinyu Lyu, Lianli Gao, Bo Dai, and Jingkuan Song. Prototype-based embedding network for scene graph generation. In CVPR (CVPR) , pages 22783– 22792, 2023. 6

    2023

  59. [68]

    Hilo: Ex- ploiting high low frequency relations for unbiased panoptic scene graph generation

    Zijian Zhou, Miaojing Shi, and Holger Caesar. Hilo: Ex- ploiting high low frequency relations for unbiased panoptic scene graph generation. In ICCV, pages 21637–21648, 2023. 1, 6

    2023

  60. [69]

    Openpsg: Open-set panoptic scene graph generation via large multimodal models

    Zijian Zhou, Zheng Zhu, Holger Caesar, and Miaojing Shi. Openpsg: Open-set panoptic scene graph generation via large multimodal models. In Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part X , pages 199–215. S...

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.