Pith. sign in

REVIEW 81 references

Generalized Visual Relation Detection with Diffusion Models

T0 review · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Diff-VRD generates visual relation phrases with a diffusion model conditioned on CLIP features, aiming to detect interactions beyond dataset labels and scoring them with text-to-image retrieval and SPICE.

arxiv 2504.12100 v1 pith:VNEGKXZT submitted 2025-04-16 cs.CV

Kaifeng Gao , Siqi Chen , Hanwang Zhang , Jun Xiao , Yueting Zhuang , Qianru Sun This is my paper · ORCID
classification cs.CV
keywords visualrelationsrelationdetectiondiff-vrddiffusiongeneralizedgeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Visual relation detection normally labels each pair of objects with one verb or preposition from a fixed list, such as 'ride' or 'hold'. But real relationships are ambiguous: a person and a horse can be described as 'ride', 'race', 'straddle', or 'sit on' depending on the viewpoint. The authors argue that forcing every prediction into a small, predefined set misses this ambiguity and makes it expensive to add new relation words later.

Their model, Diff-VRD, works in stages. First, an off-the-shelf detector finds objects and forms pairs. For each pair, CLIP extracts visual features of the subject, the object, and their union region, plus text features of the object categories. A diffusion model starts from random noise and gradually denoises a sequence of continuous embeddings until it becomes a sequence of relation words, drawn from a large vocabulary of 4,858 predicates built from image captions. A matching stage then assigns each generated relation word to a subject-object pair by computing semantic similarity. Because the vocabulary is large and the generation is not tied to the dataset's fixed labels, the model can propose relations that the original annotations never mention.

The paper confronts a real evaluation problem: standard recall metrics punish reasonable predictions that happen to be missing from the ground truth. So the authors propose two proxy tasks. In text-to-image retrieval, predicted triplets are tiled into a caption, and a vision-language model must retrieve the correct image. In the SPICE precision-recall curve, predictions are compared against human captions or captions generated by OFA. Diff-VRD clearly trails state-of-the-art models on conventional closed-set recall, but it wins on these new proxy measures.

Extended reading notes

Core claim

The paper's load-bearing assertion is that, 'Benefiting from the diffusion-based generative process, our Diff-VRD is able to generate visual relations beyond the pre-defined category labels of datasets,' and that it 'outperforms conventional state-of-the-art VRD models on both T2I retrieval and SPICE' (Abstract; Sec I). If correct, a single model trained on a large caption-derived vocabulary can propose novel, semantically reasonable predicates for arbitrary subject-object pairs.

Load-bearing premise

The weakest load-bearing premise is that the two proposed proxy metrics, text-to-image retrieval (using CLIP or X-VLM) and the SPICE PR curve (using OFA-generated captions for HICO-DET), are valid measures of semantically correct relations that go beyond ground-truth annotations. This enters at Sec IV-D, where predictions are tiled into captions and judged by a vision-language model; because the model's conditional signals, matching similarities, and training pseudo-labels are also CLIP-based (Sec III-B, III-C, IV-B), a high score may only show that the generated text is CLIP-aligned, not that the novel relations are true. If this premise is false, the claimed superiority over SOTA on generalized VRD is unsupported.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Assumptions & free parameters 5 free parameters · 7 assumptions · 0 invented entities

The central method introduces no new physical or conceptual entities. Its load-bearing inputs are the CLIP embedding space, the caption-derived vocabulary, the pseudo-labeling procedure, and the proxy evaluation metrics. The free parameters are mostly hyperparameters; none are physically motivated constants.

free parameters (5)
  • Relation sequence length L = 32
    Fixed value 'large enough' during testing; Table VII shows R@5 rises from 21.38 to 22.60 as L grows from 16 to 48, so the choice trades cost against recall.
  • Matching loss weight lambda = 1.0
    Chosen in Sec IV-B for L = Lsimple + lambda * Lmatch; no ablation is reported for this value.
  • Sigmoid temperature kappa = 0.05
    Used in Eq. (12) for BCE matching supervision; set in Sec IV-B without sensitivity analysis.
  • Verb frequency threshold for vocabulary construction = greater than 0.5
    Verbs with caption frequency greater than 0.5 are kept (Sec IV-B); this threshold determines the 4858-word vocabulary and therefore the space of possible generated relations.
  • Enhancement corruption steps T' = 250
    Used in Sec IV-E when Diff-VRD is applied on top of IEtrans; chosen without ablation.
assumptions (7)
  • standard math The DDPM forward/reverse process and variational bound (Eqs. 1-4) are valid.
    The derivation follows Ho et al. [43]; the appendix reproduces the simplified objective. This is accepted background.
  • domain assumption CLIP's joint text-image embedding space meaningfully represents fine-grained visual relations.
    The conditioning features, embedding initialization, and matching similarities all rely on CLIP (Sec III-B, III-C). If CLIP is weak at relation-level semantics, the generated 'new' relations inherit that weakness.
  • domain assumption Discrete relation words can be transported through a continuous embedding-rounding step trained jointly with the diffusion model.
    Adopted from Diffusion-LM and DiffuSeq [22,23] in Sec III-B; this is an established technique but an assumption about the suitability of continuous diffusion for discrete predicates.
  • domain assumption The caption-derived vocabulary V (4858 words and phrases) contains visually plausible and sufficiently complete relations.
    V is built by POS tagging and scene-graph parsing of captions (Sec IV-B); if V omits or over-includes predicates, the generalized relation space is biased.
  • domain assumption CLIP pseudo-labels used to pad the relation sequence during training provide reasonable supervision.
    Sec IV-B: sequence is padded with pseudo labels collected from CLIP similarity; this injects CLIP's distribution into the generative model.
  • ad hoc to paper The proposed proxy metrics (T2I retrieval and SPICE PR curve) validly assess predictions beyond ground-truth annotations.
    Introduced in Sec IV-D; they are not established benchmarks for open-set relations and depend on VLM and captioning models.
  • ad hoc to paper BERT masked-language prior pMLM gives a valid commonsense filter for plausible subject-predicate-object triplets.
    Used only in SPICE evaluation (Sec IV-D) to re-rank predictions; adds a text-only prior that may not reflect image content.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Generalized Visual Relation Detection with Diffusion Models." pith.science (2026). https://pith.science/paper/VNEGKXZT

@misc{pith2026250412100,
  author       = {Pith},
  title        = {Pith review of: Generalized Visual Relation Detection with Diffusion Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VNEGKXZT}},
  note         = {Machine review of arXiv:2504.12100}
}
read the original abstract

Visual relation detection (VRD) aims to identify relationships (or interactions) between object pairs in an image. Although recent VRD models have achieved impressive performance, they are all restricted to pre-defined relation categories, while failing to consider the semantic ambiguity characteristic of visual relations. Unlike objects, the appearance of visual relations is always subtle and can be described by multiple predicate words from different perspectives, e.g., ``ride'' can be depicted as ``race'' and ``sit on'', from the sports and spatial position views, respectively. To this end, we propose to model visual relations as continuous embeddings, and design diffusion models to achieve generalized VRD in a conditional generative manner, termed Diff-VRD. We model the diffusion process in a latent space and generate all possible relations in the image as an embedding sequence. During the generation, the visual and text embeddings of subject-object pairs serve as conditional signals and are injected via cross-attention. After the generation, we design a subsequent matching stage to assign the relation words to subject-object pairs by considering their semantic similarities. Benefiting from the diffusion-based generative process, our Diff-VRD is able to generate visual relations beyond the pre-defined category labels of datasets. To properly evaluate this generalized VRD task, we introduce two evaluation metrics, i.e., text-to-image retrieval and SPICE PR Curve inspired by image captioning. Extensive experiments in both human-object interaction (HOI) detection and scene graph generation (SGG) benchmarks attest to the superiority and effectiveness of Diff-VRD.

Figures

Figures reproduced from arXiv: 2504.12100 by the authors.

Figure 1
Figure 1. An example of visual relation detection (VRD). (a) Existing methods [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of Diff-VRD. (a): Given an image, we first detect all the objects with their categories and enumerate the subject-object pairs. (b): We extend the diffusion-denoising process with an embedding step qϕ(x0|v) and a rounding step pθ(v|x0). We introduce a tailored training objective for this extended diffusion-denoising process, as described in Sec. III-B. (c): The parameter θ in the denoising process is implem… view at source ↗
Figure 3
Figure 3. SPICE [24] precision-recall curve (PR curve) on HICO-DET [64] and V-COCO [65] datasets. We used the official caption annotations for COCO and the generated captions from OFA [71] for HICO-DET. The VRD predictions are ranked by their confidence score for SPICE precision and recall evaluation. e.g., the R@1 of GT improves from 4.51% to 7.97% on HICO-DET. 2) When using the same object detection results from a SOTA HOI … view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Relation diversity evaluation on HICO-DET [64] test set. Each relation category is a predicate-object combination. The x-axis represents category IDs. The y-axis denotes the total number of relation instances of each category. For a clear visualization, we show only th…
Figure 5
Figure 5. Figure 5: Ablation studies by diversity evaluation for the number of relations (K) of each subject-object pair, where Diff-VRD serves as an enhancement on IEtrans [27]. The experiments were conducted on VG [28] test set. We used the additionally annotated predicates from [27] as…
Figure 6
Figure 6. Figure 6: Qualitative examples of UPT [8], the CLIP [17] baseline, and our Diff-VRD on HICO-DET [64] dataset. For clarity, we only drew bounding boxes related to the displayed triplets. cost. To strike a balance between effectiveness and efficiency, we set L = 32 for all the exp…
Figure 7
Figure 7. Figure 7: Qualitative examples of Diff-VRD in V-COCO [65] dataset. For clarity, we only drew bounding boxes related to the displayed triplets. achieved generalized VRD by predicting diverse and mean￾ingful relation descriptions that are beyond the ground-truth categories. Two ne…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

81 extracted references · 55 canonical work pages

  1. [1]

    Reltr: Relation transformer for scene graph generation,

    Y . Cong, M. Y . Yang, and B. Rosenhahn, “Reltr: Relation transformer for scene graph generation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 9, pp. 11 169–11 183, 2023

    2023

  2. [2]

    Sgtr: End-to-end scene graph generation with transformer,

    R. Li, S. Zhang, and X. He, “Sgtr: End-to-end scene graph generation with transformer,” in CVPR, 2022, pp. 19 486–19 496

    work page 2022

  3. [3]

    Ted-net: Dispersal attention for perceiving interaction region in indirectly-contact hoi detection,

    Y . Wang, Q. Liu, and Y . Lei, “Ted-net: Dispersal attention for perceiving interaction region in indirectly-contact hoi detection,”IEEE Transactions on Circuits and Systems for Video Technology , vol. 34, no. 7, pp. 5603– 5615, 2024

    work page 2024

  4. [4]

    Mining the benefits of two-stage and one-stage hoi detection,

    A. Zhang, Y . Liao, S. Liu, M. Lu, Y . Wang, C. Gao, and X. Li, “Mining the benefits of two-stage and one-stage hoi detection,” NeurIPS, vol. 34, pp. 17 209–17 220, 2021

    work page 2021

  5. [5]

    Towards hard-positive query mining for detr-based human-object interaction detection,

    X. Zhong, C. Ding, Z. Li, and S. Huang, “Towards hard-positive query mining for detr-based human-object interaction detection,” in ECCV. Springer, 2022, pp. 444–460

    work page 2022

  6. [6]

    Multi-scale human-object interaction detector,

    Y . Cheng, Z. Wang, W. Zhan, and H. Duan, “Multi-scale human-object interaction detector,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 33, no. 4, pp. 1827–1838, 2023

    work page 2023

  7. [7]

    End-to-end object detection with transformers,

    N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in ECCV. Springer, 2020, pp. 213–229

    2020

  8. [8]

    Efficient two-stage detection of human-object interactions with a novel unary-pairwise transformer,

    F. Z. Zhang, D. Campbell, and S. Gould, “Efficient two-stage detection of human-object interactions with a novel unary-pairwise transformer,” in CVPR, 2022, pp. 20 104–20 112

    work page 2022

Show all 81 references
  1. [9]

    Learning self- and cross-triplet context clues for human-object interaction detec- tion,

    W. Ren, J. Luo, W. Jiang, L. Qu, Z. Han, J. Tian, and H. Liu, “Learning self- and cross-triplet context clues for human-object interaction detec- tion,” IEEE Transactions on Circuits and Systems for Video Technology (Early Access) , 2024

    2024

  2. [10]

    Label semantic knowledge distillation for unbiased scene graph generation,

    L. Li, J. Xiao, H. Shi, W. Wang, J. Shao, A.-A. Liu, Y . Yang, and L. Chen, “Label semantic knowledge distillation for unbiased scene graph generation,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 34, no. 1, pp. 195–206, 2023

    2023

  3. [11]

    Drake: Deep pair-wise relation alignment for knowledge-enhanced multimodal scene graph generation in social media posts,

    Z. Fu, C. Zheng, J. Feng, Y . Cai, X.-Y . Wei, Y . Wang, and Q. Li, “Drake: Deep pair-wise relation alignment for knowledge-enhanced multimodal scene graph generation in social media posts,” IEEE Transactions on Circuits and Systems for Video Technology , vol. 33, no. 7, pp. 3...

    2023

  4. [12]

    A novel framework for scene graph generation via prior knowledge,

    Z. Wang, J. Lian, L. Li, and J. Zhao, “A novel framework for scene graph generation via prior knowledge,” IEEE Transactions on Circuits and Systems for Video Technology , vol. 34, no. 5, pp. 3768–3781, 2024

    2024

  5. [13]

    Gaussian distribution- aware commonsense knowledge learning for scene graph generation,

    H. Tian, N. Xu, M. Kankanhalli, and A.-A. Liu, “Gaussian distribution- aware commonsense knowledge learning for scene graph generation,” IEEE Transactions on Circuits and Systems for Video Technology (Early Access), 2024

    2024

  6. [14]

    Open-vocabulary object detection using captions,

    A. Zareian, K. D. Rosa, D. H. Hu, and S.-F. Chang, “Open-vocabulary object detection using captions,” in CVPR, 2021, pp. 14 393–14 402

    2021

  7. [15]

    Open-vocabulary object detection via vision and language knowledge distillation,

    X. Gu, T.-Y . Lin, W. Kuo, and Y . Cui, “Open-vocabulary object detection via vision and language knowledge distillation,” in ICLR, 2022

    2022

  8. [16]

    Cora: Adapting clip for open- vocabulary detection with region prompting and anchor pre-matching,

    X. Wu, F. Zhu, R. Zhao, and H. Li, “Cora: Adapting clip for open- vocabulary detection with region prompting and anchor pre-matching,” in CVPR, 2023

    2023

  9. [17]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al. , “Learning transferable visual models from natural language supervision,” in ICML, 2021, pp. 8748–8763

    2021

  10. [18]

    End-to-end zero-shot hoi detection via vision and language knowledge distillation,

    M. Wu, J. Gu, Y . Shen, M. Lin, C. Chen, and X. Sun, “End-to-end zero-shot hoi detection via vision and language knowledge distillation,” in AAAI, vol. 37, no. 3, 2023, pp. 2839–2846

    2023

  11. [19]

    Towards open-vocabulary scene graph generation with prompt-based finetuning,

    T. He, L. Gao, J. Song, and Y .-F. Li, “Towards open-vocabulary scene graph generation with prompt-based finetuning,” in ECCV. Springer, 2022, pp. 56–73

    2022

  12. [20]

    From pixels to graphs: Open-vocabulary scene graph generation with vision-language models,

    R. Li, S. Zhang, D. Lin, K. Chen, and X. He, “From pixels to graphs: Open-vocabulary scene graph generation with vision-language models,” in CVPR, 2024, pp. 28 076–28 086

    2024

  13. [21]

    Learning to generate language-supervised and open-vocabulary scene graph using pre-trained visual-semantic space,

    Y . Zhang, Y . Pan, T. Yao, R. Huang, T. Mei, and C.-W. Chen, “Learning to generate language-supervised and open-vocabulary scene graph using pre-trained visual-semantic space,” in CVPR, 2023, pp. 2915–2924

    2023

  14. [22]

    Diffusion-lm improves controllable text generation,

    X. Li, J. Thickstun, I. Gulrajani, P. S. Liang, and T. B. Hashimoto, “Diffusion-lm improves controllable text generation,” NeurIPS, vol. 35, pp. 4328–4343, 2022

    2022

  15. [23]

    Diffuseq: Sequence to sequence text generation with diffusion models,

    S. Gong, M. Li, J. Feng, Z. Wu, and L. Kong, “Diffuseq: Sequence to sequence text generation with diffusion models,” in ICLR, 2023

    2023

  16. [24]

    Spice: Semantic propositional image caption evaluation,

    P. Anderson, B. Fernando, M. Johnson, and S. Gould, “Spice: Semantic propositional image caption evaluation,” in ECCV. Springer, 2016, pp. 382–398

    2016

  17. [25]

    Learning to prompt for open-vocabulary object detection with vision-language model,

    Y . Du, F. Wei, Z. Zhang, M. Shi, Y . Gao, and G. Li, “Learning to prompt for open-vocabulary object detection with vision-language model,” in CVPR, 2022, pp. 14 084–14 093

    2022

  18. [26]

    Gen-vlkt: Simplify association and enhance interaction understanding for hoi detection,

    Y . Liao, A. Zhang, M. Lu, Y . Wang, X. Li, and S. Liu, “Gen-vlkt: Simplify association and enhance interaction understanding for hoi detection,” in CVPR, 2022, pp. 20 123–20 132

    2022

  19. [27]

    Fine- grained scene graph generation with data transfer,

    A. Zhang, Y . Yao, Q. Chen, W. Ji, Z. Liu, M. Sun, and T.-S. Chua, “Fine- grained scene graph generation with data transfer,” in ECCV. Springer, 2022, pp. 409–424

    2022

  20. [28]

    Visual genome: Connecting language and vision using crowdsourced dense image annotations,

    R. Krishna, Y . Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y . Kalantidis, L.-J. Li, D. A. Shammaet al., “Visual genome: Connecting language and vision using crowdsourced dense image annotations,” International journal of computer vision , vol. 123, pp. 32–73, 2017

    2017

  21. [29]

    Scene graph inference via multi-scale context modeling,

    N. Xu, A.-A. Liu, Y . Wong, W. Nie, Y . Su, and M. Kankanhalli, “Scene graph inference via multi-scale context modeling,” IEEE Transactions on Circuits and Systems for Video Technology , vol. 31, no. 3, pp. 1031– 1041, 2021

    2021

  22. [30]

    Rr-net: Relation reasoning for end-to-end human-object interaction detection,

    D. Yang, Y . Zou, C. Zhang, M. Cao, and J. Chen, “Rr-net: Relation reasoning for end-to-end human-object interaction detection,” IEEE Transactions on Circuits and Systems for Video Technology , vol. 32, no. 6, pp. 3853–3865, 2021

    2021

  23. [31]

    Viplo: Vision transformer based pose- conditioned self-loop graph for human-object interaction detection,

    J. Park, J.-W. Park, and J.-S. Lee, “Viplo: Vision transformer based pose- conditioned self-loop graph for human-object interaction detection,” in CVPR, 2023

    2023

  24. [32]

    Semantically similarity-wise dual-branch network for scene graph generation,

    B. Zhao, Z. Mao, S. Fang, W. Zang, and Y . Zhang, “Semantically similarity-wise dual-branch network for scene graph generation,” IEEE Transactions on Circuits and Systems for Video Technology , vol. 32, no. 7, pp. 4573–4583, 2022

    2022

  25. [33]

    Glove: Global vectors for word representation,

    J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for word representation,” in Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) , 2014, pp. 1532–1543

    2014

  26. [34]

    Deep contextualized word representations,

    M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, “Deep contextualized word representations,” in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics . New Orleans, Louisiana: Asso...

    2018

  27. [35]

    Zero-shot recognition via semantic embeddings and knowledge graphs,

    X. Wang, Y . Ye, and A. Gupta, “Zero-shot recognition via semantic embeddings and knowledge graphs,” in CVPR, 2018, pp. 6857–6866

    2018

  28. [36]

    Consnet: Learning consistency graph for zero-shot human-object interaction detection,

    Y . Liu, J. Yuan, and C. W. Chen, “Consnet: Learning consistency graph for zero-shot human-object interaction detection,” in ACM MM , 2020, pp. 4235–4243

    2020

  29. [37]

    Detect- ing human-object interactions via functional generalization,

    A. Bansal, S. S. Rambhatla, A. Shrivastava, and R. Chellappa, “Detect- ing human-object interactions via functional generalization,” in AAAI, vol. 34, no. 07, 2020, pp. 10 460–10 469

    2020

  30. [38]

    Multi-grained vision language pre- training: Aligning texts with visual concepts,

    Y . Zeng, X. Zhang, and H. Li, “Multi-grained vision language pre- training: Aligning texts with visual concepts,” in International Confer- ence on Machine Learning . PMLR, 2022, pp. 25 994–26 009

    2022

  31. [39]

    Debiased contrastive learning,

    C.-Y . Chuang, J. Robinson, Y .-C. Lin, A. Torralba, and S. Jegelka, “Debiased contrastive learning,” NeurIPS, vol. 33, pp. 8765–8775, 2020

    2020

  32. [40]

    Classifier-free diffusion guidance,

    J. Ho and T. Salimans, “Classifier-free diffusion guidance,” in NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applica- tions, 2021

    2021

  33. [41]

    High- resolution image synthesis with latent diffusion models,

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High- resolution image synthesis with latent diffusion models,” in CVPR, 2022, pp. 10 684–10 695

    2022

  34. [42]

    Vdt: General-purpose video diffusion transformers via mask modeling,

    H. Lu, G. Yang, N. Fei, Y . Huo, Z. Lu, P. Luo, and M. Ding, “Vdt: General-purpose video diffusion transformers via mask modeling,” in The Twelfth International Conference on Learning Representations , 2024

    2024

  35. [43]

    Denoising diffusion probabilistic models,

    J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” NeurIPS, vol. 33, pp. 6840–6851, 2020. IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY , VOL. *, NO. *, OCTOBER 2024 12

    2020

  36. [44]

    Dif- fwave: A versatile diffusion model for audio synthesis,

    Z. Kong, W. Ping, J. Huang, K. Zhao, and B. Catanzaro, “Dif- fwave: A versatile diffusion model for audio synthesis,” arXiv preprint arXiv:2009.09761, 2020

    2009 arXiv

  37. [45]

    Guided-tts 2: A diffusion model for high- quality adaptive text-to-speech with untranscribed data,

    S. Kim, H. Kim, and S. Yoon, “Guided-tts 2: A diffusion model for high- quality adaptive text-to-speech with untranscribed data,” arXiv preprint arXiv:2205.15370, 2022

    2022 arXiv

  38. [46]

    Analog bits: Generating discrete data using diffusion models with self-conditioning,

    T. Chen, R. ZHANG, and G. Hinton, “Analog bits: Generating discrete data using diffusion models with self-conditioning,” in International Conference on Learning Representations , 2023

    2023

  39. [47]

    Semantic- conditional diffusion networks for image captioning,

    J. Luo, Y . Li, Y . Pan, T. Yao, J. Feng, H. Chao, and T. Mei, “Semantic- conditional diffusion networks for image captioning,” in CVPR, 2023, pp. 23 359–23 368

    2023

  40. [48]

    Diffusion-based multimodal video captioning,

    J. Kainulainen, Z. Guo, and J. Laaksonen, “Diffusion-based multimodal video captioning,” in Proceedings of the Asian Conference on Computer Vision, 2024, pp. 2820–2837

    2024

  41. [49]

    Exploring discrete diffusion models for image captioning,

    Z. Zhu, Y . Wei, J. Wang, Z. Gan, Z. Zhang, L. Wang, G. Hua, L. Wang, Z. Liu, and H. Hu, “Exploring discrete diffusion models for image captioning,” arXiv preprint arXiv:2211.11694 , 2022

    2022 arXiv

  42. [50]

    Image captioning with masked diffusion model,

    W. Tian, W. Xu, J. Zhao, and Z. Zhao, “Image captioning with masked diffusion model,” in International Conference on Intelligent Computing . Springer, 2024, pp. 216–227

    2024

  43. [51]

    Denoising diffusion implicit models,

    J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit models,” in International Conference on Learning Representations , 2021

    2021

  44. [52]

    Scalable diffusion models with transformers,

    W. Peebles and S. Xie, “Scalable diffusion models with transformers,” in ICCV, 2023, pp. 4195–4205

    2023

  45. [53]

    Improved denoising diffusion probabilis- tic models,

    A. Q. Nichol and P. Dhariwal, “Improved denoising diffusion probabilis- tic models,” in International conference on machine learning . PMLR, 2021, pp. 8162–8171

    2021

  46. [54]

    Auto-encoding variational bayes,

    D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” arXiv preprint arXiv:1312.6114, 2013

    2013 arXiv

  47. [55]

    Understanding diffusion models: A unified perspective,

    C. Luo, “Understanding diffusion models: A unified perspective,” arXiv preprint arXiv:2208.11970, 2022

    2022 arXiv

  48. [56]

    Variational diffusion models,

    D. Kingma, T. Salimans, B. Poole, and J. Ho, “Variational diffusion models,” Advances in neural information processing systems , vol. 34, pp. 21 696–21 707, 2021

    2021

  49. [57]

    Ladder variational autoencoders,

    C. K. Sønderby, T. Raiko, L. Maaløe, S. K. Sønderby, and O. Winther, “Ladder variational autoencoders,” Advances in neural information pro- cessing systems , vol. 29, 2016

    2016

  50. [58]

    Improved variational inference with inverse autoregressive flow,

    D. P. Kingma, T. Salimans, R. Jozefowicz, X. Chen, I. Sutskever, and M. Welling, “Improved variational inference with inverse autoregressive flow,” Advances in neural information processing systems , vol. 29, 2016

    2016

  51. [59]

    Diffusion autoencoders: Toward a meaningful and decodable represen- tation,

    K. Preechakul, N. Chatthee, S. Wizadwongsa, and S. Suwajanakorn, “Diffusion autoencoders: Toward a meaningful and decodable represen- tation,” in CVPR, 2022, pp. 10 619–10 629

    2022

  52. [60]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” NeurIPS, vol. 30, 2017

    2017

  53. [61]

    Algorithms for the assignment and transportation prob- lems,

    J. Munkres, “Algorithms for the assignment and transportation prob- lems,” Journal of the society for industrial and applied mathematics , pp. 32–38, 1957

    1957

  54. [62]

    Nicest: Noisy label correction and training for robust scene graph generation,

    L. Li, J. Xiao, H. Shi, H. Zhang, Y . Yang, W. Liu, and L. Chen, “Nicest: Noisy label correction and training for robust scene graph generation,” IEEE Transactions on Pattern Analysis and Machine Intelligence , 2024

    2024

  55. [63]

    Spatially conditioned graphs for detecting human-object interactions,

    F. Z. Zhang, D. Campbell, and S. Gould, “Spatially conditioned graphs for detecting human-object interactions,” in ICCV, 2021, pp. 13 319– 13 327

    2021

  56. [64]

    Learning to detect human-object interactions,

    Y .-W. Chao, Y . Liu, X. Liu, H. Zeng, and J. Deng, “Learning to detect human-object interactions,” in WACV. IEEE, 2018, pp. 381–389

    2018

  57. [65]

    Visual semantic role labeling,

    S. Gupta and J. Malik, “Visual semantic role labeling,” arXiv preprint arXiv:1505.04474, 2015

    2015 arXiv

  58. [66]

    Microsoft coco: Common objects in context,

    T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll ´ar, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in ECCV. Springer, 2014, pp. 740–755

    2014

  59. [67]

    Scene graph generation by iterative message passing,

    D. Xu, Y . Zhu, C. B. Choy, and L. Fei-Fei, “Scene graph generation by iterative message passing,” in CVPR, 2017, pp. 5410–5419

    2017

  60. [68]

    S. Bird, E. Klein, and E. Loper, Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit . O’Reilly Media, Inc., 2009

    2009

  61. [69]

    Generating semantically precise scene graphs from textual descriptions for improved image retrieval,

    S. Schuster, R. Krishna, A. Chang, L. Fei-Fei, and C. D. Manning, “Generating semantically precise scene graphs from textual descriptions for improved image retrieval,” in EMNLP 4th Workshop on Vision and Language, 2015, pp. 70–80

    2015

  62. [70]

    Learning transferable human-object interaction detector with natural language supervision,

    S. Wang, Y . Duan, H. Ding, Y .-P. Tan, K.-H. Yap, and J. Yuan, “Learning transferable human-object interaction detector with natural language supervision,” in CVPR, 2022, pp. 939–948

    2022

  63. [71]

    Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework,

    P. Wang, A. Yang, R. Men, J. Lin, S. Bai, Z. Li, J. Ma, C. Zhou, J. Zhou, and H. Yang, “Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework,” in ICML. PMLR, 2022, pp. 23 318–23 340

    2022

  64. [72]

    Exploring conditional multi- modal prompts for zero-shot hoi detection,

    T. Lei, S. Yin, Y . Peng, and Y . Liu, “Exploring conditional multi- modal prompts for zero-shot hoi detection,” in European Conference on Computer Vision . Springer, 2024, pp. 1–19

    2024

  65. [73]

    Pcpl: Predicate-correlation perception learning for unbiased scene graph generation,

    S. Yan, C. Shen, Z. Jin, J. Huang, R. Jiang, Y . Chen, and X.-S. Hua, “Pcpl: Predicate-correlation perception learning for unbiased scene graph generation,” in ACM MM, 2020, pp. 265–273

    2020

  66. [74]

    Neural motifs: Scene graph parsing with global context,

    R. Zellers, M. Yatskar, S. Thomson, and Y . Choi, “Neural motifs: Scene graph parsing with global context,” in CVPR, 2018, pp. 5831–5840

    2018

  67. [75]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  68. [76]

    Learning to compose dynamic tree structures for visual contexts,

    K. Tang, H. Zhang, B. Wu, W. Luo, and W. Liu, “Learning to compose dynamic tree structures for visual contexts,” in CVPR, 2019, pp. 6619– 6628

    2019

  69. [77]

    Unbiased scene graph generation from biased training,

    K. Tang, Y . Niu, J. Huang, J. Shi, and H. Zhang, “Unbiased scene graph generation from biased training,” in CVPR, 2020, pp. 3716–3725

    2020

  70. [78]

    Cogtree: Cognition tree loss for unbiased scene graph generation,

    J. Yu, Y . Chai, Y . Wang, Y . Hu, and Q. Wu, “Cogtree: Cognition tree loss for unbiased scene graph generation,” in IJCAI, 2021, pp. 1274–1280

    2021

  71. [79]

    Improved techniques for training score-based generative models,

    Y . Song and S. Ermon, “Improved techniques for training score-based generative models,” NeurIPS, vol. 33, pp. 12 438–12 448, 2020

    2020

  72. [80]

    Diffusion models beat gans on image synthesis,

    P. Dhariwal and A. Nichol, “Diffusion models beat gans on image synthesis,” NeurIPS, vol. 34, pp. 8780–8794, 2021. IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY , VOL. *, NO. *, OCTOBER 2024 13 Kaifeng Gao received the B.Eng. degree from Zhe- jiang University,...

    2021

  73. [2007]

    His cur- rent research interests include computer animation, multimedia retrieval, and machine learning

    He is currently a Professor with the College of Computer Science, Zhejiang University. His cur- rent research interests include computer animation, multimedia retrieval, and machine learning. Yueting Zhuang received his B.Sc., M.Sc. and Ph.D. degrees in Computer Science from Z...

    2018

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.