METOR: A Unified Framework for Mutual Enhancement of Objects and Relationships in Open-vocabulary Video Visual Relationship Detection

Shuo Yang; Xinxiao Wu; Yongqi Wang

REVIEW 3 major objections 4 minor 31 references

METOR: A Unified Framework for Mutual Enhancement of Objects and Relationships in Open-vocabulary Video Visual Relationship Detection

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that jointly modeling object detection and relationship classification in a single query-based framework, with context-refined text encoding and alternating mutual refinement of their representations, reduces error…

desk verdict A solid, incremental advance in Open-VidVRD with real gains on mAP and mAPo, but the central mechanism claim is undercut by a self-referential contextual loss and capacity-matched ablations are missing. read the letter →

arxiv 2505.06663 v1 pith:G3ABXARP submitted 2025-05-10 cs.CV

Yongqi Wang , Xinxiao Wu , Shuo Yang This is my paper · ORCID

classification cs.CV

keywords open-vocabularyvideovisualrelationshipdetectionquery-basedunifiedframeworkCLIPcontextualrefinementiterativeenhancementmutualofobjectsandrelationshipsscenegraphVidVRDOR

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Open-vocabulary video visual relationship detection asks a model to find objects and their relationships in videos even when the categories were unseen in training. The paper argues that the usual cascade, detecting objects first and then classifying relationships from those detections, lets object errors contaminate relationship predictions and that this error propagation is a major cause of failure. It proposes METOR, a single query-based model in which object detection and relationship classification share one transformer decoding stage and then refine each other alternately through several iterations. A pre-trained vision-language model (CLIP) is frozen, but learnable object and relationship context tokens are added so that both the text features and the object queries are adjusted to the video's actual content, which the paper credits with better generalization to novel categories. On the scene-graph-detection task, METOR reports the best mAP among compared methods on both VidVRD and VidOR, with the largest relative gains on novel-split categories.

What carries the argument

The load-bearing mechanism is the contextual refinement encoding module: learnable object and relationship context tokens are concatenated with the video patch tokens and the CLS token into the fixed CLIP ViT encoder, so the resulting context embeddings $C_o$ and $C_r$ carry video-specific object and relationship context. These embeddings refine object queries through multi-head attention and, after MLP mapping layers, are prepended to category embeddings and run through the fixed CLIP text encoder to produce instance-conditioned text features. The second mechanism is the iterative enhancement module: each layer computes relationship features with a spatio-temporal transformer and then updates the subject and object features by $\hat{O}_s^{(k)} = \alpha \hat{O}_s^{(k-1)} + (1-\alpha) M_f^{(k)}(\hat{R}^{(k)})$, with the same update for the object. This alternating loop is what operationalizes mutual enhancement and is what the ablations isolate.

What would settle it

A concrete test is to train a capacity-matched METOR variant with the context tokens, mapping layers, and contextual losses removed but with an equal number of parameters added to the transformer decoder, and compare novel-split mAP on VidVRD. If the stripped variant matches or beats METOR, the claim that CLIP context refinement is the source of generalization would be falsified.

Watch

Extended reading notes

Core claim

METOR's central claim is that objects and relationships should be recognized as one interdependent prediction problem: relationship evidence should refine object features, and refined objects should sharpen relationship features, instead of relationship prediction helplessly inheriting object mistakes. Concretely, the model appends learnable context tokens $c_o$ and $c_r$ to the input of a frozen CLIP visual encoder, producing context embeddings $C_o$ and $C_r$ that are used to refine object queries via attention and to condition the CLIP text encoder's category descriptions. An iterative enhancement module then alternates, for $N_i$ layers, between building relationship features from subject and object features through a spatio-temporal transformer and updating subject and object features from those relationship features through a mapping layer. Training is end-to-end with contrastive, trajectory, and contextual losses, and the reported evaluations show METOR ahead of prior open-vocabulary methods on SGDet mAP and object-trajectory mAP, especially for novel relationship categories.

Load-bearing premise

The framework assumes that its jointly trained context tokens make the pre-trained vision-language encoder's embeddings carry real information about which object and relationship categories are present, and that this contextual signal, not the extra trainable parameters, is what drives the open-vocabulary gains.

Editorial extensions

If this is right

Because trajectories and relationship labels come from the same query-based model, open-vocabulary video relation detection no longer depends on a separate close-set trajectory detector.
Relationship evidence is given a direct path back into object representations, so object errors can be corrected by relationship context rather than merely propagated forward.
The larger relative gains on novel-split evaluation indicate that context-conditioned text features help categories that were never seen during training.
The iteration study suggests that the mutual enhancement saturates quickly: two layers on VidVRD and three on VidOR are best, with extra layers slightly hurting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper's experiments, a capacity-matched control that removes the context tokens and mapping layers but adds an equivalent number of decoder parameters would separate the CLIP-semantics effect from the effect of simply having more trainable parameters.
The alternating object-relationship refinement is not specific to video or to CLIP; the same loop could be tried for image scene graph generation, where cascade error propagation is also a known bottleneck.
The frame-level contextual losses supervise category presence from global context embeddings, so the framework could in principle be trained on weakly labeled web video with free-text captions rather than full relationship annotations.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Quick take on METOR (arXiv:2505.06663): it's a competent, incremental extension of the same group's EOV-MMP, and it does report real gains on the SGDet task for VidVRD and VidOR, mainly on mAP and mAPo. The architecture is new in the details: a query-based unified model that alternates between enhancing object and relationship representations, plus learnable context tokens appended to CLIP to refine both object queries and text features. The ablations are consistent and the writing is clear. If you work on Open-VidVRD, this is worth reading.

What works well: the problem framing is honest—cascaded pipelines do suffer from error propagation, and the error analysis in Fig. 1 is a nice touch. The iterative enhancement module is simple and seems to help (Table 4 peaks at two iterations, and the gains are not monotonic, which is a good sanity signal). The module-level ablations in Table 2 show that each component contributes, and the qualitative T-SNE figures are supportive, though not conclusive. Code is released, which I count as a plus.

The soft spots are real and mostly in the attribution of the open-vocabulary gain. The paper's core novelty is the contextual refinement encoding module (Sec 3.3, Eq 7-10). The stress-test note is on target: the contextual losses in Eq 16 are self-referential. To is computed from Co via Eq 9, then Lobj-ctx aligns Co with To. That loss can be minimized without Co encoding any category-presence signal. Meanwhile, adding learnable tokens and mapping layers to a frozen CLIP encoder adds capacity, so the ablation "w/o CRE" (Table 3) removes the whole module and cannot separate a capacity effect from the semantic-alignment story. A capacity-matched ablation (e.g., adding the same number of parameters without the context-refinement path) would be needed to support the claim that the gain comes from CLIP semantics rather than from extra parameters. Without that, the central mechanism claim is insecure.

Other weaknesses: numbers appear to be single-run (no variance reported), and the paper omits RePro and UASAN results on VidOR due to unavailable code—fair, but it weakens the SOTA claim on that dataset. More importantly, METOR's VidOR all-split R@100 is actually slightly below EOV-MMP (9.72 vs 9.82), so "state-of-the-art" is only true on mAP and mAPo, and the recall gains are tiny (+0.11 on R@50 for VidOR all-split). That's a moderate caveat, not a fatal one. Also, hyperparameters like Ni and alpha appear to be chosen per dataset, possibly via test-set performance; that's common in the subfield but should be acknowledged. The strongest baseline is the authors' own EOV-MMP—fine in principle, but the comparison is against an incremental sibling rather than an independently implemented system, so the reader should keep that context in mind.

In short: the paper deserves a serious referee, and the empirical improvements are probably real in the narrow sense. But the causal story—that the contextual refinement improves open-vocabulary generalization through CLIP semantics—is not yet established. With a capacity-matched ablation and an independent probe of what Co/Cr encode, this could become a solid contribution. As is, I'd treat it as a useful extension rather than a conceptual breakthrough. I'd cite it if I worked on VidVRD baselines, but I wouldn't change my own approach based on the mechanism alone.

My recommendation: send it to peer review, but require a response to the capacity-confusion concern and ideally a couple of extra seeds for the main numbers.

Referee Report

3 major / 4 minor

Summary. The paper proposes METOR, a query-based unified framework for open-vocabulary video visual relationship detection (Open-VidVRD). The framework contains two main components: a CLIP-based contextual refinement encoding module that appends learnable object and relationship tokens to a frozen CLIP ViT visual encoder, uses the resulting context embeddings to refine object queries and text features, and a spatio-temporal iterative enhancement module that alternately refines subject, object, and relationship features. The model is trained end-to-end with object/relationship contrastive losses, a trajectory loss, and contextual losses. Experiments on VidVRD and VidOR under novel-split and all-split SGDet settings report improvements over the prior EOV-MMP method on most metrics, with ablations on VidVRD attributing gains to the two proposed modules.

Significance. If the claims are validated, the paper makes a useful contribution by moving Open-VidVRD away from cascaded pipelines and toward a unified query-based formulation, with a concrete mechanism for mutual enhancement of object and relationship representations. The authors provide code, an end-to-end design, and relatively large gains on the novel split of VidOR (relative mAP improvement of about 53% over EOV-MMP), which is the practically important setting for open-vocabulary generalization. However, the central attribution of these gains to the proposed contextual-refinement mechanism is not fully isolated: the ablations remove whole modules, the contextual losses do not provide an independent probe of the learned context embeddings, and all results are single-run. The comparison with the strongest baseline, EOV-MMP, is also a comparison with the authors' own prior work, which reduces the strength of the headline comparison.

major comments (3)

[Sec. 3.3, Eqs. (7)-(10) and Eq. (16)]
[Table 1]
[Tables 2-4]

minor comments (4)

[Table 1 and Related Work]
[Fig. 1(c)]
[Sec. 4.2]
[Throughout]

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the claimed gains are empirical benchmark results, and no prediction reduces by construction to a fitted input.

full rationale

The paper's derivation chain is architectural: contextual refinement encoding (Eqs. 6-10), transformer decoding (Eq. 2), iterative mutual enhancement (Eq. 11), and a supervised training objective (Eqs. 12-16). Each component is defined from learnable parameters, CLIP features, and ground-truth annotations rather than from the target metric. The contextual losses in Eq. 16 supervise Co and Cr against ground-truth category-presence sets ̃O and ̃R; although To and Tr are themselves functions of Co and Cr via Eqs. 9-10, the loss target is the external annotation, so the loss is not minimized by construction and does not make the open-vocabulary prediction equivalent to the model's own inputs. The reported improvements over EOV-MMP and other baselines are empirical comparisons on public benchmarks (VidVRD, VidOR) with standard mAP and R@K metrics. The self-citations in the paper, notably to the authors' prior EOV-MMP, are used as a baseline and as an evaluation-protocol reference, not as an unverified premise that forces the central claim. A capacity-confound concern about whole-module ablations is a legitimate experimental-design risk, but it is not circular reasoning: removing a module and observing a drop does not make the proposed mechanism definitionally identical to the measured improvement. No specific equation or cited result was found that reduces to its own inputs by construction.

Assumptions & free parameters 10 free parameters · 5 assumptions · 0 invented entities

The central claim depends on a handful of modeling and evaluation assumptions: frozen CLIP transfer, the base/novel split, the trajectory association algorithm, and the pretraining initialization. Hyperparameters are tuned per dataset. No new physical or conceptual entities are introduced beyond standard learned parameters.

free parameters (10)

alpha (Eq. 11) = 0.9
Balance between previous and mutually enhanced object features.
Ni (iterative enhancement layers) = 2 (VidVRD), 3 (VidOR)
Depth of mutual enhancement; chosen by validation performance.
theta_traj = 1.0
Weight for trajectory loss in total objective.
theta_ctx = 0.2
Weight for contextual losses.
theta_cst = 0.1
Weight for trajectory consistency loss.
Nq (object queries) = 100
Number of candidate objects decoded.
trajectory score threshold = 0.2
Detections below this average classification score are discarded.
bounding box threshold = 0.35
Boxes below this confidence are filtered.
learning rate schedule = 1e-4, decay 0.1 at epochs 15, 20, 25
Optimization schedule.
key frame sampling = 1 frame per 30, 30-frame segments
Videos subsampled into segments for training and evaluation.

assumptions (5)

domain assumption Frozen CLIP ViT-L/14 and CLIP text encoder transfer semantic knowledge to novel categories
Sec. 3.3 uses CLIP with fixed parameters; open-vocabulary gains depend on CLIP generalization.
domain assumption Base/novel split from RePro is a valid measure of open-vocabulary generalization
Sec. 4.1 follows RePro's split; no separate validation set for VidVRD.
domain assumption Greedy relation association (Shang et al. 2017) merges segment-level triplets into video-level predictions without evaluation bias
Sec. 4.2 uses this algorithm to form final predictions.
domain assumption MS-COCO pretrained decoder initialization is beneficial and does not leak novel categories
Sec. 4.2 initializes decoder with COCO-pretrained object detector excluding novel categories.
domain assumption Spatio-temporal transformer can compute relationship features from concatenated subject, object, and CLS embeddings
Sec. 3.4, Eq. 11; this is the core inductive bias of the iterative enhancement module.

how reviews work

0 comments

Cite this review

Pith. "Pith review of METOR: A Unified Framework for Mutual Enhancement of Objects and Relationships in Open-vocabulary Video Visual Relationship Detection." pith.science (2026). https://pith.science/paper/G3ABXARP

@misc{pith2026250506663,
  author       = {Pith},
  title        = {Pith review of: METOR: A Unified Framework for Mutual Enhancement of Objects and Relationships in Open-vocabulary Video Visual Relationship Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G3ABXARP}},
  note         = {Machine review of arXiv:2505.06663}
}

read the original abstract

Open-vocabulary video visual relationship detection aims to detect objects and their relationships in videos without being restricted by predefined object or relationship categories. Existing methods leverage the rich semantic knowledge of pre-trained vision-language models such as CLIP to identify novel categories. They typically adopt a cascaded pipeline to first detect objects and then classify relationships based on the detected objects, which may lead to error propagation and thus suboptimal performance. In this paper, we propose Mutual EnhancemenT of Objects and Relationships (METOR), a query-based unified framework to jointly model and mutually enhance object detection and relationship classification in open-vocabulary scenarios. Under this framework, we first design a CLIP-based contextual refinement encoding module that extracts visual contexts of objects and relationships to refine the encoding of text features and object queries, thus improving the generalization of encoding to novel categories. Then we propose an iterative enhancement module to alternatively enhance the representations of objects and relationships by fully exploiting their interdependence to improve recognition performance. Extensive experiments on two public datasets, VidVRD and VidOR, demonstrate that our framework achieves state-of-the-art performance.

Figures

Figures reproduced from arXiv: 2505.06663 by the authors.

**Figure 2.** Overview of the proposed framework. Training is only performed on base categories, while evaluation includes both base and novel categories to evaluate the model’s generalization. 3.2 Overview In this paper, we propose METOR, a query-based unified framework that jointly models object detection and relationship classification to effectively exploit their interdependence to promote mutual enhancement, streamlining t… view at source ↗

**Figure 3.** An overview of the proposed contextual refinement encoding module. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: An overview of the proposed iterative enhancement mod [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: Qualitative results of feature distributions via T-SNE. (a) [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 28 canonical work pages

[1]

Social fabric: Tubelet composi- tions for video relation detection

[Chen et al., 2021] Shuo Chen, Zenglin Shi, Pascal Mettes, and Cees GM Snoek. Social fabric: Tubelet composi- tions for video relation detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13485–13494,

work page 2021
[3]

Stacked hybrid-attention and group collaborative learning for unbiased scene graph generation

[Dong et al., 2022] Xingning Dong, Tian Gan, Xuemeng Song, Jianlong Wu, Yuan Cheng, and Liqiang Nie. Stacked hybrid-attention and group collaborative learning for unbiased scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 19427–19436,

work page 2022
[6]

Compositional prompt tuning with motion cues for open-vocabulary video rela- tion detection

[Gao et al., 2023] Kaifeng Gao, Long Chen, Hanwang Zhang, Jun Xiao, and Qianru Sun. Compositional prompt tuning with motion cues for open-vocabulary video rela- tion detection. arXiv preprint arXiv:2302.00268,

arXiv 2023
[9]

Align and prompt: Video-and-language pre-training with entity prompts

[Li et al., 2022] Dongxu Li, Junnan Li, Hongdong Li, Juan Carlos Niebles, and Steven CH Hoi. Align and prompt: Video-and-language pre-training with entity prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4953– 4963,

work page 2022
[10]

Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models

[Li et al., 2023] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In International Conference on Machine Learn- ing, pages 19730–19742. PMLR,

work page 2023
[11]

Open-vocabulary se- mantic segmentation with mask-adapted clip

[Liang et al., 2023] Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary se- mantic segmentation with mask-adapted clip. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7061–7070,

work page 2023
[12]

Microsoft coco: Com- mon objects in context

[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In Proceedings of the European Conference on Computer Vision, pages 740–755. Springer,

work page 2014
[14]

Beyond short-term snip- pet: Video relation detection with spatio-temporal global context

[Liu et al., 2020] Chenchen Liu, Yang Jin, Kehan Xu, Guo- qiang Gong, and Yadong Mu. Beyond short-term snip- pet: Video relation detection with spatio-temporal global context. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10840– 10849,

work page 2020

Show all 31 references

[16]

Decoupled weight decay regularization

[Loshchilov and Hutter, 2019] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Pro- ceedings of the International Conference on Learning Representations,

2019
[18]

Video rela- tion detection with spatio-temporal graph

[Qian et al., 2019] Xufeng Qian, Yueting Zhuang, Yimeng Li, Shaoning Xiao, Shiliang Pu, and Jun Xiao. Video rela- tion detection with spatio-temporal graph. In Proceedings of the 27th ACM International Conference on Multimedia, pages 84–93,

2019
[19]

Learning transferable visual models from nat- ural language supervision

[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from nat- ural language supervision. In International Confere...

2021
[20]

Ac- tion scene graphs for long-form understanding of egocen- tric videos

[Rodin et al., 2024] Ivan Rodin, Antonino Furnari, Kyle Min, Subarna Tripathi, and Giovanni Maria Farinella. Ac- tion scene graphs for long-form understanding of egocen- tric videos. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition , pages...

2024
[21]

Video visual relation detection

[Shang et al., 2017] Xindi Shang, Tongwei Ren, Jingfan Guo, Hanwang Zhang, and Tat-Seng Chua. Video visual relation detection. In Proceedings of the 25th ACM In- ternational Conference on Multimedia, pages 1300–1308,

2017
[23]

Video visual relation detection via iterative inference

[Shang et al., 2021] Xindi Shang, Yicong Li, Junbin Xiao, Wei Ji, and Tat-Seng Chua. Video visual relation detection via iterative inference. In Proceedings of the 29th ACM In- ternational Conference on Multimedia, pages 3654–3663,

2021
[24]

Video relationship reasoning using gated spatio- temporal energy graph

[Tsai et al., 2019] Yao-Hung Hubert Tsai, Santosh Divvala, Louis-Philippe Morency, Ruslan Salakhutdinov, and Ali Farhadi. Video relationship reasoning using gated spatio- temporal energy graph. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognitio...

2019
[25]

Actionclip: A new paradigm for video action recognition

[Wang et al., 2021] Mengmeng Wang, Jiazheng Xing, and Yong Liu. Actionclip: A new paradigm for video action recognition. arXiv preprint arXiv:2109.08472,

2021 arXiv
[26]

End-to-end open-vocabulary video vi- sual relationship detection using multi-modal prompting

[Wang et al., 2025] Yongqi Wang, Xinxiao Wu, Shuo Yang, and Jiebo Luo. End-to-end open-vocabulary video vi- sual relationship detection using multi-modal prompting. IEEE Transactions on Pattern Analysis and Machine In- telligence, pages 1–17,

2025
[27]

Meta spatio-temporal debiasing for video scene graph generation

[Xu et al., 2022] Li Xu, Haoxuan Qu, Jason Kuen, Jiuxiang Gu, and Jun Liu. Meta spatio-temporal debiasing for video scene graph generation. In European Conference on Com- puter Vision, pages 374–390. Springer,

2022
[28]

Multi-modal prompting for open- vocabulary video visual relationship detection

[Yang et al., 2024] Shuo Yang, Yongqi Wang, Xiaofeng Ji, and Xinxiao Wu. Multi-modal prompting for open- vocabulary video visual relationship detection. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 6513–6521,

2024
[29]

End-to-end video scene graph generation with temporal propagation transformer

[Zhang et al., 2023] Yong Zhang, Yingwei Pan, Ting Yao, Rui Huang, Tao Mei, and Chang-Wen Chen. End-to-end video scene graph generation with temporal propagation transformer. IEEE Transactions on Multimedia, 26:1613– 1625,

2023
[30]

Constructing holistic spatio-temporal scene graph for video semantic role labeling

[Zhao et al., 2023] Yu Zhao, Hao Fei, Yixin Cao, Bobo Li, Meishan Zhang, Jianguo Wei, Min Zhang, and Tat-Seng Chua. Constructing holistic spatio-temporal scene graph for video semantic role labeling. In Proceedings of the 31st ACM International Conference on Multimedia, pages ...

2023
[31]

Vrdformer: End-to-end video visual relation detec- tion with transformers

[Zheng et al., 2022] Sipeng Zheng, Shizhe Chen, and Qin Jin. Vrdformer: End-to-end video visual relation detec- tion with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18836–18846, 2022

2022
[2002]

Vrdone: One-stage video visual relation detection

[Jiang et al., 2024] Xinjie Jiang, Chenxi Zheng, Xuemiao Xu, Bangzhen Liu, Weiying Zheng, Huaidong Zhang, and Shengfeng He. Vrdone: One-stage video visual relation detection. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 1437–1446,

2024
[2014]

Td2-net: To- ward denoising and debiasing for video scene graph gener- ation

[Lin et al., 2024] Xin Lin, Chong Shi, Yibing Zhan, Zuopeng Yang, Yaqi Wu, and Dacheng Tao. Td2-net: To- ward denoising and debiasing for video scene graph gener- ation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 3495–3503,

2024
[2017]

Relation understanding in videos: A grand challenge overview

[Shang et al., 2019] Xindi Shang, Junbin Xiao, Donglin Di, and Tat-Seng Chua. Relation understanding in videos: A grand challenge overview. In Proceedings of the ACM In- ternational Conference on Multimedia, pages 2652–2656,

2019
[2019]

Hig: Hierarchical interlacement graph approach to scene graph generation in video understand- ing

[Nguyen et al., 2024] Trong-Thuan Nguyen, Pha Nguyen, and Khoa Luu. Hig: Hierarchical interlacement graph approach to scene graph generation in video understand- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18384– 18394,

2024
[2020]

Open-vocabulary segmenta- tion with semantic-assisted calibration

[Liu et al., 2024] Yong Liu, Sule Bai, Guanbin Li, Yitong Wang, and Yansong Tang. Open-vocabulary segmenta- tion with semantic-assisted calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 3491–3500,

2024
[2021]

Spatial-temporal transformer for dynamic scene graph generation

[Cong et al., 2021] Yuren Cong, Wentong Liao, Hanno Ack- ermann, Bodo Rosenhahn, and Michael Ying Yang. Spatial-temporal transformer for dynamic scene graph generation. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision , pages 16372– 16382,

2021
[2022]

Active open-vocabulary recognition: Let intelligent moving mitigate clip limitations

[Fan et al., 2024] Lei Fan, Jianxiong Zhou, Xiaoying Xing, and Ying Wu. Active open-vocabulary recognition: Let intelligent moving mitigate clip limitations. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16394–16403,

2024
[2023]

Stochastic neighbor embedding

[Hinton and Roweis, 2002] Geoffrey E Hinton and Sam Roweis. Stochastic neighbor embedding. In Proceedings of the Advances in Neural Information Processing Sys- tems, volume 15, pages 833–840,

2002
[2024]

Simple image-level classification improves open-vocabulary object detection

[Fang et al., 2024] Ruohuan Fang, Guansong Pang, and Xiao Bai. Simple image-level classification improves open-vocabulary object detection. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 1716–1725,

2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Social fabric: Tubelet composi- tions for video relation detection

[Chen et al., 2021] Shuo Chen, Zenglin Shi, Pascal Mettes, and Cees GM Snoek. Social fabric: Tubelet composi- tions for video relation detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13485–13494,

work page 2021

[2] [3]

Stacked hybrid-attention and group collaborative learning for unbiased scene graph generation

[Dong et al., 2022] Xingning Dong, Tian Gan, Xuemeng Song, Jianlong Wu, Yuan Cheng, and Liqiang Nie. Stacked hybrid-attention and group collaborative learning for unbiased scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 19427–19436,

work page 2022

[3] [6]

Compositional prompt tuning with motion cues for open-vocabulary video rela- tion detection

[Gao et al., 2023] Kaifeng Gao, Long Chen, Hanwang Zhang, Jun Xiao, and Qianru Sun. Compositional prompt tuning with motion cues for open-vocabulary video rela- tion detection. arXiv preprint arXiv:2302.00268,

arXiv 2023

[4] [9]

Align and prompt: Video-and-language pre-training with entity prompts

[Li et al., 2022] Dongxu Li, Junnan Li, Hongdong Li, Juan Carlos Niebles, and Steven CH Hoi. Align and prompt: Video-and-language pre-training with entity prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4953– 4963,

work page 2022

[5] [10]

Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models

[Li et al., 2023] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In International Conference on Machine Learn- ing, pages 19730–19742. PMLR,

work page 2023

[6] [11]

Open-vocabulary se- mantic segmentation with mask-adapted clip

[Liang et al., 2023] Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary se- mantic segmentation with mask-adapted clip. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7061–7070,

work page 2023

[7] [12]

Microsoft coco: Com- mon objects in context

[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In Proceedings of the European Conference on Computer Vision, pages 740–755. Springer,

work page 2014

[8] [14]

Beyond short-term snip- pet: Video relation detection with spatio-temporal global context

[Liu et al., 2020] Chenchen Liu, Yang Jin, Kehan Xu, Guo- qiang Gong, and Yadong Mu. Beyond short-term snip- pet: Video relation detection with spatio-temporal global context. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10840– 10849,

work page 2020