Pith. sign in

REVIEW 3 major objections 4 minor 85 references

Bilateral Collaboration with Large Vision-Language Models for Open Vocabulary Human-Object Interaction Detection

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read BC-HOI guides a vision-language model with the HOI detector's attention maps and feeds token-level LLM supervision back to the detector, achieving state-of-the-art results on HICO-DET and V-COCO in both open-vocabulary and closed settings.

desk verdict Strong engineering with plausible but unverified gains; the GPT-4o caption supervision may leak held-out categories and undermine the open-vocab claim. read the letter →

arxiv 2507.06510 v1 pith:NRN46TGM submitted 2025-07-09 cs.CV

Yupeng Hu , Changxing Ding , Chang Sun , Shaoli Huang , Xiangmin Xu This is my paper · ORCID
classification cs.CV
keywords openvocabularyHOIdetectionhuman-objectinteractionvision-languagemodelsattentionbiasguidanceBLIP-2LLM-basedsupervisionDETRHICO-DET
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Open-vocabulary human-object interaction (HOI) detection asks a model to recognize triplets whose verb-object combinations, objects, or verbs were never seen in training. This paper argues that large vision-language models (VLMs) provide generalizable but spatially coarse features, which clashes with detection's need for instance-level precision. To resolve that, BC-HOI sets up a bilateral collaboration: the HOI detector's cross-attention maps are injected as bias into the VLM's attention layers, making the VLM produce fine-grained, instance-aligned features, and the VLM's language component in turn supervises the detector token-by-token using image captions, improving those attention maps. Experiments on HICO-DET and V-COCO report state-of-the-art results in both open-vocabulary and closed settings; for example, it exceeds the second-best method by 11.82 mAP on unseen categories in the rare-first unseen-composition setting (RF-UC), where rare verb-object pairs are held out during training. If confirmed, the work suggests that coarse VLM features can be made detection-grade by routing detector spatial knowledge into the VLM rather than only fusing outputs late.

What carries the argument

The load-bearing object is the cross-attention map $\mathbf{A}_{ho}\in\mathbb{R}^{N_q\times h\times w}$ produced by the interaction decoder via $(\mathbf{E}_{ho},\mathbf{A}_{ho})=f_i(\mathbf{M},\mathbf{E}_h\oplus\mathbf{E}_o)$, with each query's map treated as a spatial prior for one human-object pair. ABG up-samples, reshapes, and adapts each map through an FC layer, then adds it to the self-attention scores of the corresponding duplicated class tokens inside every layer of BLIP-2's ViT, and analogously biases the Q-Former's cross-attention with the same maps. A unified attention computation with mask matrices ($0$ bias for allowed query-key pairs and $-\infty$ to exclude keys outside a query's receptive field) makes the bias injection fast and parallel. The opposite direction of the loop is carried by the auto-regressive loss $\mathcal{L}_{\mathrm{lsg}}=\frac{\gamma}{N_t}\sum_{n=1}^{N_t} w_n\,\mathrm{CE}(t_n,g_n)$ over the LLM's caption tokens, with $w_n=\alpha,\beta,1$ for nouns, verbs, and other tokens; because BLIP-2 is frozen except for the new Q-Former queries $\mathbf{Q}^q_f$, this loss back-propagates into the HOI detector and shapes $\mathbf{A}_{ho}$ into a better bias source.

What would settle it

Measure the spatial concentration of the interaction-decoder cross-attention maps in the Unseen Object (UO) setting—for example, the entropy of each map or its intersection over union with the ground-truth human-object box—and compare it between queries for seen and unseen objects; the mechanism predicts diffuse maps for unseen objects accompanying the small +0.22 mAP gain, so equally concentrated maps would falsify the claim that attention-bias quality drives the result.

Watch

Extended reading notes

Core claim

The central claim is that a one-stage HOI detector and a frozen large vision-language model can improve each other in a loop, and that this loop is what unlocks open-vocabulary generalization. The HOI detector's interaction-decoder cross-attention maps focus on individual human-object pairs; ABG duplicates the VLM's class token once per HOI query and adds each query's cross-attention map as bias into every self-attention layer of BLIP-2's ViT and into the Q-Former, so the VLM emits $N_q$ fine-grained, detector-aligned embeddings instead of one holistic embedding. LSG then adds extra queries and class tokens whose biased VLM embeddings are fed to the OPT-2.7B language model to predict captions (generated by GPT-4o) word by word, with nouns and verbs weighted more heavily; because the VLM is frozen, the resulting gradient trains the HOI detector to produce attention bias that supports caption prediction. The authors report consistent state-of-the-art results: 43.01 Full mAP on HICO-DET and 68.20 mAP#1role on V-COCO in the closed setting, and 42.31, 33.01, and 31.18 mAP on unseen categories in the RF-UC, NF-UC, and UV open-vocabulary settings respectively.

Load-bearing premise

The framework assumes that the HOI detector's cross-attention maps point at the right interaction areas even for verb-object combinations, objects, or verbs the detector has never seen, because those maps are what steer the VLM's features.

Editorial extensions

If this is right

  • Open-vocabulary HOI detection improves substantially on unseen verb-object combinations without any new external training data, because the detector's spatial prior compensates for the VLM's coarse output.
  • Frozen VLMs can be repurposed for fine-grained detection by steering their attention with detector-derived bias, avoiding expensive fine-tuning of the entire VLM.
  • LLM token-level supervision from captions is a viable alternative to pseudo-label extraction: it improves the detector's attention maps without explicit triplet labeling.
  • The framework's inference cost stays close to the baseline because the LLM component is dropped at test time, leaving only the biased VLM encoder and the detector.
  • The 11.82 mAP gain on unseen categories in the RF-UC setting and the large rare-category gains in the closed setting (45.76 versus 39.91 Rare mAP) indicate the method especially helps under-represented interactions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same attention-bias mechanism could transfer to other coarse-to-fine open-vocabulary tasks, such as open-vocabulary segmentation or referring expression grounding, where a detector or proposal network's spatial maps could steer a frozen VLM's patch attention.
  • Because the Unseen Object (UO) setting shows only a small unseen gain (+0.22 mAP), the framework's benefit appears tied to how well the detector can localize the object; pre-training LSG on large-scale image-text pairs with richer object categories, as the authors conjecture, is a directly testable next step.
  • The bilateral loop is effectively a form of online spatial prompt learning, in which the attention bias acts as a query-conditional prompt that adapts VLM features per instance, suggesting connections to parameter-efficient VLM adaptation beyond HOI detection.
  • One could test the sensitivity of the mechanism by corrupting $\mathbf{A}_{ho}$ at inference (blurring it or replacing it with uniform maps); if performance barely drops, the claimed causal role of attention bias would be weakened.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes BC-HOI, a one-stage open-vocabulary HOI detection framework that couples an HOI detector with BLIP-2 in a bilateral loop: (1) early fusion of CNN and ViT feature maps, (2) Attention Bias Guidance (ABG), which injects the detector's interaction-decoder cross-attention maps as bias into the VLM's self-attention and Q-Former to obtain instance-level interaction features, and (3) LLM-based Supervision Guidance (LSG), which trains the detector to produce attention maps that let the frozen OPT-2.7B predict token-level captions generated offline by GPT-4o. The authors report state-of-the-art results on HICO-DET and V-COCO in both closed and open-vocabulary settings, with the largest gains in the RF-UC unseen split (+11.82 mAP over the second-best method), and provide ablations of each component plus efficiency comparisons.

Significance. If the reported results hold, the bilateral-collaboration idea is a meaningful conceptual step: instead of treating the VLM as a fixed feature extractor, the detector's attention maps are used to make the VLM features instance-level, and the VLM's language head is used to provide fine-grained token-level supervision back to the detector. The paper is strong on experimental breadth: four open-vocabulary settings, closed settings on two benchmarks, and component ablations including attention-bias sources/destinations, caption sources, and token weights. The efficiency analysis is also useful. However, the central open-vocabulary claim is currently threatened by a potential test-set information leak in the LSG caption supervision, and the main comparisons are not fully controlled because the method receives an external GPT-4o caption signal that baselines do not. These issues must be resolved before the performance numbers can be taken as evidence for the proposed mechanism.

major comments (3)
  1. [Section 3.3, Eqs. (3)-(4); Supplementary Figure A] LSG trains the model to predict tokens of GPT-4o captions generated from the training images, and the captioning prompt in Supplementary Figure A explicitly requests human-object interaction descriptions (e.g., 'holding a leash and walking a dog'). In the UC, UO, and UV evaluation splits, held-out verbs, objects, and verb-object compositions can still appear in the training images, and nothing in the paper masks or filters caption tokens that belong to held-out categories. This means the model may receive direct token-level supervision for categories that are intended to be unseen, confounding the open-vocabulary evaluation and potentially explaining part of the large gains in Table 1 (e.g., RF-UC Unseen: 42.31 vs. 30.49 mAP for the second-best method). Please provide a quantitative analysis of the overlap between the caption vocabulary and the held-out category vocabulary for each split, and report results after excluding or masking all caption tokens belonging to held-out categories.
  2. [Section 4.2, Table 1; Section 3.3] The comparison in Table 1 is not fully controlled: BC-HOI is trained with an additional external supervision signal (GPT-4o captions) that none of the baselines receive. The 'w/o LSG' row in Table 7(e) is a useful internal control (NF-UC Unseen: 31.70 vs. 33.01), but the main open-vocabulary claims are made against methods without this extra signal. To separate the contribution of the bilateral collaboration from the contribution of the extra caption data, the paper should either include baselines trained with the same caption-supervision protocol or prominently report open-vocabulary numbers without LSG for the main results.
  3. [Section 4.4, Table 5; Table 7(c); Section 4.2] ABG is the core novelty, but its benefit relies on the detector's cross-attention maps being reliable semantic guidance. The UO row of Table 1 shows that when the detector's attention degrades for novel objects, the gain on unseen categories is only +0.22 mAP (19.72 to 19.94), and the authors acknowledge this limitation. However, the paper also states in Section 4.2 that ABG 'provides accurate attention bias even for unseen verb-object combinations.' These two statements are in tension, and the paper does not provide an analysis of attention-map quality as a function of seen/unseen categories. Please add such an analysis or qualify the claim to the settings where it is supported by evidence.
minor comments (4)
  1. [Section 4.1 and Table 7(f)] The hyperparameters α, β, and γ are fixed in Section 4.1, but Table 7(f) and Supplementary Table C show non-negligible sensitivity to these values (e.g., NF-UC Full ranges from 35.74 to 36.40 across weight settings). Reporting the chosen values in a table and discussing the sensitivity would improve reproducibility.
  2. [Table 4] The table shows BC-HOI's inference model size as smaller than UniHOI's even though the training model is much larger; this works because the LLM is dropped at inference. Adding a footnote to that effect would prevent confusion.
  3. [References] Reference [77] is cited for GPT-4o, but the entry points to an 'Improving language understanding by generative pre-training' URL for gpt-4; this citation appears incorrect and should be fixed.
  4. [Section 3.2, Eq. (2)] The definition of ϕ as 'the Vision Tower' is introduced after Eq. (1), where a different symbol f^i is used for the interaction decoder; the notation is understandable but could be made more uniform to avoid ambiguity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central results are evaluated on fixed external benchmarks, and the internal detector/VLM feedback loop and auxiliary caption loss do not reduce any predicted quantity to its own input.

full rationale

The paper's derivation chain is self-contained against external benchmarks. The proposed ABG component (Eqs. 1-2) uses the HOI detector's cross-attention maps Aho as attention bias for BLIP-2, and the resulting VLM features are fused back into the detector; this is an architectural feedback loop, not a derivation in which the claimed prediction equals its input, and the reported mAP numbers are measured on fixed HICO-DET/V-COCO splits. The LSG loss (Eqs. 3-4) supervises the detector via token prediction of GPT-4o-generated captions; this is an auxiliary training objective on external MLLM output, not a fit to the target HOI labels or to the test-set categories, so it does not reduce the open-vocabulary result to a fitted parameter by construction. No equation defines a target quantity in terms of the predicted quantity. The paper's citations to prior work by its own authors (e.g., DOQ [11], DP-HOI [52]) appear only as baseline comparisons and are not load-bearing for the central claim. The skeptic's concern that GPT-4o captions may leak held-out verb/object words into training is a plausible data-contamination risk for the open-vocabulary protocol, but it is not a circularity in the derivation chain and cannot be demonstrated from the paper's equations alone, so it does not raise the circularity score.

Assumptions & free parameters 6 free parameters · 3 assumptions · 0 invented entities

The central claim rests on domain assumptions about the reliability of attention maps and external captions, plus a set of hyperparameters. No new physical entities are introduced.

free parameters (6)
  • Nq = 64
    Number of HOI queries and duplicated cls tokens; affects the number of attention bias maps and model capacity.
  • Nf = 32
    Number of caption queries for LSG; set by design.
  • alpha (loss weight for nouns) = 1.5
    Weight on noun tokens in Eq. 3; selected via ablation on NF-UC (Table 7(f)).
  • beta (loss weight for verbs) = 2
    Weight on verb tokens in Eq. 3; selected via ablation on NF-UC.
  • gamma (LSG loss weight) = 0.1
    Overall weight for the captioning loss in Eq. 5.
  • initial learning rate = 1e-4
    Standard schedule with decay after 40 epochs.
assumptions (3)
  • domain assumption The cross-attention maps of HOI queries in the interaction decoder focus on the interaction area of a specific human-object pair.
    Motivates ABG (Section 3.2, Figure 2). If false, the attention bias would misguide the VLM.
  • domain assumption VLM visual features (BLIP-2 ViT) are holistic and coarse-grained, and attention bias can make them instance-level.
    This is the premise of the entire ABG design (Section 1, Figure 2).
  • domain assumption Captions generated by GPT-4o provide accurate, interaction-relevant token-level supervision.
    LSG uses these captions as ground truth for the auto-regressive loss (Section 3.3).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Bilateral Collaboration with Large Vision-Language Models for Open Vocabulary Human-Object Interaction Detection." pith.science (2026). https://pith.science/paper/NRN46TGM

@misc{pith2026250706510,
  author       = {Pith},
  title        = {Pith review of: Bilateral Collaboration with Large Vision-Language Models for Open Vocabulary Human-Object Interaction Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NRN46TGM}},
  note         = {Machine review of arXiv:2507.06510}
}
read the original abstract

Open vocabulary Human-Object Interaction (HOI) detection is a challenging task that detects all <human, verb, object> triplets of interest in an image, even those that are not pre-defined in the training set. Existing approaches typically rely on output features generated by large Vision-Language Models (VLMs) to enhance the generalization ability of interaction representations. However, the visual features produced by VLMs are holistic and coarse-grained, which contradicts the nature of detection tasks. To address this issue, we propose a novel Bilateral Collaboration framework for open vocabulary HOI detection (BC-HOI). This framework includes an Attention Bias Guidance (ABG) component, which guides the VLM to produce fine-grained instance-level interaction features according to the attention bias provided by the HOI detector. It also includes a Large Language Model (LLM)-based Supervision Guidance (LSG) component, which provides fine-grained token-level supervision for the HOI detector by the LLM component of the VLM. LSG enhances the ability of ABG to generate high-quality attention bias. We conduct extensive experiments on two popular benchmarks: HICO-DET and V-COCO, consistently achieving superior performance in the open vocabulary and closed settings. The code will be released in Github.

Figures

Figures reproduced from arXiv: 2507.06510 by the authors.

Figure 1
Figure 1. Comparisons between one-stage open vocabulary HOI [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Visualization of attention maps. (a) Self-attention maps [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of our bilateral collaboration framework between the HOI detector and VLM. The collaborations are reflected in three [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Fast computation of ABG. The zero and -inf matrices denote those with elements that are 0 and -inf, respectively. Di￾agonal elements in the ‘mask matrix’ are 0 and the others are -inf. 3.3. Guidance from VLM to the HOI Detector The ABG approach relies on the attention …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

85 extracted references · 77 canonical work pages

  1. [1]

    S. Ning, L. Qiu, Y . Liu, X. He. Hoiclip: Efficient knowledge transfer for hoi detection with vision- language models. In CVPR, 2023. 1, 2, 3, 6, 7

    2023

  2. [2]

    Y . Mao, J. Deng, W. Zhou, L. Li, Y . Fang, H. Li. CLIP4HOI: towards adapting CLIP for practical zero- shot HOI detection. In NeurIPS, 2023. 1, 2, 3, 6, 7

    2023

  3. [3]

    Y . Chao, Y . Liu, X. Liu, H. Zeng, J. Deng. Learning to detect human-object interactions. In WACV, 2018

    2018

  4. [4]

    Y . Liao, A. Zhang, M. Lu, Y . Wang, X. Li, S. Liu. Gen-vlkt: Simplify association and enhance interac- tion understanding for hoi detection. In CVPR, 2022. 1, 2, 3, 5, 6, 7

    2022

  5. [5]

    Z. Hou, X. Peng, Y . Qiao, D. Tao. Visual composi- tional learning for human-object interaction detection. In ECCV, 2020. 3, 2

    2020

  6. [6]

    M. Wu, J. Gu, Y . Shen, M. Lin, C. Chen, X. Sun. End- to-end zero-shot hoi detection via vision and language knowledge distillation. In AAAI, 2023. 1, 2, 3

    work page 2023

  7. [7]

    C. Gao, J. Xu, Y . Zou, J. Huang. Drg: Dual rela- tion graph for human-object interaction detection. In ECCV, 2020. 2

    work page 2020

  8. [8]

    M. Chen, Y . Liao, S. Liu, Z. Chen, F. Wang, C. Qian. Reformulating hoi detection as adaptive set prediction. In CVPR, 2021. 2

    work page 2021

Show all 85 references
  1. [9]

    Tamura, H

    M. Tamura, H. Ohashi, T. Yoshinaga. Qpic: Query- based pairwise human-object interaction detection with image-wide contextual information. In CVPR,

  2. [10]

    Zhang, D

    F. Zhang, D. Campbell, S. Gould. Efficient two-stage detection of human-object interactions with a novel unary-pairwise transformer. In CVPR, 2022. 6, 7, 2

    2022

  3. [11]

    X. Qu, C. Ding, X. Li, X. Zhong, D. Tao. Distillation using oracle queries for transformer-based human- object interaction detection. In CVPR, 2022. 2, 6

    2022

  4. [12]

    Zhang, Y

    A. Zhang, Y . Liao, S. Liu, M. Lu, Y . Wang, C. Gao, X. Li. Mining the benefits of two-stage and one-stage hoi detection. In NeurIPS, 2021. 2, 6, 7

    2021

  5. [13]

    C. Xie, F. Zeng, Y . Hu, S. Liang, Y . Wei. Category query learning for human-object interaction classifi- cation. In CVPR, 2023

    2023

  6. [14]

    Y . Cao, Q. Tang, X. Su, S. Chen, S. You, X. Lu, C. Xu. Detecting any human-object interaction relationship: Universal hoi detector with spatial prompt learning on foundation models. In NeurIPS, 2023. 1, 2, 3, 5, 6, 7

    2023

  7. [15]

    T. Lei, S. Yin, Y . Peng, Y . Liu. Exploring conditional multi-modal prompts for zero-shot hoi detection. In ECCV, 2024. 1, 2, 6, 7

    2024

  8. [16]

    J. Yang, B. Li, F. Yang, A. Zeng, L. Zhang, R. Zhang. Open-World Human-Object Interaction Detection via Multi-modal Prompts. In CVPR, 2024. 1, 3, 6, 7, 2

    2024

  9. [17]

    Gupta, J

    S. Gupta, J. Malik. Visual semantic role labeling. In arXiv:1505.04474, 2015. 1, 2, 6

    2015 arXiv

  10. [18]

    Ulutan, A

    O. Ulutan, A. Iftekhar, B. Manjunath. Vsgnet: Spatial attention network for detecting human object interac- tions using graph convolutions. In CVPR, 2020. 2

    2020

  11. [19]

    T. Lei, F. Caba, Q. Chen, H. Jin, Y . Peng, Y . Liu. Ef- ficient Adaptive Human-Object Interaction Detection with Concept-guided Memory. In ICCV, 2023

    2023

  12. [20]

    Gupta, A

    T. Gupta, A. Schwing, D. Hoiem. No-frills human- object interaction detection: Factorization, layout en- codings, and training techniques. In ICCV, 2019. 2

    2019

  13. [21]

    Y . Li, S. Zhou, X. Huang, L. Xu, Z. Ma, H. Fang, Y . Wang, C. Lu. Transferable interactiveness knowl- edge for human-object interaction detection. InCVPR, 2019

    2019

  14. [22]

    B. Wan, D. Zhou, Y . Liu, R. Li, X. He. Pose-aware multi-level feature network for human object interac- tion detection. In ICCV, 2019

    2019

  15. [23]

    J. Park, J. Park, J. Lee. Viplo: Vision transformer based pose-conditioned self-loop graph for human- object interaction detection. In CVPR, 2023. 2, 6, 7

    2023

  16. [24]

    Y . Liu, J. Yuan, C. Chen. Consnet: Learning consis- tency graph for zero-shot human-object interaction de- tection. In ACM MM, 2020. 2, 3

    2020

  17. [25]

    S. Qi, W. Wang, B. Jia, J. Shen, S. Zhu. Learn- ing human-object interactions by graph parsing neural networks. In ECCV, 2018. 2

    2018

  18. [26]

    H. Wang, W. Zheng, L. Yingbiao. Contextual hetero- geneous graph network for human-object interaction detection. In ECCV, 2020. 2

    2020

  19. [27]

    Y . Liao, S. Liu, F. Wang, Y . Chen, C. Qian, J. Feng. Ppdm: Parallel point detection and matching for real- time human-object interaction detection. In CVPR,

  20. [28]

    T. Wang, T. Yang, M. Danelljan, F. Khan, X. Zhang, J. Sun. Learning human-object interaction detection us- ing interaction points. In CVPR, 2020. 2

    2020

  21. [29]

    Zhong, X

    X. Zhong, X. Qu, C. Ding, D. Tao. Glance and gaze: Inferring action-aware points for one-stage human- object interaction detection. In CVPR, 2021. 2

    2021

  22. [30]

    B. Kim, T. Choi, J. Kang, H. Kim. Uniondet: Union- level detector towards real-time human-object interac- tion detection. In ECCV, 2020. 2

    2020

  23. [31]

    Radford, J

    A. Radford, J. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, Learning transferable visual models from natural lan- guage supervision. In ICML, 2021. 1

    2021

  24. [32]

    X. Gu, T. Lin, W. Kuo, Y . Cui. Open-vocabulary ob- ject detection via vision and language knowledge dis- tillation. In ICLR, 2022

    2022

  25. [33]

    L. Li, P. Zhang, H. Zhang, J. Yang, C. Li, Y . Zhong, L. Wang, L. Yuan, L. Zhang, J. Hwang, Grounded language-image pre-training. In CVPR, 2022

    2022

  26. [34]

    H. Yuan, J. Jiang, S. Albanie, T. Feng, Z. Huang, D. Ni, M. Tang. Rlip: Relational language-image pre-training for human-object interaction detection. In NeurIPS, 2022. 3

    2022

  27. [35]

    J. Li, D. Li, C. Xiong, S. Hoi. Blip: Bootstrap- 9 ping language-image pre-training for unified vision- language understanding and generation. In ICML,

  28. [36]

    J. Li, D. Li, S. Savarese, S. Hoi. Blip-2: Bootstrap- ping language-image pre-training with frozen image encoders and large language models. In ICML, 2023. 1, 3, 2

    2023

  29. [37]

    Zhong, J

    Y . Zhong, J. Yang, P. Zhang, C. Li, N. Codella, L. Li, L. Zhou, X. Dai, L. Yuan, Y . Li, Regionclip: Region- based language-image pretraining. In CVPR, 2022

    2022

  30. [38]

    Y . Lin, M. Chen, K. Zhang, H. Li, M. Li, Z. Yang, D. Lv, B. Lin, H. Liu, D. Cai. TagCLIP: A Local- to-Global Framework to Enhance Open-V ocabulary Multi-Label Classification of CLIP without Training. In AAAI, 2024. 2

    2024

  31. [39]

    Bansal, N

    H. Bansal, N. Singhi, Y . Yang, F. Yin, A. Grover, K. Chang. Cleanclip: Mitigating data poisoning attacks in multimodal contrastive learning. In ICCV, 2023

    2023

  32. [40]

    Y . Rao, W. Zhao, G. Chen, Y . Tang, Z. Zhu, G. Huang, J. Zhou, J. Lu. Denseclip: Language-guided dense prediction with context-aware prompting. In CVPR, 2022

    2022

  33. [41]

    H. Yuan, S. Zhang, X. Wang, S. Albanie, Y . Pan, T. Feng, J. Jiang, D. Ni, Y . Zhang, D. Zhao. Rlipv2: Fast scaling of relational language-image pre-training. In ICCV, 2023. 3, 6, 7, 2

    2023

  34. [42]

    Z. Khan, Y . Fu. Consistency and Uncertainty: Identi- fying Unreliable Responses From Black-Box Vision- Language Models for Selective Visual Question An- swering. In CVPR, 2024. 1

    2024

  35. [43]

    L. Li, J. Peng, H. Chen, C. Gao, X. Yang. How to configure good in-context sequence for visual ques- tion answering. In CVPR, 2024. 1

    2024

  36. [44]

    R. Li, S. Zhang, D. Lin, K. Chen, X. He. From Pixels to Graphs: Open-V ocabulary Scene Graph Generation with Vision-Language Models. In CVPR, 2024. 1

    2024

  37. [45]

    Zhang, S

    C. Zhang, S. Stepputtis, J. Campbell, K. Sycara, Y . Xie. HiKER-SGG: Hierarchical Knowledge Enhanced Robust Scene Graph Generation. In CVPR, 2024. 1

    2024

  38. [46]

    H. Kim, J. Hong, H. Kong, S. Lee. TE-TAD: Towards Full End-to-End Temporal Action Detection via Time- Aligned Coordinate Expression. In CVPR, 2024. 1

    2024

  39. [47]

    Xiong, X

    B. Xiong, X. Yang, Y . Song, Y . Wang, C. Xu. Modality-Collaborative Test-Time Adaptation for Ac- tion Recognition. In CVPR, 2024. 1

    2024

  40. [48]

    Carion, F

    N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kir- illov, S. Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020. 6

    2020

  41. [49]

    Y . Chao, Y . Liu, X. Liu, H. Zeng, J. Deng. A Bench- mark for Recognizing Human-Object Interactions in Images. In ICCV, 2015. 2, 6, 1

    2015

  42. [50]

    K. He, X. Zhang, S. Ren, J. Sun. Deep residual learn- ing for image recognition. In CVPR, 2016

    2016

  43. [51]

    Loshchilov

    I. Loshchilov. Decoupled weight decay regularization. In ICLR, 2019. 6

    2019

  44. [52]

    Z. Li, X. Li, C. Ding, X. Xu. Disentangled Pre-training for Human-Object Interaction Detection. In CVPR,

  45. [53]

    Bansal, S

    A. Bansal, S. Rambhatla, A. Shrivastava, R. Chel- lappa. Detecting human-object interactions via func- tional generalization. In AAAI, 2020. 3

    2020

  46. [54]

    Gupta, A

    T. Gupta, A. Schwing, D. Hoiem. No-frills human- object interaction detection: Factorization, layout en- codings, and training techniques. In ICCV, 2019

    2019

  47. [55]

    Z. Hou, B. Yu, Y . Qiao, X. Peng, D. Tao. Affordance transfer learning for human-object interaction detec- tion. In CVPR, 2021. 2

    2021

  48. [56]

    Z. Hou, B. Yu, Y . Qiao, X. Peng, D. Tao. Detecting human-object interaction via fabricated compositional learning. In CVPR, 2021

    2021

  49. [57]

    Z. Hou, B. Yu, D. Tao. Discovering human-object in- teraction concepts via self-compositional learning. In ECCV, 2022

    2022

  50. [58]

    Peyre, I

    J. Peyre, I. Laptev, C. Schmid, J. Sivic. Detecting un- seen visual relations using analogies. In ICCV, 2019. 3

    2019

  51. [59]

    M. Lan, C. Chen, Y . Ke, X. Wang, L. Feng, W. Zhang. Clearclip: Decomposing clip representations for dense vision-language inference. In ECCV, 2024. 2

    2024

  52. [60]

    F. Wang, J. Mei, A. Yuille. Sclip: Rethinking self-attention for dense vision-language inference. In ECCV, 2024. 2

    2024

  53. [61]

    Y . Li, X. Liu, H. Lu, S. Wang, J. Liu, J. Li, C. Lu. Detailed 2d-3d joint representation for human-object interaction. In CVPR, 2020. 2

    2020

  54. [62]

    Y . Li, X. Liu, X. Wu, Y . Li, C. Lu. Hoi analysis: In- tegrating and decomposing human-object interaction. In NeurIPS, 2020. 2

    2020

  55. [63]

    Zhang, D

    F. Zhang, D. Campbell, S. Gould. Spatially condi- tioned graphs for detecting human-object interactions. In ICCV, 2021. 2

    2021

  56. [64]

    C. Zou, B. Wang, Y . Hu, J. Liu, Q. Wu, Y . Zhao, B. Li, C. Zhang, C. Zhang, Y . Wei, End-to-end human object interaction detection with hoi transformer. In CVPR,

  57. [65]

    Iftekhar, H

    A. Iftekhar, H. Chen, K. Kundu, X. Li, J. Tighe, D. Modolo. What to look at and where: Semantic and spatial refined transformer for detecting human-object interactions. In CVPR, 2022

    2022

  58. [66]

    J. Chen, K. Yanai. Qahoi: Query-based anchors for human-object interaction detection. In MVA, 2023. 6, 2

    2023

  59. [67]

    S. Ma, Y . Wang, S. Wang, Y . Wei. Fgahoi: Fine- grained anchors for human-object interaction detec- tion. In TPAMI, 2023. 6, 7, 2

    2023

  60. [68]

    B. Kim, J. Lee, J. Kang, E. Kim, H. Kim. Hotr: End- to-end human-object interaction detection with trans- formers. In CVPR, 2021. 2

    2021

  61. [69]

    T. Zhou, W. Wang, S. Qi, H. Ling, J. Shen. Cascaded 10 human-object interaction recognition. In CVPR, 2020. 2

    2020

  62. [70]

    Y . Liu, Q. Chen, A. Zisserman. Amplifying key cues for human-object-interaction detection. In ECCV,

  63. [71]

    Q. Dong, Z. Tu, H. Liao, Y . Zhang, V . Mahadevan, S. Soatto. Visual relationship detection using part-and- sum transformers with composite queries. In ICCV, 2021

    2021

  64. [72]

    M. Chen, Y . Liao, S. Liu, Z. Chen, F. Wang, C. Qian. Reformulating hoi detection as adaptive set prediction. In CVPR, 2021

    2021

  65. [73]

    Dosovitskiy, L

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weis- senborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, N. Houlsby. An image is worth 16x16 words: Transform- ers for image recognition at scale. In ICLR, 2021. 1, 3

    2021

  66. [74]

    S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhu, Grounding dino: Marrying dino with grounded pre-training for open-set object detec- tion. In ECCV, 2024

    2024

  67. [75]

    Zhang, S

    S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. Lin, Opt: Open pre-trained transformer language models. In arXiv:2205.01068, 2022. 3

    2022 arXiv

  68. [76]

    A. Radford. Improving language understanding by generative pre-training. In https : / / openai . com/blog/chatgpt/, 2018. 5

    2018

  69. [77]

    A. Radford. Improving language understanding by generative pre-training. In https : / / openai . com/blog/gpt-4/, 2023. 5, 8, 1

    2023

  70. [78]

    J. Gao, C. Cai, R. Wang, W. Liu, K. Yap, K. Garg, B. Han. CL-HOI: Cross-Level Human-Object Interaction Distillation from Vision Large Language Models. In arXiv:2410.15657, 2024. 2, 6

    2024 arXiv

  71. [79]

    T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, Microsoft coco: Common objects in con- text. In ECCV, 2014. 6

    2014

  72. [80]

    J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, J. Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. In arXiv:2308.12966, 2023. 8

    2023 arXiv

  73. [81]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, I. Polosukhin. Attention is all you need. In NeurIPS, 2017. 3

    2017

  74. [82]

    G. Wang, Y . Guo, Z. Xu, M. Kankanhalli. Bilateral adaptation for human-object interaction detection with occlusion-robustness. In CVPR, 2024. 2, 3, 6, 7

    2024

  75. [83]

    L. Li, W. Wang, Y . Yang. Human-object interaction detection collaborated with large relation-driven dif- fusion models. In NeurIPS, 2024

    2024

  76. [84]

    Q. Lei, B. Wang, R. Tan. EZ-HOI: VLM Adaptation via Guided Prompt Learning for Zero-Shot HOI De- tection. In NeurIPS, 2024. 2, 3, 6, 7

    2024

  77. [85]

    EF Only” and “EF+LSG

    R. Zeng, W. Huang, M. Tan, Y . Rong, P. Zhao, J. Huang, C. Gan. Graph Convolutional Module for Temporal Action Localization in Videos. In TPAMI, 2022. 11 Bilateral Collaboration with Large Vision-Language Models for Open V ocabulary Human-Object Interaction Detection Supplemen...

    2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.