Pith. sign in

REVIEW 5 major objections 6 minor 57 references

VLM-HOI: Vision Language Models for Interpretable Human-Object Interaction Analysis

T0 review · 5 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A frozen vision-language model can serve as a training objective for HOI detection, beating CLIP-based knowledge transfer.

desk verdict A plausible VLM-distillation extension with a real practical upside, but the central loss is under-specified and the reported numbers don't line up; fixable, not trustworthy as written. read the letter →

arxiv 2411.18038 v1 pith:MAA6EBNC submitted 2024-11-27 cs.CV cs.AI

Donggoo Kang , Dasol Jeong , Hyunmin Lee , Sangwoo Park , Hasil Park , Sunkyu Kwon , Yeongjoon Kim , Joonki Paik This is my paper · ORCID
classification cs.CVcs.AI
keywords vision-languagemodelhuman-objectinteractiondetectionknowledgedistillationcontrastivelearningimage-textmatchingBLIPHOIDETR
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a frozen vision-language model can serve as the objective function for human-object interaction (HOI) detection, not just as a feature extractor. The proposed VLM-HOI pipeline converts each predicted HOI triplet into a template sentence like "A person hold a tennis racket", scores it against the image with BLIP's image-text matching, and uses those scores in a contrastive loss. The authors report state-of-the-art mAP on HICO-DET and V-COCO, surpassing GEN-VLKT, a CLIP-based knowledge-transfer method. If the claim holds, it shows that object-centric VLMs encode priors that can regularize and improve HOI prediction, especially for rare categories, without adding any computation at inference time.

What carries the argument

The key machinery is the image-text matching (ITM) score $\mathrm{sim}(I, S)$ of a frozen BLIP model, applied to template sentences "A person [verb] a [object]" constructed from predicted HOI triplets. These scores feed a contrastive loss $\mathcal{L}_{\mathrm{ITM}} = \sum_i \max(0, \alpha - \mathrm{sim}(I, s^+_i)) + \sum_j \mathrm{sim}(I, s^-_j)$, which regularizes a DETR-based HOI detector during training only. BLIP is chosen over CLIP because its object-centric, sub-word-level matching is claimed to separate positive from negative triplet sentences more sharply, as illustrated in the paper's comparison figures.

What would settle it

One concrete check would be to evaluate the frozen BLIP teacher alone on the HICO-DET test set: compute ITM scores for the ground-truth triplets (positive sentences) and for the same triplets with the verb randomly replaced (negative sentences), and measure the separation between the two score distributions. If the distributions overlap heavily, the contrastive loss cannot carry meaningful supervision, and the reported gains would not replicate. A second direct test is to replace the BLIP teacher with a randomly initialized BLIP of the same architecture; if VLM-HOI retains its advantage, the gains come from the extra loss term rather than from language priors.

Watch

Extended reading notes

Core claim

On the paper's terms, the discovery is that BLIP's image-text matching score, computed for short grounded template sentences describing predicted human-object-verb triplets, is a usable training signal for HOI detection. The score is turned into a contrastive loss with a positive margin that pulls matched triplets toward the image and pushes unmatched triplets away, and this loss is added to the standard HOI detection loss while the VLM stays frozen. Experiments on HICO-DET and V-COCO show improvements over prior methods, including the CLIP-based GEN-VLKT, with the largest gains on rare interaction categories. The authors argue this is the first use of a VLM's language abilities as an objective function for HOI detection.

Load-bearing premise

The load-bearing premise is that BLIP's image-text matching scores on short, grammatically awkward template sentences are reliable and discriminative enough to separate correct from incorrect HOI triplets, even though the sentences are unlike BLIP's natural-caption training data and the construction of negative sentences is left unspecified.

Editorial extensions

If this is right

  • A frozen VLM can act as a training-time teacher for HOI detection, so the deployed model has the same parameter count and inference speed as the baseline.
  • Object-centric VLMs such as BLIP may be better-suited teachers than CLIP for tasks that require localizing subjects and objects, not just global image-text alignment.
  • Rare HOI categories stand to gain the most from the VLM's language priors, since the contrastive signal regularizes categories with few training examples.
  • The template-based ITM objective could extend to other structured prediction tasks that can be expressed as (subject, verb, object) text.
  • The choice of the positive margin $\alpha$ is tied to the scale of the teacher's scores, so a principled setting of $\alpha$ depends on the score distribution of the chosen VLM.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same training-time objective could be applied to video HOI detection, using a video-language model's matching scores across frames, if such scores are discriminative enough.
  • Because the paper never specifies how negative sentences are built, an immediate test is whether hard negatives (verbs swapped among objects present in the image) give larger gains than random negatives; the contrastive loss's behavior will hinge on that choice.
  • A natural extension would be to use a larger, instruction-tuned VLM as the teacher; the paper's own argument implies gains should scale with the teacher's language understanding, which is testable by replacing BLIP with a larger VLM.
  • If BLIP's scores are sensitive to the awkward sentence template, then dataset-specific verb and object vocabulary changes could break the transfer, making prompt engineering part of the method's deployment cost.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes VLM-HOI, a method that uses a frozen BLIP image-text matching (ITM) model as an external objective for training a DETR-based HOI detector. Predicted HOI triplets are converted into template sentences, and BLIP's ITM scores for positive and negative sentences are combined into a contrastive loss. The authors report state-of-the-art results on HICO-DET and V-COCO. However, as written, the objective is underspecified and internally contradictory: Eq. (2) defines positive and negative sentences identically, Eq. (4) contradicts its own prose, and reported numbers disagree between text and tables. These issues prevent the reader from reproducing or verifying the claimed contribution.

Significance. The underlying idea--using a frozen, object-centric VLM as a training objective that is not needed at inference time--is of interest to the HOI community and is a natural extension of CLIP-based distillation methods. The qualitative evidence in Fig. 1 that BLIP ITM is more discriminative than CLIP for short, ungrammatical triplets is suggestive. If the objective were properly specified and the numbers reconciled, the method could be a useful baseline for future work. The current manuscript, however, does not yet provide a well-defined training loss, and the reported gains cannot be attributed to the proposed mechanism. No code is released, which further limits reproducibility.

major comments (5)
  1. [Section 3.3, Eq. (2)] The definition of positive and negative sentences is identical: s+_i and s-_i are both ["A person" + v_i + " a " + o_i], differing only in the condition T in H vs T not in H. Since no actual construction of Sneg is given--e.g., which verbs/objects are swapped, whether negatives are sampled from the dataset, or whether they are the same template with random labels--the loss in Eq. (4) is not a well-defined function of the model predictions. This must be specified before the reported gains can be attributed to the proposed objective.
  2. [Section 3.4, Eq. (4) and surrounding text] The text states that positive similarity scores should be "as close to zero as possible" and negative scores "as high as possible," which is the opposite of Eq. (4), where positive scores are pushed above margin alpha and negative scores are minimized. The direction of the objective must be corrected and made consistent with Figure 1, which shows positive scores as high. As written, the training objective is ambiguous and cannot be used to interpret the experimental results.
  3. [Section 4.4, Table 1 and text] The text reports 33.64% mAP on HICO-DET Default, while Table 1 reports 34.25; similarly, the V-COCO margins over GEN-VLKT are stated as 7.1 and 7.7 AP, whereas Table 2 implies 5.3 and 6.5 AP. These numeric inconsistencies affect the central claim of improvement over GEN-VLKT and must be reconciled before the results can be verified.
  4. [Section 4.4, Tables 1-2] The state-of-the-art claim is based on an incomplete baseline set. The tables omit RLIPv2 and HOICLIP, both of which are closely related language-supervised HOI methods cited in the text, and the exclusion of Swin-based models is stated only in the table caption while the abstract and title make an unqualified SOTA claim. The comparison should be completed or the claim narrowed to match the actual baselines.
  5. [Section 4.5, Table 3] The positive margin alpha is selected using V-COCO test performance, with no validation set described; the reported AP values in Table 3 are test-set numbers. This selection-on-test procedure can inflate the reported gains and should be clarified or replaced by validation-based hyperparameter selection, especially since alpha is a key component of the proposed loss.
minor comments (6)
  1. [Section 2.2 heading] The heading "Vision-Lauguage Model" contains a typo; it should read "Vision-Language Model."
  2. [Section 2.1] The sentence "Another content [16,18,26,34,47,52] recent work has focused on improving both branches of HOI detection" is ungrammatical and should be rephrased.
  3. [Section 3.3, Eq. (2)] The notation is inconsistent: the text defines object and interaction class names as o_i and v_i, but Eq. (2) uses hat v_i and hat o_i; unify the notation.
  4. [Figures 1 and 5] Figures 1 and 5 lack axis labels and score scales, so the claimed typical positive-score range of 1 to 2 in Section 4.5 cannot be verified.
  5. [Abstract] The claim of "first utilization of VLM language abilities for HOI detection" is too strong given the cited GEN-VLKT and RLIP works; the claim should be qualified, e.g., "first to use a frozen image-text matching VLM as a training objective for HOI detection."
  6. [Table 5] The parameter counts are confusing: the table lists 430M (DETR+BLIP) and 28M (Learnable), while the text says BLIP has 361M and the baseline has 41M DETR plus 28M decoder; clarify the composition of these numbers.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the BLIP ITM supervision is an external, frozen signal and the reported gains are not forced by construction.

full rationale

The paper's claimed derivation is: predicted HOI triplets are converted into grounded text prompts; a frozen, pretrained BLIP model computes image-text matching scores; these scores enter a contrastive loss that trains the HOI detector. None of these steps reduce to the paper's own outputs or conclusions. The BLIP model is external, pretrained on general image-text data, and kept frozen during training, so its scores are not a function of the HOI model's parameters or of the benchmark labels in a way that would make the reported mAP a tautology. The contrastive loss is a genuine training signal, even if, as written, Eq. (2) defines positive and negative sentences identically and Eq. (4)'s direction contradicts the surrounding prose; those are specification and reproducibility problems, not circular reasoning. The margin alpha is tuned on the validation split (Table 3), which is standard hyperparameter selection rather than a fitted input renamed as a prediction. There are no load-bearing self-citations: the cited BLIP work, GEN-VLKT, and baseline methods are external prior work by other authors. The paper's comparisons are against external benchmarks and prior methods, so the central claim is self-contained against outside evidence. The internal inconsistencies are serious correctness risks, but they do not constitute circularity under the definitions used here.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method relies on standard supervised HOI training plus a distillation signal from a frozen BLIP model. The main free parameter is the margin alpha in the contrastive loss, tuned on V-COCO validation. The key domain assumptions concern the reliability of BLIP ITM scores for template sentences and the unspecified construction of negative examples. No new entities are introduced.

free parameters (2)
  • positive margin alpha = 1 (tuned on V-COCO)
    Table 3 reports that alpha=1 gives the best V-COCO AP; the loss in Eq. (4) depends directly on this margin, so the central result is partly tuned on the evaluation benchmark.
  • loss weighting coefficients lambda_i = 2.5, 1, 1, 1
    These weights for L1, GIoU, object classification, and interaction classification are set in Section 4.1 following prior work; they are chosen hyperparameters but not the focus of the paper.
assumptions (4)
  • domain assumption BLIP's image-text matching scores on template HOI sentences provide a valid, transferable supervision signal for HOI detection.
    The entire distillation loss in Eq. (4) assumes that higher ITM scores correlate with correct HOI triplets, even for awkward prompts (Section 3.3-3.4).
  • domain assumption The simple template 'A person [verb] a [object]' preserves enough semantics for the VLM to discriminate correct from incorrect interactions.
    Section 3.3 constructs all positive and negative sentences with this template; no linguistic analysis is provided.
  • domain assumption The negative sentences Sneg are sampled in a way that provides useful contrastive signal.
    The paper never specifies how negative sentences are generated, yet the loss in Eq. (4) sums scores over an unstated negative set.
  • domain assumption HOI annotations in HICO-DET and V-COCO are correct and complete for training and evaluation.
    Standard benchmark assumption; the method and all comparisons rely on these labels (Section 4.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of VLM-HOI: Vision Language Models for Interpretable Human-Object Interaction Analysis." pith.science (2026). https://pith.science/paper/MAA6EBNC

@misc{pith2026241118038,
  author       = {Pith},
  title        = {Pith review of: VLM-HOI: Vision Language Models for Interpretable Human-Object Interaction Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MAA6EBNC}},
  note         = {Machine review of arXiv:2411.18038}
}
read the original abstract

The Large Vision Language Model (VLM) has recently addressed remarkable progress in bridging two fundamental modalities. VLM, trained by a sufficiently large dataset, exhibits a comprehensive understanding of both visual and linguistic to perform diverse tasks. To distill this knowledge accurately, in this paper, we introduce a novel approach that explicitly utilizes VLM as an objective function form for the Human-Object Interaction (HOI) detection task (\textbf{VLM-HOI}). Specifically, we propose a method that quantifies the similarity of the predicted HOI triplet using the Image-Text matching technique. We represent HOI triplets linguistically to fully utilize the language comprehension of VLMs, which are more suitable than CLIP models due to their localization and object-centric nature. This matching score is used as an objective for contrastive optimization. To our knowledge, this is the first utilization of VLM language abilities for HOI detection. Experiments demonstrate the effectiveness of our method, achieving state-of-the-art HOI detection accuracy on benchmarks. We believe integrating VLMs into HOI detection represents important progress towards more advanced and interpretable analysis of human-object interactions.

Figures

Figures reproduced from arXiv: 2411.18038 by the authors.

Figure 1
Figure 1. Comparison of Image-Text Matching scores and CLIP similarity for various HOI triplets. While CLIP struggles to capture semantic relationships due to its reliance on simple prompts, VLM models effectively distinguish between positive and negative HOI triplets despite not receiving complete sentences as input. that describes the relationship between them. For example, in an image of a person riding a bike, the subject… view at source ↗
Figure 2
Figure 2. Visualization of attention maps for BLIP [21] and CLIP [35] models. Both models exhibit accurate localization in the first example. However, in the second ex￾ample, CLIP focuses solely on individual word locations, failing to capture the broader context of the input sentence. In light of this, we propose a novel approach that leverages the capabilities of large VLM as a distillation form for the HOI detection task (… view at source ↗
Figure 3
Figure 3. The overview of our proposed VLM-HOI. The network consists of a DETR￾based encoder and a query-based transformer decoder. Predicted HOI triplets are matched positive and negative Then these sets are converted into text form. The image￾text matching task of VLM computes the matching score of these text sets. distillation in localization tasks. Prior works [26, 32] have utilized CLIP as a teacher model, achieving nota… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Comparison of Baseline [18] and proposed method with example queries. Given a verb query q v i , we visualize the top two most confident predictions, including their corresponding activation maps and bounding boxes [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Analysis of Image-Text Matching Scores on HOI Detection Benchmarks. This figure visualizes the image-text similarity scores computed between visual input and corresponding grounded sentence prompts [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 31 canonical work pages

  1. [1]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., Zhang, L.: Bottom-up and top-down attention for image captioning and visual question answering. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 6077–6086 (2018)

    2018

  2. [2]

    In: Proceedings of the IEEE international conference on computer vision

    Antol,S.,Agrawal,A.,Lu,J.,Mitchell,M.,Batra,D.,LawrenceZitnick,C.,Parikh, D.: Vqa: Visual question answering. In: Proceedings of the IEEE international conference on computer vision. pp. 2425–2433 (2015)

    work page 2015

  3. [3]

    IEEE Transactions on Pattern Analysis and Machine Intelligence 41(2), 423–443 (2019)

    Baltrušaitis, T., Ahuja, C., Morency, L.P.: Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence 41(2), 423–443 (2019)

    work page 2019

  4. [4]

    Advances in neural information processing systems33, 1877–1901 (2020)

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Nee- lakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems33, 1877–1901 (2020)

    2020

  5. [5]

    In: European conference on computer vision

    Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End- to-end object detection with transformers. In: European conference on computer vision. pp. 213–229. Springer (2020)

    2020

  6. [6]

    Chen, J., Yanai, K.: Qahoi: Query-based anchors for human-object interaction de- tection.In:202318thInternationalConferenceonMachineVisionandApplications (MVA). pp. 1–5. IEEE (2023)

    work page 2023

  7. [7]

    arXiv preprint arXiv:1504.00325 (2015)

    Chen, X., Fang, H., Lin, T.Y., Vedantam, R., Gupta, S., Dollar, P., Zitnick, C.L.: Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325 (2015)

    arXiv 2015

  8. [8]

    In: European conference on computer vision

    Chen, Y.C., Li, L., Yu, L., El Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., Liu, J.: Uniter: Universal image-text representation learning. In: European conference on computer vision. pp. 104–120. Springer (2020)

    2020

Show all 57 references
  1. [9]

    arXiv preprint arXiv:2204.02311 (2022)

    Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H.W., Sutton, C., Gehrmann, S., et al.: Palm: Scaling lan- guage modeling with pathways. arXiv preprint arXiv:2204.02311 (2022)

    2022 arXiv

  2. [10]

    Dai, W., Li, J., Li, D., Tiong, A.M.H., Zhao, J., Wang, W., Li, B., Fung, P., Hoi, S.: Instructblip: Towards general-purpose vision-language models with instruction tuning (2023)

    2023

  3. [11]

    arXiv preprint arXiv:1810.04805 (2018)

    Devlin,J.,Chang,M.W.,Lee,K.,Toutanova,K.:Bert:Pre-trainingofdeepbidirec- tional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)

    2018 arXiv

  4. [12]

    In: British Machine Vision Conference (2018) 16 Kang et al

    Gao, C., Zou, Y., Huang, J.B.: ican: Instance-centric attention network for human- object interaction detection. In: British Machine Vision Conference (2018) 16 Kang et al

    2018

  5. [13]

    Gupta,S.,Malik,J.:Visualsemanticrolelabeling.arXivpreprintarXiv:1505.04474 (2015)

    2015 arXiv

  6. [14]

    In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XV 16

    Hou, Z., Peng, X., Qiao, Y., Tao, D.: Visual compositional learning for human- object interaction detection. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XV 16. pp. 584–

    2020

  7. [15]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Karpathy, A., Fei-Fei, L.: Deep visual-semantic alignments for generating image de- scriptions. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3128–3137 (2015)

    2015

  8. [16]

    In: Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition

    Kim, B., Lee, J., Kang, J., Kim, E.S., Kim, H.J.: Hotr: End-to-end human-object interaction detection with transformers. In: Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition. pp. 74–83 (2021)

    2021

  9. [17]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Kim, B., Mun, J., On, K.W., Shin, M., Lee, J., Kim, E.S.: Mstr: Multi-scale trans- former for end-to-end human-object interaction detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19578– 19587 (2022)

    2022

  10. [18]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Kim, S., Jung, D., Cho, M.: Relational context learning for human-object interac- tion detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2925–2934 (2023)

    2023

  11. [19]

    In: arXiv preprint arXiv:1411.2539 (2014)

    Kiros, R., Salakhutdinov, R., Zemel, R.S.: Unifying visual-semantic embeddings with multimodal neural language models. In: arXiv preprint arXiv:1411.2539 (2014)

    2014 arXiv

  12. [20]

    arXiv preprint arXiv:2301.12597 (2023)

    Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 (2023)

    2023 arXiv

  13. [21]

    In: International Con- ference on Machine Learning

    Li, J., Li, D., Xiong, C., Hoi, S.: Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In: International Con- ference on Machine Learning. pp. 12888–12900. PMLR (2022)

    2022

  14. [22]

    Advances in neural information processing systems34, 9694–9705 (2021)

    Li,J.,Selvaraju,R.,Gotmare,A.,Joty,S.,Xiong,C.,Hoi,S.C.H.:Alignbeforefuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems34, 9694–9705 (2021)

    2021

  15. [23]

    arXiv preprint arXiv:1908.03557 (2019)

    Li, L.H., Yatskar, M., Yin, D., Hsieh, C.J., Chang, K.W.: Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557 (2019)

    2019 arXiv

  16. [24]

    Advances in Neural Information Processing Systems33, 5011–5022 (2020)

    Li, Y.L., Liu, X., Wu, X., Li, Y., Lu, C.: Hoi analysis: Integrating and decomposing human-object interaction. Advances in Neural Information Processing Systems33, 5011–5022 (2020)

    2020

  17. [25]

    In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion

    Liao, Y., Liu, S., Wang, F., Chen, Y., Qian, C., Feng, J.: Ppdm: Parallel point detection and matching for real-time human-object interaction detection. In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion. pp. 482–490 (2020)

    2020

  18. [26]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Liao, Y., Zhang, A., Lu, M., Wang, Y., Li, X., Liu, S.: Gen-vlkt: Simplify asso- ciation and enhance interaction understanding for hoi detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 20123–20132 (2022)

    2022

  19. [27]

    In: Computer Vision– ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13

    Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Computer Vision– ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. pp. 740–755...

    2014

  20. [28]

    arXiv preprint arXiv:2304.08485 (2023) VLM-HOI: Vision Language Model for Human-Object Interaction 17

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. arXiv preprint arXiv:2304.08485 (2023) VLM-HOI: Vision Language Model for Human-Object Interaction 17

    2023 arXiv

  21. [29]

    arXiv preprint arXiv:1711.05101 (2017)

    Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)

    2017 arXiv

  22. [30]

    In: Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition

    Lu, J., Goswami, V., Rohrbach, M., Parikh, D., Lee, S.: 12-in-1: Multi-task vision and language representation learning. In: Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition. pp. 10437–10446 (2019)

    2019

  23. [31]

    arXiv preprint arXiv:2301.04019 (2023)

    Ma, S., Wang, Y., Wang, S., Wei, Y.: Fgahoi: Fine-grained anchors for human- object interaction detection. arXiv preprint arXiv:2301.04019 (2023)

    2023 arXiv

  24. [32]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Ning, S., Qiu, L., Liu, Y., He, X.: Hoiclip: Efficient knowledge transfer for hoi de- tection with vision-language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 23507–23517 (2023)

    2023

  25. [33]

    OpenAI: Gpt-4 technical report (2023)

    2023

  26. [34]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Park, J., Park, J.W., Lee, J.S.: Viplo: Vision transformer based pose-conditioned self-loop graph for human-object interaction detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 17152– 17162 (2023)

    2023

  27. [35]

    In: International conference on machine learning

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021)

    2021

  28. [36]

    arXiv preprint arXiv:1908.08530 (2019)

    Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., Dai, J.: Vl-bert: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv:1908.08530 (2019)

    2019 arXiv

  29. [37]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Tamura, M., Ohashi, H., Yoshinaga, T.: Qpic: Query-based pairwise human-object interaction detection with image-wide contextual information. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10410–10419 (2021)

    2021

  30. [38]

    In: arXiv preprint arXiv:1908.07490 (2019)

    Tan,H.,Bansal,M.:Lxmert:Learningcross-modalityencoderrepresentationsfrom transformers. In: arXiv preprint arXiv:1908.07490 (2019)

    2019 arXiv

  31. [39]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Vinyals, O., Toshev, A., Bengio, S., Erhan, D.: Show and tell: A neural image caption generator. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3156–3164 (2015)

    2015

  32. [40]

    In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion

    Wang, T., Yang, T., Danelljan, M., Khan, F.S., Zhang, X., Sun, J.: Ipnet. In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion. pp. 4116–4125 (2020)

    2020

  33. [41]

    In: Proceedings of the European conference on computer vision (ECCV)

    Xiao, T., Liu, Y., Zhou, B., Jiang, Y., Sun, J.: Unified perceptual parsing for scene understanding. In: Proceedings of the European conference on computer vision (ECCV). pp. 418–434 (2018)

    2018

  34. [42]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Xie, X., Bhatnagar, B.L., Pons-Moll, G.: Visibility aware human-object interaction tracking from single rgb camera. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4757–4768 (2023)

    2023

  35. [43]

    In: International conference on machine learning

    Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., Zemel, R., Bengio, Y.: Show, attend and tell: Neural image caption generation with visual attention. In: International conference on machine learning. pp. 2048–2057 (2015)

    2015

  36. [44]

    In: Proceedings of the AAAI conference on ar- tificial intelligence

    Yan, S., Xiong, Y., Lin, D.: Spatial temporal graph convolutional networks for skeleton-based action recognition. In: Proceedings of the AAAI conference on ar- tificial intelligence. vol. 32 (2018)

    2018

  37. [45]

    In: Proceedings of the European conference on computer vision (ECCV)

    Yao,T.,Pan,Y.,Li,Y.,Mei,T.:Exploringvisualrelationshipforimagecaptioning. In: Proceedings of the European conference on computer vision (ECCV). pp. 684– 699 (2018)

    2018

  38. [46]

    arXiv preprint arXiv:2205.01917 (2022) 18 Kang et al

    Yu, J., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., Wu, Y.: Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917 (2022) 18 Kang et al

    2022 arXiv

  39. [47]

    Ad- vances in Neural Information Processing Systems35, 37416–37431 (2022)

    Yuan, H., Jiang, J., Albanie, S., Feng, T., Huang, Z., Ni, D., Tang, M.: Rlip: Re- lational language-image pre-training for human-object interaction detection. Ad- vances in Neural Information Processing Systems35, 37416–37431 (2022)

    2022

  40. [48]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Yuan, H., Zhang, S., Wang, X., Albanie, S., Pan, Y., Feng, T., Jiang, J., Ni, D., Zhang, Y., Zhao, D.: Rlipv2: Fast scaling of relational language-image pre-training. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 21649–21661 (2023)

    2023

  41. [49]

    Advances in Neural Information Processing Systems 34, 17209–17220 (2021)

    Zhang, A., Liao, Y., Liu, S., Lu, M., Wang, Y., Gao, C., Li, X.: Mining the ben- efits of two-stage and one-stage hoi detection. Advances in Neural Information Processing Systems 34, 17209–17220 (2021)

    2021

  42. [50]

    In: Proceedings of the IEEE/CVF International Con- ference on Computer Vision

    Zhang, F.Z., Campbell, D., Gould, S.: Spatially conditioned graphs for detecting human-object interactions. In: Proceedings of the IEEE/CVF International Con- ference on Computer Vision. pp. 13319–13327 (2021)

    2021

  43. [51]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Zhang, F.Z., Campbell, D., Gould, S.: Efficient two-stage detection of human- object interactions with a novel unary-pairwise transformer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 20104– 20112 (2022)

    2022

  44. [52]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Zhang, F.Z., Yuan, Y., Campbell, D., Zhong, Z., Gould, S.: Exploring predicate visual context in detecting of human-object interactions. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10411–10421 (2023)

    2023

  45. [53]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition

    Zhang, Y., Pan, Y., Yao, T., Huang, R., Mei, T., Chen, C.W.: Exploring structure- aware transformer over interaction proposals for human-object interaction detec- tion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition. pp. 19548–19557 (2022)

    2022

  46. [54]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Zhong, X., Qu, X., Ding, C., Tao, D.: Glance and gaze: Inferring action-aware points for one-stage human-object interaction detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13234– 13243 (2021)

    2021

  47. [55]

    Zhou,D.,Liu,Z.,Wang,J.,Wang,L.,Hu,T.,Ding,E.,Wang,J.:Human-objectin- teractiondetectionviadisentangledtransformer.In:ProceedingsoftheIEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19568–19577 (2022)

    2022

  48. [56]

    In: Proceedings of the AAAI conference on artificial intelligence

    Zhou, L., Palangi, H., Zhang, L., Hu, H., Corso, J., Gao, J.: Unified vision-language pre-training for image captioning and vqa. In: Proceedings of the AAAI conference on artificial intelligence. vol. 34, pp. 13041–13049 (2020)

    2020

  49. [57]

    In: Proceedings of the IEEE/CVF conference on computer vision and pat- tern recognition

    Zou, C., Wang, B., Hu, Y., Liu, J., Wu, Q., Zhao, Y., Li, B., Zhang, C., Zhang, C., Wei, Y., et al.: End-to-end human object interaction detection with hoi trans- former. In: Proceedings of the IEEE/CVF conference on computer vision and pat- tern recognition. pp. 11825–11834 (2021)

    2021

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.