Pith. sign in

REVIEW 5 major objections 5 minor 70 references

HOLa: Zero-Shot HOI Detection with Low-Rank Decomposed VLM Feature Adaptation

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Decomposing VLM text features into class-shared basis and class-specific weights, guided by LLM action descriptions, lifts zero-shot HOI detection to a new state of the art: 27.91 unseen-verb mAP on HICO-DET.

desk verdict Solid new SOTA for zero-shot HOI with a genuinely novel low-rank decomposition, but the key KL regularization in Eq. (5) does not type-check as written and the LLM-alignment assumption is unexamined. read the letter →

arxiv 2507.15542 v2 pith:2GKKQEHL submitted 2025-07-21 cs.CV

Qinqian Lei , Bo Wang , Robby T. Tan This is my paper
classification cs.CV
keywords zero-shotHOIdetectionhuman-objectinteractionvision-languagemodelslow-rankfeaturedecompositionLLM-derivedregularizationactiondistinctionHICO-DETunseengeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that zero-shot human-object interaction (HOI) detection — recognizing actions in images that never appeared during training — improves sharply when VLM text features are restructured rather than used directly. HOLa factorizes the text features of each HOI class into a class-shared basis and per-class weights, so the shared basis carries information that transfers to unseen classes while the per-class weights, adapted under supervision and steered by LLM-generated action descriptions, separate actions that share the same object. The paper reports a new state of the art across all four zero-shot HOI settings on HICO-DET, led by an unseen-class mAP of 27.91 in the unseen-verb setting. The central claim a sympathetic reader should take away is that the low-rank decomposition, not the VLM itself, is what unlocks unseen-action generalization.

What carries the argument

The engine is the pair of factorization identities $F = WB^{\top}$ for HOI text features and $F_a = W_a B_a^{\top}$ for action text features, coupled by sharing the basis: $B_a$ is a subset sampled from $B$. The class-shared basis $B$, kept out of the classification loss and updated only by reconstruction, is what preserves unseen-class information; the adapted per-class weights $W$, regularized toward $W_a$ by the KL term $D_{\text{KL}}[W_{ar} \| W_a]$, are what sharpen same-object action distinction. On the vision side, human-object tokens $T_{ho} = (f_h + f_o)/2 + f_{\text{spatial}}$ — mean human and object appearance features plus an MLP-encoded spatial vector — are pushed through the adapted VLM visual encoder so that interaction structure enters the similarity scores of Eq. (7).

What would settle it

Swap the LLM-generated action descriptions for generic, action-free templates (e.g., "a person doing something with an object") while keeping every other module and loss fixed; if unseen-verb mAP stays near 27.91 instead of dropping, the KL regularization is not actually injecting action semantics, and the alignment premise of the method fails.

Watch

Extended reading notes

Core claim

HOLa's central claim is that the tension in zero-shot HOI detection — generalizing to unseen classes while distinguishing actions on the same object — is resolved by writing VLM text features as a low-rank product $F \approx WB^{\top}$ and letting the two factors play different roles. The basis $B$ is shared by all HOI classes, is updated only by the reconstruction loss, and thus preserves the representation of unseen classes; the weights $W$ are per-class and are adapted by a text adapter under the action classification loss, letting them carry the details that separate "ride bike" from "jump bike". To stop this adaptation from overfitting seen actions, action text features are factorized into weights $W_a$ on a basis $B_a$ drawn from $B$, and the adapted HOI weights are pulled toward $W_a$ by KL divergence, injecting LLM-derived action knowledge into classes never annotated. In the vision branch, human-object tokens combining appearance and spatial cues enrich the visual encoder's interaction representation. The evidence offered is the HICO-DET tables: state-of-the-art unseen mAP in all four zero-shot settings, including 27.91 unseen-verb mAP, plus ablations attributing the gain to each component.

Load-bearing premise

The load-bearing premise is that the LLM's action descriptions, once encoded by the VLM and factorized into $W_a$, capture action structure that matches the visual encoder's notion of those actions; if the descriptions are generic, object-biased, or misaligned with visual features, the KL regularization pulls the adapted weights in a direction that does not help unseen actions, and the paper only checks this indirectly through end-task ablations.

Editorial extensions

If this is right

  • If the central claim holds, zero-shot HOI methods built on frozen VLM feature spaces have a large margin to recover: HOLa reports a +6.40 gain in unseen-verb mAP over the previous best.
  • The rank of the decomposition becomes a practical trade-off knob: lower ranks (e.g., 17) emphasize class-shared information and raise unseen mAP, while higher ranks (e.g., 119) retain more class detail and raise seen mAP.
  • The design carries over to fully supervised HOI detection, where the paper reports 35.41 mAP on HICO-DET with a ViT-B backbone and 39.05 mAP with ViT-L, above existing two-stage HOI methods.
  • The method needs only class names and LLM descriptions, not visual data of unseen classes, during training — a practical property when collecting interaction annotations is expensive.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The decomposition-plus-coupling recipe is a candidate template for other fine-grained VLM tasks, such as attribute-object composition or fine-grained action recognition, where the same failure mode appears: VLM features under-differentiate categories that share an object or noun.
  • The paper never directly measures whether the action weights $W_a$ are semantically aligned with visual action features, so a useful extension is to probe that alignment; if it is weak, the KL term may be acting as a generic regularizer rather than a carrier of action semantics.
  • An adaptive rank per action or per HOI class, informed by the rank-ablation trade-off, could plausibly push unseen-verb mAP above the fixed rank $m = 71$ used here.
  • The authors themselves flag that HOLa depends on predefined unseen class names; that constraint makes open-vocabulary HOI — where the class set is not fixed at training time — the natural stress test of whether the decomposition generalizes beyond enumerated names.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes HOLa, a zero-shot HOI detection method that decomposes VLM text features of HOI class descriptions into a low-rank class-shared basis B and per-class weights W, adapts W through a text adapter, regularizes the adapted weights with LLM-generated action descriptions via KL divergence, fuses action and object text features, and enriches visual features with human-object tokens. The method is evaluated on HICO-DET under unseen-verb, unseen-composition, and unseen-object zero-shot settings, reporting an unseen-verb mAP of 27.91 and claiming new state-of-the-art results across all settings, with ablations attributing the gains to the proposed modules.

Significance. If the results hold, HOLa makes a strong empirical contribution: it improves unseen-class mAP substantially over prior zero-shot HOI methods (e.g., +1.68 over CMMP in the unseen-verb setting) while using only 4.0M trainable parameters and releasing code. The low-rank decomposition of VLM text features is a principled way to preserve class-shared information, and the LLM-derived action regularization is a novel mechanism for transferring external action knowledge. The paper includes extensive ablations, visualizations, and comparisons with recent methods, and it reports competitive fully-supervised results as a bonus. However, the central regularization term in Eq. (5) is not well specified, an inconsistency exists regarding which LLM generates the HOI descriptions, and the reported numbers lack error bars; these issues currently prevent the stated state-of-the-art claim from being fully reproducible and statistically supported.

major comments (5)
  1. [Sec. 3.2, Eq. (5)] The KL regularization in Eq. (5) is not executable as written. W is N×m, W_a is N_a×k, and the index set I = {i | b_i ∈ B_a} selects basis vectors; the text states that W_ar contains the i-th column of the adapted matrix W, which has dimension N (or m, depending on the row/column convention), while W_a rows have dimension k. DKL[W_ar ∥ W_a] therefore compares objects of incompatible shape, and no normalization to a common probability space is specified. This is load-bearing: Table 5 shows that LLM-derived action regularization contributes +1.22 unseen mAP and is required to surpass CMMP's 26.23 in the unseen-verb setting. Please specify the exact shapes, the row/column convention for B and W, how indices in I map to rows of W_a, and how the KL is computed (e.g., row-wise softmax over which axis).
  2. [Sec. 3.2, Eq. (6)] The notation for the second action-regularization term is ambiguous: W_a is first introduced as the factorized action weights in Eq. (4), but then 'adapted action weights W_a' are regularized toward W_a in Eq. (6). It is unclear which copy is trainable, which is the frozen target, and whether the KL is over rows, columns, or the full matrix. Please clarify the roles of the two W_a symbols and align the notation with Eq. (5).
  3. [Sec. 3.2, Table 5] The mechanism assumes that CLIP text encodings of LLaMA-3 action descriptions are semantically aligned with visual action features, so that pushing W toward W_a improves unseen-action discrimination. The paper provides no direct evidence for this alignment; the Fig. 8 visualization shows weight similarities before and after adaptation but not similarity to W_a. Since the +1.22 mAP gain is the decisive margin, please include a control experiment (e.g., permuted action descriptions, generic descriptions, or random W_a targets) and a quantitative alignment measure between W_ar and W_a to demonstrate that the gain comes from action semantics rather than from generic regularization.
  4. [Sec. 3.1 vs. Supplementary C.1] Section 3.1 states that an LLM [9] (LLaMA-3) generates detailed HOI class descriptions that are encoded into F, but Supplementary C.1 states that HOI class descriptions are taken from EZ-HOI and generated with LLaVA, while LLaMA-3-8B is used only for action descriptions and vision-branch prior knowledge. This contradiction affects the reproducibility of the main text and the attribution of the method's gains; specify exactly which model generates which set of descriptions.
  5. [Sec. 4, Tables 1-4] All quantitative results are single runs, and numerous hyperparameters (α, β1-β4, γ1-γ2, τ, m, k) are tuned on the test benchmark without a validation split or error bars. Given that several SOTA margins are small (e.g., Table 3 unseen +1.16 over CMMP and Table 1 seen +1.60 over EZ-HOI), report mean±std over at least three seeds and describe the hyperparameter selection protocol to substantiate the claimed improvements.
minor comments (5)
  1. [Sec. 4, Implementation Details; Sec. 3.4] The implementation details and the inference paragraph refer to 'Eq. (13)' for the action score, but Eq. (13) is in the supplementary; the main-text score equation is Eq. (7) and the final HOI score is Eq. (10). Please fix the cross-references.
  2. [Sec. 3.2] B is defined as R^{d×m} with basis vectors as columns, yet b_i is called the i-th row of B; please clarify the indexing convention to avoid confusion with the subsequent column-based extraction of W.
  3. [Sec. 4.1, RF-UC paragraph] The text says HOLa exceeds CMMP by 1.59 mAP on unseen classes in the RF-UC setting, but Table 3 shows a difference of 1.16 (30.61 vs. 29.45); the 1.59 figure is the NF-UC margin and should be corrected.
  4. [Sec. 4.2, Table 5 reference] The main text refers to 'Table 15' for the VLM feature decomposition and adaptation ablation, but Table 15 appears only in the supplementary; renumber or explicitly state that the table is in the appendix.
  5. [Supplementary Table 16] The supplementary reports that varying β1 to 0.5 or β2 to 0.5 yields higher unseen mAP (28.32 and 28.81) than the chosen configuration (27.91), yet the default is selected based on full/seen mAP; please state the selection criterion explicitly so the hyperparameter choice is transparent.

Circularity Check

0 steps flagged · score 0.0 of 10

HOLa's SOTA claim rests on held-out unseen-class mAP and external baselines; the only self-citation is a description source, not a load-bearing proof.

full rationale

The claimed derivation is not circular. The target claim is unseen-class mAP on HICO-DET, measured against held-out HOI classes and published baselines; no equation in the paper defines this metric in terms of the fitted weights or basis features. The decomposition losses (Eqs. 1-4) reconstruct VLM text features and act as representation regularizers; the action-regularization terms (Eqs. 5-6) align adapted weights to an auxiliary W_a derived from action text, which is an input feature, not the evaluation score. The only same-author citation is the use of EZ-HOI [22] as the source of LLaVA-generated HOI descriptions (Supp. C.1); this supplies input text, not a load-bearing proof, and Table 7 ablates the contribution of LLM descriptions. The dimension mismatch in Eq. (5) and the unvalidated CLIP/LLM action alignment are correctness and reproducibility risks, not circular reductions; likewise C.6's reliance on predefined unseen class names is a scope limitation. Since the reported gains come from external held-out evaluation and component ablations (Tables 5 and 7), the central claim is not equivalent to its inputs by construction.

Assumptions & free parameters 6 free parameters · 5 assumptions · 2 invented entities

The method's behavior depends heavily on hand-set hyperparameters and factorization ranks, which are selected via ablations and reconstruction criteria on HICO-DET rather than derived from theory. The low-rank projection is the core modeling choice: all generalization claims pass through it. The main domain assumptions are the standard zero-shot protocol and the alignment of LLM text knowledge with visual action semantics.

free parameters (6)
  • m, rank of HOI basis features and weights = 71
    Selected from reconstruction percentages and HICO-DET ablations (Supp. Table 9) to balance seen and unseen performance.
  • k, rank of action basis features and weights = 35
    Set in implementation details; not ablated independently from the action decomposition.
  • gamma1 and gamma2 in action prediction score = 2.66, 2.66
    Hand-set coefficients in Eq. 7 that weight the two similarity terms in the action score.
  • alpha, weight of semantic loss = 80
    Chosen, according to Supp. Table 16, to keep loss terms on a comparable scale; affects final performance.
  • beta1, beta2, beta3, beta4 in feature decomposition loss = 0.1, 0.1, 0.001, 50
    Hand-set weights for reconstruction, sparsity, orthogonality, and action regularization losses; sensitivity shown in Supp. Table 16.
  • tau, temperature at inference = 2.8
    Set to 1 during training and 2.8 during inference following prior work; directly scales the final HOI score.
assumptions (5)
  • domain assumption Unseen HOI class names and their text descriptions are available during training, but visual annotations for unseen classes are not.
    This is the standard zero-shot protocol stated in Sec. 3, and it makes the text-side decomposition and regularization possible. If unseen class names were unavailable, the method could not be used as described.
  • domain assumption CLIP text features, after low-rank factorization and weight adaptation, provide a reliable semantic space for actions and objects.
    The entire language branch builds on this premise in Sec. 3.1. If the VLM text space does not separate actions, the decomposition and regularization cannot recover them.
  • domain assumption LLM-generated descriptions of actions faithfully capture visual action semantics.
    Action regularization in Eqs. 5-6 pushes adapted weights toward weights derived from LLaMA-3 descriptions. Incorrect or overly generic descriptions would misguide adaptation.
  • ad hoc to paper A low-rank subspace B of dimension 71 preserves enough class-shared and action-discriminative information.
    The rank is justified only by reconstruction percentages and HICO-DET ablations (Supp. Table 9), not by an independent principle. The entire generalization argument depends on this factorization being informative.
  • domain assumption Pretrained DETR supplies accurate human and object detections.
    The vision branch and the final HOI score in Eq. 10 multiply detector scores, so detector errors propagate to HOI predictions.
invented entities (2)
  • Human-object tokens (Tho)
    purpose: Encode appearance and spatial cues for each human-object pair and feed through the VLM visual encoder to refine interaction representations (Sec. 3.3).
    The tokens are internal model constructs; their effectiveness is shown only by internal ablations (Table 5 and Supp. Table 13), with no external falsifiable prediction.
  • Low-rank basis feature matrices B and Ba
    purpose: Serve as class-shared semantic bases for reconstructing VLM text features and regularizing action weights.
    These shared latent bases are the core new representation. There is no outside evidence that they correspond to interpretable action or object concepts, and the paper itself notes they are not directly interpretable (Supp. C.5).

how reviews work

0 comments
Cite this review

Pith. "Pith review of HOLa: Zero-Shot HOI Detection with Low-Rank Decomposed VLM Feature Adaptation." pith.science (2026). https://pith.science/paper/2GKKQEHL

@misc{pith2026250715542,
  author       = {Pith},
  title        = {Pith review of: HOLa: Zero-Shot HOI Detection with Low-Rank Decomposed VLM Feature Adaptation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2GKKQEHL}},
  note         = {Machine review of arXiv:2507.15542}
}
read the original abstract

Zero-shot human-object interaction (HOI) detection remains a challenging task, particularly in generalizing to unseen actions. Existing methods address this challenge by tapping Vision-Language Models (VLMs) to access knowledge beyond the training data. However, they either struggle to distinguish actions involving the same object or demonstrate limited generalization to unseen classes. In this paper, we introduce HOLa (Zero-Shot HOI Detection with Low-Rank Decomposed VLM Feature Adaptation), a novel approach that both enhances generalization to unseen classes and improves action distinction. In training, HOLa decomposes VLM text features for given HOI classes via low-rank factorization, producing class-shared basis features and adaptable weights. These features and weights form a compact HOI representation that preserves shared information across classes, enhancing generalization to unseen classes. Subsequently, we refine action distinction by adapting weights for each HOI class and introducing human-object tokens to enrich visual interaction representations. To further distinguish unseen actions, we guide the weight adaptation with LLM-derived action regularization. Experimental results show that our method sets a new state-of-the-art across zero-shot HOI settings on HICO-DET, achieving an unseen-class mAP of 27.91 in the unseen-verb setting. Our code is available at https://github.com/ChelsieLei/HOLa.

Figures

Figures reproduced from arXiv: 2507.15542 by the authors.

Figure 1
Figure 1. (a) HOI detection methods utilizing a frozen VLM fea [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of our HOLa. In the language branch, VLM text HOI features are decomposed into HOI basis features [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. VLM text feature decomposition illustration. We first [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Human-object token illustration in the vision branch. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Action dissimilarity comparison. The action dissimilar [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Visualization of unseen HOI predictions in the unseen [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Overview of our vision branch. The goal is to provide a comprehensive pool of de￾scriptive information for the human, object, and their pos￾sible configurations in various scenarios. Avoid limiting the scope by naming specific interactions or actions. Fo￾cus instead on…
Figure 8
Figure 8. Figure 8: (a) Weight subset similarity visualization related to [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Visualization of HOI predictions in the unseen-verb setting on HICO-DET. The purple bar indicates predictions for seen HOI [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Visualization of HOI predictions in the rare-first unseen-composition setting on HICO-DET. The purple bar indicates predictions [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Visualization of HOI predictions in the non-rare-first unseen-composition setting on HICO-DET. The purple bar indicates [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: Visualization of HOI predictions in the unseen-object setting on HICO-DET. The purple bar indicates predictions for seen HOI [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

70 extracted references · 50 canonical work pages

  1. [9]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 ,

    arXiv

  2. [1]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems , 35:23716–23736,

  3. [2]

    Detecting human-object interactions via functional generalization

    Ankan Bansal, Sai Saketh Rambhatla, Abhinav Shrivastava, and Rama Chellappa. Detecting human-object interactions via functional generalization. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 10460–10469,

    work page

  4. [3]

    Detecting any human-object interaction relationship: Universal hoi detector with spatial prompt learning on foundation models

    Yichao Cao, Qingfei Tang, Xiu Su, Song Chen, Shan You, Xiaobo Lu, and Chang Xu. Detecting any human-object interaction relationship: Universal hoi detector with spatial prompt learning on foundation models. Advances in Neural Information Processing Systems, 36, 2024. 1, 2, 3, 6, 7, 14

    work page 2024

  5. [4]

    End-to- end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to- end object detection with transformers. In European confer- ence on computer vision, pages 213–229. Springer, 2020. 3, 5, 6, 7, 12

    work page 2020

  6. [5]

    Learning to detect human-object interactions

    Yu-Wei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng. Learning to detect human-object interactions. In 2018 ieee winter conference on applications of computer vision (wacv), pages 381–389. IEEE, 2018. 6, 14

    work page 2018

  7. [6]

    Qahoi: query-based an- chors for human-object interaction detection

    Junwen Chen and Keiji Yanai. Qahoi: query-based an- chors for human-object interaction detection. arXiv preprint arXiv:2112.08647, 2021. 2

    arXiv 2021

  8. [7]

    Bottom-up 2d pose estimation via dual anatomical centers for small-scale persons

    Yu Cheng, Yihao Ai, Bo Wang, Xinchao Wang, and Robby T Tan. Bottom-up 2d pose estimation via dual anatomical centers for small-scale persons. Pattern Recognition, 139: 109403, 2023. 1

    work page 2023

Show all 70 references
  1. [8]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2021

  2. [10]

    Clip-adapter: Better vision-language models with feature adapters

    Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision, 132(2): 581–595, 2024. 3

    2024

  3. [11]

    Detecting and recognizing human-object interactions

    Georgia Gkioxari, Ross Girshick, Piotr Doll ´ar, and Kaiming He. Detecting and recognizing human-object interactions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8359–8367, 2018. 2

    2018

  4. [12]

    Open-vocabulary object detection via vision and language knowledge distillation

    Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.13921 ,

    arXiv

  5. [13]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 6

    2016

  6. [14]

    Vi- sual compositional learning for human-object interaction de- tection

    Zhi Hou, Xiaojiang Peng, Yu Qiao, and Dacheng Tao. Vi- sual compositional learning for human-object interaction de- tection. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceed- ings, Part XV 16 , pages 584–600. Springer, 2020. 1, 2...

    2020

  7. [15]

    Affordance transfer learning for human-object interaction detection

    Zhi Hou, Baosheng Yu, Yu Qiao, Xiaojiang Peng, and Dacheng Tao. Affordance transfer learning for human-object interaction detection. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 495–504, 2021. 3, 14, 19

    2021

  8. [16]

    Detecting human-object interaction via fab- ricated compositional learning

    Zhi Hou, Baosheng Yu, Yu Qiao, Xiaojiang Peng, and Dacheng Tao. Detecting human-object interaction via fab- ricated compositional learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14646–14655, 2021. 1, 2, 3, 14, 17

    2021

  9. [17]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 3

    2021 arXiv

  10. [18]

    Vi- sual prompt tuning

    Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Vi- sual prompt tuning. In European Conference on Computer Vision, pages 709–727. Springer, 2022. 2, 3

    2022

  11. [19]

    Maple: Multi-modal prompt learning

    Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19113–19122, 2023. 2, 3

    2023

  12. [20]

    Relational context learning for human-object interaction detection

    Sanghyun Kim, Deunsol Jung, and Minsu Cho. Relational context learning for human-object interaction detection. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 2925–2934, 2023. 2

    2023

  13. [21]

    Human action recognition and predic- tion: A survey

    Yu Kong and Yun Fu. Human action recognition and predic- tion: A survey. International Journal of Computer Vision , 130(5):1366–1401, 2022. 1

    2022

  14. [22]

    Qinqian Lei, Bo Wang, and Robby T. Tan. Ez-hoi: Vlm adaptation via guided prompt learning for zero-shot hoi de- tection. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 1, 2, 3, 6, 7, 14

    2024

  15. [23]

    Few-shot learn- ing from augmented label-uncertain queries in bongard-hoi

    Qinqian Lei, Bo Wang, and Robby T Tan. Few-shot learn- ing from augmented label-uncertain queries in bongard-hoi. In Proceedings of the AAAI Conference on Artificial Intelli- gence, pages 2974–2982, 2024. 1

    2024

  16. [24]

    Efficient adaptive human-object interac- tion detection with concept-guided memory

    Ting Lei, Fabian Caba, Qingchao Chen, Hailin Jin, Yuxin Peng, and Yang Liu. Efficient adaptive human-object interac- tion detection with concept-guided memory. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 6480–6490, 2023. 1, 2, 3, 6, 7, 1...

    2023

  17. [25]

    Exploring the po- tential of large foundation models for open-vocabulary hoi 9 detection

    Ting Lei, Shaofeng Yin, and Yang Liu. Exploring the po- tential of large foundation models for open-vocabulary hoi 9 detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 16657–16667, 2024. 18

    2024

  18. [26]

    Explor- ing conditional multi-modal prompts for zero-shot hoi de- tection

    Ting Lei, Shaofeng Yin, Yuxin Peng, and Yang Liu. Explor- ing conditional multi-modal prompts for zero-shot hoi de- tection. In European Conference on Computer Vision, pages 1–19. Springer, 2024. 1, 2, 3, 6, 7, 14

    2024

  19. [27]

    Blip: Bootstrapping language-image pre-training for uni- fied vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for uni- fied vision-language understanding and generation. In In- ternational Conference on Machine Learning, pages 12888– 12900. PMLR, 2022. 3

    2022

  20. [28]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023. 3

    2023 arXiv

  21. [29]

    Neural- logic human-object interaction detection

    Liulei Li, Jianan Wei, Wenguan Wang, and Yi Yang. Neural- logic human-object interaction detection. Advances in Neu- ral Information Processing Systems, 36, 2024. 2, 6, 7, 14

    2024

  22. [30]

    Gen-vlkt: Simplify association and enhance interaction understanding for hoi detection

    Yue Liao, Aixi Zhang, Miao Lu, Yongliang Wang, Xiaobo Li, and Si Liu. Gen-vlkt: Simplify association and enhance interaction understanding for hoi detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 20123–20132, 2022....

    2022

  23. [31]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceeding...

    2014

  24. [32]

    Focal loss for dense object detection

    Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Doll´ar. Focal loss for dense object detection. In Pro- ceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017. 5

    2017

  25. [33]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023. 3

    2023 arXiv

  26. [34]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916, 2023. 3, 14

    2023

  27. [35]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 14

    2017 arXiv

  28. [36]

    Discovering syntactic interaction clues for human-object interaction detection

    Jinguo Luo, Weihong Ren, Weibo Jiang, Xi’ai Chen, Qiang Wang, Zhi Han, and Honghai Liu. Discovering syntactic interaction clues for human-object interaction detection. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 28212–28222, 2024. 2

    2024

  29. [37]

    Clip4hoi: Towards adapting clip for practi- cal zero-shot hoi detection

    Yunyao Mao, Jiajun Deng, Wengang Zhou, Li Li, Yao Fang, and Houqiang Li. Clip4hoi: Towards adapting clip for practi- cal zero-shot hoi detection. Advances in Neural Information Processing Systems, 36, 2024. 1, 2, 3, 6, 7, 14

    2024

  30. [38]

    Simple open-vocabulary object detection

    Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, et al. Simple open-vocabulary object detection. In European Conference on Computer Vision, pages 728–755. S...

    2022

  31. [39]

    Hoiclip: Efficient knowledge transfer for hoi detection with vision-language models

    Shan Ning, Longtian Qiu, Yongfei Liu, and Xuming He. Hoiclip: Efficient knowledge transfer for hoi detection with vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 23507–23517, 2023. 1, 2, 3, 6, 7, 8, 14, 17

    2023

  32. [40]

    Viplo: Vision transformer based pose-conditioned self-loop graph for human-object interaction detection

    Jeeseung Park, Jin-Woo Park, and Jong-Seok Lee. Viplo: Vision transformer based pose-conditioned self-loop graph for human-object interaction detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17152–17162, 2023. 2

    2023

  33. [41]

    Distillation using oracle queries for transformer-based human-object interaction detection

    Xian Qu, Changxing Ding, Xingao Li, Xubin Zhong, and Dacheng Tao. Distillation using oracle queries for transformer-based human-object interaction detection. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 19558–19567, 2022. 2

    2022

  34. [42]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  35. [43]

    Regularizing cnns with locally constrained decorrelations

    Pau Rodr ´ıguez, Jordi Gonzalez, Guillem Cucurull, Josep M Gonfaus, and Xavier Roca. Regularizing cnns with locally constrained decorrelations. arXiv preprint arXiv:1611.01967, 2016. 4

    2016 arXiv

  36. [44]

    Test- time prompt tuning for zero-shot generalization in vision- language models

    Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test- time prompt tuning for zero-shot generalization in vision- language models. Advances in Neural Information Process- ing Systems, 35:14274–14289, 2022. 2, 3

    2022

  37. [45]

    Energy-based learning for scene graph generation

    Mohammed Suhail, Abhay Mittal, Behjat Siddiquie, Chris Broaddus, Jayan Eledath, Gerard Medioni, and Leonid Si- gal. Energy-based learning for scene graph generation. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 13936–13945, 2021. 1

    2021

  38. [46]

    Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks

    Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5227–5237,

  39. [47]

    Qpic: Query-based pairwise human-object interaction detec- tion with image-wide contextual information

    Masato Tamura, Hiroki Ohashi, and Tomoaki Yoshinaga. Qpic: Query-based pairwise human-object interaction detec- tion with image-wide contextual information. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10410–10419, 2021. 2

    2021

  40. [48]

    Bilateral adaptation for human-object interac- tion detection with occlusion-robustness

    Guangzhi Wang, Yangyang Guo, Ziwei Xu, and Mohan Kankanhalli. Bilateral adaptation for human-object interac- tion detection with occlusion-robustness. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27970–27980, 2024. 2

    2024

  41. [49]

    Exploring pose-aware human-object interaction via hybrid learning

    Eastman ZY Wu, Yali Li, Yuan Wang, and Shengjin Wang. Exploring pose-aware human-object interaction via hybrid learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17815– 17825, 2024. 2

    2024

  42. [50]

    End-to-end zero-shot hoi detec- 10 tion via vision and language knowledge distillation

    Mingrui Wu, Jiaxin Gu, Yunhang Shen, Mingbao Lin, Chao Chen, and Xiaoshuai Sun. End-to-end zero-shot hoi detec- 10 tion via vision and language knowledge distillation. In Pro- ceedings of the AAAI Conference on Artificial Intelligence , pages 2839–2846, 2023. 3, 14, 17

    2023

  43. [51]

    Category query learning for human-object interaction classification

    Chi Xie, Fangao Zeng, Yue Hu, Shuang Liang, and Yichen Wei. Category query learning for human-object interaction classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15275– 15284, 2023. 2

    2023

  44. [52]

    Chore: Contact, human and object reconstruction from a sin- gle rgb image

    Xianghui Xie, Bharat Lal Bhatnagar, and Gerard Pons-Moll. Chore: Contact, human and object reconstruction from a sin- gle rgb image. In European Conference on Computer Vision, pages 125–145. Springer, 2022. 1

    2022

  45. [53]

    Open-world human-object interaction detection via multi-modal prompts

    Jie Yang, Bingliang Li, Ailing Zeng, Lei Zhang, and Ruimao Zhang. Open-world human-object interaction detection via multi-modal prompts. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 16954–16964, 2024. 18

    2024

  46. [54]

    Mma: Multi-modal adapter for vision-language models

    Lingxiao Yang, Ru-Yuan Zhang, Yanchen Wang, and Xiao- hua Xie. Mma: Multi-modal adapter for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 23826– 23837, 2024. 3

    2024

  47. [55]

    Rlip: Rela- tional language-image pre-training for human-object interac- tion detection

    Hangjie Yuan, Jianwen Jiang, Samuel Albanie, Tao Feng, Ziyuan Huang, Dong Ni, and Mingqian Tang. Rlip: Rela- tional language-image pre-training for human-object interac- tion detection. Advances in Neural Information Processing Systems, 35:37416–37431, 2022. 2

    2022

  48. [56]

    Unified vision and language prompt learning

    Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Unified vision and language prompt learning. arXiv preprint arXiv:2210.07225, 2022. 2, 3

    2022 arXiv

  49. [57]

    Open-vocabulary object detection using captions

    Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih- Fu Chang. Open-vocabulary object detection using captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14393–14402, 2021. 1

    2021

  50. [58]

    Spatially conditioned graphs for detecting human-object in- teractions

    Frederic Z Zhang, Dylan Campbell, and Stephen Gould. Spatially conditioned graphs for detecting human-object in- teractions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13319–13327, 2021. 2, 3, 6

    2021

  51. [59]

    Efficient two-stage detection of human-object interactions with a novel unary-pairwise transformer

    Frederic Z Zhang, Dylan Campbell, and Stephen Gould. Efficient two-stage detection of human-object interactions with a novel unary-pairwise transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20104–20112, 2022. 2, 3, 6

    2022

  52. [60]

    Exploring predicate visual con- text in detecting of human-object interactions

    Frederic Z Zhang, Yuhui Yuan, Dylan Campbell, Zhuoyao Zhong, and Stephen Gould. Exploring predicate visual con- text in detecting of human-object interactions. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 10411–10421, 2023. 2

    2023

  53. [61]

    Perceiving 3d human-object spatial arrangements from a single image in the wild

    Jason Y Zhang, Sam Pepose, Hanbyul Joo, Deva Ramanan, Jitendra Malik, and Angjoo Kanazawa. Perceiving 3d human-object spatial arrangements from a single image in the wild. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceed- ings, P...

    2020

  54. [62]

    Tip-adapter: Training-free clip-adapter for better vision- language modeling

    Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free clip-adapter for better vision- language modeling. arXiv preprint arXiv:2111.03930, 2021. 3

    2021 arXiv

  55. [63]

    Xin Zhang and Robby T. Tan. Mamba as a bridge: Where vision foundation models meet vision language models for domain-generalized semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 14527–14537, 2025. 1

    2025

  56. [64]

    Conditional prompt learning for vision-language mod- els

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language mod- els. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition , pages 16816–16825,

  57. [65]

    Learning to prompt for vision-language models

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. In- ternational Journal of Computer Vision, 130(9):2337–2348,

  58. [66]

    End-to-end human object interaction detection with hoi transformer

    Cheng Zou, Bohan Wang, Yue Hu, Junqi Liu, Qian Wu, Yu Zhao, Boxun Li, Chenguang Zhang, Chi Zhang, Yichen Wei, et al. End-to-end human object interaction detection with hoi transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages...

    2021

  59. [67]

    - High- light the possible roles of specific body parts (e.g., hands gripping, feet pressing, or knees bending) without specify- ing actions

    **Human Body Description:** - Describe the posi- tioning and orientation of key body parts (e.g., hands, feet, arms, legs, torso, head) in relation to the object. - High- light the possible roles of specific body parts (e.g., hands gripping, feet pressing, or knees bending) wi...

  60. [68]

    LLM input humancupFFN DETR pairing … : finetuning : frozen : features : boxes : human-object tokens Cross Attention Layer 𝑉

    **Object Description:** - Provide a clear and con- cise description of the object, focusing on its relative size, shape, and structure compared to the human in the image. - Include details about key components (e.g., wheels, deck from skateboard) and their spatial relationship...

  61. [69]

    The head’s angle might vary, possibly tilted up or down, depending on the human’s position relative to the car’s height, such as looking over the roof or under the chassis

    Human Body Description: (a) Positioning and Orien- tation: - Head : The human’s head can be oriented in vari- ous directions relative to the car, such as facing towards the car, away from it, or to the sides. The head’s angle might vary, possibly tilted up or down, depending o...

  62. [70]

    hosing a car

    Object Description: (a) Size, Shape, and Structure: - General Structure : The car is a relatively large object com- pared to a human, with a defined structure typically includ- ing a body, wheels, and windows. It is generally elongated with a front (hood), middle (cabin), and ...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.