Pith. sign in

REVIEW 3 major objections 5 minor 71 references

Dynamic Scoring with Enhanced Semantics for Training-Free Human-Object Interaction Detection

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A training-free HOI detector can beat all training-free rivals and most supervised methods on rare interactions by enriching CLIP's verb semantics with LLM-generated interaction signatures and dynamically reweighting four attention heads…

desk verdict Useful training-free HOI recipe, but the rare-class win leans on a labeled registry and the headline numbers have a reproducibility wrinkle. read the letter →

arxiv 2507.17456 v1 pith:MELIFXSH submitted 2025-07-23 cs.CV

Francesco Tonini , Lorenzo Vaquero , Alessandro Conti , Cigdem Beyan , Elisa Ricci This is my paper
classification cs.CV
keywords human-objectinteractiondetectiontraining-freevision-languagemodelssignaturesmulti-headattentionmultimodalregistryrareinteractionszero-shot
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that human-object interaction (HOI) detection can be done without training a dedicated detector, and that the main obstacle—vision-language models like CLIP represent objects well but verbs poorly—can be fixed by enriching the textual side. The proposed DYSCO generates, for each interaction, a set of 'signatures': numerous LLM-written descriptions of what the action looks like, encoded by CLIP's text encoder and used as keys in an attention-based classifier. Prediction is recast as a four-head attention process whose contributions are reweighted per interaction by a small orchestrator module. DYSCO reports the best results among training-free methods on HICO-DET, reaching 28.24% Full mAP with a ViT-L/14 backbone and beating most fully supervised methods on rare interactions. That matters because HOI annotations are expensive and inconsistent, and rare interactions are exactly where supervised training struggles.

What carries the argument

The load-bearing pieces are the interaction signatures and the multi-head predictor. A signature is a matrix of M=50 textual descriptions per interaction, produced by an LLM from parameterized templates and encoded by CLIP's text encoder, giving each verb-object pair a set of action-centric key vectors instead of one fixed label embedding. The predictor is a four-head attention mechanism where each head computes a_h = (q_h k_h^T) v_h; keys come from the signatures (textual fine and coarse heads) or from a small visual registry of up to J=8 labeled human-object crop pairs per interaction plus union-context crops (visual instance and contextual heads). A negative bias N_h = -(q_h k_h^T)(1 - v_h) sharpens the contrast between visually similar interactions. The Multi-head Orchestrator Module then reweights the four heads per interaction via a temperature-scaled softmax contribution matrix, so that, for example, 'drink with' leans on visual instance evidence while 'pour' favors contextual and coarse text cues.

What would settle it

Run DYSCO with the registry emptied (J=0, equivalent to keeping only text heads, or use the label-free variant dysco-lf) on the 138 rare HICO-DET categories; if Rare mAP drops to or below the ADA-CM baseline of 27.24%, then the reported rare-class advantage is carried by the labeled memory rather than by the semantic enrichment, and the training-free claim as stated fails. A cleaner test: evaluate on a held-out set of novel verb-object compositions with no registry entries at all and compare against text-head-only performance.

Watch

Extended reading notes

Core claim

On its own terms, the central claim is that a training-free HOI detector can match supervised performance by fixing two weaknesses of prior zero-shot pipelines: CLIP's weak grasp of verbs, and the static, equal weighting of visual and textual evidence. DYSCO's interaction signatures replace the single verb label with a matrix of M=50 LLM-generated scene descriptions, giving each interaction a stratified manifold in CLIP space rather than a single point. Its multi-head predictor treats each human-object pair as a query against four key sets—the fine-grained signatures, their average, a small visual registry of labeled crop pairs, and union-context crops—and the Multi-head Orchestrator Module computes a temperature-scaled contribution matrix so that, per interaction, the most informative heads are amplified. With negative bias added to the visual heads to separate visually similar actions, the method reaches 34.22% Rare and 28.24% Full mAP on HICO-DET with ViT-L/14, surpassing the previous training-free state of the art and all training-based methods except SICHOI and BCOM on the Rare split.

Load-bearing premise

The visual heads depend on a small registry of labeled human-object crop pairs (up to J=8 per interaction); if no labeled examples exist for an interaction or a new domain, those heads contribute no useful signal and the claimed training-free advantage rests entirely on the text heads' assumption that CLIP's text space aligns with visual actions.

Editorial extensions

If this is right

  • DYSCO is the new training-free state of the art on HICO-DET, surpassing ADA-CM on Full, Rare, and Non-Rare mAP across every tested backbone.
  • On the Rare split, DYSCO outperforms all training-based methods in the comparison except SICHOI and BCOM, showing that a training-free pipeline can close most of the gap on long-tailed interactions.
  • The visual registry needs only a handful of labeled pairs per interaction (J=8), and performance keeps improving up to J=128, so adapting to a new domain may require only a tiny annotation effort.
  • The label-free variant dysco-lf, which builds its registry from confident text-head predictions or MLLM-generated pseudolabels, still reaches 24.38% Full mAP, indicating graceful degradation when all labels are removed.
  • Injecting the interaction signatures into plain CLIP and LongCLIP baselines improves their HOI performance, suggesting the signature procedure is a portable plug-in independent of the multi-head predictor.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The signature-generation recipe may transfer to other verb-centric vision-language tasks such as action recognition and video captioning, where CLIP-style encoders also under-represent actions and LLM-written descriptions could supply the missing verb semantics.
  • The per-interaction reweighting pattern learned by MhOM—'drink with' trusting visual instance heads while 'pour' favors contextual and coarse text—suggests that even supervised HOI models could benefit from verb-dependent fusion weights rather than a single fixed combination.
  • Because DYSCO's performance keeps improving with registry size up to J=128, unlike ADA-CM which degrades beyond J=16, the honest comparison with prior work may understate what the visual heads can do; a version with a larger registry might climb further on rare classes.
  • Several qualitative failure cases are instances where DYSCO's prediction looks more plausible than the ground-truth label (reading vs opening a book, jumping with vs training a horse), which hints that part of the remaining 'error' is annotation noise rather than model weakness.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents DYSCO, a training-free human-object interaction (HOI) detection method that combines a frozen DETR object detector, CLIP visual and textual encoders, LLM-generated interaction signatures, and a multimodal registry of labeled human-object crops. Candidate human-object pairs are scored by four attention heads (textual fine-grained, textual coarse, visual instance, visual contextual) with an additional negative bias, and the head outputs are combined by a Multi-head Orchestrator Module (MhOM) with a temperature parameter. On HICO-DET and V-COCO, the authors report that DYSCO outperforms prior training-free methods and is competitive with training-based methods, especially on rare HOI classes. The paper also reports ablations, backbone comparisons, zero-shot results, and a label-free variant (dysco-lf).

Significance. If the reported numbers are correct, DYSCO would be a meaningful advance for training-free HOI detection: it is one of the first methods to systematically combine LLM-derived textual signatures, a visual registry, and sample-adaptive head weighting, and the paper provides code, multiple backbone comparisons, and component ablations. However, the strength of the empirical claims is tempered by an internal numerical inconsistency and by the fact that the rare-class advantage relies on a registry that, for HICO-DET rare categories, contains nearly the entire labeled training support. The label-free variant falls below ADA-CM, so the 'training-free' advantage is not yet cleanly established.

major comments (3)
  1. [Table 8; Sec. 4.5] The full DYSCO configuration is reported as 25.75% Full mAP in Table 8, but Tables 3 and 5 report 26.21% for the same ViT-B/16 configuration. This inconsistency directly affects the central comparison with ADA-CM (25.20%) and the interpretation of the ablations, because the claimed gain in Sec. 4.3 is either 1.01 points or 0.55 points depending on which number is correct. Please reconcile the two numbers, clarify which configuration corresponds to the headline result, and state whether one table uses a different evaluation protocol.
  2. [Sec. 4.1, Sec. 4.2, Eq. (6), Table 1, Table 8] The registry R is set to J=8 visual samples per interaction (Sec. 4.2), while rare HICO-DET categories are defined as having fewer than 10 training samples (Sec. 4.1). For a typical rare interaction, R therefore contains 80-100% of the available training support, and Eq. (6) reads out one-hot interaction labels from exactly that support. The reported rare-class mAP of 34.22% (Table 1) is thus partially a transductive retrieval result rather than an independent generalization result. The paper's own label-free ablation supports this reading: dysco-lf drops to 24.38% Full mAP (Table 8), below ADA-CM's 25.20%, and its rare mAP falls from 30.53% to 29.29%. To substantiate the central claims, please report rare-class mAP for dysco-lf together with the main comparison, show the effect of J separately for rare categories (e.g., J=1,2), and state clearly which 'surpasses' claims require the labeled registry.
  3. [Sec. 4.2, Sec. 4.4, Supp. Fig. 6, Table 4] The free hyperparameters are selected directly on the HICO-DET test set: tau is tuned in Table 4, M is 'empirically determined' in Sec. 4.4, and J is explored in Supp. Fig. 6 with J=8 chosen in line with prior art rather than on a validation split. This selection on the test benchmark can inflate the reported gains and weakens the comparison with ADA-CM, whose hyperparameters were likely chosen on the same benchmark. Please either provide a validation-split protocol for choosing tau, M, and J, or explicitly acknowledge this limitation and report the sensitivity of the main conclusion over the explored ranges.
minor comments (5)
  1. [Table 7] In the M=10 row, the AFull value 24.80% is inconsistent with neighboring rows (27.45-27.73%) and is likely a typographical error; please correct it.
  2. [Table 8] The table caption contains the typo 'Performace'; it should be 'Performance'.
  3. [Sec. 4.3] The sentence claiming that DYSCO 'surpasses all training-based methods but [28,50]' should be qualified by the backbone (ViT-L/14 in Table 1) and by the registry setting, since the ablations in Tables 3 and 5 use ViT-B/16.
  4. [Supp. Fig. 6] The plots show performance continuing to improve up to J=128, yet the paper fixes J=8; please justify this choice with a validation protocol or discuss why the larger registry is not used.
  5. [Sec. 1 / Sec. 3] The term 'training-free' is used for a method that requires a registry of labeled interaction examples; the paper should define this term explicitly to avoid ambiguity about what 'training-free' means.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation found: DYSCO's registry-based attention is a transductive support-set classifier, not an equation that reduces to its own inputs.

full rationale

I checked the derivation chain in Sections 3.1-3.3 and the experimental protocol in Sections 4.2-4.5. The central scoring formula, Eq. (6), computes an attention-weighted average of one-hot interaction labels over a support set, with the test pair as query. This is a transductive nearest-neighbor-style classifier: the labels in v_h are inputs, and the predicted probabilities in Eq. (8) are functions of similarities between test features and registry/signature features. The output is not forced to equal the support labels by construction, because the query features come from held-out test images and the similarity weights are nonzero across classes. The interaction signatures are generated from GPT-4 descriptions of verb/object labels and then matched to CLIP image features; this is a standard zero-shot label-to-vision mapping, not a self-referential reduction. The MhOM module adaptively reweights the same head outputs; it introduces no fitted parameter that would make the final score equal to a benchmark answer by definition. The registry size J=8, temperature tau=0.1, and signature dimensionality M=50 are selected empirically, which raises a possible benchmark-tuning/overfitting concern, but that is a separate experimental-validity issue, not derivational circularity. I also note an internal reporting inconsistency: the full DYSCO configuration is reported as 26.21% in Tables 3 and 5 but as 25.75% in Table 8; this is a numerical inconsistency that should be corrected, but it does not constitute a circular step. No load-bearing self-citations, imported uniqueness theorems, or ansatz-by-citation chains are present in the paper. The claim that the method is 'training-free' while using a labeled support registry is a terminology and generalization question, not a circularity of the derivation.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on three families of assumptions: (1) CLIP's text encoder aligns LLM-generated verb descriptions with visual features; (2) a small labeled registry is representative; (3) DETR proposals are sufficient. The main reported numbers also depend on hyperparameters selected on the HICO-DET test set.

free parameters (4)
  • MhOM temperature tau = 0.1
    Controls sharpness of the contribution matrix (Eq. 7); value chosen based on HICO-DET results in Table 4, with no separate validation split.
  • Signature dimensionality M = 50
    Number of templates per interaction in Sec. 3.1; 'empirically determined' in Sec. 4.4 via Table 7 on HICO-DET.
  • Registry size J = 8
    Maximum labeled samples per interaction in visual heads (Sec. 4.2); supplement Fig. 6 shows J=128 gives higher Full mAP, so J=8 is a selected trade-off.
  • DETR confidence threshold and sample bounds = 0.2; 3 to 15
    Detection filtering and sampling choices in Sec. 4.2; they affect pair enumeration and final scores.
assumptions (4)
  • domain assumption CLIP's shared embedding space is invariant to non-semantic content and linearly encodes semantic concepts (Eq. 1).
    Sec. 3.1 states 'we can reasonably infer that CLIP captures semantic concepts while remaining invariant'; this is unproved and load-bearing for the text-head scores.
  • domain assumption GPT-4-generated interaction descriptions are semantically grounded and align CLIP text embeddings with visual interaction features.
    Sec. 3.1 uses GPT-4 as psi to produce descriptions; no evidence beyond qualitative t-SNE and end-task gains, and no exact prompt or model version is specified.
  • domain assumption A small labeled registry of human-object crops with one-hot labels is representative of each interaction class.
    Sec. 3.3 uses registry R to build visual keys and label values; with J=8 samples per class, coverage of rare poses and contexts is not assured.
  • domain assumption Frozen DETR proposals contain all relevant humans and objects; missed detections cannot be recovered by later heads.
    Sec. 3.2 constructs pairs only from DETR outputs; a missed human or object removes all related interactions from consideration.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dynamic Scoring with Enhanced Semantics for Training-Free Human-Object Interaction Detection." pith.science (2026). https://pith.science/paper/MELIFXSH

@misc{pith2026250717456,
  author       = {Pith},
  title        = {Pith review of: Dynamic Scoring with Enhanced Semantics for Training-Free Human-Object Interaction Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MELIFXSH}},
  note         = {Machine review of arXiv:2507.17456}
}
read the original abstract

Human-Object Interaction (HOI) detection aims to identify humans and objects within images and interpret their interactions. Existing HOI methods rely heavily on large datasets with manual annotations to learn interactions from visual cues. These annotations are labor-intensive to create, prone to inconsistency, and limit scalability to new domains and rare interactions. We argue that recent advances in Vision-Language Models (VLMs) offer untapped potential, particularly in enhancing interaction representation. While prior work has injected such potential and even proposed training-free methods, there remain key gaps. Consequently, we propose a novel training-free HOI detection framework for Dynamic Scoring with enhanced semantics (DYSCO) that effectively utilizes textual and visual interaction representations within a multimodal registry, enabling robust and nuanced interaction understanding. This registry incorporates a small set of visual cues and uses innovative interaction signatures to improve the semantic alignment of verbs, facilitating effective generalization to rare interactions. Additionally, we propose a unique multi-head attention mechanism that adaptively weights the contributions of the visual and textual features. Experimental results demonstrate that our DYSCO surpasses training-free state-of-the-art models and is competitive with training-based approaches, particularly excelling in rare interactions. Code is available at https://github.com/francescotonini/dysco.

Figures

Figures reproduced from arXiv: 2507.17456 by the authors.

Figure 1
Figure 1. We introduce dysco, a training-free Human-Object Interaction (HOI) detector that leverages a multimodal registry enriched with fine-grained interaction representations denoted as signatures. Unlike ADA-CM [18], the only existing training-free model, which relies mainly on visual features, dysco integrates multimodal data and adaptively reweights multimodal head scores based on the unique characteristics of each test… view at source ↗
Figure 2
Figure 2. Our dysco. We begin by generating novel interaction signatures , which enhance the semantic information of textual categories. We also utilize a object detector to identify humans and objects in the image and extract visual features from the crops of the detected human, object, and their union bounding box. A set of attention heads then processes the features of the test sample alongside those of the registry of int… view at source ↗
Figure 3
Figure 3. Using standard HOI textual information for the pre [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: shows the interactions predicted from both our dysco and ADA￾CM [18] on some samples of the HICO-DET dataset. Our predictions consistently outperform those of ADA-CM [18], aligning with the quantitative results. Please refer to the Supp. Mat.) for additional examples …
Figure 5
Figure 5. Figure 5: Behavior of our MhOM applied to objects and verbs of the HICO-DET dataset. dataset. For example, in the case of the object “bottle”, the verb “drink with” gets the most attention from the visual heads, while “lick” relies on all heads, and “pouring” favors contextual v…
Figure 6
Figure 6. Figure 6: Effect of the registry R size on dysco. Tested on HICO￾DET [9] using ViT-B/16 as backbone. 8 Additional experiments on label-free HOI Following Sec. 4.5, we present supplementary experiments eval￾uating dysco-lf under varying confidence thresholds. As shown in Tab. 9, …
Figure 7
Figure 7. Figure 7: T-SNE representations of interaction signatures for the objects (a) chair, (b) sheep, (c) person, (d) bird, (e) book, (f) broccoli, [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Qualitative results of our dysco (top) and ADA-CM [18] (bottom). Bold is ground-truth, while red bar is the top-1 prediction [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Failure cases of our dysco (top) and ADA-CM [18] (bottom). Bold is ground-truth, while red bar is the top-1 prediction [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

71 extracted references · 67 canonical work pages

  1. [1]

    Abdelrahman Abdelhamed, Mahmoud Afifi, and Alec Go. 2024. What Do You See? Enhancing Zero-Shot Image Classification with Multimodal Large Language Models. CoRR abs/2405.15668 (2024), 1–13

    arXiv 2024

  2. [2]

    Calmon, and Himabindu Lakkaraju

    Usha Bhalla, Alex Oesterling, Suraj Srinivas, Flávio P. Calmon, and Himabindu Lakkaraju. 2024. Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE).In Adv.NeuralInf.Process.Syst.(NeurIPS) ,Vol.37.CurranAssociates, Inc., Vancouver, BC, Canada, 84298–84328

    work page 2024

  3. [3]

    Yichao Cao, Qingfei Tang, Xiu Su, Song Chen, Shan You, Xiaobo Lu, and Chang Xu. 2023. Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Models. InAdv. Neural Inf. Process. Syst. (NeurIPS), Vol. 36. Curran Associates, Inc., New Orleans, LA, USA, 739–751

    work page 2023

  4. [4]

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020. End-to-End Object Detection with Trans- formers. InEuropean Conf. Comput. Vis. (ECCV), Vol. 12346. Springer, Glasgow, UK, 213–229

    work page 2020

  5. [5]

    Learning to Detect Human-Object Interactions

    Yu-WeiChao,YunfanLiu,XieyangLiu,HuayiZeng,andJiaDeng.2018. Learning to Detect Human-Object Interactions. InIEEE Winter Conf. Appl. Comp. Vis. (WACV). IEEE Computer Society, Lake Tahoe, NV, USA, 381–389

    work page 2018

  6. [6]

    Yuan Chen, Zi-han Ding, Ziqin Wang, Yan Wang, Lijun Zhang, and Si Liu. 2024. Asynchronous large language model enhanced planner for autonomous driving. In European Conf. Comput. Vis. (ECCV), Vol. 15094. Springer, Milan, Italy, 22–38

    work page 2024

  7. [7]

    Thomas Fel, Victor Boutin, Louis Béthune, Rémi Cadène, Mazda Moayeri, Léo Andéol, Mathieu Chalvidal, and Thomas Serre. 2023. A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimation. In Adv. Neural Inf. Process. Syst. (NeurIPS), Vol. 36. Curran Associates, Inc., New Orleans, LA, USA, 54805–54818

    work page 2023

  8. [8]

    Chen Gao, Jiarui Xu, Yuliang Zou, and Jia-Bin Huang. 2020. DRG: Dual Relation Graph for Human-Object Interaction Detection. InEuropean Conf. Comput. Vis. (ECCV), Vol. 12357. Springer, Glasgow, UK, 696–712

    work page 2020

Show all 71 references
  1. [9]

    Saurabh Gupta and Jitendra Malik. 2015. Visual Semantic Role Labeling.CoRR abs/1505.04474 (2015), 1–11

    2015 arXiv

  2. [10]

    Zhi Hou, Baosheng Yu, Yu Qiao, Xiaojiang Peng, and Dacheng Tao. 2021. Detectinghuman-objectinteractionviafabricatedcompositionallearning.In IEEE Conf. Comput. Vis. Pattern Recog. (CVPR). Computer Vision Foundation / IEEE, Virtual, 14646–14655

    2021

  3. [11]

    Kyle Hsu, William Dorrell, James C. R. Whittington, Jiajun Wu, and Chelsea Finn

  4. [12]

    Bumsoo Kim, Taeho Choi, Jaewoo Kang, and Hyunwoo J. Kim. 2020. UnionDet: Union-Level Detector Towards Real-Time Human-Object Interaction Detection. InEuropean Conf. Comput. Vis. (ECCV). Springer, Glasgow, UK, 498–514

    2020

  5. [13]

    Bumsoo Kim, Junhyun Lee, Jaewoo Kang, Eun-Sol Kim, and Hyunwoo J. Kim

  6. [14]

    Bumsoo Kim, Jonghwan Mun, Kyoung-Woon On, Minchul Shin, Junhyun Lee, and Eun-Sol Kim. 2022. MSTR: Multi-Scale Transformer for End-to-End Human- Object Interaction Detection. InIEEE Conf. Comput. Vis. Pattern Recog. (CVPR). IEEE, New Orleans, LA, USA, 19556–19565

    2022

  7. [15]

    Dong-Jin Kim, Xiao Sun, Jinsoo Choi, Stephen Lin, and In So Kweon. 2020. De- tectingHuman-ObjectInteractionswithActionCo-occurrencePriors.In European Conf. Comput. Vis. (ECCV). Springer, Glasgow, UK, 718–736

    2020

  8. [16]

    RelationalContextLearning for Human-Object Interaction Detection

    SanghyunKim,DeunsolJung,andMinsuCho.2023. RelationalContextLearning for Human-Object Interaction Detection. InIEEE Conf. Comput. Vis. Pattern Recog. (CVPR). IEEE, Vancouver, BC, Canada, 2925–2934

    2023

  9. [17]

    ConceptBottleneckModels.In Int.Conf.Mach

    PangWeiKoh,ThaoNguyen,YewSiangTang,StephenMussmann,EmmaPierson, BeenKim,andPercyLiang.2020. ConceptBottleneckModels.In Int.Conf.Mach. Learn. (ICML). PMLR, Virtual, 5338–5348

    2020

  10. [18]

    Ting Lei, Fabian Caba, Qingchao Chen, Hailin Jin, Yuxin Peng, and Yang Liu

  11. [19]

    Llava-onevision: Easy visual task transfer.CoRR abs/2408.03326 (2024), 1–43

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al.2024. Llava-onevision: Easy visual task transfer.CoRR abs/2408.03326 (2024), 1–43

    2024 arXiv

  12. [20]

    Huadong Li, Ying Wei, Shuailei Ma, Mingyu Chen, and Ge Li. 2024. Ripple Transformer:AHuman-ObjectInteractionBackboneandaNewPredictionStrategy for Smart Surveillance Devices.IEEE Trans. Consumer Electron.70, 1 (2024), 2257–2268

    2024

  13. [21]

    InIEEE Int

    Efficient Adaptive Human-Object Interaction Detection with Concept- guided Memory. InIEEE Int. Conf. Comput. Vis. (ICCV). IEEE, Paris, France, 6457–6467

  14. [22]

    Yong-Lu Li, Xinpeng Liu, Han Lu, Shiyi Wang, Junqi Liu, Jiefeng Li, and Cewu Lu. 2020. Detailed 2D-3D Joint Representation for Human-Object Interaction. In IEEE Conf. Comput. Vis. Pattern Recog. (CVPR). Computer Vision Foundation / IEEE, Seattle, WA, USA, 10163–10172

    2020

  15. [23]

    Yong-Lu Li, Xinpeng Liu, Xiaoqian Wu, Xijie Huang, Liang Xu, and Cewu Lu

  16. [24]

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. 2022. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Under- standing and Generation. InInt. Conf. Mach. Learn. (ICML), Vol. 162. PMLR, Baltimore, Maryland, USA, 12888–12900

    2022

  17. [25]

    Yue Liao, Aixi Zhang, Miao Lu, Yongliang Wang, Xiaobo Li, and Si Liu. 2022. GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detection. InIEEE Conf. Comput. Vis. Pattern Recog. (CVPR). IEEE, New Orleans, LA, USA, 20091–20100

    2022

  18. [26]

    InACM Multimedia (ACMMM)

    YeLiu,JunsongYuan,andChangWenChen.2020.ConsNet:LearningConsistency Graph for Zero-Shot Human-Object Interaction Detection. InACM Multimedia (ACMMM). ACM, Seattle, WA, USA, 4235–4243

    2020

  19. [27]

    Stich, and Martin Jaggi

    Francesco Locatello, Anant Raj, Sai Praneeth Karimireddy, Gunnar Rätsch, Bernhard Schölkopf, Sebastian U. Stich, and Martin Jaggi. 2018. On Matching Pursuit and Coordinate Descent. InInt. Conf. Mach. Learn. (ICML). PMLR, Stockholm, Sweden, 3204–3213

    2018

  20. [28]

    PPDM: Parallel Point Detection and Matching for Real-Time Human-Object Interaction Detection

    YueLiao,SiLiu,FeiWang,YanjieChen,ChenQian,andJiashiFeng.2020. PPDM: Parallel Point Detection and Matching for Real-Time Human-Object Interaction Detection. InIEEE Conf. Comput. Vis. Pattern Recog. (CVPR). Computer Vision Foundation / IEEE, Seattle, WA, USA, 479–487

    2020

  21. [29]

    Valentino Maiorca, Luca Moschella, Antonio Norelli, Marco Fumero, Francesco Locatello, and Emanuele Rodolà. 2023. Latent Space Translation via Semantic Alignment.In Adv.NeuralInf.Process.Syst.(NeurIPS) ,Vol.36.CurranAssociates, Inc., New Orleans, LA, USA, 55394–55414

    2023

  22. [30]

    Yunyao Mao, Jiajun Deng, Wengang Zhou, Li Li, Yao Fang, and Houqiang Li

  23. [31]

    Tomás Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013. Linguistic Regularities in Continuous Space Word Representations. InConfer. North Americ. Chap. Assoc. Comp. Ling.: Human Lang. Tech. (NAACL-HLT). The Association for Computational Linguistics, Atlanta, Georgia, USA, 746–751

    2013

  24. [32]

    Jinguo Luo, Weihong Ren, Weibo Jiang, Xi’ai Chen, Qiang Wang, Zhi Han, and Honghai Liu. 2024. Discovering Syntactic Interaction Clues for Human-Object Interaction Detection. InIEEE Conf. Comput. Vis. Pattern Recog. (CVPR). IEEE, Seattle, WA, USA, 28212–28222

    2024

  25. [33]

    Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodolà. 2023. Relative representations enable zero-shot latent space communication. InInt. Conf. Learn. Represent. (ICLR). OpenRe- view.net, Kigali, Rwanda, 1–26

    2023

  26. [34]

    HOICLIP:Efficient Knowledge Transfer for HOI Detection with Vision-Language Models

    ShanNing,LongtianQiu,YongfeiLiu,andXumingHe.2023. HOICLIP:Efficient Knowledge Transfer for HOI Detection with Vision-Language Models. InIEEE Conf. Comput. Vis. Pattern Recog. (CVPR). IEEE, Vancouver, BC, Canada, 23507–23517

    2023

  27. [35]

    InAdv.NeuralInf.Process.Syst.(NeurIPS) ,Vol.36.CurranAssociates,Inc.,New Orleans, LA, USA, 45895–45906

    CLIP4HOI: Towards Adapting CLIP for Practical Zero-Shot HOI Detection. InAdv.NeuralInf.Process.Syst.(NeurIPS) ,Vol.36.CurranAssociates,Inc.,New Orleans, LA, USA, 45895–45906

  28. [36]

    OpenAI. 2023. GPT-4 Technical Report.CoRR abs/2303.08774 (2023), 1–100

    2023 arXiv

  29. [37]

    LilianeMomeni,MathildeCaron,ArshaNagrani,AndrewZisserman,andCordelia Schmid. 2023. Verbs in Action: Improving verb understanding in video-language models. InIEEE Int. Conf. Comput. Vis. (ICCV). IEEE, Paris, France, 15533– 15545

    2023

  30. [38]

    Kiho Park, Yo Joong Choe, and Victor Veitch. 2024. The Linear Representation Hypothesis and the Geometry of Large Language Models. InInt. Conf. Mach. Learn. (ICML). OpenReview.net, Vienna, Austria, 1–24

    2024

  31. [39]

    Judea Pearl. 2021. Radical empiricism and machine learning research.J. Causal Inference9, 1 (2021), 78–82

    2021

  32. [40]

    ASIF:CoupledDataTurnsUnimodalMod- els to Multimodal without Training

    Antonio Norelli, Marco Fumero, Valentino Maiorca, Luca Moschella, Emanuele Rodolà,andFrancescoLocatello.2023. ASIF:CoupledDataTurnsUnimodalMod- els to Multimodal without Training. InAdv. Neural Inf. Process. Syst. (NeurIPS), Vol. 36. Curran Associates, Inc., New Orleans, LA, U...

    2023

  33. [41]

    Whatdoesaplaty- pus look like? Generating customized prompts for zero-shot image classification

    SarahM.Pratt,IanCovert,RosanneLiu,andAliFarhadi.2023. Whatdoesaplaty- pus look like? Generating customized prompts for zero-shot image classification. InIEEE Int. Conf. Comput. Vis. (ICCV). IEEE, Paris, France, 15645–15655

    2023

  34. [42]

    Jeeseung Park, Jin-Woo Park, and Jong-Seok Lee. 2023. ViPLO: Vision Trans- former Based Pose-Conditioned Self-Loop Graph for Human-Object Interaction Detection. InIEEE Conf. Comput. Vis. Pattern Recog. (CVPR). IEEE, Vancouver, BC, Canada, 17152–17162

    2023

  35. [43]

    Girshick, and Jian Sun

    Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. 2017. Faster R-CNN: TowardsReal-TimeObjectDetectionwithRegionProposalNetworks. IEEETrans. Pattern Anal. Mach. Intell.39, 6 (2017), 1137–1149

    2017

  36. [44]

    Paul Hongsuck Seo, Arsha Nagrani, Anurag Arnab, and Cordelia Schmid. 2022. End-to-End Generative Pretraining for Multimodal Video Captioning. InIEEE Conf. Comput. Vis. Pattern Recog. (CVPR). IEEE, New Orleans, LA, USA, 17959–17968

    2022

  37. [45]

    Erika Petersen and Christopher Potts. 2023. Lexical Semantics with Large Language Models: A Case Study of English "break". InFind. Assoc. Comp. Linguist. (EACL). Association for Computational Linguistics, Dubrovnik, Croatia, 490–511. MM ’25, October 27–31, 2025, Dublin, Irelan...

    2023

  38. [46]

    Frederik Träuble, Anirudh Goyal, Nasim Rahaman, Michael Curtis Mozer, Kenji Kawaguchi, Yoshua Bengio, and Bernhard Schölkopf. 2023. Discrete Key-Value Bottleneck. InInt. Conf. Mach. Learn. (ICML). PMLR, Honolulu, Hawaii, USA, 34431–34455

    2023

  39. [47]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. InInt. Conf. M...

    2021

  40. [48]

    Oytun Ulutan, A. S. M. Iftekhar, and B. S. Manjunath. 2020. VSGNet: Spatial Attention Network for Detecting Human Object Interactions Using Graph Convo- lutions. InIEEE Conf. Comput. Vis. Pattern Recog. (CVPR). Computer Vision Foundation / IEEE, Seattle, WA, USA, 13614–13623

    2020

  41. [49]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. InAdv. Neural Inf. Process. Syst. (NIPS), Vol. 30. Curran Associates, Inc., Long Beach, CA, USA, 5998–6008

    2017

  42. [50]

    Masato Tamura, Hiroki Ohashi, and Tomoaki Yoshinaga. 2021. QPIC: Query- BasedPairwiseHuman-ObjectInteractionDetectionWithImage-WideContextual Information.In IEEEConf.Comput.Vis.PatternRecog.(CVPR) .ComputerVision Foundation / IEEE, Virtual, 10410–10419

    2021

  43. [51]

    RecoveryofSparse Signals via Generalized Orthogonal Matching Pursuit: A New Analysis.IEEE Trans

    JianWang,SuhyukKwon,PingLi,andByonghyoShim.2016. RecoveryofSparse Signals via Generalized Orthogonal Matching Pursuit: A New Analysis.IEEE Trans. Signal Process.64, 4 (2016), 1076–1089

    2016

  44. [52]

    Danyang Tu, Xiongkuo Min, Huiyu Duan, Guodong Guo, Guangtao Zhai, and Wei Shen. 2022. Iwin: Human-Object Interaction Detection via Transformer with Irregular Windows. InEuropean Conf. Comput. Vis. (ECCV). Springer, Tel Aviv, Israel, 87–103

    2022

  45. [53]

    Tiancai Wang, Tong Yang, Martin Danelljan, Fahad Shahbaz Khan, Xiangyu Zhang, and Jian Sun. 2020. Learning Human-Object Interaction Detection Using Interaction Points. InIEEE Conf. Comput. Vis. Pattern Recog. (CVPR). Computer Vision Foundation / IEEE, Seattle, WA, USA, 4115–4124

    2020

  46. [54]

    ExploringPose- Aware Human-Object Interaction via Hybrid Learning

    EastmanZ.Y.Wu,YaliLi,YuanWang,andShengjinWang.2024. ExploringPose- Aware Human-Object Interaction via Hybrid Learning. InIEEE Conf. Comput. Vis. Pattern Recog. (CVPR). IEEE, Seattle, WA, USA, 17815–17825

    2024

  47. [55]

    In IEEE Conf

    GuangzhiWang,YangyangGuo,ZiweiXu,andMohanKankanhalli.2024.Bilateral adaptation for human-object interaction detection with occlusion-robustness. In IEEE Conf. Comput. Vis. Pattern Recog. (CVPR). IEEE, Seattle, WA, USA, 27970–27980

    2024

  48. [56]

    Mingrui Wu, Yuqi Liu, Jiayi Ji, Xiaoshuai Sun, and Rongrong Ji. 2024. Toward Open-Set Human Object Interaction Detection. InAAAI Conf. Artif. Intell. (AAAI). AAAI Press, Vancouver, Canada, 6066–6073

    2024

  49. [57]

    Suchen Wang, Yueqi Duan, Henghui Ding, Yap-Peng Tan, Kim-Hui Yap, and Junsong Yuan. 2022. Learning Transferable Human-Object Interaction Detector with Natural Language Supervision. InIEEE Conf. Comput. Vis. Pattern Recog. (CVPR). IEEE, New Orleans, LA, USA, 929–938

    2022

  50. [58]

    DeconfoundedImageCaptioning: ACausalRetrospect

    XuYang,HanwangZhang,andJianfeiCai.2023. DeconfoundedImageCaptioning: ACausalRetrospect. IEEETrans.PatternAnal.Mach.Intell. 45,11(2023),12996– 13010

    2023

  51. [59]

    Mert Yüksekgönül, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. 2023. When and Why Vision-Language Models Behave like Bags-Of-Words, andWhattoDoAboutIt?.In Int.Conf.Learn.Represent.(ICLR) .OpenReview.net, Kigali, Rwanda, 1–20

    2023

  52. [60]

    End-to-EndZero-ShotHOIDetectionviaVisionandLanguageKnowl- edge Distillation

    Mingrui Wu, Jiaxin Gu, Yunhang Shen, Mingbao Lin, Chao Chen, and Xiaoshuai Sun.2023. End-to-EndZero-ShotHOIDetectionviaVisionandLanguageKnowl- edge Distillation. InAAAI Conf. Artif. Intell. (AAAI). AAAI Press, Washington, DC, USA, 2839–2846

    2023

  53. [61]

    Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. 2024. Long-CLIP: Unlocking the Long-Text Capability of CLIP. InEuropean Conf. Comput. Vis. (ECCV), Vol. 15109. Springer, Milan, Italy, 310–325

    2024

  54. [62]

    Chi Xie, Shuang Liang, Jie Li, Zhao Zhang, Feng Zhu, Rui Zhao, and Yichen Wei. 2025. RelationLMM: Large Multimodal Model as Open and Versatile Visual Relationship Generalist.IEEE Trans. Pattern Anal. Mach. Intell.47, 5 (2025), 3515–3529

    2025

  55. [63]

    EfficientTwo-Stage DetectionofHuman-ObjectInteractionswithaNovelUnary-PairwiseTransformer

    FredericZ.Zhang,DylanCampbell,andStephenGould.2022. EfficientTwo-Stage DetectionofHuman-ObjectInteractionswithaNovelUnary-PairwiseTransformer. In IEEE Conf. Comput. Vis. Pattern Recog. (CVPR). IEEE, New Orleans, LA, USA, 20072–20080

    2022

  56. [64]

    Long Zhao, Liangzhe Yuan, Boqing Gong, Yin Cui, Florian Schroff, Ming-Hsuan Yang, Hartwig Adam, and Ting Liu. 2023. Unified Visual Relationship Detection with Vision and Language Models. InIEEE Int. Conf. Comput. Vis. (ICCV). IEEE, Paris, France, 6939–6950

    2023

  57. [65]

    Aixi Zhang, Yue Liao, Si Liu, Miao Lu, Yongliang Wang, Chen Gao, and Xiaobo Li. 2021. Mining the Benefits of Two-stage and One-stage HOI Detection. InAdv. Neural Inf. Process. Syst. (NeurIPS), Vol. 34. Curran Associates, Inc., Virtual, 17209–17220

    2021

  58. [66]

    Yes”| “Is the person{verb} the{object}?

    Desen Zhou, Zhichao Liu, Jian Wang, Leshan Wang, Tao Hu, Errui Ding, and Jingdong Wang. 2022. Human-Object Interaction Detection via Disentangled Transformer. InIEEE Conf. Comput. Vis. Pattern Recog. (CVPR). IEEE, New Orleans, LA, USA, 19546–19555. Dynamic Scoring with Enhance...

    2022

  59. [67]

    Zhang, Dylan Campbell, and Stephen Gould

    Frederic Z. Zhang, Dylan Campbell, and Stephen Gould. 2021. Spatially Con- ditioned Graphs for Detecting Human-Object Interactions. InIEEE Int. Conf. Comput. Vis. (ICCV). IEEE, Montreal, QC, Canada, 13299–13307

    2021

  60. [70]

    TowardsHard- Positive Query Mining for DETR-Based Human-Object Interaction Detection

    XubinZhong,ChangxingDing,ZijianLi,andShaoliHuang.2022. TowardsHard- Positive Query Mining for DETR-Based Human-Object Interaction Detection. In European Conf. Comput. Vis. (ECCV). Springer, Tel Aviv, Israel, 444–460

    2022

  61. [2021]

    InIEEE Conf

    HOTR:End-to-EndHuman-ObjectInteractionDetectionWithTransformers. InIEEE Conf. Comput. Vis. Pattern Recog. (CVPR). Computer Vision Foundation / IEEE, Virtual, 74–83

  62. [2022]

    IEEE Trans

    Transferable Interactiveness Knowledge for Human-Object Interaction Detection. IEEE Trans. Pattern Anal. Mach. Intell.44, 7 (2022), 3870–3882

    2022

  63. [2023]

    Disentanglement via Latent Quantization. InAdv. Neural Inf. Process. Syst. (NeurIPS),Vol.36.CurranAssociates,Inc.,NewOrleans,LA,USA,45463–45488

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.