Pith. sign in

REVIEW 3 major objections 4 minor 91 references

The paper argues that a 'Toxic Siblings' bias—similar HOI triplets interfering at input and output—degrades interaction-decoder learning, and that two auxiliary objectives, contrastive-then-calibration and merge-then-split, remove enough of

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A detection transformer for human-object interactions gains 9.18 mAP on HICO-DET by adding contrastive-then-calibration and merge-then-split training objectives against a diagnosed 'toxic siblings' interference bias.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Strong results, thin causal story; the text-similarity proxy needs a control, but this deserves a serious referee. the 3 major comments →

arxiv 2509.00760 v1 pith:FQAYACJJ submitted 2025-08-31 cs.CV

No More Sibling Rivalry: Debiasing Human-Object Interaction Detection

Bin Yang , Yulin Zhang , Hong-Yu Zhou , Sibei Yang This is my paper
classification cs.CV
keywords human-object interaction detectionToxic Siblings biascontrastive learningmerge-then-splitlong-tail recognitionDETRCLIPHICO-DET
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a large share of HOI-detection errors comes from 'Toxic Siblings': HOI triplets that look alike, either because they co-occur in the same image sharing an object or verb, or because their category representations are semantically close. It claims this bias makes similar categories compete, so that improving one sibling degrades the other, and it proposes two auxiliary losses—contrastive-then-calibration and merge-then-split—that remove enough of the bias to lift mAP from 33.75 to 42.93 on HICO-DET with the same backbone and CLIP. The paper frames the losses as a drop-in debiasing layer, not a new architecture, and shows transfer to HOICLIP and to zero-shot settings. The reader should care because HOI detection is a bottleneck for scene understanding, and this is a path to improving it without extra data or heavy external models.

Core claim

The paper's central claim is that the dominant remaining error source in DETR-style HOI detectors is not lack of capacity but a sibling-interference bias, and that this bias is reducible by two training objectives attached to the interaction decoder. The input-side objective, contrastive-then-calibration, builds positive/negative sets from triplets that share object or verb labels within an image, pushes features apart with a contrastive loss, then feeds the decoder mislabeled queries with correct spatial boxes and trains it to reconstruct the true semantic feature. The output-side objective, merge-then-split, clusters CLIP text features into superclasses, trains a superclass head to learn s

What carries the argument

The central objects are the two 'sibling' relation definitions: input-level siblings are triplets in the same image that share an object or verb, and output-level siblings are categories with high cosine similarity between their CLIP text features. The carrying mechanism is two auxiliary losses in the interaction decoder: contrastive-then-calibration (C2C) reconstructs correct triplets from sibling-like wrong inputs using spatial priors, and merge-then-split (M2S) first classifies into text-derived superclasses, then runs a contrastive loss over the top-k nearest categories. The CLIP text encoder supplies the feature geometry on which siblinghood and superclasses are built, and the human-obj

Load-bearing premise

The load-bearing premise is that CLIP text-feature similarity between HOI category names tracks the visual confusion the interaction decoder actually experiences, because the output-side sibling definition, the merge superclass clusters, and the split top-k competitors all come from those text features.

What would settle it

Train the same baseline with siblings defined by a confusion matrix from a pretrained detector instead of CLIP text similarity; if the mAP gains disappear while text-similarity-sibling gains remain, the stated mechanism is wrong. Or, hold CLIP similarity fixed but use visually unrelated categories as 'siblings': if mAP still rises by roughly nine points, the losses are generic extra supervision rather than debiasing.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Existing DETR-style HOI detectors could gain several mAP points by adding C2C and M2S as auxiliary losses, with no new pretraining data or external models.
  • The rare-category gain (+13.16 mAP on HICO-DET Rare) suggests the merge-then-split loss specifically relieves long-tail confusion among similar categories.
  • The HOICLIP transfer (+8.42 mAP) indicates the losses are not tuned to one detector and can be bolted onto other interaction decoders.
  • Zero-shot improvements across UV, UO, NF, and RF settings imply sibling bias also hurts unseen-category generalization, so debiasing helps open-vocabulary HOI detection.
  • The method's output-side debiasing is tied to the initial CLIP text-feature geometry, so its benefit may scale with the quality and alignment of the text encoder.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's output-level sibling definition is entirely text-based; a natural test is whether defining siblings from visual confusion matrices of a pretrained detector changes the magnitude of the gains.
  • Because C2C relies on sibling triplets co-occurring within the same image, its input-side benefit should be concentrated in dense images; sparse images would see little contrastive pressure.
  • The merge-then-split recipe—cluster labels, learn shared structure, then discriminate within top-k neighbors—is a generic prescription for large-vocabulary fine-grained recognition, and could be tested outside HOI detection.
  • The split objective's top-k selection is computed from query-to-text similarity; a per-image or per-query adaptive k2 might reduce noise and computation further than the fixed global k2 used here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper introduces the concept of "Toxic Siblings" bias in transformer-based HOI detection: similar HOI triplets within an image (input level) and similar HOI categories (output level) interfere with interaction-decoder learning. To counter this, the authors propose two auxiliary objectives: (i) C2C, which samples sibling-like negative triplets, applies a contrastive loss, and then uses a calibration/reconstruction loss with strong spatial priors; and (ii) M2S, which clusters HOI categories into superclasses using CLIP text features (merge loss) and then applies a contrastive split loss over the top-k similar categories per query. On HICO-DET with a GEN-VLKT R50/CLIP baseline, the method improves mAP from 33.75 to 42.93 (Tables 1 and 3), outperforms the prior CLIP-based SOTA BCOM by 3.59 mAP, transfers to HOICLIP (+8.42, Supp. Table 7), improves V-COCO, and shows gains in zero-shot settings (Table 2). The ablation chain is monotonic (33.75, 35.37, 36.82, 38.03, 39.88, 42.93) and hyperparameters are swept (Table 4, Supp. Tables 9-10).

Significance. If the result holds, the contribution is practically significant: two lightweight auxiliary losses that can be plugged into existing DETR-style HOI detectors without external data or heavy models, with sizeable gains on rare classes (42.41 vs 29.25 baseline). The paper is transparent in its diagnostics and supplies useful qualitative and supplementary material. I credit the authors for a complete ablation chain, explicit hyperparameter sweeps, transfer experiments to HOICLIP and zero-shot settings, and detailed supplementary analyses (Fig. 6, Tables 11-13). However, the paper's central mechanistic interpretation—that the gains come from removing sibling interference—is not fully established because the sibling structure is defined entirely through CLIP text-feature similarity, and no control separates the structured losses from generic auxiliary supervision. This concern is the main reason for a major revision rather than acceptance.

major comments (3)
  1. [Sec. 3.3, Eqs. (9)-(12); Table 3] The central claim that M2S removes output-level sibling bias is not separated from generic auxiliary supervision. Output-level siblings are defined by cosine similarity of CLIP text features (Supp. Sec. 6.1); the same similarity builds the superclass hierarchy and selects k1/k2 in Eq. (10). The +4.90 mAP from lines 5 to 7 of Table 3 (and the HOICLIP transfer, Supp. Table 7) could therefore be produced by any extra superclass CE loss plus any contrastive loss, regardless of sibling structure. Please add a control in which the k1/k2 candidate categories in Eq. (10) are sampled randomly (or clustered with shuffled/random features), and a control in which the merge clusters are formed without text-similarity structure. If the gain persists, the "Toxic Siblings" mechanism is not supported.
  2. [Fig. 2(d), Supp. Sec. 6.1] The diagnostic correlation is confounded. The scatter plots average initial classification-head cosine similarity against final AP for "non-head classes" only, without controlling for class frequency, object frequency, or image complexity. Rare classes are likely to have both lower AP and different similarity distributions. Please report a partial correlation controlling for log frequency and/or stratified analysis within rare and non-rare subsets. Fig. 2(c) similarly compares samples with/without input-level siblings without controlling for the number of instances or overlapping boxes; the causal language "increases error rate" is stronger than the evidence.
  3. [Tables 3, 4; Supp. Tables 9, 10] All reported numbers are single runs. The ablation increments are small (e.g., +1.21 for calibration, +1.85 for merge, +1.65 for k2=5 vs 10 in Table 4) and could be within seed variation. Please report mean ± std over at least 3 seeds for the main ablation chain and for the key comparisons in Tables 1, 3, and 4, and state the number of seeds used.
minor comments (4)
  1. [Eq. (8)] The L_cal loss appears to compare the decoder output \tilde{v}^{tri}_i with the original v^{tri}_i, but Eq. (8) prints the same symbol for both. Add the tilde and define the target explicitly.
  2. [Eq. (12)] The split loss uses q^a_i in the numerator although the subset is defined as \tilde{q}^a_i. Clarify the notation to distinguish query features before and after the decoder.
  3. [Sec. 3.3.1 / Sec. 4.1] The clustering algorithm is not described (k-means? agglomerative? which distance metric?), and Sec. 4.1 reports M=25 while Table 4 uses M1=5, M2=5. State both M1 and M2 and provide the clustering implementation details, including whether the cluster assignment is fixed before training.
  4. [Supp. Tables 11-13; Fig. 9] Several verb phrases appear truncated or incomplete (e.g., "and a donut", "and a suitcase" in Fig. 9 and Table 13). Please correct these or indicate if they are rendering artifacts.

Circularity Check

0 steps flagged

No circularity: the proposed losses are auxiliary training objectives validated by external mAP benchmarks, not by-construction re-statements of the target quantity.

full rationale

The paper's derivation chain is self-contained with respect to circularity. The central claim is that two auxiliary objectives, C2C and M2S, reduce learning interference among similar HOI triplets/categories; the evidence is HICO-DET and V-COCO mAP computed from ground-truth annotations, which is an external measurement not forced by the loss definitions. The M2S loss uses CLIP text features to define superclasses and select top-k similar categories, and the paper also uses initial classifier-head cosine similarity (which is derived from the same CLIP text features) to identify output-level siblings. This overlap raises a substantive validity question about whether the gains come from the sibling structure specifically or from generic auxiliary supervision, and it should be probed with a randomized-sibling control. However, this is not a circularity by the standard used here: no equation reduces to its input, no fitted parameter is renamed as a prediction, and no load-bearing conclusion is justified solely by self-citation. The test-time mAP is independent of the text-feature construction of the losses, so the empirical improvements could in principle falsify the authors' mechanism claim. Thus the correct circularity score is 0.

Axiom & Free-Parameter Ledger

6 free parameters · 6 axioms · 1 invented entities

The paper's contribution is measured against free parameters tuned on the test benchmark (five loss weights, tau, k1, k2, M) and three domain assumptions: CLIP-text-as-visual-proxy, same-image sibling causality, and the efficacy of the L_cal reconstruction task. None of these is independently verified. Only one invented entity, the Toxic Siblings construct, is introduced, and it lacks independent evidence as a mechanism.

free parameters (6)
  • loss weights lambda_1..lambda_5 = 1, 1, 0.5, 1, 0.5
    Weights of detector, contrastive, calibration, merge, and split losses in Eq. (13); ablations on V-COCO in Supp. Table 9.
  • temperature tau (contrastive and split losses)
    Scales dot-product logits in Eqs. (6) and (12); no value reported anywhere.
  • k1 (top-k similar categories per query) = 2
    Eq. (10); ablated over 1, 2, 3, 5 in Supp. Table 10.
  • k2 (most frequent category features fed to decoder) = 10
    Eq. (11); ablated over 5, 10, 15 in Table 4.
  • cluster counts M1 x M2 for supercategories = 5 x 5 = 25 super interaction-object classes
    Sec. 3.3.1; ablated over 3x3, 5x5, 7x7 in Table 4.
  • baseline detector losses lambda_b, lambda_u, lambda_c = 2.5, 1, 1
    Inherited from GEN-VLKT [41] in Eq. (3).
axioms (6)
  • domain assumption CLIP text-feature similarity between HOI descriptions is a faithful proxy for visual confusion among categories.
    Defines output-level siblings (Supp. Sec. 6.1), the merge hierarchy (Sec. 3.3.1), and the split top-k set (Sec. 3.3.2).
  • domain assumption Co-occurring sibling triplets in the same image causally reduce accuracy.
    Fig. 2(c) shows an error-rate correlation (+27.59) but no control for image difficulty; the C2C objective assumes spatial cues can resolve the interference.
  • ad hoc to paper Reconstructing a corrupted-semantic triplet feature toward the original feature (L_cal) improves downstream classification.
    Auxiliary denoising task in Sec. 3.2, Eq. (8); justified only by the ablation gain of +1.21 mAP.
  • domain assumption Hungarian bipartite matching and DETR set-prediction losses from GEN-VLKT carry over unchanged.
    Baseline construction, Sec. 3.1.2, Eq. (3); inherited rather than re-derived.
  • ad hoc to paper Superclass classification (merge) learned jointly with fine classification does not wash out fine distinctions.
    Motivates the split loss in Sec. 3.3.2; supported only by ablation.
  • domain assumption Human-object union bounding box masks focus interaction attention correctly.
    HOR mask, Supp. Sec. 6.3; contributes +1.62 mAP alone but is presented largely in the appendix.
invented entities (1)
  • 'Toxic Siblings' bias (input-level and output-level) no independent evidence
    purpose: Conceptual label for mutual interference among similar HOI triplets; organizes the two proposed losses.
    The construct is inferred from correlational diagnostics (Fig. 2) on the same benchmarks used to tune and evaluate the method. The mAP gains are falsifiable on public benchmarks, but the bias mechanism itself has no external falsification in the paper.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of No More Sibling Rivalry: Debiasing Human-Object Interaction Detection." pith.science (2026). https://pith.science/paper/FQAYACJJ

@misc{pith2026250900760,
  author       = {Pith},
  title        = {Pith review of: No More Sibling Rivalry: Debiasing Human-Object Interaction Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FQAYACJJ}},
  note         = {Machine review of arXiv:2509.00760}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Detection transformers have been applied to human-object interaction (HOI) detection, enhancing the localization and recognition of human-action-object triplets in images. Despite remarkable progress, this study identifies a critical issue-"Toxic Siblings" bias-which hinders the interaction decoder's learning, as numerous similar yet distinct HOI triplets interfere with and even compete against each other both input side and output side to the interaction decoder. This bias arises from high confusion among sibling triplets/categories, where increased similarity paradoxically reduces precision, as one's gain comes at the expense of its toxic sibling's decline. To address this, we propose two novel debiasing learning objectives-"contrastive-then-calibration" and "merge-then-split"-targeting the input and output perspectives, respectively. The former samples sibling-like incorrect HOI triplets and reconstructs them into correct ones, guided by strong positional priors. The latter first learns shared features among sibling categories to distinguish them from other groups, then explicitly refines intra-group differentiation to preserve uniqueness. Experiments show that we significantly outperform both the baseline (+9.18% mAP on HICO-Det) and the state-of-the-art (+3.59% mAP) across various settings.

Figures

Figures reproduced from arXiv: 2509.00760 by Bin Yang, Hong-Yu Zhou, Sibei Yang, Yulin Zhang.

Figure 1
Figure 1. Figure 1: Illustration of “Toxic Siblings” biases in input and output level between highly similar HOI triplets and classes. (a)- (b): Input level biases mislead HOI recognition by introducing similar HOIs within images. For example, in (a), the interaction between the marked person and the stool is misclassified from standing to sitting, influenced by other nearby sibling HOIs humans who are all seated with the sto… view at source ↗
Figure 2
Figure 2. Figure 2: Illustrates the training patterns and effects of Toxic Siblings Bias in HOI Detection. (a) & (b): The training mAP curves for sibling HOI pairs demonstrate a mutually detrimental learning dynamic (red lines) at both the input-level (a) and output￾level (b). (c) presents a statistical analysis of the impact of input￾level ”Toxic Siblings” bias, which significantly increases the error rate. (d) demonstrates … view at source ↗
Figure 3
Figure 3. Figure 3: Overview of overall framework addressing “Toxic Siblings” biases. Middle: we use the one-stage two-branch detector [41] as baseline (Sec. 3.1), incorporating additional learning objectives in the interaction decoder to address biases. Left (Input “Toxic Siblings”): For each matched query: ① we first identify the positive and negative samples and represent their spatial and semantic features (define in Equ … view at source ↗
Figure 4
Figure 4. Figure 4: Visualization of our prediction results and comparison [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Visualization of our prediction results and comparison [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Experimental results of our method compared to the [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Qualitative comparisons of the first type of input level “Toxic Siblings” bias between our method and the baseline. [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Qualitative comparisons of the second type of input level “Toxic Siblings” bias between our method and the baseline. [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Qualitative comparisons of output level “Toxic Siblings” bias between our method and the baseline. [PITH_FULL_IMAGE:figures/full_fig_p019_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

91 extracted references · 71 canonical work pages · 3 internal anchors

  1. [1]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision , pages 2425– 2433, 2015. 1

    2015

  2. [2]

    Detecting human-object interactions via functional generalization

    Ankan Bansal, Sai Saketh Rambhatla, Abhinav Shrivastava, and Rama Chellappa. Detecting human-object interactions via functional generalization. In AAAI, 2020. 2, 3

    2020

  3. [3]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neu- ral information processing systems, 33:1877–1901, 2020. 2, 7

    1901

  4. [4]

    Detecting any human-object interaction relationship: Universal hoi detector with spatial prompt learning on foundation models

    Yichao Cao, Qingfei Tang, Xiu Su, Song Chen, Shan You, Xiaobo Lu, and Chang Xu. Detecting any human-object interaction relationship: Universal hoi detector with spatial prompt learning on foundation models. Advances in Neural Information Processing Systems, 36:739–751, 2023. 1, 2, 3, 4, 7

    2023

  5. [5]

    End-to- end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to- end object detection with transformers. In ECCV, 2020. 1, 3, 4, 6

    2020

  6. [6]

    A short note on the kinetics-700 human action dataset

    Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zisser- man. A short note on the kinetics-700 human action dataset. arXiv preprint arXiv:1907.06987, 2019. 2

    Pith/arXiv arXiv 1907

  7. [7]

    Learning to detect human-object interactions

    Yu-Wei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng. Learning to detect human-object interactions. 2018. 3, 6, 7, 8, 15

    2018

  8. [8]

    A survey on graph neural networks and graph transformers in computer vision: A task-oriented perspective

    Chaoqi Chen, Yushuang Wu, Qiyuan Dai, Hong-Yu Zhou, Mutian Xu, Sibei Yang, Xiaoguang Han, and Yizhou Yu. A survey on graph neural networks and graph transformers in computer vision: A task-oriented perspective. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 2024. 2

    2024

  9. [9]

    QAHOI: Query-Based Anchors for Human-Object Interaction Detection

    Junwen Chen and Keiji Yanai. Qahoi: query-based an- chors for human-object interaction detection. arXiv preprint arXiv:2112.08647, 2021. 3

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  10. [10]

    Schwing, Alexan- der Kirillov, and Rohit Girdhar

    Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexan- der Kirillov, and Rohit Girdhar. Masked-attention mask trans- former for universal image segmentation. 2022. 13

    2022

  11. [11]

    Curriculum point prompting for weakly-supervised referring image segmentation

    Qiyuan Dai and Sibei Yang. Curriculum point prompting for weakly-supervised referring image segmentation. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13711–13722, 2024. 1

    2024

  12. [12]

    Free on the fly: Enhancing flexibility in test-time adaptation with online em, 2025

    Qiyuan Dai and Sibei Yang. Free on the fly: Enhancing flexibility in test-time adaptation with online em, 2025. 3

    2025

  13. [13]

    Adap- tive part learning for fine-grained generalized category dis- covery: A plug-and-play enhancement, 2025

    Qiyuan Dai, Hanzhuo Huang, Yu Wu, and Sibei Yang. Adap- tive part learning for fine-grained generalized category dis- covery: A plug-and-play enhancement, 2025. 3

    2025

  14. [14]

    Cg-hoi: Contact-guided 3d human-object interaction generation

    Christian Diller and Angela Dai. Cg-hoi: Contact-guided 3d human-object interaction generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19888–19901, 2024. 1

    work page 2024

  15. [15]

    Category-aware transformer network for better human-object interaction detection

    Leizhen Dong, Zhimin Li, Kunlun Xu, Zhijun Zhang, Luxin Yan, Sheng Zhong, and Xu Zou. Category-aware transformer network for better human-object interaction detection. In CVPR, 2022. 2, 3

    work page 2022

  16. [16]

    Dirv: Dense interaction region voting for end-to-end human-object interaction detection

    Hao-Shu Fang, Yichen Xie, Dian Shao, and Cewu Lu. Dirv: Dense interaction region voting for end-to-end human-object interaction detection. In AAAI, 2021. 3

    work page 2021

  17. [17]

    Drg: Dual relation graph for human-object interaction detection

    Chen Gao, Jiarui Xu, Yuliang Zou, and Jia-Bin Huang. Drg: Dual relation graph for human-object interaction detection. In ECCV, 2020. 3

    work page 2020

  18. [18]

    Multi-evidence fil- tering and fusion for multi-label classification, object detec- tion and semantic segmentation based on weakly supervised learning

    Weifeng Ge, Sibei Yang, and Yizhou Yu. Multi-evidence fil- tering and fusion for multi-label classification, object detec- tion and semantic segmentation based on weakly supervised learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1277–1286, 2018. 1

    work page 2018

  19. [19]

    Detecting and recognizing human-object interactions

    Georgia Gkioxari, Ross Girshick, Piotr Doll ´ar, and Kaiming He. Detecting and recognizing human-object interactions. In CVPR, 2018. 3

    work page 2018

  20. [20]

    Visual semantic role labeling

    Saurabh Gupta and Jitendra Malik. Visual semantic role labeling. arXiv preprint arXiv:1505.04474, 2015. 3, 6, 7, 14

    Pith/arXiv arXiv 2015

  21. [21]

    No- frills human-object interaction detection: Factorization, lay- out encodings, and training techniques

    Tanmay Gupta, Alexander Schwing, and Derek Hoiem. No- frills human-object interaction detection: Factorization, lay- out encodings, and training techniques. In ICCV, 2019. 3

    work page 2019

  22. [22]

    Non-local context encoder: Robust biomedical image segmentation against adversarial attacks

    Xiang He, Sibei Yang, Guanbin Li, Haofeng Li, Huiyou Chang, and Yizhou Yu. Non-local context encoder: Robust biomedical image segmentation against adversarial attacks. In Proceedings of the AAAI Conference on Artificial Intelli- gence, pages 8417–8424, 2019. 1

    work page 2019

  23. [23]

    Denoising diffu- sion probabilistic models, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models, 2020. 2

    2020

  24. [24]

    Visual compositional learning for human-object interaction detec- tion

    Zhi Hou, Xiaojiang Peng, Yu Qiao, and Dacheng Tao. Visual compositional learning for human-object interaction detec- tion. In ECCV, 2020. 3

    work page 2020

  25. [25]

    Hand-object interaction controller (hoic): Deep reinforce- ment learning for reconstructing interactions with physics

    Haoyu Hu, Xinyu Yi, Zhe Cao, Jun-Hai Yong, and Feng Xu. Hand-object interaction controller (hoic): Deep reinforce- ment learning for reconstructing interactions with physics. In ACM SIGGRAPH 2024 Conference Papers , pages 1–10,

    2024

  26. [26]

    Free-bloom: Zero-shot text-to-video generator with llm director and ldm animator

    Hanzhuo Huang, Yufan Feng, Cheng Shi, Lan Xu, Jingyi Yu, and Sibei Yang. Free-bloom: Zero-shot text-to-video generator with llm director and ldm animator. Advances in Neural Information Processing Systems , 36:26135–26158,

    work page

  27. [27]

    Mvtokenflow: High-quality 4d content generation using multiview token flow

    Hanzhuo Huang, Yuan Liu, Ge Zheng, Jiepeng Wang, Zhiyang Dou, and Sibei Yang. Mvtokenflow: High-quality 4d content generation using multiview token flow. arXiv preprint arXiv:2502.11697, 2025. 2

    Pith/arXiv arXiv 2025

  28. [28]

    What to look at and where: Semantic and spatial refined transformer for detecting human- object interactions

    ASM Iftekhar, Hao Chen, Kaustav Kundu, Xinyu Li, Joseph Tighe, and Davide Modolo. What to look at and where: Semantic and spatial refined transformer for detecting human- object interactions. In CVPR, 2022. 2, 3

    work page 2022

  29. [29]

    Uniondet: Union-level detector towards real-time human-object interaction detection

    Bumsoo Kim, Taeho Choi, Jaewoo Kang, and Hyunwoo J Kim. Uniondet: Union-level detector towards real-time human-object interaction detection. In ECCV, 2020. 1, 3

    work page 2020

  30. [30]

    Detecting human-object interactions with action co- occurrence priors

    Dong-Jin Kim, Xiao Sun, Jinsoo Choi, Stephen Lin, and In So Kweon. Detecting human-object interactions with action co- occurrence priors. In ECCV, 2020. 3

    work page 2020

  31. [31]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 4015–4026, 2023. 7

    work page 2023

  32. [32]

    Visual genome: Connecting language and vision using crowdsourced dense image annotations

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalan- tidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32–73, 2017. 2

    work page 2017

  33. [33]

    Efficient adaptive human-object inter- action detection with concept-guided memory

    Ting Lei, Fabian Caba, Qingchao Chen, Hailin Jin, Yuxin Peng, and Yang Liu. Efficient adaptive human-object inter- action detection with concept-guided memory. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pages 6480–6490, 2023. 7, 15

    work page 2023

  34. [34]

    Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Interna- tional conference on machine learning, pages 19730–19742. PMLR, 2023. 1, 2, 4, 6

    work page 2023

  35. [35]

    Neural- logic human-object interaction detection.Advances in Neural Information Processing Systems, 36, 2024

    Liulei Li, Jianan Wei, Wenguan Wang, and Yi Yang. Neural- logic human-object interaction detection.Advances in Neural Information Processing Systems, 36, 2024. 14

    work page 2024

  36. [36]

    Transferable interactiveness knowledge for human-object interaction de- tection

    Yong-Lu Li, Siyuan Zhou, Xijie Huang, Liang Xu, Ze Ma, Hao-Shu Fang, Yanfeng Wang, and Cewu Lu. Transferable interactiveness knowledge for human-object interaction de- tection. In CVPR, 2019. 3

    work page 2019

  37. [37]

    Detailed 2d-3d joint representation for human-object interaction

    Yong-Lu Li, Xinpeng Liu, Han Lu, Shiyi Wang, Junqi Liu, Jiefeng Li, and Cewu Lu. Detailed 2d-3d joint representation for human-object interaction. In CVPR, 2020. 2, 3

    work page 2020

  38. [38]

    Hoi analysis: Integrating and decomposing human- object interaction

    Yong-Lu Li, Xinpeng Liu, Xiaoqian Wu, Yizhuo Li, and Cewu Lu. Hoi analysis: Integrating and decomposing human- object interaction. In NeurIPS, 2020. 3

    work page 2020

  39. [39]

    Disentangled pre-training for human-object interaction detec- tion

    Zhuolong Li, Xingao Li, Changxing Ding, and Xiangmin Xu. Disentangled pre-training for human-object interaction detec- tion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 28191–28201,

    work page

  40. [40]

    Ppdm: Parallel point detection and matching for real-time human-object interaction detection

    Yue Liao, Si Liu, Fei Wang, Yanjie Chen, Chen Qian, and Jiashi Feng. Ppdm: Parallel point detection and matching for real-time human-object interaction detection. In CVPR,

    work page

  41. [41]

    Gen-vlkt: Simplify association and enhance interaction understanding for hoi detection

    Yue Liao, Aixi Zhang, Miao Lu, Yongliang Wang, Xiaobo Li, and Si Liu. Gen-vlkt: Simplify association and enhance interaction understanding for hoi detection. In CVPR, 2022. 1, 2, 3, 4, 6, 7, 8, 13, 15

    work page 2022

  42. [42]

    Structured attention network for refer- ring image segmentation

    Liang Lin, Pengxiang Yan, Xiaoqian Xu, Sibei Yang, Kun Zeng, and Guanbin Li. Structured attention network for refer- ring image segmentation. IEEE Transactions on Multimedia, 24:1922–1932, 2021. 1

    work page 1922

  43. [43]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision , pages 740–755. Springer, 2014. 2, 6

    work page 2014

  44. [44]

    Am- plifying key cues for human-object-interaction detection

    Yang Liu, Qingchao Chen, and Andrew Zisserman. Am- plifying key cues for human-object-interaction detection. In ECCV, 2020. 3

    work page 2020

  45. [45]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 6

    Pith/arXiv arXiv 2017

  46. [46]

    Discovering syntactic interaction clues for human-object interaction detection

    Jinguo Luo, Weihong Ren, Weibo Jiang, Xi’ai Chen, Qiang Wang, Zhi Han, and Honghai Liu. Discovering syntactic interaction clues for human-object interaction detection. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 28212–28222, 2024. 2, 7

    work page 2024

  47. [47]

    FGAHOI: Fine-Grained Anchors for Human-Object Interaction Detection

    Shuailei Ma, Yuefeng Wang, Shanze Wang, and Ying Wei. Fgahoi: Fine-grained anchors for human-object interaction detection. arXiv preprint arXiv:2301.04019, 2023. 3

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  48. [48]

    Hoiclip: Efficient knowledge transfer for hoi detection with vision-language models

    Shan Ning, Longtian Qiu, Yongfei Liu, and Xuming He. Hoiclip: Efficient knowledge transfer for hoi detection with vision-language models. In CVPR, 2023. 2, 3, 7, 15

    work page 2023

  49. [49]

    Repre- sentation learning with contrastive predictive coding

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Repre- sentation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. 5

    Pith/arXiv arXiv 2018

  50. [50]

    Viplo: Vision transformer based pose-conditioned self-loop graph for human-object interaction detection

    Jeeseung Park, Jin-Woo Park, and Jong-Seok Lee. Viplo: Vision transformer based pose-conditioned self-loop graph for human-object interaction detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17152–17162, 2023. 7, 15

    work page 2023

  51. [51]

    Human-computer interac- tion

    Jenny Preece, Yvonne Rogers, Helen Sharp, David Benyon, Simon Holland, and Tom Carey. Human-computer interac- tion. Addison-Wesley Longman Ltd., 1994. 1

    work page 1994

  52. [52]

    Learning human-object interactions by graph parsing neural networks

    Siyuan Qi, Wenguan Wang, Baoxiong Jia, Jianbing Shen, and Song-Chun Zhu. Learning human-object interactions by graph parsing neural networks. In ECCV, 2018. 3

    work page 2018

  53. [53]

    Distillation using oracle queries for transformer-based human-object interaction detection

    Xian Qu, Changxing Ding, Xingao Li, Xubin Zhong, and Dacheng Tao. Distillation using oracle queries for transformer-based human-object interaction detection. In CVPR, 2022. 2, 3

    work page 2022

  54. [54]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision

    work page

  55. [55]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022. 2, 7

    work page 2022

  56. [56]

    Prompting large language models with answer heuristics for knowledge- based visual question answering

    Zhenwei Shao, Zhou Yu, Meng Wang, and Jun Yu. Prompting large language models with answer heuristics for knowledge- based visual question answering. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recog- nition, pages 14974–14983, 2023. 1

    work page 2023

  57. [57]

    Edadet: Open-vocabulary object detection using early dense alignment

    Cheng Shi and Sibei Yang. Edadet: Open-vocabulary object detection using early dense alignment. In Proceedings of the IEEE/CVF international conference on computer vision, pages 15724–15734, 2023. 1

    work page 2023

  58. [58]

    Logoprompt: Synthetic text im- ages can be good visual prompts for vision-language models

    Cheng Shi and Sibei Yang. Logoprompt: Synthetic text im- ages can be good visual prompts for vision-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2932–2941, 2023. 3

    work page 2023

  59. [59]

    The devil is in the object bound- ary: Towards annotation-free instance segmentation using foundation models

    Cheng Shi and Sibei Yang. The devil is in the object bound- ary: Towards annotation-free instance segmentation using foundation models. arXiv preprint arXiv:2404.11957, 2024. 1

    Pith/arXiv arXiv 2024

  60. [60]

    Part2object: Hierarchical unsupervised 3d instance segmentation

    Cheng Shi, Yulin Zhang, Bin Yang, Jiajin Tang, Yuexin Ma, and Sibei Yang. Part2object: Hierarchical unsupervised 3d instance segmentation. InEuropean Conference on Computer Vision, pages 1–18. Springer, 2024. 3

    work page 2024

  61. [61]

    Plain-det: A plain multi-dataset object detector

    Cheng Shi, Yuchen Zhu, and Sibei Yang. Plain-det: A plain multi-dataset object detector. In European Conference on Computer Vision, pages 210–226. Springer, 2024. 1

    work page 2024

  62. [62]

    Qpic: Query-based pairwise human-object interaction de- tection with image-wide contextual information

    Masato Tamura, Hiroki Ohashi, and Tomoaki Yoshinaga. Qpic: Query-based pairwise human-object interaction de- tection with image-wide contextual information. In CVPR,

    work page

  63. [63]

    Con- trastive grouping with transformer for referring image seg- mentation

    Jiajin Tang, Ge Zheng, Cheng Shi, and Sibei Yang. Con- trastive grouping with transformer for referring image seg- mentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 23570– 23580, 2023. 1

    work page 2023

  64. [64]

    Temporal collection and distribution for referring video object segmentation

    Jiajin Tang, Ge Zheng, and Sibei Yang. Temporal collection and distribution for referring video object segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15466–15476, 2023. 1

    work page 2023

  65. [65]

    Vsgnet: Spatial attention network for detecting human object interactions using graph convolutions

    Oytun Ulutan, ASM Iftekhar, and Bangalore S Manjunath. Vsgnet: Spatial attention network for detecting human object interactions using graph convolutions. In CVPR, 2020. 3

    work page 2020

  66. [66]

    Show and tell: A neural image caption generator

    Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3156–3164, 2015. 1

    work page 2015

  67. [67]

    Pose-aware multi-level feature network for human object interaction detection

    Bo Wan, Desen Zhou, Yongfei Liu, Rongjie Li, and Xuming He. Pose-aware multi-level feature network for human object interaction detection. In ICCV, 2019. 3

    work page 2019

  68. [68]

    Bilateral adaptation for human-object interac- tion detection with occlusion-robustness

    Guangzhi Wang, Yangyang Guo, Ziwei Xu, and Mohan Kankanhalli. Bilateral adaptation for human-object interac- tion detection with occlusion-robustness. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27970–27980, 2024. 7, 15

    work page 2024

  69. [69]

    Contextual heterogeneous graph network for human-object interaction detection

    Hai Wang, Wei-shi Zheng, and Ling Yingbiao. Contextual heterogeneous graph network for human-object interaction detection. In ECCV, 2020. 3

    work page 2020

  70. [70]

    Learning transferable human-object interaction detector with natural language su- pervision

    Suchen Wang, Yueqi Duan, Henghui Ding, Yap-Peng Tan, Kim-Hui Yap, and Junsong Yuan. Learning transferable human-object interaction detector with natural language su- pervision. In CVPR, 2022. 2, 3

    work page 2022

  71. [71]

    Learning human-object interaction detection using interaction points

    Tiancai Wang, Tong Yang, Martin Danelljan, Fahad Shahbaz Khan, Xiangyu Zhang, and Jian Sun. Learning human-object interaction detection using interaction points. InCVPR, 2020. 3

    work page 2020

  72. [72]

    CycleHOI: Improving Human-Object Interaction Detection with Cycle Consistency of Detection and Generation

    Yisen Wang, Yao Teng, and Limin Wang. Cyclehoi: Improving human-object interaction detection with cycle consistency of detection and generation. arXiv preprint arXiv:2407.11433, 2024. 7

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  73. [73]

    End-to-end zero-shot hoi detection via vision and language knowledge distillation

    Mingrui Wu, Jiaxin Gu, Yunhang Shen, Mingbao Lin, Chao Chen, and Xiaoshuai Sun. End-to-end zero-shot hoi detection via vision and language knowledge distillation. In Proceed- ings of the AAAI Conference on Artificial Intelligence, pages 2839–2846, 2023. 3

    work page 2023

  74. [74]

    Learning to detect human-object interac- tions with knowledge

    Bingjie Xu, Yongkang Wong, Junnan Li, Qi Zhao, and Mo- han S Kankanhalli. Learning to detect human-object interac- tions with knowledge. In CVPR, 2019. 2, 3

    work page 2019

  75. [75]

    Boosting human-object inter- action detection with text-to-image diffusion model

    Jie Yang, Bingliang Li, Fengyu Yang, Ailing Zeng, Lei Zhang, and Ruimao Zhang. Boosting human-object inter- action detection with text-to-image diffusion model. arXiv preprint arXiv:2305.12252, 2023. 7

    Pith/arXiv arXiv 2023

  76. [76]

    Open-world human-object interaction detection via multi-modal prompts

    Jie Yang, Bingliang Li, Ailing Zeng, Lei Zhang, and Ruimao Zhang. Open-world human-object interaction detection via multi-modal prompts. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 16954–16964, 2024. 3

    work page 2024

  77. [77]

    Bottom-up shift and reasoning for referring image segmentation

    Sibei Yang, Meng Xia, Guanbin Li, Hong-Yu Zhou, and Yizhou Yu. Bottom-up shift and reasoning for referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11266– 11275, 2021. 1

    work page 2021

  78. [78]

    Rlip: Rela- tional language-image pre-training for human-object interac- tion detection

    Hangjie Yuan, Jianwen Jiang, Samuel Albanie, Tao Feng, Ziyuan Huang, Dong Ni, and Mingqian Tang. Rlip: Rela- tional language-image pre-training for human-object interac- tion detection. In NeurIPS, 2022. 2, 3

    work page 2022

  79. [79]

    Rlipv2: Fast scaling of re- lational language-image pre-training

    Hangjie Yuan, Shiwei Zhang, Xiang Wang, Samuel Al- banie, Yining Pan, Tao Feng, Jianwen Jiang, Dong Ni, Yingya Zhang, and Deli Zhao. Rlipv2: Fast scaling of re- lational language-image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 21649–21661, 2023. 2, 3, 7

    work page 2023

  80. [80]

    Spa- tially conditioned graphs for detecting human-object interac- tions

    Frederic Z Zhang, Dylan Campbell, and Stephen Gould. Spa- tially conditioned graphs for detecting human-object interac- tions. In ICCV, 2021. 3

    work page 2021

Showing first 80 references.

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.