Pith. sign in

REVIEW 3 major objections 5 minor 23 references

Prototype Embedding Optimization for Human-Object Interaction Detection in Livestreaming

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Prototype embedding optimization improves livestreaming HOI detection by embedding object-category word vectors into HO pair union features and reweighting noisy samples.

desk verdict A useful new dataset and a sensible system whose central claim about prototype embedding is not supported by the paper's own ablation. read the letter →

arxiv 2505.22011 v2 pith:EJDCV26U submitted 2025-05-28 cs.CV

Menghui Zhang , Jing Zhang , Lin Chen , Li Zhuo This is my paper
classification cs.CV
keywords human-objectinteractiondetectionprototypeembeddingobjectbiaspropensity-reweightedlosslivestreamingvideoHOIspatio-temporalcontextViddataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes PeO-HOI, a method for detecting interactions between a streamer and objects in livestreaming video, a task where existing HOI detectors over-focus on object category and under-weight the interaction. The central claim is that embedding semantic prototype features, drawn from pretrained word vectors of object categories, into the union features of human-object pairs makes those features more discriminative and reduces this object bias. Experiments on the VidHOI benchmark and a self-built livestreaming dataset, BJUT-HOI, report 37.19% and 45.13% mAP@full, improving on the prior gaze-based method by 1.55 and 1.60 percentage points respectively. The paper further claims that a propensity-reweighted cross-entropy loss activates the prototype module and handles label noise in livestreaming annotations, and that a sliding-window spatio-temporal encoder improves context modeling.

What carries the argument

The load-bearing mechanism is the prototype embedding module paired with the propensity-reweighted cross-entropy loss (LPWCE). Object-category word vectors serve as semantic prototypes: each HO pair union feature is concatenated with the prototype of its object category, passed through multi-head self-attention with a residual connection and a feedforward network, and then classified under LPWCE. LPWCE computes a label-frequency-based propensity score for each interaction class and uses it to reweight samples, so noisy or unimportant labels do not dominate training; the full objective is the class-balanced focal loss plus a weighted LPWCE term. The spatial attention encoder and temporal window encoder then fuse context for final HOI prediction. The paper observes that the prototype module alone can deactivate during training and that adding LPWCE is what activates its contribution.

What would settle it

Run PeO-HOI with the LPWCE loss but replace the semantic word-embedding prototypes with random vectors of the same dimension; if full mAP on BJUT-HOI stays near 45.13%, then the semantic content of the prototypes is not what drives the gain, and the object-bias explanation is not settled. An LPWCE-only arm without the prototype module would show whether the loss alone accounts for the improvement.

Watch

Extended reading notes

Core claim

PeO-HOI claims that object bias in livestreaming HOI detection can be mitigated by prototype embedding optimization. The method takes HO pair union features from an existing detection-and-tracking pipeline, concatenates them with category-level word-embedding vectors acting as prototypes, refines the concatenation through a self-attention module and feedforward network, and trains a classifier on the resulting prototype-embedded union features with a propensity-reweighted cross-entropy loss. After spatial attention and a sliding-window temporal encoder fuse context, the model predicts spatial and action relations. The paper reports that this pipeline raises full mAP from 35.64% to 37.19% on VidHOI and from 43.53% to 45.13% on BJUT-HOI relative to the gaze-based baseline [10], with the largest gains on non-rare classes, and presents t-SNE plots as evidence that prototype-embedded features form tighter clusters.

Load-bearing premise

The load-bearing premise is that word-embedding vectors of object names are good semantic prototypes for interactions, so that injecting them into HO pair union features shifts the feature space away from object bias; the ablation table shows the prototype module alone slightly reduces full and rare accuracy before the reweighting loss is added.

Editorial extensions

If this is right

  • If PeO-HOI's object-bias claim is right, livestreaming content review systems should become more reliable at flagging streamer-object interactions such as holding, lifting, and touching rather than merely detecting the object present.
  • The larger non-rare gains imply that common interaction patterns in livestreaming benefit most, while rare interactions improve only slightly, so the method does not fully solve long-tail imbalance.
  • Because the full loss requires LPWCE to activate the prototype module, future prototype-based HOI methods should pair semantic prototypes with sample-reweighted objectives instead of plain cross-entropy.
  • The sliding-window temporal encoder fuses nearby frames, which means the HOI representation can be updated incrementally as frames arrive, making the approach compatible with streaming rather than batch-only analysis.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's Table IV suggests a test it does not run: an LPWCE-only variant without the prototype embedding module. If that variant matches 45.13% mAP@full on BJUT-HOI, the prototype module is not the causal driver of the improvement, and the object-bias explanation would need different evidence.
  • The same component recipe could be transplanted to image-based HOI detectors by concatenating category word vectors with union features and reweighting with a propensity loss; image HOI also suffers from object bias, so this would be a direct transfer test of the claimed mechanism.
  • Since the prototypes are frozen word vectors, the method's domain transfer likely depends on how well the pretrained embedding space separates livestreaming object categories; a domain-adapted embedding or object-function prototype could change results.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes PeO-HOI, a video HOI detection method aimed at livestreaming scenarios. It builds HO-pair features from detector/tracker outputs, visual appearance, GloVe word embeddings, and gaze cues; embeds a semantic prototype (the GloVe vector of the object/preposition category) into the HO-pair union feature via a self-attention module; and optimizes the resulting feature space with a propensity-reweighted cross-entropy loss, followed by spatial and temporal encoders. Experiments are reported on the public VidHOI benchmark and on a self-built BJUT-HOI dataset, with gains over the Ni et al. baseline (37.19 vs. 35.64 mAP@full on VidHOI; 45.13 vs. 43.53 on BJUT-HOI). The central claim is that prototype embedding optimization mitigates object bias and that this is the cause of the improvements.

Significance. If the central attribution were supported, the contribution would be useful: a relatively light-weight prototype-embedding mechanism plus a propensity-reweighted loss that improves both rare and non-rare HOI categories on an external benchmark would be a clean, practical debiasing recipe. The paper also makes a reasonable effort to compare against a range of published video HOI methods and to ablate its two main components. However, the current evidence does not support the headline causal claim: the ablation omits the crucial baseline-plus-LPWCE control, the main hyperparameter is tuned directly on the evaluation set, and the t-SNE evidence is qualitative. The underlying pipeline is a standard supervised setup with no circularity, so these issues are fixable in revision, but the paper as written overstates what the experiments establish.

major comments (3)
  1. [Section III.G, Table IV] The ablation is incomplete for the paper's central claim. Group 1 adds only the prototype embedding module (PEN) to the baseline and it slightly decreases mAP@full (43.53 to 43.45) and mAP@rare (30.09 to 29.84); group 2 adds PEN plus LPWCE and produces the large gains (45.13/62.78/30.37). There is no baseline-plus-LPWCE control, so the observed gains in group 2 could come entirely from the propensity-reweighted loss in Eq. (8), which is itself a debiasing mechanism that down-weights frequent/noisy labels. The text in Section III.G even states that the prototype module is deactivated without LPWCE. To support the abstract's claim that prototype embedding optimization 'mitigates the effect of object bias,' the authors must add the missing baseline-plus-LPWCE condition and, ideally, a control with random or non-semantic prototype vectors; if the gains persist without the semantic prototypes, the causal claim should be revised.
  2. [Section III.F, Table III] The hyperparameter λ is selected directly on the BJUT-HOI evaluation set. The reported final numbers (45.13, 62.78, 30.37) are exactly the numbers obtained at λ=0.8 in Table III, and no separate validation split or cross-validation procedure is described. Tuning on the test set can inflate the reported gains and makes the 1.60-percentage-point improvement over Ni et al. unreliable. Please report results with λ chosen on a validation split, and provide mean and standard deviation over multiple training runs; without error bars or repeated runs, the reported differences of 1.55 and 1.60 points in Tables I and II may be within run-to-run noise.
  3. [Section III.E, Fig. 4] The t-SNE visualization is not quantitative evidence for the causal role of prototype embedding. The samples are filtered by frequency thresholds, 'cluster noise' is manually removed, and the panels show only selected representative clusters. No metric (e.g., silhouette score, intra-class vs. inter-class distance, or nearest-class accuracy) is reported, and the LPWCE-only condition is absent from the visualization. Please add quantitative feature-separation metrics computed on fixed, unpruned sample sets for the baseline, PEN-only, LPWCE-only, and PEN+LPWCE conditions, so the t-SNE can actually discriminate between the effects of the two components.
minor comments (5)
  1. [Section II.B.2, Eq. (7)] Equation (7) is corrupted in the manuscript: the formula for the propensity score α_l is unreadable and the definition of C is inconsistent with the displayed expression. Please rewrite it with standard notation (e.g., α_l = 1 / (1 + exp(-C log(N_l))) or the exact form from Jain et al.).
  2. [Throughout] There are several typos and grammatical slips, including 'smaples' for 'samples' in Section II.B.2, 'categoriy' in Section II.A, 'to to' in Section II.C, 'STUDAY' in the Table IV caption, and 'Sin position encoding' presumably for 'sinusoidal position encoding' in Algorithm 2.
  3. [Section III.A] The BJUT-HOI dataset is not released and the description is brief: only the number of videos and categories is given. Please provide details of the annotation protocol, label definitions, video sources, and a plan to release the dataset, since the paper's second benchmark is self-built and cannot currently be reproduced.
  4. [Section II.B.2, Eq. (10)] The notation in the class-balanced focal loss is confusing: p_y^i is used both as the predicted probability for class i and as the estimated probability y. Please define all variables explicitly, especially the role of the ground-truth label in the focal term.
  5. [Section II.B] Calling the GloVe word vector a 'prototype' is a strong use of the term: the embedding is a fixed pretrained vector rather than a learned category prototype. Clarifying that this is an injected semantic prior, rather than a learned prototype representation, would make the method description more accurate and the comparison with prior prototype-learning literature more precise.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: benchmark gains are genuine held-out predictions; the missing ablation control affects attribution, not derivational circularity.

full rationale

The derivation chain is self-contained: features are extracted with fixed pretrained components (YOLOv5, DeepSort, ResNet-101, GloVe), the prototype embedding module is a generic self-attention plus feedforward block, and the propensity-reweighted loss (Eq. 8) uses label frequencies from the training set. The reported VidHOI and BJUT-HOI mAP numbers are measured on held-out test data against external baselines; no fitted parameter is renamed as a prediction, no uniqueness theorem is imported, and no self-citation is load-bearing. The ablation in Table IV omits a baseline-plus-LPWCE control, so the causal attribution of the gains to the prototype module is under-supported, but that is an experimental-design and attribution issue, not circularity: the comparison itself does not reduce to its own inputs by construction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method relies on pretrained word embeddings as prototypes, off-the-shelf detectors and trackers, and several loss hyperparameters that are tuned on the evaluation datasets. No new physical or conceptual entities are introduced.

free parameters (3)
  • lambda = 0.8
    Weight balancing class-balanced focal loss and LPWCE; chosen by sensitivity analysis on BJUT-HOI in Section III.F, so test-set performance is parameter-selected.
  • beta = not reported
    Class-balanced focal loss exponent in Eq. (10); the value is not given in the paper.
  • gamma = not reported
    Focal loss focusing parameter in Eq. (10); the value is not reported.
assumptions (4)
  • domain assumption Word embeddings of object categories provide meaningful semantic prototypes for interactions
    The prototype embedding module concatenates the GloVe vector of the object category with union features; no evidence is given that these vectors align with interaction semantics.
  • domain assumption The object detector, tracker, and gaze module produce sufficiently accurate HO pair candidates
    Feature extraction relies on YOLOv5, DeepSORT, and a gaze module (Section II.A); errors in detection and tracking propagate to HOI features.
  • domain assumption The BJUT-HOI dataset annotations are correctly labeled
    The paper notes label noise in livestreaming data (Section I) but still uses BJUT-HOI to evaluate the method; noisy labels may inflate or deflate reported mAP.
  • standard math Class-balanced focal loss is an appropriate baseline for long-tailed HOI distributions
    Adopted from prior work; not derived here.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Prototype Embedding Optimization for Human-Object Interaction Detection in Livestreaming." pith.science (2026). https://pith.science/paper/EJDCV26U

@misc{pith2026250522011,
  author       = {Pith},
  title        = {Pith review of: Prototype Embedding Optimization for Human-Object Interaction Detection in Livestreaming},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EJDCV26U}},
  note         = {Machine review of arXiv:2505.22011}
}
read the original abstract

Livestreaming often involves interactions between streamers and objects, which is critical for understanding and regulating web content. While human-object interaction (HOI) detection has made some progress in general-purpose video downstream tasks, when applied to recognize the interaction behaviors between a streamer and different objects in livestreaming, it tends to focuses too much on the objects and neglects their interactions with the streamer, which leads to object bias. To solve this issue, we propose a prototype embedding optimization for human-object interaction detection (PeO-HOI). First, the livestreaming is preprocessed using object detection and tracking techniques to extract features of the human-object (HO) pairs. Then, prototype embedding optimization is adopted to mitigate the effect of object bias on HOI. Finally, after modelling the spatio-temporal context between HO pairs, the HOI detection results are obtained by the prediction head. The experimental results show that the detection accuracy of the proposed PeO-HOI method has detection accuracies of 37.19%@full, 51.42%@non-rare, 26.20%@rare on the publicly available dataset VidHOI, 45.13%@full, 62.78%@non-rare and 30.37%@rare on the self-built dataset BJUT-HOI, which effectively improves the HOI detection performance in livestreaming.

Figures

Figures reproduced from arXiv: 2505.22011 by the authors.

Figure 1
Figure 1. The proposed PeO-HOI detection. A. Feature Extraction of Human-Object Pairs In this paper, we model the HO pair feature representation from multiple dimensions such as visual appearance, word features, and streamer's gaze cues. We adopt an interactive system developed by Ni et al. [10] to produce the HO pair representation, including YOLOv5 [15] for object detection, DeepSort [16] for tracking, GloVe [17] for word e… view at source ↗
Figure 2
Figure 2. Prototype embedding optimization mechanism. The steps are as follows: a) Concate the matched HO pair union feature v u t,<i,j> with the corresponding interaction predicate embedding feature st,i to obtain the concated feature Ft: ( ) u F v s t t i j t i = Concat , . , , ,  (2) b) Feed Ft into a self-attention module with residual structure to capture the correlation between features to obtain the correlated featur… view at source ↗
Figure 4
Figure 4. (a), (b) illustrate the feature distributions on the VidHOI via t-SNE projections before using our method. These visualizations indicate that features from different classes overlap with one another, while features belonging to the same class are scattered, suggesting potential limitations in HOI detection. A comparable projection pattern is also observed in the BJUT-HOI dataset, asin [PITH_FULL_IMAGE:figures/full_… view at source ↗
Figures from the paper (1 more)
Figure 3
Figure 3. Figure 3: Visual results on BJUT-HOI dataset. E. t-SNE To demonstrate that prototype embedding union features are more discriminative for the HO pairs, we apply t-SNE on the VidHOI and BJUT-HOI datasets before and after prototype embedding optimization, as shown in [PITH_FULL_I…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

23 extracted references · 21 canonical work pages

  1. [1]

    Learning human activities and object affordances from RGB-D videos,

    H. S. Koppula, R. Gupta, and A. Saxena “Learning human activities and object affordances from RGB-D videos,” International Journal of Robotics Research, vol 32, pp. 951–970, 2013

    work page 2013

  2. [2]

    Learning human-object interactions by graph parsing neural networks,

    S. Qi, W. Wang, B. Jia, J. Shen, and S. Zhu “Learning human-object interactions by graph parsing neural networks,” European Conference on Computer Vision, Munich, Germany, Sep. 2018, pp. 401–417

    work page 2018

  3. [3]

    Ramakrishnan G. Lighten: Learning interactions with graph and hierarchical temporal networks for HOI in videos ,

    S. P. R. Sunkesula, and R. Dabral, “Ramakrishnan G. Lighten: Learning interactions with graph and hierarchical temporal networks for HOI in videos ,” ACM International Conference on Multimedia , Seattle, USA, Oct. 2020, pp. 691–699

    work page 2020

  4. [4]

    QPIC: Query -based pairwise human -object interaction detection with image -wide contextual information,

    M. Tamura, H . Ohashi, and T. Yoshinaga “QPIC: Query -based pairwise human -object interaction detection with image -wide contextual information,” IEEE Conference on Computer Vision and Pattern Recognition, Nashville, USA, Jun. 2021, pp.10410–10419

    work page 2021

  5. [5]

    Spatial-temporal transformer for dynamic scene graph generation ,

    Y. Cong, W . Liao, H . Ackermann, B . Rosenhahn, and M. Yang “Spatial-temporal transformer for dynamic scene graph generation ,” IEEE International Conference on Computer Vision, Montreal, Canada, Oct. 2022, pp. 16372–16382

    work page 2022

  6. [6]

    Video-based human- object interaction detection from tubelet tokens ,

    D. Tu, W. Sun, X. Min, G. Zhai, and W. Shen “Video-based human- object interaction detection from tubelet tokens ,” Proc. of Neural Information Processing Systems, New Orleans, USA, Nov. 2022, pp. 23345–23357

    work page 2022

  7. [7]

    End-to-end video scene graph generation with temporal propagation Transformer,

    Y. Zhang, Y. Pan, T. Yao, R. Huang, T. Mei, and C. Chen “End-to-end video scene graph generation with temporal propagation Transformer,” IEEE Transactions on Multimedia, vol. 26, no. 6, pp. 1613–1625, 2024

    work page 2024

  8. [8]

    ST- HOI: A spatial -temporal baseline for human -object interaction detection in videos ,

    M. J. Chiou, C. Y. Liao, L. Wang, R. Zimmermann, and J. Feng “ST- HOI: A spatial -temporal baseline for human -object interaction detection in videos ,” Intelligent Cross-Data Analysis and Retrieval, Taipei, China, Aug. 2021, pp. 9–17

    work page 2021

Show all 23 references
  1. [9]

    From detection to understanding: A survey on representation learning for human -object interaction,

    T. Luo, S . Guan, R . Yang, and J. Smith “From detection to understanding: A survey on representation learning for human -object interaction,” Neurocomputing, vol. 543, pp. 126243, 2023

    2023

  2. [10]

    Human-object interaction prediction in videos through gaze following ,

    Z. Ni, EV. Mascaró, H. Ahn, and D. Lee “Human-object interaction prediction in videos through gaze following ,” Computer Vision and Image Understanding, vol. 233, 103741, 2023

    2023

  3. [11]

    Highlighting object category immunity for the generalization of human -object interaction detection ,

    X. Liu, Y. L. Li, and C. Lu “Highlighting object category immunity for the generalization of human -object interaction detection ,” AAAI Conference on Artificial Intelligence , Virtual, Feb. 2022, pp. 1819– 1827

    2022

  4. [12]

    Chairs can be stood on: Overcoming object bias in human -object interaction detection ,

    G. Wang, Y. Guo, Y. Wong, and M. Kankanhalli “Chairs can be stood on: Overcoming object bias in human -object interaction detection ,” European Conference on Computer Vision, Tel-Aviv, Israel, Oct. 2022, pp. 654–672

    2022

  5. [13]

    Prototype rectification for few -shot learning,

    J. Liu, L. Song, and Y. Qin “Prototype rectification for few -shot learning,” European Conference on Computer Vision, Glasgow, UK, Aug. 2020, pp. 741–756

    2020

  6. [14]

    Prototype contrastive learning for point - supervised temporal action detection ,

    P. Li, J . Cao, X. Ye and “Prototype contrastive learning for point - supervised temporal action detection ,” Expert Systems with Applications, vol. 213, pp. 118965 2023

    2023

  7. [15]

    Ultralytics YOLOv5,

    G. Jocher, "Ultralytics YOLOv5," Software, Version 7.0, Zenodo,

  8. [16]

    Simple online and realtime tracking with a deep association metric,

    N. Wojke, A . Bewley, and D. Paulus “Simple online and realtime tracking with a deep association metric,” IEEE International Conference on Image Processing, Beijing, China, Sep. 2017, pp. 3645– 3649

    2017

  9. [17]

    GloVe: Global vectors for word representation ,

    J. Pennington, R. Socher, and C.D. Manning “GloVe: Global vectors for word representation ,” Empirical Methods in Natural Language Processing, Doha, Qatar, Oct. 2014, pp. 1532–1543

    2014

  10. [18]

    Extreme multi-label loss functions for recommendation, tagging, ranking & other missing label applications,

    H. Jain, Y. Prabhu, and M. Varma “Extreme multi-label loss functions for recommendation, tagging, ranking & other missing label applications,” ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, New York, USA, Aug. 2016, pp. 935–944

    2016

  11. [19]

    Ramakrishnan G. Skew -robust human-object interactions in videos ,

    A. Agarwal, R. Dabral, and A. Jain, “Ramakrishnan G. Skew -robust human-object interactions in videos ,” IEEE Winter Conference on Applications of Computer Vision , Waikoloa, USA, Jan. 2023, pp. 5098–5107

    2023

  12. [20]

    CrowdHuman: A benchmark for detecting human in a crowd,

    S. Shao, Z . Zhao, B . Li, T . Xiao, Yu G, Zhang X, and Sun J. “CrowdHuman: A benchmark for detecting human in a crowd,” arXiv preprint, arXiv:1805.00123,2018

    2018 arXiv

  13. [21]

    Detecting attended visual targets in video ,

    E. Chong, Y. Wang, N. Ruiz, and J. Rehg “Detecting attended visual targets in video ,” IEEE Conference on Computer Vision and Pattern Recognition, Seattle, USA, Jun. 2020, pp. 5396–5406

    2020

  14. [22]

    Open Set Video HOI detection from Action-centric Chain -of-Look Prompting ,

    N. Xi , J . Meng, and J. Yuan “Open Set Video HOI detection from Action-centric Chain -of-Look Prompting ,” IEEE International Conference on Computer Vision , Paris, France, Oct. 2024, pp. 3079– 3089

    2024

  15. [2020]

    Available: https://doi.org/10.5281/zenodo.3908559

    [Online]. Available: https://doi.org/10.5281/zenodo.3908559

    doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.