Pith. sign in

REVIEW 4 major objections 4 minor 38 references

OV-HHIR: Open Vocabulary Human Interaction Recognition Using Cross-modal Integration of Large Language Models

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proposes OV-HHIR, a three-branch vision-language model that feeds person and background video streams to LLaMA 2, generating open-vocabulary interaction descriptions and recognizing interactions never seen in training.

desk verdict A genuine multi-branch architecture and a useful dataset idea are undermined by an evaluation that lacks a held-out split and uses a metric the authors themselves call misleading. read the letter →

arxiv 2501.00432 v1 pith:XPPNOAND submitted 2024-12-31 cs.CV cs.LG

Lala Shakti Swarup Ray , Bo Zhou , Sungho Suh , Paul Lukowicz This is my paper · ORCID
classification cs.CVcs.LG
keywords humanactivityrecognitionhuman-to-humaninteractionopen-vocabularyvideounderstandinglargelanguagemodelscross-modallearningsegmentationLLaMA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to establish that an open-vocabulary, LLM-backed video model can recognize human-to-human interactions, including interactions it has never seen, by converting each video into separate person and background streams and producing free-form textual descriptions. The motivation is that real-world surveillance and monitoring demand systems that adapt to novel, unpredictable interactions rather than being confined to a predetermined label set. To make this trainable, the paper pools ten existing datasets into one 86,623-video benchmark, HHIRChat, with 103 interaction classes and GPT-4-generated descriptive labels. The reported results put the model ahead of VideoLLaMA-2 LoRA (cosine similarity 0.628 vs. 0.414) and ahead of fixed-vocabulary classifiers on several datasets, and it is the only method that can attempt the unseen-class evaluation at all. If these numbers hold up under a clean evaluation protocol, the practical consequence is a single, general-purpose interaction recognizer for open-world video.

What carries the argument

The machinery is a multi-branch cross-modal alignment built on Video-LLaMA. Track Anything first segments each video into a Person 1 stream, a Person 2 stream, and an optional background stream. Each stream is encoded by a frozen feature extractor—ViTPose for the person branches, a vanilla ViT for the background—then passed through a video Q-Former (borrowed from BLIP-2) and linear projection layers that map the video embeddings into the same space as text embeddings. The three projected embeddings are concatenated and fed into the frozen LLaMA 2 13B chat model, which generates open-vocabulary captions. Training uses cross-entropy between the model's embedding and the GPT-4-generated soft labels, updating only the Q-Formers and the linear layers; all encoders and the LLM stay frozen.

What would settle it

Run the exact OV-HHIR pipeline on HHIRChat with an explicit, documented train/test split (for instance, 80/20 at the video level) and compute Table II's cosine similarity only on held-out videos; if the score drops to the level of VideoLLaMA-2 LoRA or below, the advertised open-vocabulary advantage is not demonstrated. Independently, have a small panel of human annotators rate whether the GPT-4 soft labels match the video content; a high mismatch rate would indicate that the training signal itself is unreliable.

Watch

Extended reading notes

Core claim

The paper's central claim is that splitting an interaction video into three branches—Person 1, Person 2, and background—and feeding the separately encoded streams into a frozen LLaMA 2 13B chat model unlocks open-vocabulary interaction recognition. On the HHIRChat benchmark, the model obtains a mean cosine similarity of $0.628 \pm 0.019$ between its generated captions and ground-truth descriptions, against $0.414 \pm 0.033$ for VideoLLaMA-2 LoRA, and it outperforms or matches the fixed-vocabulary baselines on most of the ten constituent datasets. When evaluated through a prompt-engineered classifier on six interaction classes that were deliberately excluded from training, it reaches a macro-F1 of $0.315 \pm 0.023$; the paper notes that any fixed-vocabulary classifier would necessarily score zero on such classes. The qualitative examples further indicate that the model's captions track the actual interaction more closely than vanilla VideoLLaMA-2 or its LoRA adaptation.

Load-bearing premise

The load-bearing premise is that the 103-class unified label set produced by GPT-4 is accurate and consistent across the ten source datasets, and that the reported scores are computed on a held-out split; the paper does not document the label-merging process, validate the generated descriptions, or state whether the cosine-similarity numbers come from training or test videos.

Editorial extensions

If this is right

  • A fixed-vocabulary classifier cannot assign any label to an interaction it has not been trained on, whereas the OV-HHIR classifier reports a macro-F1 of 0.315 on six such unseen classes.
  • Because the model outputs free-form text, it can be used to retrieve or filter surveillance footage with natural-language queries instead of a fixed label list.
  • The unified HHIRChat dataset (86,623 sequences, 103 categories) gives the field a single common benchmark, making results from different labs comparable.
  • The per-person branches reduce interference in crowded scenes, since the model sees each individual as a separate stream rather than one fused frame.
  • The reported cosine similarity of 0.628 over caption embeddings suggests the generated descriptions are semantically close to ground truth, not just label-matched.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: A sharper test of the open-vocabulary claim would be a zero-shot evaluation where the classifier is trained only on a randomly chosen subset of the 103 classes and tested on the rest, rather than on six hand-picked YouTube clips; this would give a controlled measure of how recognition degrades as the label gap widens.
  • Inference: Because the GPT-4 soft labels are not human-validated, a useful extension is a human-annotation study on a random sample of HHIRChat to measure label noise; if label noise is high, some of the reported similarity gain may reflect the labels rather than the architecture.
  • Inference: The three-branch design suggests a direct route to group interactions (three or more people) by adding one branch per tracked individual; the paper lists this as future work but the mechanism is already modular.
  • Inference: The model's natural-language output could be consumed by downstream surveillance queries (e.g., 'flag any video where one person strikes another'), which the paper motivates but does not demonstrate.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes OV-HHIR, an open-vocabulary human-to-human interaction recognition framework. Videos are first decomposed into Person 1, Person 2, and background streams using Track Anything; each stream is fed to a separate vision-language branch using ViTPose or ViT encoders, Q-Formers, and linear projection layers; the fused embeddings are passed to a frozen LLaMA-2 13B chat model that is trained to generate GPT-4-produced textual descriptions. The authors also introduce HHIRChat, a unified dataset of 86,623 video sequences covering 103 interaction classes assembled from ten existing datasets. Quantitative evaluation reports cosine similarity between generated and reference embeddings on HHIRChat, macro-F1 scores on per-dataset classification, and an open-set experiment with six unseen activity classes.

Significance. If the claimed results are correct, the paper makes two useful contributions: a large unified human-interaction dataset with GPT-4-generated soft labels, and a multi-branch architecture that decouples segmentation, feature extraction, and language generation for interaction recognition. The open-set evaluation idea is also valuable, as fixed-vocabulary classifiers are indeed limited for emerging interaction types. However, the quantitative evidence in the current manuscript is not sufficient to support the central claims. The headline cosine-similarity comparison is computed without a stated train/test split, and the paper itself concedes that this metric can be misleading. Because the training objective is to reproduce the GPT-4 captions on the same HHIRChat data, an in-sample evaluation would strongly favor OV-HHIR over baselines that did not train on those captions. The open-set experiment likewise lacks essential details. The dataset and architecture are plausible resources, but the current evaluation does not establish superiority over existing methods.

major comments (4)
  1. [Section III-A, Table II] The paper does not state whether the videos used for the cosine-similarity evaluation in Table II are held out from training. Since OV-HHIR is trained with cross-entropy between LLaMA-generated embeddings and GPT-4 ground-truth embeddings on the same HHIRChat data, an in-sample evaluation would be heavily biased in its favor. The problem is compounded by Section III-B, which explicitly states that 'Purely relying on cosine similarity can lead to incorrect evaluations' and calls the metric 'inaccurate.' Therefore the headline comparison (0.628 vs 0.414) is not established as a fair measure of generalization. The authors must specify the exact split, report which videos or classes are used for evaluation, and either justify cosine similarity as a valid metric despite their own caveat or replace it with a metric they consider reliable.
  2. [Section II-A, Table I] The procedure for merging ten datasets with heterogeneous label vocabularies into 103 unified interaction classes is not described. The paper simply states that GPT-4 was used to convert hard labels into descriptive soft labels, but does not provide the taxonomy mapping, the prompt used, or any validation of the generated descriptions. Since HHIRChat is both a claimed contribution and the training signal for all experiments, inconsistent or erroneous soft labels would undermine every reported result. The authors should provide a detailed label-merging protocol, examples of original and converted labels, and an analysis of label consistency and errors.
  3. [Section III-A, open-set evaluation] The open-vocabulary claim is supported only by a macro-F1 score of 0.315 +/- 0.023 on 'a set of 6 uncommon activities' collected from YouTube. The paper does not report the number of videos per activity, the class distribution, or whether these videos were used in any way during development of the classifier prompt. Moreover, the assertion that 'The same approach would always result in 0 for any classification model' is not demonstrated and is not generally true, since a fixed-vocabulary classifier can be designed to reject out-of-vocabulary inputs. To support the central contribution, the authors should provide per-class results, dataset statistics, and a comparison against a reasonable open-set-capable baseline.
  4. [Section III-B, Fig. 3] The qualitative example in Fig. 3 illustrates a deeper problem with the chosen evaluation metric: in the second example, OV-HHIR achieves a lower cosine similarity (0.42) than VideoLLaMA-2 LoRA (0.58), even though the OV-HHIR caption is semantically correct and the LoRA caption is wrong. This directly contradicts the use of cosine similarity as the primary quantitative metric in Table II. The paper should either adopt an evaluation metric that ranks semantic correctness appropriately or explain why Table II's cosine-similarity ranking is still meaningful despite this counterexample.
minor comments (4)
  1. [Figure 1] The JSON example contains a malformed field: `"bg:"0_bg.mp4"` is missing the colon-space and proper quoting; it should be `"bg": "0_bg.mp4"`.
  2. [Section III-A] The text lists '6 uncommon activities, i.e., bowing down, ear whispering, forehead kissing, lifting, and taunting' but only five activities are listed; the sixth should be added.
  3. [Table I] The row for DeepMind Kinetics appears to have only one action count and one sample count despite including multiple kinetics classes, and the NTU RGB+D row is formatted ambiguously; please clarify the table entries.
  4. [References] Reference [14] is titled 'Als-har' in the bibliography but the text refers to 'ASL-HAR'; the inconsistency should be fixed.

Circularity Check

1 steps flagged · score 6.0 of 10

Headline cosine-similarity score is the direct training objective measured on an unspecified split, and the paper itself disowns the metric.

  1. fitted input called prediction [Section III (Experiments), first paragraph; Section III-A, Table II]
    "Cross Entropy loss is calculated between embeddings generated by LLaMA 2 and the ground truth."

    The training objective is to make the model's generated embedding match the GPT-4 ground-truth embedding. Table II then reports the mean cosine similarity of generated embeddings on the HHIRChat dataset without ever stating a train/test split, and Section III-A says the OV-HHIR model spans the entire HHIRChat dataset. The reported score is therefore the same quantity the loss optimizes, measured on data the model was trained on, so it is consistent with being an in-sample value of the fitted objective rather than an external prediction. The paper itself concedes in Section III-B that 'purely relying on cosine similarity can lead to incorrect evaluations,' further weakening the headline comparison.

full rationale

This is an empirical systems paper, so the circularity check centers on evaluation design rather than mathematical derivation. The clearest circular step is the headline quantitative claim: the model is trained by cross-entropy between its generated embeddings and GPT-4 ground-truth embeddings, and the main result in Table II is the cosine similarity between those same classes of embeddings on HHIRChat, with no held-out split described and with the model described as spanning the entire HHIRChat dataset. Under that reading, the 0.628 score is a direct measure of the training objective, not a generalization result. The Table III macro-F1 comparisons share the same missing-split ambiguity, while the six-class YouTube evaluation is a genuinely external open-vocabulary check and is not circular. Self-citations to T2P [8] and ALS-HAR [14] are methodological rather than load-bearing: the soft-label ground truth is generated by GPT-4 [22], an external model, and no central result is forced by a self-citation chain. The architecture itself is composed of standard, independently published components. Overall, the central SOTA comparison partially reduces to the fitted objective on an unspecified split, giving a score of 6; the open-vocabulary idea retains independent content from the held-out YouTube evaluation.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method relies on several unvalidated domain assumptions: the accuracy of Track Anything masks, the correctness of GPT-4-generated captions as ground truth, the semantic consistency of merging ten label sets into 103 classes, and the suitability of aligning ViTPose/ViT features to LLaMA 2's text space. None of these are tested or discussed.

free parameters (3)
  • sampled frames per video = 16
    Hand-chosen to manage compute for dynamic video lengths; no sensitivity analysis or ablation is reported (Section III).
  • HHIRChat action class count = 103
    The number of classes resulting from merging ten public datasets is an arbitrary granularity; the mapping procedure from the original labels is not described (Table I).
  • Classification prompt template = not specified
    The classifier prompts the LLM to output one of N classes, but the exact template and parsing logic are not given in the paper (Section III-A).
assumptions (4)
  • domain assumption Track Anything produces accurate masks separating person 1, person 2, and the background
    The entire multi-branch pipeline depends on correct person separation; no evaluation of mask quality or failure modes is provided (Section II-B).
  • domain assumption GPT-4-generated soft descriptions are correct and consistent targets for training
    Hard labels from ten datasets are converted to sentences via GPT-4 with no human verification, consistency check, or conflict resolution (Section II-A).
  • domain assumption The 103-class taxonomy from merging ten datasets is semantically coherent
    The paper does not explain how overlapping or conflicting labels (e.g., 'punching' vs 'hitting') are mapped to a single class, which affects both training and evaluation (Table I).
  • domain assumption The visual embeddings can be aligned to the LLaMA 2 text embedding space via Q-Former and FC layers
    The alignment is trained with a cross-entropy loss on the generated captions, but the paper gives no analysis of the alignment quality or of the representation gap (Section II-B).

how reviews work

0 comments
Cite this review

Pith. "Pith review of OV-HHIR: Open Vocabulary Human Interaction Recognition Using Cross-modal Integration of Large Language Models." pith.science (2026). https://pith.science/paper/XPPNOAND

@misc{pith2026250100432,
  author       = {Pith},
  title        = {Pith review of: OV-HHIR: Open Vocabulary Human Interaction Recognition Using Cross-modal Integration of Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XPPNOAND}},
  note         = {Machine review of arXiv:2501.00432}
}
read the original abstract

Understanding human-to-human interactions, especially in contexts like public security surveillance, is critical for monitoring and maintaining safety. Traditional activity recognition systems are limited by fixed vocabularies, predefined labels, and rigid interaction categories that often rely on choreographed videos and overlook concurrent interactive groups. These limitations make such systems less adaptable to real-world scenarios, where interactions are diverse and unpredictable. In this paper, we propose an open vocabulary human-to-human interaction recognition (OV-HHIR) framework that leverages large language models to generate open-ended textual descriptions of both seen and unseen human interactions in open-world settings without being confined to a fixed vocabulary. Additionally, we create a comprehensive, large-scale human-to-human interaction dataset by standardizing and combining existing public human interaction datasets into a unified benchmark. Extensive experiments demonstrate that our method outperforms traditional fixed-vocabulary classification systems and existing cross-modal language models for video understanding, setting the stage for more intelligent and adaptable visual understanding systems in surveillance and beyond.

Figures

Figures reproduced from arXiv: 2501.00432 by the authors.

Figure 1
Figure 1. HHIRChat data format that mixes cross-modal tokens. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the proposed OV-HHIR framework that uses three video language branches and LLaMA 2 13B chat to generate open-ended natural [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Examples showing the ground-truth vs predictions by Vanilla VideoLLaMA 2, VideoLLaMA 2 LoRA, and OV-HHIR. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 23 canonical work pages

  1. [1]

    Llama: Open and efficient foundation language models,

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi `ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al., “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971 , 2023

    arXiv 2023

  2. [2]

    Gemma: Open models based on gemini research and technology,

    Gemma Team, Thomas Mesnard, et al., “Gemma: Open models based on gemini research and technology,” arXiv preprint arXiv:2403.08295 , 2024

    arXiv 2024

  3. [3]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” in International conference on machine learning. PMLR, 2023, pp. 19730–19742

    work page 2023

  4. [4]

    Language-grounded dynamic scene graphs for interactive object search with mobile manipulation,

    Daniel Honerkamp, Martin B ¨uchner, Fabien Despinoy, Tim Welschehold, and Abhinav Valada, “Language-grounded dynamic scene graphs for interactive object search with mobile manipulation,” IEEE Robotics and Automation Letters , 2024

    work page 2024

  5. [5]

    Instruct2act: Mapping multi-modality instruc- tions to robotic actions with large language model,

    Siyuan Huang, Zhengkai Jiang, Hao Dong, Yu Qiao, Peng Gao, and Hongsheng Li, “Instruct2act: Mapping multi-modality instruc- tions to robotic actions with large language model,” arXiv preprint arXiv:2305.11176, 2023

    arXiv 2023

  6. [6]

    Code llama: Open foundation models for code,

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, J ´er´emy Rapin, et al., “Code llama: Open foundation models for code,” arXiv preprint arXiv:2308.12950, 2023

    arXiv 2023

  7. [7]

    Codegen: An open large language model for code with multi-turn program synthesis,

    Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong, “Codegen: An open large language model for code with multi-turn program synthesis,” arXiv preprint arXiv:2203.13474, 2022

    arXiv 2022

  8. [8]

    Text me the data: Generating ground pressure sequence from textual descriptions for har,

    Lala Shakti Swarup Ray, Bo Zhou, Sungho Suh, Lars Krupp, Vi- tor Fortes Rey, and Paul Lukowicz, “Text me the data: Generating ground pressure sequence from textual descriptions for har,” in 2024 IEEE International Conference on Pervasive Computing and Commu- nications Workshops and other Affiliated Events (PerCom Workshops) . IEEE, 2024, pp. 461–464

    work page 2024

Show all 38 references
  1. [9]

    Video-llama: An instruction- tuned audio-visual language model for video understanding,

    Hang Zhang, Xin Li, and Lidong Bing, “Video-llama: An instruction- tuned audio-visual language model for video understanding,” arXiv preprint arXiv:2306.02858, 2023

    2023 arXiv

  2. [10]

    Bliva: A simple multimodal llm for better handling of text-rich visual questions,

    Wenbo Hu, Yifan Xu, Yi Li, Weiyue Li, Zeyuan Chen, and Zhuowen Tu, “Bliva: A simple multimodal llm for better handling of text-rich visual questions,” in Proceedings of the AAAI Conference on Artificial Intelligence, 2024, vol. 38, pp. 2256–2264

    2024

  3. [11]

    Infogcn: Representation learning for human skeleton-based action recognition,

    Hyung-gun Chi, Myoung Hoon Ha, Seunggeun Chi, Sang Wan Lee, Qixing Huang, and Karthik Ramani, “Infogcn: Representation learning for human skeleton-based action recognition,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 20186–20196

    2022

  4. [12]

    Mujo: Multimodal joint feature space learning for human activity recognition,

    Stefan Gerd Fritsch, Cennet Oguz, Vitor Fortes Rey, Lala Ray, Maximil- ian Kiefer-Emmanouilidis, and Paul Lukowicz, “Mujo: Multimodal joint feature space learning for human activity recognition,” arXiv preprint arXiv:2406.03857, 2024

    2024 arXiv

  5. [13]

    Decoupled spatial- temporal attention network for skeleton-based action-gesture recogni- tion,

    Lei Shi, Yifan Zhang, Jian Cheng, and Hanqing Lu, “Decoupled spatial- temporal attention network for skeleton-based action-gesture recogni- tion,” in Proceedings of the Asian conference on computer vision , 2020

    2020

  6. [14]

    Als-har: Harnessing wearable ambient light sensors to enhance imu-based har,

    Lala Shakti Swarup Ray, Daniel Geißler, Mengxi Liu, Bo Zhou, Sungho Suh, and Paul Lukowicz, “Als-har: Harnessing wearable ambient light sensors to enhance imu-based har,” arXiv preprint arXiv:2408.09527 , 2024

    2024 arXiv

  7. [15]

    Human- to-human interaction detection,

    Zhenhua Wang, Kaining Ying, Jiajun Meng, and Jifeng Ning, “Human- to-human interaction detection,” in Neural Information Processing: 30th International Conference, ICONIP 2023, Changsha, China, Novem- ber 20–23, 2023, Proceedings, Part IV , Berlin, Heidelberg, 2023, p. 120–132...

    2023

  8. [16]

    A two-stream hybrid cnn-transformer network for skeleton-based human interaction recognition,

    Ruoqi Yin and Jianqin Yin, “A two-stream hybrid cnn-transformer network for skeleton-based human interaction recognition,” arXiv preprint arXiv:2401.00409, 2023

    2023 arXiv

  9. [17]

    Hargpt: Are llms zero-shot human activity recognizers?,

    Sijie Ji, Xinzhe Zheng, and Chenshu Wu, “Hargpt: Are llms zero-shot human activity recognizers?,” 2024

    2024

  10. [18]

    Unsupervised human activity recognition through two-stage prompting with chatgpt,

    Qingxin Xia, Takuya Maekawa, and Takahiro Hara, “Unsupervised human activity recognition through two-stage prompting with chatgpt,” arXiv preprint arXiv:2306.02140 , 2023

    2023 arXiv

  11. [19]

    Segment anything,

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al., “Segment anything,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 4015–4026

    2023

  12. [20]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    Alexey Dosovitskiy, Lucas Beyer, et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” 2021

    2021

  13. [21]

    Llama 2: Open foundation and fine- tuned chat models,

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Alma- hairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al., “Llama 2: Open foundation and fine- tuned chat models,” arXiv preprint arXiv:2307.09288 , 2023

    2023 arXiv

  14. [22]

    Gpt-4 technical report,

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  15. [23]

    Track anything: Segment anything meets videos,

    Jinyu Yang, Mingqi Gao, Zhe Li, Shang Gao, Fangjing Wang, and Feng Zheng, “Track anything: Segment anything meets videos,” arXiv preprint arXiv:2304.11968, 2023

    2023 arXiv

  16. [24]

    Actions in context,

    Marcin Marszalek, Ivan Laptev, and Cordelia Schmid, “Actions in context,” in 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2009, pp. 2929–2936

    2009

  17. [25]

    Sportshhi: A dataset for human-human interaction detection in sports videos,

    Tao Wu, Runyu He, Gangshan Wu, and Limin Wang, “Sportshhi: A dataset for human-human interaction detection in sports videos,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 18537–18546

    2024

  18. [26]

    High five: Recognising human interactions in tv shows.,

    Alonso Patron-Perez, Marcin Marszalek, Andrew Zisserman, and Ian Reid, “High five: Recognising human interactions in tv shows.,” in BMVC, 2010, vol. 1, p. 33

    2010

  19. [27]

    Two-person interaction detection using body- pose features and multiple instance learning,

    Kiwon Yun, Jean Honorio, Debaleena Chattopadhyay, Tamara L Berg, and Dimitris Samaras, “Two-person interaction detection using body- pose features and multiple instance learning,” in 2012 IEEE computer society conference on computer vision and pattern recognition work- shops. ...

    2012

  20. [28]

    First-person activity recognition: What are they doing to me?,

    M. S. Ryoo and L. Matthies, “First-person activity recognition: What are they doing to me?,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Portland, OR, June 2013

    2013

  21. [29]

    Interaction relational net- work for mutual action recognition,

    Mauricio Perez, Jun Liu, and Alex C. Kot, “Interaction relational net- work for mutual action recognition,” IEEE Transactions on Multimedia (TMM), 2021

    2021

  22. [30]

    The kinetics human action video dataset,

    Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al., “The kinetics human action video dataset,” arXiv preprint arXiv:1705.06950, 2017

    2017 arXiv

  23. [31]

    Air- act2act: Human–human interaction dataset for teaching non-verbal social behaviors to robots,

    Woo-Ri Ko, Minsu Jang, Jaeyeon Lee, and Jaehong Kim, “Air- act2act: Human–human interaction dataset for teaching non-verbal social behaviors to robots,” The International Journal of Robotics Research , vol. 40, no. 4-5, pp. 691–697, 2021

    2021

  24. [32]

    Human behavior under- standing,

    Hyun Soo Park, Albert Ali Salah, Yong Jae Lee, Louis-Philippe Morency, Yaser Sheikh, and Rita Cucchiara, “Human behavior under- standing,” in Human Behavior Understanding , sept 2014

    2014

  25. [33]

    Ntu rgb+ d 120: A large-scale benchmark for 3d human activity understanding,

    Jun Liu, Amir Shahroudy, Mauricio Perez, Gang Wang, Ling-Yu Duan, and Alex C Kot, “Ntu rgb+ d 120: A large-scale benchmark for 3d human activity understanding,” IEEE transactions on pattern analysis and machine intelligence , vol. 42, no. 10, pp. 2684–2701, 2019

    2019

  26. [34]

    Caption anything: Interactive image description with diverse multimodal controls,

    Teng Wang, Jinrui Zhang, Junjie Fei, Hao Zheng, Yunlong Tang, Zhe Li, Mingqi Gao, and Shanshan Zhao, “Caption anything: Interactive image description with diverse multimodal controls,” arXiv preprint arXiv:2305.02677, 2023

    2023 arXiv

  27. [35]

    Vitpose: Sim- ple vision transformer baselines for human pose estimation,

    Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao, “Vitpose: Sim- ple vision transformer baselines for human pose estimation,” Advances in Neural Information Processing Systems , vol. 35, pp. 38571–38584, 2022

    2022

  28. [36]

    Tokens- to-token vit: Training vision transformers from scratch on imagenet,

    Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan, “Tokens- to-token vit: Training vision transformers from scratch on imagenet,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021...

    2021

  29. [37]

    Vide- ollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms,

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al., “Vide- ollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms,” arXiv preprint arXiv:2406.07476 , 2024

    2024 arXiv

  30. [38]

    Lora: Low-rank adaptation of large language models,

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.