Pith. sign in

REVIEW 4 major objections 5 minor 55 references

HyperGLM: HyperGraph for Video Scene Graph Generation and Anticipation

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A unified hypergraph of spatial and causal relationships, injected into an LLM, improves video scene graph generation, anticipation, and reasoning across five tasks.

desk verdict Useful hypergraph idea and a big new dataset, but the five-task SOTA claim is undermined by an unspecified test-time graph and the paper's own captioning numbers. read the letter →

arxiv 2411.18042 v2 pith:WTKQAT5T submitted 2024-11-27 cs.CV

Trong-Thuan Nguyen , Pha Nguyen , Jackson Cothren , Alper Yilmaz , Khoa Luu This is my paper · ORCID
classification cs.CV
keywords videoscenegraphgenerationanticipationhypergraphmultimodallargelanguagemodelsquestionansweringcaptioningrelationreasoningdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that video scene understanding improves when object relationships are represented as a unified hypergraph—hyperedges connecting multiple entities and relationship types—and when that hypergraph is injected into a large language model. Prior video scene graph methods, the authors argue, model only pairwise relationships and miss multi-way interactions and the causal evolution of relations across frames. The authors build the VSGR dataset (1.9 million frames, three viewpoints, five tasks) and present HyperGLM as consistently outperforming state-of-the-art baselines on scene graph generation, scene graph anticipation, video question answering, video captioning, and relation reasoning. If the claim holds, higher-order graph structure would be a useful intermediate representation for LLM-based video reasoning.

What carries the argument

The central object is the unified HyperGraph $H = (V_H, E_H)$, which combines entity scene graphs $G_t$ (spatial subject–relationship–object triples per frame) with a procedural graph $P$ (weighted transitions between relationship categories, with probabilities $w(r_m, r_n)$ estimated from observed frequencies). A random-walk algorithm (Alg. 1) samples hyperedges from this combined structure—e.g., $\{person, holding, placing, releasing\}$—capturing multi-object and multi-step interactions. This hypergraph is injected into a Mistral-7B instruction-tuned LLM through the conditioning $p(A \mid V, Q, H)$, and the procedural graph's transition probabilities are what allow the model to anticipate future relationships.

What would settle it

A decisive experiment: evaluate the same trained HyperGLM on VSGR VQA and Relation Reasoning twice, once with the hypergraph injected and once with a video-only prompt. If accuracy collapses when the graph is withheld, and matches the reported numbers when the LLM is given the graph text alone without video, the reported gains reflect reading injected labels rather than video reasoning.

Watch

Extended reading notes

Core claim

HyperGLM's central claim is that a unified HyperGraph, merging per-frame entity scene graphs with a procedural graph of relationship transitions, enables a multimodal LLM to reason about multi-object interactions better than pairwise scene graphs alone. The hypergraph is constructed by random walks that sample hyperedges (for instance, person–sitting–holding–playing guitar), and the generation objective $p(A \mid V, Q, H)$ conditions the language model directly on this structure. The paper reports consistent gains over transformer-based and NeuralODE/NeuralSDE baselines, with SGG Recall@20 of 7.5% on PVSG and 35.8% on VSGR, SGA Recall@10 of 35.7% on Action Genome and 25.1% on VSGR, 45.4% accuracy on VQA, and 47.2% accuracy on Relation Reasoning.

Load-bearing premise

The load-bearing premise is that the hypergraph fed to the LLM at test time contains only information the model could legitimately infer from the video, not the ground-truth relationships used to write the VSGR questions and captions.

Editorial extensions

If this is right

  • If HyperGLM's claim holds, replacing pairwise scene-graph modules with hypergraph construction should improve downstream video tasks that depend on multi-object relationships, such as event forecasting and human–object interaction understanding.
  • The VSGR dataset gives the community a single benchmark with 1.9M frames and annotations for all five tasks across third-person, egocentric, and drone views, enabling direct comparison of generation, anticipation, and reasoning methods.
  • The reported gap between HyperGLM and the hypergraph-only variant indicates that injecting the structured graph into an LLM yields additional reasoning gains beyond the graph representation alone.
  • Because the procedural graph is built from observed transition frequencies, the method's anticipation ability should carry over to new videos whose relationship dynamics follow the same regularities as the training data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not state whether the hypergraph injected at test time for VQA, VC, and RR is model-predicted or ground truth; if it is ground truth, the reported gains over video-only baselines likely overstate the model's reasoning ability.
  • Because the VSGR questions and captions are generated from the same <subject, relation, object> triplets that the hypergraph encodes, a decisive test is to withhold the hypergraph at test time and measure how much accuracy survives—this separates graph-reading from genuine video reasoning.
  • The random-walk parameters ($N_w = 60$, $N_l = 7$) are fixed; a testable extension would make the walk count and length adaptive to scene complexity rather than constant across datasets.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes HyperGLM, a multimodal LLM-based framework for video scene graph generation (SGG), scene graph anticipation (SGA), video question answering (VQA), video captioning (VC), and relation reasoning (RR). The method constructs a unified HyperGraph H that merges per-frame entity scene graphs with a procedural graph modeling relationship transitions, samples hyperedges via random walks, and injects H into an LLM (Mistral-7B with LoRA). The authors also introduce the VSGR dataset with 1.9M frames from third-person, egocentric, and drone viewpoints, supporting all five tasks. The paper claims that HyperGLM consistently outperforms state-of-the-art methods across all five tasks on PVSG, Action Genome, and VSGR. The manuscript includes algorithm pseudocode, implementation details, ablation plots for hypergraph parameters, and comparison tables for each task.

Significance. If validated, the central idea of representing higher-order spatial-temporal interactions as a hypergraph and injecting it into an LLM could be a useful direction for video scene graph reasoning. The VSGR dataset is a potentially valuable resource, being larger than prior video scene graph datasets and covering five tasks from multiple viewpoints. The random-walk construction in Algorithm 1 is clearly specified, and the authors attempt to address long-tail predicate bias through a procedural graph. However, the significance of the empirical claims is currently undermined by an underspecified test-time protocol for H and by evaluation results that do not uniformly support the 'consistently outperforms' statement. The paper's contributions are credible only after these issues are resolved with additional experiments or explicit clarifications.

major comments (4)
  1. [Sec. 4.2, Eq. (7) and Sec. 5.1] The test-time provenance of the HyperGraph H is not specified. In Sec. 5.1, the VQA, VC, and RR items are generated from <subject, relation, object> triplets, and in Sec. 4.1, H is constructed by integrating those same entity scene graphs G_t and the procedural graph P. If H at test time is the ground-truth graph, then a question such as 'what is the relation between person and cup?' can be answered by reading the corresponding hyperedge, and the comparison against video-only baselines in Tables 5-7 does not measure reasoning. The paper never states whether H is model-predicted or ground-truth, and Fig. 5 appears to show ground-truth relations in the hypergraph. Please state explicitly how H is obtained at inference for each of the five tasks, and provide an ablation without H and with a model-predicted graph built from detected objects and predicted pairwise relations. Without this, the central five-task SOTA claim cannot be evaluated.
  2. [Table 6 and Table 2] The abstract and conclusion claim that HyperGLM 'consistently outperforms state-of-the-art methods' across five tasks, but Table 6 directly contradicts this: HyperGLM's CIDEr of 54.5 is below MV-GPT's 57.1, and its MENTOR of 30.7 is below MV-GPT's 37.5. Similarly, Table 2 shows HyperGLM with lower mean Recall than SceneSayerSDE in multiple VSGR cells (e.g., F=0.3, R/mR@20: 16.5 vs 17.0; F=0.7, R/mR@10: 13.5 vs 14.5). The claim should be revised to reflect that HyperGLM achieves the best or near-best results on some metrics, or the authors should provide a statistical aggregation justifying 'consistently outperforms'.
  3. [Tables 3-7] The evaluation reports only point estimates without error bars, multiple seeds, or significance tests. Many claimed improvements are small; for example, Table 5 shows VQA accuracy of 45.4% for HyperGLM versus 44.3% for Chat-UniVi-7B, and Table 7 shows RR accuracy of 47.2% versus 44.1% for LLaMA-VID-7B. The reader cannot determine whether these gaps are above run-to-run noise, especially for LLM-based systems with decoding stochasticity. Please report variance across at least three runs or provide significance tests for the key SOTA comparisons.
  4. [Sec. 4.1 and Sec. 6.1] The construction of H appears to require relationship labels for the entity scene graphs G_t (Eq. (6)), but for the SGG task the model is supposed to predict those relationships from raw video frames. It is unclear where the relationships used to build H come from at test time. If ground-truth relationships are used to build H during SGG inference, then the R@20 gains in Table 3 are not meaningful. If H is instead built from a base model's predictions, that pipeline should be described explicitly and trained end-to-end or with a comparable protocol. Please clarify the inference-time construction of H for SGG and SGA.
minor comments (5)
  1. [Sec. 6.1] The implementation details state 'We train for two epochs with a batch size of 128 over 16 iterations on 4 × GPUs' — 16 iterations is inconsistent with two epochs on a dataset of this size; this is likely a typo and should be corrected.
  2. [Main text and Appendices] The text refers to 'theoretical foundations and mathematical properties are detailed in the Appendices' and to additional parameter experiments in the Appendices, but the submitted version does not contain these appendices. Please include them or remove the references.
  3. [Tables 5 and 7] The baseline name 'Video-LLaV A-7B' in Table 5 appears to be a typo for 'Video-LLaVA-7B'; please make the baseline naming consistent throughout all tables.
  4. [Fig. 6] The ablation in Fig. 6 shows performance versus the number of hyperedges, but no variance or confidence intervals are provided; adding error bars with multiple runs would make the choice of Nw=60 and Nl=7 more convincing.
  5. [Sec. 5.1] The dataset construction states that 'we exclude questions that LLMs can answer correctly,' but it is not specified which LLM is used for this filtering or how that choice affects the difficulty and validity of the remaining questions; please provide this detail and the proportion of questions removed.

Circularity Check

1 steps flagged · score 4.0 of 10

Reasoning gains on VSGR may be read from an oracle hypergraph: H and the VQA/VC/RR answers are both built from the same annotated scene-graph triplets, and the paper never states that H is model-predicted.

  1. self definitional [Sec. 4.1 (Eq. 6), Sec. 4.2 (Eq. 7), Sec. 5.1 Dataset Construction]
    "Comprehension Tasks via Question-Answering. We introduce tasks that leverage fine-grained relationships from scene graphs, extending Scene Graph Generation to focus on relation understanding and subject/object interpretation using <subject, relation, object> triplets ... Using the annotated scene graphs, we produce 61,120 relation reasoning tasks by selecting partial information as an incomplete input. -- with Eq. (6): H = (⋃_t V_{G_t} ∪ V_P, ⋃_t E_{G_t} ∪ E_P) and Eq. (7): p(A|V,Q,H)."

    Equation (6) constructs the test-time HyperGraph H from the scene-graph relationship sets E_{G_t}; Eq. (7) then generates every answer conditioned on H. Section 5.1 creates the VQA, VC, and RR questions and captions from those same <subject, relation, object> triplets and from the annotated scene graphs. If H at inference is the ground-truth annotated hypergraph -- and the paper never states that H is model-predicted, specifying raw frames plus a detector only for SGG/SGA in Sec. 6.1 -- then answering 'what is the relation between person and cup?' is retrieval of a hyperedge already present in H, not reasoning. The reported gains over video-only LLMs would reflect reading the answer from the input, and the Limitations section does not acknowledge this ambiguity.

full rationale

The SGG and SGA results on PVSG and Action Genome are anchored to external benchmarks and are not circular: HyperGLM is compared against published systems using standard recall metrics, and the procedural-graph transition statistics are fit on training relationships, not on the test answers. The circularity risk is concentrated in the three reasoning tasks evaluated only on the self-built VSGR dataset. There, the paper's own equations couple the answer distribution p(A|V,Q,H) to an H whose hyperedges are exactly the relationship triplets from which the VQA/VC/RR items were generated (Sec. 5.1). If H is the annotated scene graph at test time, the reasoning tasks reduce by construction to reading the queried relation out of the conditioning input, making the comparison against Video-ChatGPT, Video-LLaVA, MovieChat, Chat-UniVi, MA-LMM, and LLaMA-VID a test of input formatting rather than relational reasoning. The paper's language that the process 'constructs' G_t suggests a model-predicted H, but no experiment ablates H, replaces H with model-predicted scene graphs, or states the inference-time H source for VQA/VC/RR; Sec. 6.1 gives that specification only for SGG/SGA. This is a conditional self-definitional structure, not a demonstrated equivalence, so the score is 4 rather than higher: the external SGG/SGA contributions retain independent content. Separately, Table 6 shows HyperGLM's CIDEr (54.5) and MENTOR (30.7) below MV-GPT (57.1, 37.5), contradicting the Abstract's 'consistently outperforms' wording; that is a consistency issue, not a circularity, but it further weakens the five-task superiority claim.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central method rests on two fitted components: the transition matrix of the procedural graph (Eq. 3) and the random-walk hyperparameters Nw,Nl. The validity of the reasoning tasks rests on the assumption that LLM-generated annotations are reliable and that the injected hypergraph does not leak ground-truth relationships. No new physical entities are introduced.

free parameters (2)
  • Number of random walks Nw and walk length Nl = Nw=60, Nl=7
    Selected based on SGG performance in Fig. 6; no held-out validation described, so these may be tuned to the test benchmark.
  • Procedural graph transition probabilities w(rm,rn) = Empirical frequencies from training data (Eq. 3)
    The central prior for relationship anticipation; estimated from observed transitions in the training set and normalized per Eq. 4.
assumptions (5)
  • domain assumption The procedural graph P with transition frequencies w(rm,rn) faithfully models causal relationship evolution across frames.
    Invoked in Eq. (3)-(5) and Sec 4.1; assumes Markovian transitions between relationship categories are sufficient for anticipation and bias reduction.
  • ad hoc to paper Random walks over the unified hypergraph preserve higher-order semantic structure and approximate subgraph matching.
    Stated in Sec 4.1; no proof or validation in the main text, deferred to appendices.
  • domain assumption Injecting the unified hypergraph as tokens into an LLM improves reasoning without changing the task distribution.
    Core to the method in Sec 4.2; the paper provides no analysis of what the LLM learns from the hypergraph versus the video tokens.
  • domain assumption GPT-4/GPT-3.5 generated questions, captions, and reasoning tasks, after human refinement, are valid ground truth.
    Sec 5.1 relies on LLM generation for all language annotations; the validation is described qualitatively.
  • domain assumption Pre-trained Faster R-CNN detections are reliable enough to build entity scene graphs for SGG/SGA.
    Sec 6.1 'Settings' adopts this standard tooling assumption without analysis of detection errors.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HyperGLM: HyperGraph for Video Scene Graph Generation and Anticipation." pith.science (2026). https://pith.science/paper/WTKQAT5T

@misc{pith2026241118042,
  author       = {Pith},
  title        = {Pith review of: HyperGLM: HyperGraph for Video Scene Graph Generation and Anticipation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WTKQAT5T}},
  note         = {Machine review of arXiv:2411.18042}
}
read the original abstract

Multimodal LLMs have advanced vision-language tasks but still struggle with understanding video scenes. To bridge this gap, Video Scene Graph Generation (VidSGG) has emerged to capture multi-object relationships across video frames. However, prior methods rely on pairwise connections, limiting their ability to handle complex multi-object interactions and reasoning. To this end, we propose Multimodal LLMs on a Scene HyperGraph (HyperGLM), promoting reasoning about multi-way interactions and higher-order relationships. Our approach uniquely integrates entity scene graphs, which capture spatial relationships between objects, with a procedural graph that models their causal transitions, forming a unified HyperGraph. Significantly, HyperGLM enables reasoning by injecting this unified HyperGraph into LLMs. Additionally, we introduce a new Video Scene Graph Reasoning (VSGR) dataset featuring 1.9M frames from third-person, egocentric, and drone views and supports five tasks: Scene Graph Generation, Scene Graph Anticipation, Video Question Answering, Video Captioning, and Relation Reasoning. Empirically, HyperGLM consistently outperforms state-of-the-art methods across five tasks, effectively modeling and reasoning complex relationships in diverse video scenes.

Figures

Figures reproduced from arXiv: 2411.18042 by the authors.

Figure 1
Figure 1. Our HyperGLM framework supports Video Scene Graph Generation, Anticipation, and Reasoning. HyperGLM constructs scene graphs from [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. (a) To model the temporal transition, a simple approach can be [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Our Video Scene HyperGraph, including entity graphs and a procedural graph, as defined in Sec. 4.1. Blue nodes represent entities, while green nodes denote relationships. The entity graph captures spatial relationships (subject ⊸ relationship ⊸ object), whereas the procedural graph models relationship transitions (→). Hyperedges are visualized as polygons, encapsulating interactions through chains of relationships. … view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: An example of the diversified context within the streaming dialog in our VSGR dataset. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Comparison of Recall (R) and mean Recall (mR) at different [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Qualitative comparison of our HyperGLM approach versus SceneSayerSDE [ [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 47 canonical work pages

  1. [1]

    Hypergraph convolution and hypergraph attention

    Song Bai, Feihu Zhang, and Philip HS Torr. Hypergraph convolution and hypergraph attention. Pattern Recognition, 110:107637, 2021. 2

    work page 2021

  2. [2]

    Videollm-online: Online video large language model for streaming video

    Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia-Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. Videollm-online: Online video large language model for streaming video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18407–18418, 2024. 1

    work page 2024

  3. [3]

    More knowledge, less bias: Unbiasing scene graph generation with explicit ontological adjustment

    Zhanwen Chen, Saed Rezayi, and Sheng Li. More knowledge, less bias: Unbiasing scene graph generation with explicit ontological adjustment. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 4023–4032, 2023. 2

    work page 2023

  4. [4]

    Spatial-temporal transformer for dynamic scene graph generation

    Yuren Cong, Wentong Liao, Hanno Ackermann, Bodo Rosen- hahn, and Michael Ying Yang. Spatial-temporal transformer for dynamic scene graph generation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 16372–16382, 2021. 1, 2, 3

    work page 2021

  5. [5]

    Ackermann, M

    Yuren Cong, Wentong Liao, H. Ackermann, M. Yang, and B. Rosenhahn. Spatial-temporal transformer for dynamic scene graph generation. IEEE International Conference on Computer Vision, 2021. 7

    work page 2021

  6. [6]

    Reltr: Relation transformer for scene graph generation

    Yuren Cong, Michael Ying Yang, and Bodo Rosenhahn. Reltr: Relation transformer for scene graph generation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9):11169–11183, 2023. 2

    work page 2023

  7. [7]

    Hierarchical memory learning for fine-grained scene graph generation

    Youming Deng, Yansheng Li, Yongjun Zhang, Xiang Xi- ang, Jian Wang, Jingdong Chen, and Jiayi Ma. Hierarchical memory learning for fine-grained scene graph generation. In European Conference on Computer Vision, pages 266–283. Springer, 2022. 2

    work page 2022

  8. [8]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representa- tions, 2021. 6

    2021

Show all 55 references
  1. [9]

    Scenegenie: Scene graph guided diffusion models for image synthesis

    Azade Farshad, Yousef Yeganeh, Yu Chi, Chengzhi Shen, Böjrn Ommer, and Nassir Navab. Scenegenie: Scene graph guided diffusion models for image synthesis. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pages 88–98, 2023. 2

    2023

  2. [10]

    Video-of-thought: Step-by-step video reasoning from perception to cognition

    Hao Fei, Shengqiong Wu, Wei Ji, Hanwang Zhang, Meishan Zhang, Mong-Li Lee, and Wynne Hsu. Video-of-thought: Step-by-step video reasoning from perception to cognition. In Forty-first International Conference on Machine Learning,

  3. [11]

    Exploiting long-term de- pendencies for generating dynamic scene graphs

    Shengyu Feng, Hesham Mostafa, Marcel Nassar, Somdeb Majumdar, and Subarna Tripathi. Exploiting long-term de- pendencies for generating dynamic scene graphs. IEEE Work- shop/Winter Conference on Applications of Computer Vision,

  4. [12]

    Hgnn+: General hypergraph neural networks

    Yue Gao, Yifan Feng, Shuyi Ji, and Rongrong Ji. Hgnn+: General hypergraph neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3181–3199,

  5. [13]

    Dsgg: Dense relation transformer for an end-to-end scene graph generation

    Zeeshan Hayder and Xuming He. Dsgg: Dense relation transformer for an end-to-end scene graph generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 28317–28326, 2024. 2

    2024

  6. [14]

    Ma-lmm: Memory-augmented large multimodal model for long-term video understanding

    Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xue- fei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. Ma-lmm: Memory-augmented large multimodal model for long-term video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni...

    2024

  7. [15]

    To- wards open-vocabulary scene graph generation with prompt- based finetuning

    Tao He, Lianli Gao, Jingkuan Song, and Yuan-Fang Li. To- wards open-vocabulary scene graph generation with prompt- based finetuning. In European Conference on Computer Vision, pages 56–73. Springer, 2022. 2

    2022

  8. [16]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. 6

    2022

  9. [17]

    Egtr: Extracting graph from transformer for scene graph generation

    Jinbae Im, JeongYeon Nam, Nokyung Park, Hyungmin Lee, and Seunghyun Park. Egtr: Extracting graph from transformer for scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 24229–24238, 2024. 2

    2024

  10. [18]

    Action genome: Actions as compositions of spatio- temporal scene graphs

    Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Carlos Niebles. Action genome: Actions as compositions of spatio- temporal scene graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10236–10247, 2020. 1, 2, 5, 6, 7

    2020

  11. [19]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825,

    arXiv

  12. [20]

    Chat-univi: Unified visual representation em- powers large language models with image and video un- derstanding

    Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation em- powers large language models with image and video un- derstanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 137...

    2024

  13. [21]

    Fast contextual scene graph generation with unbiased context aug- mentation

    Tianlei Jin, Fangtai Guo, Qiwei Meng, Shiqiang Zhu, Xi- angming Xi, Wen Wang, Zonghao Mu, and Wei Song. Fast contextual scene graph generation with unbiased context aug- mentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6302–6311,

  14. [22]

    Hypergraph attention networks for multimodal learning

    Eun-Sol Kim, Woo Young Kang, Kyoung-Woon On, Yu-Jung Heo, and Byoung-Tak Zhang. Hypergraph attention networks for multimodal learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14581–14590, 2020. 2

    2020

  15. [23]

    Llm4sgg: Large language models for weakly supervised scene graph generation

    Kibum Kim, Kanghoon Yoon, Jaehyeong Jeon, Yeonjun In, Jinyoung Moon, Donghyun Kim, and Chanyoung Park. Llm4sgg: Large language models for weakly supervised scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2830...

    2024

  16. [24]

    Meltr: Meta loss transformer for learning to fine-tune video foundation models

    Dohwan Ko, Joonmyung Choi, Hyeong Kyu Choi, Kyoung- 9 Woon On, Byungseok Roh, and Hyunwoo J Kim. Meltr: Meta loss transformer for learning to fine-tune video foundation models. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 20105–20115,

  17. [25]

    Is-ggt: Iter- ative scene graph generation with generative transformers

    Sanjoy Kundu and Sathyanarayanan N Aakur. Is-ggt: Iter- ative scene graph generation with generative transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6292–6301, 2023. 2

    2023

  18. [26]

    Sgtr: End-to- end scene graph generation with transformer

    Rongjie Li, Songyang Zhang, and Xuming He. Sgtr: End-to- end scene graph generation with transformer. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19486–19496, 2022. 2

    2022

  19. [27]

    From pixels to graphs: Open-vocabulary scene graph generation with vision-language models

    Rongjie Li, Songyang Zhang, Dahua Lin, Kai Chen, and Xuming He. From pixels to graphs: Open-vocabulary scene graph generation with vision-language models. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 28076–28086, 2024. 2

    2024

  20. [28]

    Multi-hyperedge hypergraph for group activity recog- nition

    Wanxin Li, Wei Xie, Zhigang Tu, Wei Wang, and Lianghao Jin. Multi-hyperedge hypergraph for group activity recog- nition. In 2022 International Joint Conference on Neural Networks (IJCNN), pages 01–07. IEEE, 2022. 2

    2022

  21. [29]

    Llama-vid: An im- age is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An im- age is worth 2 tokens in large language models. In European Conference on Computer Vision, pages 323–340. Springer,

  22. [30]

    Video-llava: Learning united visual rep- resentation by alignment before projection

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual rep- resentation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023. 1, 8

    2023 arXiv

  23. [31]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 6

    2024

  24. [32]

    Video-ChatGPT: Towards detailed video un- derstanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Video-ChatGPT: Towards detailed video un- derstanding via large vision and language models. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar, editors, Pro- ceedings of the 62nd Annual Meeting of the Association for ...

    2024

  25. [33]

    Unbiased scene graph generation in videos

    Sayak Nag, Kyle Min, Subarna Tripathi, and Amit K Roy- Chowdhury. Unbiased scene graph generation in videos. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 22803–22813, 2023. 2

    2023

  26. [34]

    Hig: Hier- archical interlacement graph approach to scene graph genera- tion in video understanding

    Trong-Thuan Nguyen, Pha Nguyen, and Khoa Luu. Hig: Hier- archical interlacement graph approach to scene graph genera- tion in video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ,

  27. [35]

    CYCLO: Cyclic graph transformer approach to multi-object relationship modeling in aerial videos

    Trong-Thuan Nguyen, Pha Nguyen, Li Xin, Cothren Jack- son, Yilmaz Alper, and Khoa Luu. CYCLO: Cyclic graph transformer approach to multi-object relationship modeling in aerial videos. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 1, 2, 5, 7

    2024

  28. [36]

    Towards scene graph anticipation

    Rohith Peddi, Saksham Singh, Parag Singla, Vibhav Gogate, et al. Towards scene graph anticipation. In European Confer- ence on Computer Vision. Springer, 2024. 1, 2, 3, 7, 8

    2024

  29. [37]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  30. [38]

    End-to-end generative pretraining for mul- timodal video captioning

    Paul Hongsuck Seo, Arsha Nagrani, Anurag Arnab, and Cordelia Schmid. End-to-end generative pretraining for mul- timodal video captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17959–17968, 2022. 1, 8

    2022

  31. [39]

    Video visual relation detection via iterative inference

    Xindi Shang, Yicong Li, Junbin Xiao, Wei Ji, and Tat-Seng Chua. Video visual relation detection via iterative inference. In Proceedings of the 29th ACM international conference on Multimedia, pages 3654–3663, 2021. 2

    2021

  32. [40]

    Accurate and fast compressed video captioning

    Yaojie Shen, Xin Gu, Kai Xu, Heng Fan, Longyin Wen, and Libo Zhang. Accurate and fast compressed video captioning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15558–15567, 2023. 1, 8

    2023

  33. [41]

    Moviechat: From dense token to sparse memory for long video understanding

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patt...

    2024

  34. [42]

    Target adaptive context aggregation for video scene graph generation

    Yao Teng, Limin Wang, Zhifeng Li, and Gangshan Wu. Target adaptive context aggregation for video scene graph generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13688–13697, 2021. 1, 2, 3

    2021

  35. [43]

    Graph (graph): A nested graph-based framework for early accident anticipation

    Nupur Thakur, PrasanthSai Gouripeddi, and Baoxin Li. Graph (graph): A nested graph-based framework for early accident anticipation. In Proceedings of the IEEE/CVF Winter Confer- ence on Applications of Computer Vision, pages 7533–7541,

  36. [44]

    Learn- ing situation hyper-graphs for video question answering

    Aisha Urooj, Hilde Kuehne, Bo Wu, Kim Chheu, Walid Bous- selham, Chuang Gan, Niels Lobo, and Mubarak Shah. Learn- ing situation hyper-graphs for video question answering. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 14879–14889, 2023. 2

    2023

  37. [45]

    Memory-and-anticipation transformer for online action understanding

    Jiahao Wang, Guo Chen, Yifei Huang, Limin Wang, and Tong Lu. Memory-and-anticipation transformer for online action understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13824–13835, 2023. 1

    2023

  38. [46]

    Multi-object event graph repre- sentation learning for video question answering

    Yanan Wang, Shuichiro Haruta, Donghuo Zeng, Julio Viz- carra, and Mori Kurokawa. Multi-object event graph repre- sentation learning for video question answering. In Meeting on Image Recognition and Understanding, 2024. 2

    2024

  39. [47]

    Sportshhi: A dataset for human-human interaction detection in sports videos

    Tao Wu, Runyu He, Gangshan Wu, and Limin Wang. Sportshhi: A dataset for human-human interaction detection in sports videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. 2, 5

    2024

  40. [48]

    Video question answering via gradually refined attention over appearance and motion

    Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM international conference on Multimedia, pages 1645–1653, 2017. 6

    2017

  41. [49]

    Msr-vtt: A large video description dataset for bridging video and language

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and 10 pattern recognition, pages 5288–5296, 2016. 6

    2016

  42. [50]

    Modeling semantic composition with syntac- tic hypergraph for video question answering

    Zenan Xu, Wanjun Zhong, Qinliang Su, Zijing Ou, and Fuwei Zhang. Modeling semantic composition with syntac- tic hypergraph for video question answering. arXiv preprint arXiv:2205.06530, 2022. 2

    2022 arXiv

  43. [51]

    Panoptic scene graph gen- eration

    Jingkang Yang, Yi Zhe Ang, Zujin Guo, Kaiyang Zhou, Wayne Zhang, and Ziwei Liu. Panoptic scene graph gen- eration. In European Conference on Computer Vision, pages 178–196. Springer, 2022. 2

    2022

  44. [52]

    Panoptic video scene graph generation

    Jingkang Yang, Wenxuan Peng, Xiangtai Li, Zujin Guo, Liangyu Chen, Bo Li, Zheng Ma, Kaiyang Zhou, Wayne Zhang, Chen Change Loy, et al. Panoptic video scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18675– 1868...

    2023

  45. [53]

    Com- monscenes: Generating commonsense 3d indoor scenes with scene graphs

    Guangyao Zhai, Evin Pınar Örnek, Shun-Cheng Wu, Yan Di, Federico Tombari, Nassir Navab, and Benjamin Busam. Com- monscenes: Generating commonsense 3d indoor scenes with scene graphs. Advances in Neural Information Processing Systems, 36, 2024. 2

    2024

  46. [54]

    Deep hypergraph structure learning

    Zizhao Zhang, Yifan Feng, Shihui Ying, and Yue Gao. Deep hypergraph structure learning. arXiv preprint arXiv:2208.12547, 2022. 2

    2022 arXiv

  47. [55]

    Dynamical attention hypergraph convolutional network for group activity recognition

    Xiaolin Zhu, Dongli Wang, Jianxun Li, Rui Su, Qin Wan, and Yan Zhou. Dynamical attention hypergraph convolutional network for group activity recognition. IEEE Transactions on Neural Networks and Learning Systems, 2024. 2 11

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.