Pith. sign in

REVIEW 3 major objections 4 minor 60 references

DyGEnc: Encoding a Sequence of Textual Scene Graphs to Reason and Answer Questions in Dynamic Scenes

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DyGEnc compresses a sequence of textual scene graphs into a single latent token and shows that a frozen LLM, tuned with LoRA, can then answer interaction and sequencing questions about the scene at high accuracy, outperforming visual…

desk verdict Solid graph-sequence encoder, but the headline 15-25% margin over visual methods rests on oracle scene-graph input and does not hold for automatically extracted graphs. read the letter →

arxiv 2505.03581 v1 pith:GS2GEUNG submitted 2025-05-06 cs.CV

Sergey Linok , Vadim Semenov , Anastasia Trunova , Oleg Bulichev , Dmitry Yudin This is my paper · ORCID
classification cs.CV
keywords dynamicscenegraphsvideoquestionansweringtextualgraphencodinglargelanguagemodelsLoRAfine-tuningQ-Formerroboticmemory
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DyGEnc aims to show that a long sequence of textual scene graphs describing a changing environment can be compressed into a fixed handful of latent tokens—one in the default configuration—and still give a large language model enough temporal information to answer questions about what happened, in what order, and to which objects. The method fuses a graph encoder with a Q-Former sequence encoder and injects the compressed tokens as a soft prompt into a LoRA-tuned frozen LLM. On the STAR benchmark this reaches 0.89 average accuracy with Llama3.2-3B and 0.73 on AGQA2.0, beating video-based baselines by 15–25 percent on human–object interaction and sequence queries. The same pipeline can be run on raw video when scene graphs are extracted by foundation models, which the authors demonstrate on a wheeled-manipulator robot, though accuracy on that out-of-distribution setup drops to 0.30–0.34.

What carries the argument

The load-bearing mechanism is the Q-Former sequence encoder acting as a learned compressor. For each graph in the sequence, a graph encoder produces a token enriched with rotary positional encoding; then a fixed set of learnable query tokens attends across all graph tokens via cross-attention, collapsing an arbitrarily long graph sequence into a constant number of latent tokens. Because the number of output tokens does not depend on sequence length, the LLM context stays fixed regardless of how many scene graphs are observed. The graph encoder and Q-Former are trained together with a LoRA-tuned LLM, so the compression is optimized for the downstream question-answering objective rather than for reconstruction.

What would settle it

Run DyGEnc on the STAR validation videos using the same foundation-model graph extraction pipeline as the DRobot experiment instead of the provided annotations. If average accuracy falls to the 0.30–0.34 range rather than remaining near 0.89, then the method's advantage over visual baselines does not survive real-world graph extraction.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the temporal content of a dynamic scene—who touched what, in what order—survives aggressive compression into a single learned latent token, provided the token is produced by a graph-aware encoder trained end-to-end with a language model. Given a sequence of textual scene graphs, DyGEnc encodes node and edge text with ModernBert, aggregates each graph with a GraphTransformer into one graph token, adds rotary positional encoding, and lets a Q-Former compress the whole sequence into $k$ query tokens (by default $k=1$). These tokens are projected into the LLM embedding space between <graph> and </graph> markers, and only a LoRA adapter plus the encoders are trained. With this setup DyGEnc reports 0.97 interaction and 0.90 sequence accuracy on STAR (0.89 average) and 0.73 average on AGQA2.0, outperforming the visual baselines it compares against by 15–25 percent on the interaction and sequence question types. Cross-attention maps show the single token attending to the frames that answer the query, evidence that reasoning is driven by relevant frames rather than memorized answers.

Load-bearing premise

The central assumption is that the ground-truth scene graph annotations used in the STAR and AGQA evaluations are similar in quality to the graphs the system would receive from automatic extraction in a real deployment; the 15–25 percent margin is measured against visual methods under gold annotations, and drops to 0.30–0.34 accuracy when graphs are extracted by foundation models on the robot benchmark.

Editorial extensions

If this is right

  • With gold scene-graph annotations, DyGEnc answers interaction questions at 0.97 and sequence questions at 0.90 on STAR, so temporal relational reasoning does not require per-frame pixel features.
  • Constant token count means graph sequences of arbitrary length fit into a fixed LLM context, which is a prerequisite for long-horizon robotic memory.
  • The 3B Llama model matches or beats the 8B model in most DyGEnc configurations, so the approach's accuracy is not dependent on the largest available LLM.
  • On AGQA2.0 open questions, DyGEnc outperforms prior methods by a wide margin, indicating the method distinguishes unique textual graph features rather than exploiting binary answer biases.
  • Using foundation-model graph extraction, the same DyGEnc model can answer questions from raw video on a robot, showing the architecture transfers beyond curated graph annotations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 0.30–0.34 accuracy on automatically extracted graphs versus 0.89 on gold annotations suggests the practical bottleneck is graph extraction quality, not the encoder; improving automatic textual scene graphs would likely transfer directly to DyGEnc accuracy.
  • A single latent token appears to preserve enough information for interaction and sequence questions, but the ablation shows prediction and feasibility questions improve markedly with more tokens (16 tokens raise prediction from 0.77 to 0.91), implying different question types need different compression budgets.
  • If graph extraction improves to near-gold quality, graph-based memory could replace implicit video representations in embodied QA, because the reasoning component already reaches near-ceiling accuracy on curated graphs.
  • A direct test of the deployment claim would be to run DyGEnc on raw video from STAR rather than gold graphs, measuring the true end-to-end margin over visual baselines.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces DyGEnc, a method for encoding a sequence of textual scene graphs into a small number of latent tokens that are injected into a frozen, LoRA-tuned large language model for visual question answering. The architecture combines a text encoder (ModernBERT), a graph encoder (GraphTransformer), a sequence encoder (Q-Former with a fixed number of query tokens), and an MLP projector. The authors evaluate on the STAR and AGQA2.0 benchmarks, reporting average accuracies of 0.89 and 0.73 respectively, and additionally describe a robotics experiment (DRobot) in which scene graphs are extracted automatically from video via foundation models. The central claim, stated in the abstract and Section I, is that DyGEnc outperforms existing visual methods by 15–25% on questions about the history of human-to-object interactions.

Significance. If the headline comparison were fair, the contribution would be significant: the paper demonstrates that a dynamic scene can be compressed into a single latent graph token while retaining enough temporal information for an LLM to answer interaction and sequence questions with high accuracy, and it provides a thorough ablation of the encoder components. The architecture is clearly specified and the ablations are sensible. However, the practical significance is heavily contingent on the quality of automatically extracted scene graphs; the DRobot experiment (Section IV-F) shows accuracy of only 0.30–0.34 with foundation-model-extracted graphs, far below the 0.89 reported on STAR. This discrepancy substantially weakens the real-world applicability claim. The method may still be valuable as a graph-sequence encoder, but the paper's central 'large margin over visual methods' claim is not supported under a protocol where DyGEnc receives oracle annotations while baselines receive raw video.

major comments (3)
  1. [Sec. IV-D.4 and Table IV] The comparison with visual baselines is unfair: DyGEnc receives ground-truth scene graph annotations for STAR and AGQA2.0, while all baselines (STEP, Q-ViD, MIST, SeViLA, ViLA, VidF4, LRR) receive raw video or video features. The 15–25% margin over visual methods claimed in the abstract is therefore a comparison of oracle-augmented reasoning against end-to-end visual perception, not a head-to-head test of the same input modality. Section IV-F and Table VI reinforce this concern: when graphs are extracted automatically with foundation models on the DRobot benchmark, DyGEnc accuracy falls to 0.30–0.34, below the 0.71 reported for LRR in Table IV. To support the headline claim, the authors should either (a) compare against visual baselines using predicted scene graphs as input to DyGEnc, or (b) explicitly reframe the claim as 'DyGEnc outperforms visual methods when given ground-truth scene graphs', and move the video-input results to a prominent position.
  2. [Sec. IV-B] The evaluation metric is more permissive than the exact-match accuracy typically used by the baselines: a prediction is counted correct if the ground-truth answer 'contains' the generated response. This 'contains' criterion can inflate scores relative to exact-match or normalized-answer evaluation. Since Tables IV and V compare against numbers reported by prior papers, the paper must verify that the baselines used the same evaluation protocol; if not, the margins are not directly comparable. At minimum, the authors should report both 'contains' and exact-match accuracy for DyGEnc on STAR and AGQA2.0 to quantify the inflation.
  3. [Sec. IV-F (DRobot)] The DRobot experiment is presented as evidence that DyGEnc can be 'seamlessly extended to process raw input images', but the reported accuracy (0.30–0.34) is barely above random for a multi-class setting and far below the benchmark numbers. The paper does not compare DRobot results against any visual baseline, so it is unclear whether DyGEnc with extracted graphs is competitive with even simple video QA methods on this data. The limitations section acknowledges the keyframe extraction issue but does not address the graph-construction quality gap. This is a load-bearing issue for the real-world applicability claim; the paper should either provide a baseline comparison on DRobot or temper the claim to 'the full pipeline is feasible but currently underperforms on automatic graphs'.
minor comments (4)
  1. [Throughout] There are numerous typos and grammatical slips, e.g., 'positivly skewed' (Sec. IV-A.2), 'appliend' (Sec. III-B), 'unconstraned' (Sec. III-B), 'effietefly' (Sec. IV-D.2), 'Recall metric' where accuracy is used (Sec. IV-D.1, Sec. IV-D.2), and 'AQGA' instead of AGQA in Sec. IV-E heading. These should be corrected.
  2. [Sec. IV-E / Table V] The column abbreviations in Table V (Obj.-Rel., Rel.-Act., etc.) are not fully expanded in the table; the expansion is only partially given in the text. Please add a caption or footnote that spells out all abbreviations (e.g., Obj.-Rel. = Object-Relationship, Sup. = Superlative, E. = Exists, Act.-Rec. = Activity Recognition, B. = Binary, O. = Open, A. = All).
  3. [Sec. II-B / Reference [10]] The STAR benchmark reference appears in the related work as [10] but is cited as a 2024 arXiv preprint; the published version (if any) should be cited instead. Also, the paper inconsistently refers to 'STAR' as both a dataset and a benchmark; please standardize.
  4. [Sec. IV-C.1] The preprocessing step that 'retains unique graphs' is described briefly; it would be helpful to state whether duplicates are removed globally or within each sequence, and how this affects the temporal index t for duration reasoning.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; DyGEnc's headline numbers are held-out predictions and its graph-sequence encoding is not derived from the benchmark answer labels.

full rationale

DyGEnc is an empirical architecture paper, not a derivation: the model is trained on STAR/AGQA train splits and evaluated on held-out validation/test splits, so the reported accuracies are genuine predictions rather than values fitted to the test labels. The method in Eqs. (1)-(6) maps textual scene graph attributes into latent tokens via a text encoder, GNN, Q-Former, and MLP; nowhere is the answer distribution or the benchmark label used to define the input representation. The only self-citation is ref. [2] (BBQ), cited in related work for 3D scene-graph retrieval, and it is not load-bearing for any of DyGEnc's components or conclusions. The comparison against video baselines uses ground-truth graph annotations while baselines see raw video, and the accuracy rule counts a prediction as correct if the response is contained in the ground-truth answer; these are protocol-equivalence and external-validity concerns, not circularity, because the DyGEnc numbers are still produced by a model that never sees test answers during training. Sec. IV-F and Table VI further show that when graphs are automatically extracted from video on the DRobot benchmark, accuracy drops to 0.30-0.34, demonstrating that the method's success is not tautologically forced by its inputs. No circular step can be exhibited, so the circularity score is 0.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the availability of high-quality textual scene graphs and on the learned compression retaining task-relevant temporal information. The only hand-chosen numeric hyperparameter with direct impact on the claim is the number of Q-Former query tokens (K=1). No new physical or conceptual entities are introduced. The most fragile assumption is the transferability of gold-annotation performance to automatically extracted graphs, which the paper's own DRobot experiment calls into question.

free parameters (1)
  • Q-Former query token count K = 1
    The number of latent tokens used to compress the graph sequence is a hyperparameter selected by validation on STAR (Table II). K=1 is used in the final model, trading compression against reasoning accuracy; larger K slightly improves prediction and feasibility categories but increases token count.
assumptions (3)
  • domain assumption STAR and AGQA ground-truth scene graphs contain sufficient information to answer the benchmark questions.
    The full-text fine-tuning baseline reaches 0.99 average on STAR (Table III), indicating the textual graphs are nearly sufficient for the task. DyGEnc's 0.89 is therefore bounded by its compression and reasoning, not by missing information in the graphs.
  • domain assumption A single Q-Former query token can retain the temporally relevant information across a graph sequence.
    The method's compression to 0.03x relies on this assumption. It is empirically validated only on STAR validation, and the DRobot transfer shows significant degradation when graph inputs change distribution, suggesting the assumption is fragile.
  • domain assumption The Llama LLM, with LoRA fine-tuning, can reason correctly over latent graph tokens.
    The whole approach depends on the LLM grounding its answers in the implanted soft prompt. The paper provides no mechanism guaranteeing this beyond the empirical benchmark results, which are themselves dependent on the gold-graph input setting.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DyGEnc: Encoding a Sequence of Textual Scene Graphs to Reason and Answer Questions in Dynamic Scenes." pith.science (2026). https://pith.science/paper/GS2GEUNG

@misc{pith2026250503581,
  author       = {Pith},
  title        = {Pith review of: DyGEnc: Encoding a Sequence of Textual Scene Graphs to Reason and Answer Questions in Dynamic Scenes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GS2GEUNG}},
  note         = {Machine review of arXiv:2505.03581}
}
read the original abstract

The analysis of events in dynamic environments poses a fundamental challenge in the development of intelligent agents and robots capable of interacting with humans. Current approaches predominantly utilize visual models. However, these methods often capture information implicitly from images, lacking interpretable spatial-temporal object representations. To address this issue we introduce DyGEnc - a novel method for Encoding a Dynamic Graph. This method integrates compressed spatial-temporal structural observation representation with the cognitive capabilities of large language models. The purpose of this integration is to enable advanced question answering based on a sequence of textual scene graphs. Extended evaluations on the STAR and AGQA datasets indicate that DyGEnc outperforms existing visual methods by a large margin of 15-25% in addressing queries regarding the history of human-to-object interactions. Furthermore, the proposed method can be seamlessly extended to process raw input images utilizing foundational models for extracting explicit textual scene graphs, as substantiated by the results of a robotic experiment conducted with a wheeled manipulator platform. We hope that these findings will contribute to the implementation of robust and compressed graph-based robotic memory for long-horizon reasoning. Code is available at github.com/linukc/DyGEnc.

Figures

Figures reproduced from arXiv: 2505.03581 by the authors.

Figure 1
Figure 1. DyGEnc compactly encodes a dynamic graph (sequence of textual [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the DyGEnc pipeline. Given a dynamic scene graph - a sequence of textual scene graphs, where nodes and edges carry attributes [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Example of cross-attention visualization from [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Example of cross-attention visualization from [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Illustration for a robotic experiment setup - to the left: mobile platform Husky with UR5 manipulator equipped to perform MOVE-AND-PICK [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 24 canonical work pages

  1. [1]

    Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning,

    Q. Gu, A. Kuwajerwala, S. Morin, K. M. Jatavallabhula, B. Sen, A. Agarwal, C. Rivera, W. Paul, K. Ellis, R. Chellappa, et al. , “Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning,” in 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 5021–5028

    work page 2024

  2. [2]

    Beyond bare queries: Open-vocabulary object retrieval with 3d scene graph,

    S. Linok, T. Zemskova, S. Ladanova, R. Titkov, and D. Yudin, “Beyond bare queries: Open-vocabulary object retrieval with 3d scene graph,” arXiv e-prints, pp. arXiv–2406, 2024

    work page 2024

  3. [3]

    Search3d: Hierarchical open-vocabulary 3d segmenta- tion,

    A. Takmaz, A. Delitzas, R. W. Sumner, F. Engelmann, J. Wald, and F. Tombari, “Search3d: Hierarchical open-vocabulary 3d segmenta- tion,” IEEE Robotics and Automation Letters , 2025

    work page 2025

  4. [4]

    Hierarchical open-vocabulary 3d scene graphs for language-grounded robot navigation,

    A. Werby, C. Huang, M. B ¨uchner, A. Valada, and W. Burgard, “Hierarchical open-vocabulary 3d scene graphs for language-grounded robot navigation,” in First Workshop on Vision-Language Models for Navigation and Manipulation at ICRA 2024 , 2024

    2024

  5. [5]

    Clio: Real-time task-driven open-set 3d scene graphs,

    D. Maggio, Y . Chang, N. Hughes, M. Trang, D. Griffith, C. Dougherty, E. Cristofalo, L. Schmid, and L. Carlone, “Clio: Real-time task-driven open-set 3d scene graphs,” IEEE Robotics and Automation Letters , 2024

    2024

  6. [6]

    4d panoptic scene graph generation,

    J. Yang, J. Cen, W. Peng, S. Liu, F. Hong, X. Li, K. Zhou, Q. Chen, and Z. Liu, “4d panoptic scene graph generation,” Advances in Neural Information Processing Systems , vol. 36, pp. 69 692–69 705, 2023

    work page 2023

  7. [7]

    G-retriever: Retrieval-augmented generation for textual graph understanding and question answering,

    X. He, Y . Tian, Y . Sun, N. Chawla, T. Laurent, Y . LeCun, X. Bresson, and B. Hooi, “G-retriever: Retrieval-augmented generation for textual graph understanding and question answering,” Advances in Neural Information Processing Systems , vol. 37, pp. 132 876–132 907, 2025

    work page 2025

  8. [8]

    Let your graph do the talking: Encoding structured data for llms,

    B. Perozzi, B. Fatemi, D. Zelle, A. Tsitsulin, M. Kazemi, R. Al-Rfou, and J. Halcrow, “Let your graph do the talking: Encoding structured data for llms,” arXiv preprint arXiv:2402.05862 , 2024

    arXiv 2024

Show all 60 references
  1. [9]

    Can llms enhance performance prediction for deep learning models?

    K. P. Selvam, P. M. Phothilimthana, S. Abu-El-Haija, B. Perozzi, and M. Brorsson, “Can llms enhance performance prediction for deep learning models?”

  2. [10]

    Star: A benchmark for situated reasoning in real-world videos,

    B. Wu, S. Yu, Z. Chen, J. B. Tenenbaum, and C. Gan, “Star: A benchmark for situated reasoning in real-world videos,” arXiv preprint arXiv:2405.09711, 2024

    2024 arXiv

  3. [11]

    Agqa 2.0: An updated benchmark for compositional spatio-temporal reasoning,

    M. Grunde-McLaughlin, R. Krishna, and M. Agrawala, “Agqa 2.0: An updated benchmark for compositional spatio-temporal reasoning,”

  4. [12]

    Visual genome: Connecting language and vision using crowdsourced dense image annotations,

    R. Krishna, Y . Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y . Kalantidis, L.-J. Li, D. A. Shamma, et al. , “Visual genome: Connecting language and vision using crowdsourced dense image annotations,” International journal of computer vision , vol. 123, pp. 32–73, 2017

    2017

  5. [13]

    Gqa: A new dataset for real- world visual reasoning and compositional question answering,

    D. A. Hudson and C. D. Manning, “Gqa: A new dataset for real- world visual reasoning and compositional question answering,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 6700–6709

    2019

  6. [14]

    Panoptic scene graph generation,

    J. Yang, Y . Z. Ang, Z. Guo, K. Zhou, W. Zhang, and Z. Liu, “Panoptic scene graph generation,” in European Conference on Computer Vision. Springer, 2022, pp. 178–196

    2022

  7. [15]

    Action genome: Actions as compositions of spatio-temporal scene graphs,

    J. Ji, R. Krishna, L. Fei-Fei, and J. C. Niebles, “Action genome: Actions as compositions of spatio-temporal scene graphs,” in Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 10 236–10 247

    2020

  8. [16]

    Panoptic video scene graph generation,

    J. Yang, W. Peng, X. Li, Z. Guo, L. Chen, B. Li, Z. Ma, K. Zhou, W. Zhang, C. C. Loy, et al., “Panoptic video scene graph generation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 18 675–18 685

    2023

  9. [17]

    Egtr: Extracting graph from transformer for scene graph generation,

    J. Im, J. Nam, N. Park, H. Lee, and S. Park, “Egtr: Extracting graph from transformer for scene graph generation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 24 229–24 238

    2024

  10. [18]

    Reltr: Relation transformer for scene graph generation,

    Y . Cong, M. Y . Yang, and B. Rosenhahn, “Reltr: Relation transformer for scene graph generation,” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 45, no. 9, pp. 11 169–11 183, 2023

    2023

  11. [19]

    Oed: towards one-stage end-to- end dynamic scene graph generation,

    G. Wang, Z. Li, Q. Chen, and Y . Liu, “Oed: towards one-stage end-to- end dynamic scene graph generation,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 27 938–27 947

    2024

  12. [20]

    Sam 2: Segment anything in images and videos,

    N. Ravi, V . Gabeur, Y .-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. R¨adle, C. Rolland, L. Gustafson, et al., “Sam 2: Segment anything in images and videos,” arXiv preprint arXiv:2408.00714 , 2024

    2024 arXiv

  13. [21]

    Llavanext: Improved reasoning, ocr, and world knowledge,

    H. Liu, C. Li, Y . Li, B. Li, Y . Zhang, S. Shen, and Y . J. Lee, “Llavanext: Improved reasoning, ocr, and world knowledge,” 2024

    2024

  14. [22]

    Yolo-world: Real-time open-vocabulary object detection,

    T. Cheng, L. Song, Y . Ge, W. Liu, X. Wang, and Y . Shan, “Yolo-world: Real-time open-vocabulary object detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 16 901–16 911

    2024

  15. [23]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    2023 arXiv

  16. [24]

    Yi: Open foundation models by 01. ai,

    A. Young, B. Chen, C. Li, C. Huang, G. Zhang, G. Zhang, G. Wang, H. Li, J. Zhu, J. Chen, et al., “Yi: Open foundation models by 01. ai,” arXiv preprint arXiv:2403.04652 , 2024

    2024 arXiv

  17. [25]

    Nvila: Efficient frontier visual language models,

    Z. Liu, L. Zhu, B. Shi, Z. Zhang, Y . Lou, S. Yang, H. Xi, S. Cao, Y . Gu, D. Li,et al., “Nvila: Efficient frontier visual language models,” arXiv preprint arXiv:2412.04468 , 2024

    2024 arXiv

  18. [26]

    Qwen2. 5-vl technical report,

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, et al., “Qwen2. 5-vl technical report,” arXiv preprint arXiv:2502.13923, 2025

    2025 arXiv

  19. [27]

    Graph- based captioning: Enhancing visual descriptions by interconnecting region captions,

    Y .-G. Hsieh, C.-Y . Hsieh, S.-Y . Yeh, L. B ´ethune, H. Pouransari, P. K. A. Vasu, C.-L. Li, R. Krishna, O. Tuzel, and M. Cuturi, “Graph- based captioning: Enhancing visual descriptions by interconnecting region captions,” arXiv preprint arXiv:2407.06723 , 2024

    2024 arXiv

  20. [28]

    Provision: Programmatically scaling vision-centric instruction data for multimodal language models,

    J. Zhang, L. Xue, L. Song, J. Wang, W. Huang, M. Shu, A. Yan, Z. Ma, J. C. Niebles, C. Xiong, et al., “Provision: Programmatically scaling vision-centric instruction data for multimodal language models,” arXiv preprint arXiv:2412.07012, 2024

    2024 arXiv

  21. [29]

    Llm4sgg: large language models for weakly supervised scene graph generation,

    K. Kim, K. Yoon, J. Jeon, Y . In, J. Moon, D. Kim, and C. Park, “Llm4sgg: large language models for weakly supervised scene graph generation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 28 306–28 316

    2024

  22. [30]

    Thinking in space: How multimodal large language models see, remember, and recall spaces,

    J. Yang, S. Yang, A. W. Gupta, R. Han, L. Fei-Fei, and S. Xie, “Thinking in space: How multimodal large language models see, remember, and recall spaces,” arXiv preprint arXiv:2412.14171, 2024

    2024 arXiv

  23. [31]

    From seconds to hours: Reviewing multimodal large language models on comprehensive long video understanding,

    H. Zou, T. Luo, G. Xie, F. Lv, G. Wang, J. Chen, Z. Wang, H. Zhang, H. Zhang, et al., “From seconds to hours: Reviewing multimodal large language models on comprehensive long video understanding,” arXiv preprint arXiv:2409.18938, 2024

    2024 arXiv

  24. [32]

    Visual large language models for generalized and specialized applications,

    Y . Li, Z. Lai, W. Bao, Z. Tan, A. Dao, K. Sui, J. Shen, D. Liu, H. Liu, and Y . Kong, “Visual large language models for generalized and specialized applications,” arXiv preprint arXiv:2501.02765, 2025

    2025 arXiv

  25. [33]

    (2.5+ 1) d spatio- temporal scene graphs for video question answering,

    A. Cherian, C. Hori, T. K. Marks, and J. Le Roux, “(2.5+ 1) d spatio- temporal scene graphs for video question answering,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 36, no. 1, 2022, pp. 444–453

    2022

  26. [34]

    Action scene graphs for long-form understanding of egocentric videos,

    I. Rodin, A. Furnari, K. Min, S. Tripathi, and G. M. Farinella, “Action scene graphs for long-form understanding of egocentric videos,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 18 622–18 632

    2024

  27. [35]

    Hyper- glm: Hypergraph for video scene graph generation and anticipation,

    T.-T. Nguyen, P. Nguyen, J. Cothren, A. Yilmaz, and K. Luu, “Hyper- glm: Hypergraph for video scene graph generation and anticipation,” arXiv preprint arXiv:2411.18042 , 2024

    2024 arXiv

  28. [36]

    Step: Enhancing video-llms’ composi- tional reasoning by spatio-temporal graph-guided self-training,

    H. Qiu, M. Gao, L. Qian, K. Pan, Q. Yu, J. Li, W. Wang, S. Tang, Y . Zhuang, and T.-S. Chua, “Step: Enhancing video-llms’ composi- tional reasoning by spatio-temporal graph-guided self-training,” arXiv preprint arXiv:2412.00161, 2024

    2024 arXiv

  29. [37]

    Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference,

    B. Warner, A. Chaffin, B. Clavi ´e, O. Weller, O. Hallstrom, S. Taghadouini, A. Gallagher, R. Biswas, F. Ladhak, T. Aarsen, et al., “Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference,” arXiv pr...

    2024 arXiv

  30. [38]

    Benchmarking graph neural networks,

    V . P. Dwivedi, C. K. Joshi, A. T. Luu, T. Laurent, Y . Bengio, and X. Bresson, “Benchmarking graph neural networks,” Journal of Machine Learning Research , vol. 24, no. 43, pp. 1–48, 2023

    2023

  31. [39]

    Masked label prediction: Unified message passing model for semi-supervised classification,

    Y . Shi, Z. Huang, S. Feng, H. Zhong, W. Wang, and Y . Sun, “Masked label prediction: Unified message passing model for semi-supervised classification,” arXiv preprint arXiv:2009.03509 , 2020

    2009 arXiv

  32. [40]

    Roformer: En- hanced transformer with rotary position embedding,

    J. Su, M. Ahmed, Y . Lu, S. Pan, W. Bo, and Y . Liu, “Roformer: En- hanced transformer with rotary position embedding,” Neurocomputing, vol. 568, p. 127063, 2024

    2024

  33. [41]

    Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,” in International conference on machine learning . PMLR, 2023, pp. 19 730–19 742

    2023

  34. [42]

    Agqa: A benchmark for compositional spatio-temporal reasoning,

    M. Grunde-McLaughlin, R. Krishna, and M. Agrawala, “Agqa: A benchmark for compositional spatio-temporal reasoning,” in Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 11 287–11 297

    2021

  35. [43]

    Lora: Low-rank adaptation of large language models

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, W. Chen,et al., “Lora: Low-rank adaptation of large language models.” ICLR, vol. 1, no. 2, p. 3, 2022

    2022

  36. [44]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. , “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783 , 2024

    2024 arXiv

  37. [45]

    Do we really need complicated model architectures for temporal networks?

    W. Cong, S. Zhang, J. Kang, B. Yuan, H. Wu, X. Zhou, H. Tong, and M. Mahdavi, “Do we really need complicated model architectures for temporal networks?” arXiv preprint arXiv:2302.11636 , 2023

    2023 arXiv

  38. [46]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017

    2017

  39. [47]

    Question-instructed visual descrip- tions for zero-shot video question answering,

    D. Romero and T. Solorio, “Question-instructed visual descrip- tions for zero-shot video question answering,” arXiv preprint arXiv:2402.10698, 2024

    2024 arXiv

  40. [48]

    Mist: Multi-modal iterative spatial-temporal transformer for long-form video question answering,

    D. Gao, L. Zhou, L. Ji, L. Zhu, Y . Yang, and M. Z. Shou, “Mist: Multi-modal iterative spatial-temporal transformer for long-form video question answering,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2023, pp. 14 773–14 783

    2023

  41. [49]

    Self-chained image-language model for video localization and question answering,

    S. Yu, J. Cho, P. Yadav, and M. Bansal, “Self-chained image-language model for video localization and question answering,” Advances in Neural Information Processing Systems , vol. 36, pp. 76 749–76 771, 2023

    2023

  42. [50]

    Vila: Efficient video-language alignment for video question answering,

    X. Wang, J. Liang, C.-K. Wang, K. Deng, Y . Lou, M. C. Lin, and S. Yang, “Vila: Efficient video-language alignment for video question answering,” in European Conference on Computer Vision . Springer, 2024, pp. 186–204

    2024

  43. [51]

    End- to-end video question answering with frame scoring mechanisms and adaptive sampling,

    J. Liang, X. Meng, Y . Wang, C. Liu, Q. Liu, and D. Zhao, “End- to-end video question answering with frame scoring mechanisms and adaptive sampling,” arXiv preprint arXiv:2407.15047 , 2024

    2024 arXiv

  44. [52]

    Look, remember and reason: Grounded reasoning in videos with language models,

    A. Bhattacharyya, S. Panchal, M. Lee, R. Pourreza, P. Madan, and R. Memisevic, “Look, remember and reason: Grounded reasoning in videos with language models,”arXiv preprint arXiv:2306.17778, 2023

    2023 arXiv

  45. [53]

    Glance and focus: Memory prompt- ing for multi-event video question answering,

    Z. Bai, R. Wang, and X. Chen, “Glance and focus: Memory prompt- ing for multi-event video question answering,” Advances in Neural Information Processing Systems , vol. 36, pp. 34 247–34 259, 2023

    2023

  46. [54]

    Learning to reason iteratively and parallelly for complex visual reasoning scenarios,

    S. Jaiswal, D. Roy, B. Fernando, and C. Tan, “Learning to reason iteratively and parallelly for complex visual reasoning scenarios,” Advances in Neural Information Processing Systems , vol. 37, pp. 137 965–137 998, 2025

    2025

  47. [55]

    Efficient temporal extrapolation of multimodal large language models with temporal grounding bridge,

    Y . Wang, Y . Wang, P. Wu, J. Liang, D. Zhao, Y . Liu, and Z. Zheng, “Efficient temporal extrapolation of multimodal large language models with temporal grounding bridge,” arXiv preprint arXiv:2402.16050 , 2024

    2024 arXiv

  48. [56]

    Learning fine-grained visual understanding for video question answering via decoupling spatial-temporal modeling,

    H.-Y . Lee, H.-T. Su, B.-C. Tsai, T.-H. Wu, J.-F. Yeh, and W. H. Hsu, “Learning fine-grained visual understanding for video question answering via decoupling spatial-temporal modeling,” arXiv preprint arXiv:2210.03941, 2022

    2022 arXiv

  49. [57]

    G-retriever: Retrieval-augmented generation for textual graph understanding and question answering,

    X. He, Y . Tian, Y . Sun, N. V . Chawla, T. Laurent, Y . LeCun, X. Bres- son, and B. Hooi, “G-retriever: Retrieval-augmented generation for textual graph understanding and question answering,” arXiv preprint arXiv:2402.07630, 2024

    2024 arXiv

  50. [58]

    A note on the prize collecting traveling salesman problem,

    D. Bienstock, M. X. Goemans, D. Simchi-Levi, and D. Williamson, “A note on the prize collecting traveling salesman problem,” Mathe- matical programming, vol. 59, no. 1-3, pp. 413–420, 1993

    1993

  51. [59]

    Factual: A benchmark for faithful and consistent textual scene graph parsing,

    Z. Li, Y . Chai, T. Y . Zhuo, L. Qu, G. Haffari, F. Li, D. Ji, and Q. H. Tran, “Factual: A benchmark for faithful and consistent textual scene graph parsing,” arXiv preprint arXiv:2305.17497 , 2023

    2023 arXiv

  52. [2022]

    Available: https://arxiv.org/abs/2204.06105

    [Online]. Available: https://arxiv.org/abs/2204.06105

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.