Pith. sign in

REVIEW 5 major objections 6 minor 57 references

Foundation Models and Adaptive Feature Selection: A Synergistic Approach to Video Question Answering

T0 review · 5 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LGQAVE achieves new state-of-the-art VideoQA accuracy by grounding object graphs in the question, averaging 9.29% over the strongest non-LLM baseline and 6.61% over the strongest LLM baseline.

desk verdict Plausible architecture, but the SOTA claim is contradicted by the paper's own tables and the load-bearing MiniGPT grounding step is never validated. read the letter →

arxiv 2412.09230 v1 pith:2URI6GK3 submitted 2024-12-12 cs.CV cs.AI

Sai Bhargav Rongali , Mohamad Hassan N C , Ankit Jha , Neha Bhargava , Saurabh Prasad , Biplab Banerjee This is my paper · ORCID
classification cs.CVcs.AI
keywords videoquestionansweringquestion-awareframeselectionvisualgroundingdynamicgraphtransformerlocal-globalrepresentationcross-attentionfoundationmodelsreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that video question answering works better when every modeling choice is driven by the question itself rather than by generic video features. The proposed LGQAVE system picks only the frames a question is relevant to, asks MiniGPT-4 to box the objects the question depends on, and builds per-frame spatial graphs from those boxes. A question-conditioned dynamic graph transformer then produces both frame-level local representations and a global video representation, and a cross-attention fusion step refines the global view with local detail before a language model produces the answer. On NExT-QA, TGIF-FrameQA, MSRVTT-QA, ActivityNet-QA, STAR-QA, and Causal-VidQA, the authors report average accuracy gains of 9.29% over CoVGT and 6.61% over VideoLlava. If the results hold, question-conditioned selection is a practical way to remove redundant video content while sharpening answers to both multiple-choice and open-ended questions.

What carries the argument

The load-bearing mechanism is a cascade of three question-conditioned modules. First, a learnable cross-attention between projected CLIP frame features and RoBERTa question features computes a relevance score $s_t$ for each frame, and only frames whose score exceeds a threshold $\beta$ are kept for further processing. Second, MiniGPT-4 performs visual grounding on each selected frame together with its two temporal neighbors on either side, returning up to $m \leq 10$ bounding boxes around question-relevant objects; these boxes become the nodes of a frame-specific spatial graph whose edge weights are learned from appearance and spatial features. Third, the question-aware dynamic graph transformer (Q-DGT) processes the graphs with a masked question embedding to emit per-frame local representations and a mean-pooled global representation, and a final cross-attention uses the local representations as keys and values to refine the global one with mixing weight $\gamma$ before answer decoding.

What would settle it

Take a subset of NExT-QA frames with human-annotated ground-truth boxes for the objects named in each question, compute the per-question recall of MiniGPT-4's bounding boxes, and check whether accuracy on questions with full grounding recall is noticeably higher than on questions with missed objects; if it is not, the grounding stage is not doing the causal work the paper attributes to it.

Watch

Extended reading notes

Core claim

The central claim is that LGQAVE, the Local-Global Question Aware Video Embedding model, establishes a new state of the art in VideoQA by making frame selection, object selection, and representation fusion all question-aware. The authors show that a cross-attention score between question tokens and frame features identifies the frames worth keeping; that grounding those frames with MiniGPT-4 and building object interaction graphs yields richer local representations than using all detected objects; and that combining these local representations with a global video representation through cross-attention beats either representation used alone. The ablation on NExT-QA attributes steady gains to each component: full model Acc@All 66.69 versus 57.40 without sampling, MiniGPT-4 grounding, and local representations. The paper's own summary of the result is that LGQAVE surpasses previous state-of-the-art methods on all tested benchmarks, improving accuracy on average by 9.29% over non-LLM methods like CoVGT and 6.61% over LLM models like VideoLlava.

Load-bearing premise

The whole pipeline assumes that MiniGPT-4 reliably finds and boxes every object the question depends on; any object it misses never enters the object graphs and cannot be recovered at a later stage.

Editorial extensions

If this is right

  • Because only question-relevant frames are processed, VideoQA systems can avoid the computational cost and redundancy of analyzing every frame while focusing attention where the question points.
  • Question-grounded object graphs let the model answer fine-grained causal and temporal questions that global-only representations miss, which is where the reported gains over CoVGT concentrate.
  • The frame-selection and grounding modules are separable: the authors show they improve HQGA and CoVGT when added to those architectures, so the same selection idea could be reused in other VideoQA pipelines.
  • Cross-attention fusion of local and global representations outperforms pooling or concatenating them, so the interaction pattern, not just the features themselves, contributes to the accuracy gain.
  • The pipeline does not rely on video captions or transcript-style supervision, so it can be applied to videos whose content is outside the distribution of caption-based training data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension beyond the paper is to use the frame-selection cross-attention scores as a temporal relevance map and check them against human key-frame annotations; if they align, the same module could serve video grounding and highlight generation.
  • The ceiling of the whole pipeline is likely set by the recall of MiniGPT-4's visual grounding, since an object it fails to box never enters the graphs; benchmarking per-question grounding recall would show where the method loses accuracy.
  • If the gains generalize to longer videos, question-conditioned frame selection could become the default front-end for other video-language tasks such as moment retrieval and video captioning, where most frames are also irrelevant to the target output.
  • One implicit claim worth testing is whether the average gain is a pure frame-sampling effect; rerunning the strongest baseline with the same sampled frames and boxes would isolate the contribution of the graph and local-global modules.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper presents LGQAVE, a video question answering framework that combines three components: a cross-attention-based question-aware frame selection module operating on CLIP frame features and RoBERTa question embeddings; construction of spatial graphs from bounding boxes returned by a MiniGPT-4-based visual grounding step; and a question-conditioned dynamic graph transformer (Q-DGT) that produces local and global video representations refined by cross-attention and used for answer generation. The authors report results on NExT-QA, TGIF-FrameQA, MSRVTT-QA, ActivityNet-QA, STAR-QA, and Causal-VidQA, and claim in Section 4.1 that LGQAVE surpasses previous state-of-the-art methods on all tasks, with average gains of 9.29% over CoVGT and 6.61% over VideoLlava.

Significance. The paper addresses a relevant problem and proposes an interesting integration of question-guided frame selection, visual grounding, and graph-based reasoning for VideoQA. The inclusion of ablations (Table 3), comparisons with multiple baselines, and analysis of the beta and gamma parameters (Figure 3) is a strength. If the performance claims were fully supported, the work would be a useful contribution. However, the manuscript contains numerical inconsistencies that directly contradict the headline claim, and the most novel component, the MiniGPT-4 grounding step, is neither specified nor validated, so the significance of the reported improvements cannot be assessed from the presented evidence.

major comments (5)
  1. [Section 4.1 / Table 1] The claim that "LGQA VE significantly surpasses the previous SOTAs on all tasks" is contradicted by the paper's own Table 1: on MSRVTT-QA, LGQAVE scores 44.81, below VideoLlava's 45.30, and on ActivityNet-QA it scores 61.48, below VideoLlava's 62.25. The text should be revised to acknowledge these exceptions, and the reported average improvements of 9.29% and 6.61% need to be recomputed or justified with a precise definition of the averaging procedure.
  2. [Section 3.2 / Table 3] The entire object-graph pathway depends on MiniGPT-4's visual grounding to produce the bounding boxes that become graph nodes in Eq. 3, but the paper provides no details on how grounding is performed: no prompt template, confidence threshold, grounding head, or post-processing is described, and no grounding accuracy or failure analysis is reported. Since ablation C-4, which removes MiniGPT grounding, drops Acc@All from 66.69 to 58.13 on NExT-QA, the claimed gains are concentrated in this unvalidated component. The authors must supply implementation details and a quantitative validation of the grounding step, or the central performance claim cannot be supported.
  3. [Section 4.1 / Table 2] The claimed benefit of fine frame selection (FFS) is not visible in the reported numbers. The text states "An average increase of 4.23% is observed with this method alone," but Table 2 shows HQGA+FFS at 49.40 Acc@All, which is below the HQGA baseline of 51.34 in Table 1, while CoVGT+FFS at 61.47 is above CoVGT's 57.40. This inconsistency undermines the conclusion that FFS helps existing graph-based methods and should be explained or corrected.
  4. [Section 3.1 / Eq. (2)] The cross-attention score st is defined as Mean(Softmax(˜Et_i · ˜Q^⊤_i) · ˜Qi). Since ˜Et_i is N×C and ˜Qi is M×C, the product is N×M; after softmax and multiplication by ˜Qi, the result is N×C, and the mean over N yields a C-dimensional vector, not the scalar score used to compare frames. The reduction dimension and the intended scalarization need to be specified precisely.
  5. [Section 3.6 / Eqs. (11-12)] The loss functions Lvqa and Lvq are used in Eqs. (11) and (12) but are never defined in the paper; the text says the formulation resembles prior work but does not give the exact functional form. Without these definitions, the training objective is not reproducible.
minor comments (6)
  1. [Abstract] The abstract uses "miniGPT model" while the rest of the paper refers to "MiniGPT-4"; the naming should be consistent.
  2. [Section 3.2] The statement that including two preceding and two subsequent frames was "fixed through empirical validation" is not supported by any reported experiment; please provide the relevant ablation or remove the claim.
  3. [Section 3.3] The notation Q-DGTcm is introduced in Eqs. (6) and (7) without being defined in the text; please clarify what this cross-modal encoder is and how it differs from the Q-DGT module described earlier.
  4. [Table 1] The ActivityNet-QA value for VGT (PT) is reported as 3.70, which appears to be a typographical error; please verify and correct this entry.
  5. [Section 4.1] The computational cost figures (289 GFlops training, 138 GFlops testing) are reported without context or comparison to the baselines; please provide a comparison or state the hardware and batch size used.
  6. [References] Reference [46] lists three authors in a single string; the author list should be separated properly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the pipeline is trained end-to-end against external answer labels; all cited components are external prior work.

full rationale

The derivation chain is not circular. Frame selection (Eq. 2), MiniGPT-4-based graph construction (Eq. 3), Q-DGT refinement (Eqs. 4-7), cross-attention fusion (Eq. 8), and answer scoring (Eqs. 9-10) are all trained with the supervised losses in Eqs. 11-12 using ground-truth answer annotations. Nothing in these equations defines the target answer in terms of its own prediction; the answer labels are external to the model. The MiniGPT-4 grounding step uses the question to select question-relevant bounding boxes, but the final answer is not read off those boxes—it is produced by learned transformations and evaluated against benchmark labels, so this is a standard attention/grounding mechanism rather than a definitional shortcut. No fitted parameter is renamed as a prediction, and no self-citation chain is load-bearing: CoVGT [44], MiniGPT-4 [56], and the frame-selection methods [10, 33] are all external prior work by other authors. The paper's headline claim that LGQAVE surpasses SOTA on all tasks is contradicted by its own Table 1 on MSRVTT-QA and ActivityNet-QA, and the hyperparameters beta and gamma are selected on the same benchmarks, but those are factual and evaluation-protocol concerns, not evidence that a result reduces to its inputs by construction. Accordingly, no circular step meets the required evidentiary bar.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central empirical claim rests on several hand-set hyperparameters and imported modules; the paper provides no code, data, or grounding-quality metrics to evaluate these choices independently.

free parameters (4)
  • frame selection threshold beta = 0.4
    Chosen by sweeping on the target datasets in Figure 3; controls how many frames survive cross-attention selection.
  • fusion weight gamma = 0.9
    Chosen by sweeping in Figure 3; weights global evidence vs cross-attended local evidence in Eq. 8.
  • context window size = 2 preceding + 2 following frames
    The authors state this was fixed through empirical validation, a free choice affecting temporal continuity in graphs.
  • max bounding boxes per frame m = 10
    Cap on number of MiniGPT-4 grounding boxes; no sensitivity analysis is reported.
assumptions (4)
  • domain assumption Softmax cross-attention score st in Eq. 2 measures question relevance of each frame, so thresholding it selects useful frames.
    The paper provides no evaluation of frame-selection quality; the entire pipeline depends on this score.
  • domain assumption MiniGPT-4 bounding-box grounding returns question-relevant objects with sufficient reliability for graph construction.
    No quantitative grounding accuracy is reported; graphs are built from these boxes.
  • domain assumption DGT from CoVGT [44] can be reused as a question-conditioned module without modification.
    Q-DGT is imported from [44] with masking added; no architecture details or re-implementation verification are provided.
  • domain assumption The evaluation protocol in prior benchmark papers is directly comparable, and the reported baseline numbers are accurate.
    The paper reuses baseline accuracies from earlier publications without rerunning them; this is standard but an unverified premise.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Foundation Models and Adaptive Feature Selection: A Synergistic Approach to Video Question Answering." pith.science (2026). https://pith.science/paper/2URI6GK3

@misc{pith2026241209230,
  author       = {Pith},
  title        = {Pith review of: Foundation Models and Adaptive Feature Selection: A Synergistic Approach to Video Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2URI6GK3}},
  note         = {Machine review of arXiv:2412.09230}
}
read the original abstract

This paper tackles the intricate challenge of video question-answering (VideoQA). Despite notable progress, current methods fall short of effectively integrating questions with video frames and semantic object-level abstractions to create question-aware video representations. We introduce Local-Global Question Aware Video Embedding (LGQAVE), which incorporates three major innovations to integrate multi-modal knowledge better and emphasize semantic visual concepts relevant to specific questions. LGQAVE moves beyond traditional ad-hoc frame sampling by utilizing a cross-attention mechanism that precisely identifies the most relevant frames concerning the questions. It captures the dynamics of objects within these frames using distinct graphs, grounding them in question semantics with the miniGPT model. These graphs are processed by a question-aware dynamic graph transformer (Q-DGT), which refines the outputs to develop nuanced global and local video representations. An additional cross-attention module integrates these local and global embeddings to generate the final video embeddings, which a language model uses to generate answers. Extensive evaluations across multiple benchmarks demonstrate that LGQAVE significantly outperforms existing models in delivering accurate multi-choice and open-ended answers.

Figures

Figures reproduced from arXiv: 2412.09230 by the authors.

Figure 1
Figure 1. Qualitative analysis of LGQAVE. We present the answers produced by various state-of-the-art VideoQA models in response to a specific question paired with a sequence of frames from a given video in the NextQA [40] dataset. Our findings indicate that the answers generated by our LGQAVE model are notably more direct and precise in their semantic content. Abstract This paper tackles the intricate challenge of video ques… view at source ↗
Figure 2
Figure 2. Schematic of the model diagram for LGQAVE. Given a question and its corresponding video, our process begins with a question-aware frame sampling module that identifies the pertinent frames from the video. Subsequently, a miniGPT4-based visual grounding module constructs object relation graphs from these selected frames. The Q-DGT module then processes these graphs along with masked question embeddings to produce loc… view at source ↗
Figure 3
Figure 3. Performance of LGQAVE with change in β and γ parameters on various datasets are shown in the first two plots. Performance of LGQAVE with usage of different combinations of Flocal and Fglobal. is shown at the end. What does the women in green jacket doing ? Sampling Module Complete Graph Question￾Aware Graph Global Features Local Features Answer Running in a playground Running with dog Training the dog Spectating the… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Qualitative answers† by LGQAVE model for various ablation configurations on a video from the NextQA dataset. sion of a sampling strategy markedly enhances our model’s performance. Without sampling (Configuration C-1), the model depends solely on global representations,…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 31 canonical work pages

  1. [33]

    Coarse to fine frame selection for online open-ended video question an- swering

    Vidyaranya Nuthalapati and Anirudh Tunga. Coarse to fine frame selection for online open-ended video question an- swering. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops , pages 353–361, October 2023. 7

    work page 2023

  2. [1]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736,

  3. [2]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Dosovitskiy Alexey. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint arXiv: 2010.11929, 2020. 2

    arXiv 2010

  4. [3]

    Revisiting the “Video” in Video-Language Understanding

    Shyamal Buch, Cristobal Eyzaguirre, Adrien Gaidon, Jia- jun Wu, Li Fei-Fei, and Juan Carlos Niebles. Revisiting the “Video” in Video-Language Understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), 2022. 2, 7

    work page 2022

  5. [4]

    Activitynet: A large-scale video benchmark for human activity understanding

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015. 6, 7

    work page 2015

  6. [5]

    Mobilevlm: A fast, strong and open vi- sion language assistant for mobile devices

    Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang, Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, et al. Mobilevlm: A fast, strong and open vi- sion language assistant for mobile devices. arXiv preprint arXiv:2312.16886, 2023. 2

    arXiv 2023

  7. [6]

    Mobilevlm v2: Faster and stronger baseline for vision language model

    Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang, Xiaofei Sun, Yiming Hu, Xinyang Lin, Bo Zhang, and Chunhua Shen. Mobilevlm v2: Faster and stronger baseline for vision language model. ArXiv, abs/2402.03766, 2024. 2

    arXiv 2024

  8. [7]

    Heterogeneous memory en- hanced multimodal attention model for video question an- swering

    Chenyou Fan, Xiaofan Zhang, Shu Zhang, Wensheng Wang, Chi Zhang, and Heng Huang. Heterogeneous memory en- hanced multimodal attention model for video question an- swering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1999–2007,

    work page 1999

Show all 57 references
  1. [8]

    Uatvr: Uncertainty-adaptive text-video retrieval

    Bo Fang, Wenhao Wu, Chang Liu, Yu Zhou, Yuxin Song, Weiping Wang, Xiangbo Shu, Xiangyang Ji, and Jingdong Wang. Uatvr: Uncertainty-adaptive text-video retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13723–13733, 2023. 3

    2023

  2. [9]

    Motion-appearance co-memory networks for video ques- tion answering

    Jiyang Gao, Runzhou Ge, Kan Chen, and Ram Nevatia. Motion-appearance co-memory networks for video ques- tion answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6576–6585,

  3. [10]

    Bridgeformer: Bridging video- text retrieval with multiple choice questions

    Yuying Ge, Yixiao Ge, Xihui Liu, Dian Li, Ying Shan, Xi- aohu Qie, and Ping Luo. Bridgeformer: Bridging video- text retrieval with multiple choice questions. arXiv preprint arXiv:2201.04850, 2022. 7

    2022 arXiv

  4. [11]

    Hierarchical relational atten- tion for video question answering

    Muhammad Iqbal Hasan Chowdhury, Kien Nguyen, Sridha Sridharan, and Clinton Fookes. Hierarchical relational atten- tion for video question answering. In 2018 25th IEEE In- ternational Conference on Image Processing (ICIP) , pages 599–603, 2018. 3

    2018

  5. [12]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016. 2

    2016

  6. [13]

    Lita: Language instructed temporal-localization assistant

    De-An Huang, Shijia Liao, Subhashree Radhakrishnan, Hongxu Yin, Pavlo Molchanov, Zhiding Yu, and Jan Kautz. Lita: Language instructed temporal-localization assistant. CoRR, abs/2403.19046, 2024. 2

    2024 arXiv

  7. [14]

    Tgif-qa: Toward spatio-temporal reasoning in visual question answering

    Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio-temporal reasoning in visual question answering. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 2758–2766, 2017. 3, 6, 7

    2017

  8. [15]

    Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V . Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. ArXiv, abs/2102.05918,

    arXiv

  9. [16]

    Reasoning with heterogeneous graph alignment for video question answering

    Pin Jiang and Yahong Han. Reasoning with heterogeneous graph alignment for video question answering. Proceedings of the AAAI Conference on Artificial Intelligence, 34:11109– 11116, 04 2020. 7

    2020

  10. [17]

    Chat-univi: Unified visual representation em- powers large language models with image and video under- standing

    Peng Jin, Ryuichi Takanobu, Caiwan Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation em- powers large language models with image and video under- standing. ArXiv, abs/2311.08046, 2023. 2

    2023 arXiv

  11. [18]

    Prompting visual-language models for efficient video understanding

    Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. Prompting visual-language models for efficient video understanding. In European Conference on Computer Vi- sion, pages 105–124. Springer, 2022. 3

    2022

  12. [19]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 3

    2023

  13. [20]

    From representa- tion to reasoning: Towards both evidence and commonsense reasoning for video question-answering

    Jiangtong Li, Li Niu, and Liqing Zhang. From representa- tion to reasoning: Towards both evidence and commonsense reasoning for video question-answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 21273–21282, 2022. 6

    2022

  14. [21]

    Videochat: Chat-centric video understanding

    Kunchang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023. 7

    2023 arXiv

  15. [22]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. arXiv preprint arXiv:2311.17043, 2023. 2, 3

    2023 arXiv

  16. [23]

    Video-llava: Learning united visual represen- tation by alignment before projection

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual represen- tation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023. 6, 7

    2023 arXiv

  17. [24]

    Frozen clip models are efficient video learners

    Ziyi Lin, Shijie Geng, Renrui Zhang, Peng Gao, Gerard De Melo, Xiaogang Wang, Jifeng Dai, Yu Qiao, and Hong- sheng Li. Frozen clip models are efficient video learners. In European Conference on Computer Vision, pages 388–404. Springer, 2022. 2, 3

    2022

  18. [25]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 3

    2024

  19. [26]

    Roberta: A robustly optimized bert pretraining approach

    Y Liu, M Ott, N Goyal, J Du, M Joshi, D Chen, O Levy, M Lewis, L Zettlemoyer, and V Stoyanov. Roberta: A robustly optimized bert pretraining approach. arxiv [preprint](2019). arXiv preprint arXiv:1907.11692, 1907. 4

    2019 arXiv

  20. [27]

    Lgdn: Language-guided denoising network for video- language modeling

    Haoyu Lu, Mingyu Ding, Nanyi Fei, Yuqi Huo, and Zhiwu Lu. Lgdn: Language-guided denoising network for video- language modeling. Advances in Neural Information Pro- cessing Systems, 35:25198–25211, 2022. 2

    2022

  21. [28]

    Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning

    Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neu- rocomputing, 508:293–304, 2022. 2, 3

    2022

  22. [29]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fa- had Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023. 3

    2023 arXiv

  23. [30]

    Khan, and Fahad Shahbaz Khan

    Muhammad Maaz, Hanoona Abdul Rasheed, Salman H. Khan, and Fahad Shahbaz Khan. Videogpt+: Integrating im- age and video encoders for enhanced video understanding. ArXiv, abs/2406.09418, 2024. 3

    2024 arXiv

  24. [31]

    Foundation mod- els for video understanding: A survey

    Neelu Madan, Andreas Møgelmose, Rajat Modi, Yogesh S Rawat, and Thomas B Moeslund. Foundation mod- els for video understanding: A survey. arXiv preprint arXiv:2405.03770, 2024. 2

    2024 arXiv

  25. [32]

    Expanding language-image pretrained models for gen- eral video recognition

    Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, and Haibin Ling. Expanding language-image pretrained models for gen- eral video recognition. In European Conference on Com- puter Vision, pages 1–18. Springer, 2022. 3

    2022

  26. [34]

    St-adapter: Parameter-efficient image-to-video transfer learning

    Junting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao, and Hong- sheng Li. St-adapter: Parameter-efficient image-to-video transfer learning. Advances in Neural Information Process- ing Systems, 35:26462–26477, 2022. 3

    2022

  27. [35]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Con...

    2021

  28. [36]

    Faster r-cnn: Towards real-time object detection with region proposal networks

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6):1137–1149, 2017. 2

    2017

  29. [37]

    Annotating objects and relations in user- generated videos

    Xindi Shang, Donglin Di, Junbin Xiao, Yu Cao, Xun Yang, and Tat-Seng Chua. Annotating objects and relations in user- generated videos. In Proceedings of the 2019 on Interna- tional Conference on Multimedia Retrieval, pages 279–287,

    2019

  30. [38]

    Star: A benchmark for situated rea- soning in real-world videos

    Bo Wu and Shoubin Yu. Star: A benchmark for situated rea- soning in real-world videos. ArXiv, abs/2405.09711, 2024. 6

    2024 arXiv

  31. [39]

    Florence-2: Advancing a unified representation for a variety of vision tasks

    Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a unified representation for a variety of vision tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pag...

    2024

  32. [40]

    Next-qa: Next phase of question-answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 9777–9786, 2021. 1, 6, 7

    2021

  33. [41]

    Can i trust your answer? visually grounded video question answering

    Junbin Xiao, Angela Yao, Yicong Li, and Tat-Seng Chua. Can i trust your answer? visually grounded video question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13204– 13214, 2024. 2

    2024

  34. [42]

    Video as conditional graph hierarchy for multi-granular question answering

    Junbin Xiao, Angela Yao, Zhiyuan Liu, Yicong Li, Wei Ji, and Tat-Seng Chua. Video as conditional graph hierarchy for multi-granular question answering. In Proceedings of the 36th AAAI Conference on Artificial Intelligence (AAAI) , pages 2804–2812, 2022. 7

    2022

  35. [43]

    Video graph transformer for video question answering

    Junbin Xiao, Pan Zhou, Tat-Seng Chua, and Shuicheng Yan. Video graph transformer for video question answering. In European Conference on Computer Vision , pages 39–58. Springer, 2022. 7

    2022

  36. [44]

    J. Xiao, P. Zhou, A. Yao, Y . Li, R. Hong, S. Yan, and T. Chua. Contrastive video question answering via video graph transformer. IEEE Transactions on Pattern Analysis & Machine Intelligence, 45(11):13265–13280, nov 2023. 2, 3, 4, 5, 6, 7

    2023

  37. [45]

    Video question answer- ing via gradually refined attention over appearance and mo- tion

    Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answer- ing via gradually refined attention over appearance and mo- tion. In ACM Multimedia, 2017. 6, 7

    2017

  38. [46]

    Videoclip: Con- trastive pre-training for zero-shot video-text understanding

    Hu Xu, Gargi Ghosh, Po-Yao (Bernie) Huang, Dmytro Okhonko, Armen Aghajanyan, and Florian Metze Luke Zettlemoyer Christoph Feichtenhofer. Videoclip: Con- trastive pre-training for zero-shot video-text understanding. In Conference on Empirical Methods in Natural Language Process...

    2021

  39. [47]

    Just ask: Learning to answer questions from millions of narrated videos

    Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Just ask: Learning to answer questions from millions of narrated videos. In ICCV, 2021. 7

    2021

  40. [48]

    Learning to answer visual questions from web videos

    Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Learning to answer visual questions from web videos. IEEE transactions on pattern analysis and ma- chine intelligence, PP, 2022. 2

    2022

  41. [49]

    Self-chained image-language model for video localization and question answering

    Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. Self-chained image-language model for video localization and question answering. Advances in Neural Information Processing Systems, 36, 2024. 2

    2024

  42. [50]

    Lever- aging video descriptions to learn video question answering

    Kuo-Hao Zeng, Tseng-Hung Chen, Ching-Yao Chuang, Yuan-Hong Liao, Juan Carlos Niebles, and Min Sun. Lever- aging video descriptions to learn video question answering. In Proceedings of the AAAI conference on artificial intelli- gence, volume 31, 2017. 1

    2017

  43. [51]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding. arXiv preprint arXiv:2306.02858, 2023. 3, 7

    2023 arXiv

  44. [52]

    Instruction tuning for large language models: A survey

    Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xi- aofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308.10792, 2023. 3

    2023

  45. [53]

    Video question answering via hierarchi- cal spatio-temporal attention networks

    Zhou Zhao, Qifan Yang, Deng Cai, Xiaofei He, Yueting Zhuang, Zhou Zhao, Qifan Yang, Deng Cai, Xiaofei He, and Yueting Zhuang. Video question answering via hierarchi- cal spatio-temporal attention networks. In IJCAI, volume 2, page 8, 2017. 1

    2017

  46. [54]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonza- lez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In A. Oh, T. Naumann, A. Globerson, K. Saen...

    2023

  47. [55]

    Video question answer- ing: Datasets, algorithms and challenges

    Yaoyao Zhong, Junbin Xiao, Wei Ji, Yicong Li, Wei- hong Deng, and Tat-Seng Chua. Video question answer- ing: Datasets, algorithms and challenges. arXiv preprint arXiv:2203.01225, 2022. 1

    2022 arXiv

  48. [56]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 2, 3

    2023 arXiv

  49. [57]

    Actbert: Learning global-local video-text representations

    Linchao Zhu and Yi Yang. Actbert: Learning global-local video-text representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020. 2

    2020

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.