Pith. sign in

REVIEW 5 major objections 4 minor 1 cited by

TOGA: Temporally Grounded Open-Ended Video QA with Weak Supervision

T0 review · 5 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TOGA grounds open-ended video answers in time using only weak question-answer supervision, with no temporal labels.

desk verdict A genuinely useful idea for weak-supervised grounding, undermined by an oversold SOTA claim and an underspecified pseudo-label pipeline. read the letter →

arxiv 2506.09445 v1 pith:ZJVVLSV4 submitted 2025-06-11 cs.CV cs.AI

Ayush Gupta , Anirban Roy , Rama Chellappa , Nathaniel D. Bastian , Alvaro Velasquez , Susmit Jha This is my paper
classification cs.CVcs.AI
keywords videoquestionansweringtemporalgroundingweaksupervisionvision-languagemodelpseudo-labelingconsistencyconstraintopen-endedmulti-scalefeatures
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TOGA claims that a vision-language model can answer open-ended questions about video and simultaneously output the start and end times of the evidence, without ever being trained on temporal annotations. The paper's route is to generate pseudo-label intervals by cropping segments of training videos, asking a stage-one model to describe each segment, and then keeping only those labels whose answer agrees with a paired question referring to the same segment. A single instruction-tuned decoder is trained to emit the answer and the interval together, and the paper reports that this joint generation improves both answer accuracy and grounding localization. On the grounded-QA benchmark NExT-GQA the method reaches 24.4 mean IoU, and on open-ended MSVD-QA and ActivityNet-QA it reaches 73.8 and 52.0 accuracy. If correct, temporal grounding becomes obtainable from question-answer data alone, which removes an expensive annotation bottleneck.

What carries the argument

The load-bearing object is the multi-scale vision-language connector (MS-VLC), a two-branch module that reads 16 densely sampled frames for high-frequency, short-term temporal cues and 4 sparsely sampled frames for low-frequency, long-term cues, implemented as two shared-parameter RegNet stages separated by a 3D convolution. It is the only visual component trained in all stages, and it aligns video features with question features for the frozen Mistral-7B decoder. The other key mechanism is the three-stage training loop: first, align MS-VLC on caption and question-answer video-text data; second, crop segments, describe them with the stage-one model, and instruction-tune on the resulting pseudo answer-plus-interval pairs using prompts that specify the output format; third, apply a consistency constraint between a grounding question and a referring question about the same segment, while also requiring the answer to match the dataset's question-answer annotation. Together these let the model learn interval prediction entirely from weak question-answer supervision.

What would settle it

Take a small set of NExT-GQA training videos for which human annotators provide the true evidence interval, run the pseudo-labeling pipeline, and compare the intervals that pass the consistency filter with those that fail; if accepted intervals are no more accurate than rejected ones, the filter is selecting on format rather than content. A complementary test is to remove the consistency stage and finetune only with unfiltered pseudo labels; the paper reports mIoU dropping from 24.4 to 12.1, so a reader could also check whether a random pairing of referring questions produces a similar drop, which would indicate the gain comes from consistency itself.

Watch

Extended reading notes

Core claim

The central claim is that temporal grounding does not need dedicated annotation if the model is trained to answer and localize in one pass. TOGA instruction-tunes a Mistral-7B decoder, fed features from a frozen CLIP encoder through a multi-scale vision-language connector, to produce responses of the form “answer [start, end]” with start and end scaled to [0,100]. Since no grounding labels exist, the training pipeline manufactures them: random segments are cropped, described by the stage-one model as if they were complete videos, and then filtered by requiring consistency between a grounding question (what is happening?) and a referring question (what happens in [start,end]?) for the same segment. The paper reports that this consistency stage lifts mean IoU on NExT-GQA from 12.1 to 24.4, and that joint answer-plus-interval decoding outperforms separately predicting the two. Its strongest stated result is state-of-the-art grounding (24.4 mIoU, 40.5 mIoP on NExT-GQA), together with state-of-the-art open-ended answering (73.8 accuracy on MSVD-QA, 52.0 on ActivityNet-QA), plus zero-shot grounding on ReXTime at 25.53 mIoU.

Load-bearing premise

The load-bearing premise is that the first-stage model, trained on ordinary caption and question-answer data, can describe arbitrary cropped video segments accurately enough that the automatically generated start and end times are useful training signal, and that the consistency filter removes bad labels rather than keeping a systematically biased subset.

Editorial extensions

If this is right

  • Grounding can be learned from ordinary question-answer pairs: any dataset that has QA annotations but no interval labels can be used to teach both answering and localization.
  • Generating the answer and the interval together yields better grounding than separate modules; abandoning joint decoding gives up measurable IoU on NExT-GQA.
  • Open-ended answers can be evaluated without giving the model candidate options, so grounded video QA no longer requires a closed-set design.
  • The consistency filter is a concrete denoiser for pseudo labels: removing it drops mIoU from 24.4 to 12.1 on NExT-GQA.
  • Multi-scale temporal features are the main reason both short and long events are localized; single-scale branches perform worse on both extremes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the same answer-pairing filter could be applied to pseudo-label any structured output, such as evidence sentences, frame indices, or reasoning chains, whenever two question phrasings can be generated over the same unit.
  • Beyond the paper: the supplement's finding that predictions are biased toward starting at 0 and ending at 100 suggests a boundary- or length-aware training objective as a direct next experiment, since it might recover short and medium intervals without new labels.
  • Beyond the paper: the large gain from the consistency stage could be audited by comparing accepted and rejected pseudo labels against human interval annotations on a small sample; this would tell whether the filter is selecting for accuracy or simply for format.
  • Beyond the paper: since (16,4) outperforms (16,8) and (8,2), the sparse branch should be much coarser than the dense branch; a systematic search over the sparse-to-dense ratio on longer videos could sharpen the design.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The manuscript proposes TOGA, a vision-language model for weakly supervised, open-ended video question answering with temporal grounding. The model combines a frozen CLIP vision encoder, a Mistral-7B language decoder, and a multi-scale vision-language connector (MS-VLC) that processes videos at sparse and dense frame rates. Training proceeds in three stages: vision-text alignment on Video-ChatGPT data; instruction tuning on pseudo-labeled temporal segments obtained by cropping videos and generating descriptions with the stage-1 model; and a consistency stage that requires grounding answers and referring answers for the same segment to match. Evaluation covers NExT-GQA and ReXTime for grounded QA, and MSVD-QA and ActivityNet-QA for open-ended QA. The paper claims state-of-the-art performance for both tasks, with mIoU 24.4 and Acc@GQA 24.6 on NExT-GQA, and accuracy 73.8 on MSVD-QA and 52.0 on ActivityNet-QA.

Significance. If the pseudo-labeling and consistency mechanism is validated, the contribution is significant: it removes the need for temporal annotations, supports open-ended answers, and demonstrates that joint generation of answer and grounding improves both tasks. The MS-VLC ablation, question-type analysis, and zero-shot ReXTime evaluation are useful contributions. The paper also includes multiple evaluation protocols (GPT-based, Llama-based, and METEOR), which strengthens reproducibility of the QA comparisons. However, the central validation is incomplete because pseudo-label quality is never measured, and the state-of-the-art claim for grounded QA is not supported by the combined metric.

major comments (5)
  1. [Abstract; Section 4.2, Table 1] The abstract's state-of-the-art claim for grounded QA is not supported by the combined metric on NExT-GQA: TOGA obtains Acc@GQA of 24.6, while Grounded-VideoLLM obtains 26.7. Since Acc@GQA is the headline metric for jointly correct answer and grounding, the claim that TOGA outperforms the state of the art on this task should be restricted to the IoU/IoP metrics or accompanied by a direct explanation of why the combined metric is lower.
  2. [Section 3, Multi-stage training] The pseudo-labeling procedure is underspecified. The text states that the authors 'crop temporal segments in videos with known start and end times' and generate descriptions by considering these segments as full videos, but the source of the crop boundaries is not described, and the quality of the generated pseudo-labels is never measured. The manuscript should report the crop-selection procedure, the pseudo-label retention rate, the agreement of retained pseudo-answers with ground-truth QA pairs, and the IoU of retained pseudo-intervals against manually annotated intervals. Without these measurements, the central claim that consistency-filtered pseudo-labels yield accurate weakly supervised grounding is not established.
  3. [Section 3, Figure 3] The consistency constraint assumes that the answer to a grounding question such as 'What is the boy doing?' and the answer to a referring question such as 'What is happening in [25,73]?' should match exactly. This assumption is strong and may not hold in general, because a referring question can require a scene-level answer that is broader than the answer to a specific grounding question. The paper should quantify how often the two questions actually have compatible answers in the training data, and how often the consistency filter discards valid labels or retains only generic answers.
  4. [Section 4.3, Analysis of temporal representation and Analysis of the number of frames] The choices of the [0,100] temporal range and the (16,4) frame configuration are selected by comparing mIoU values on the NExT-GQA test set. Because no validation split is used, the reported grounding results reflect test-set tuning. The authors should either use a held-out validation split for these design choices or state explicitly that the reported numbers are tuned on the test set, which is a form of test-set leakage and weakens the generality of the conclusions.
  5. [Supplementary Material, Section 2.3, Table 3] The predicted boundaries are noticeably more biased toward the video edges than the ground truth: 21.2% of predictions start at time 0 versus 14.4% for the ground truth, and 14.05% end at time 100 versus 10.1% for the ground truth. This pattern is consistent with the consistency stage learning a prior over segment boundaries rather than learning to localize the evidence for the answer. The ablation showing an mIoU improvement from 12.1 to 24.4 with the consistency constraint should be accompanied by a breakdown of improvements on interior versus boundary events, and by error analysis conditional on pseudo-label quality.
minor comments (4)
  1. [Supplementary Material, Section 2.6] Supplementary Section 2.6 reports Acc@QA of 67.0, which is below LLoVi's 67.7. The text says this is 'close to SOTA', but this is in tension with the abstract's state-of-the-art claim. Please reconcile these statements.
  2. [Table 2 versus Supplementary Material, Section 2.10] Main Table 2 reports a ReXTime test mIoU of 25.53, while Supplementary Section 2.10 reports a validation mIoU of 27.4 without clarifying that these are different splits. Please clarify which numbers correspond to which split.
  3. [Section 4.3, Analysis of the number of frames] The (32,8) configuration is said to require 'three times higher training time', but no training time is reported for the (16,4) configuration, making the comparison vague. Please report the actual times.
  4. [Supplementary Material, Section 2.9] A METEOR score of 0.498 on ActivityNet-QA is reported without comparison to any baseline. Either add comparisons or remove this number, since it does not provide context for the model's quality.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: TOGA's pseudo-label self-training is evaluated on external human-annotated grounding and QA benchmarks, so the reported predictions do not reduce to the training inputs.

full rationale

The pipeline is a self-training/self-consistency scheme: a stage-1 model trained on Video-ChatGPT produces descriptions for cropped segments, and a final consistency constraint filters pseudo-labels using the model's own referring answers. This is self-referential in a training-signal sense, but it is not circular in the logical sense required here. Every headline result (NExT-GQA mIoU 24.4, MSVD-QA 73.8, ActivityNet-QA 52.0, ReXTime zero-shot 25.53) is measured against externally provided human ground truth, not against the pseudo-labels or the consistency criterion. The consistency ablation (12.1 -> 24.4 mIoU) is an empirical comparison, not an identity. The paper does not invoke any load-bearing self-citation, uniqueness theorem, or fitted parameter renamed as a prediction. The concern that pseudo-label quality is unmeasured (e.g., no retention rate, no agreement with GT intervals) is a real robustness/correctness risk, and the supplementary statistics even show a bias toward start=0 and end=100; however, that is a limitation of the weak-supervision signal, not a circular derivation. The limitations section explicitly acknowledges the single-interval assumption, which further confirms the claims are empirical rather than definitional.

Assumptions & free parameters 4 free parameters · 5 assumptions · 1 invented entities

The central claim rests on several hand-chosen hyperparameters and unstated details in the pseudo-labeling pipeline. No new physical entities are introduced; the only invented component is the MS-VLC connector.

free parameters (4)
  • Temporal representation range [0,100] = integers 0 to 100
    Chosen by hand; the [0,1] float representation gives mIoU 19.0 versus 24.4 with [0,100], so this choice is fitted to test performance.
  • Frame sampling configuration (16,4) = 16 dense, 4 sparse
    Selected via test-set ablations: (8,2) gives mIoU 20.8 and (32,8) gives 21.5, both below the chosen (16,4) at 24.4.
  • Consistency filtering threshold = not reported
    The paper does not specify how many pseudo labels pass the consistency filter or the threshold used, a hidden choice that affects training.
  • Pseudo-segment selection strategy = not reported
    How temporal segments are cropped or sampled for pseudo-label generation is not described, affecting the distribution of training labels.
assumptions (5)
  • domain assumption CLIP-ViT-Large features are sufficient frame-level representations for video QA.
    Invoked in Section 3, Vision encoder; the encoder is kept frozen and relied on for text-aligned visual features.
  • domain assumption Mistral-7B Instruct can be instruction-tuned to generate grounded answers.
    Invoked in Section 3, Text decoder; the decoder is trained on next-token prediction to emit answer and grounding.
  • domain assumption Video-ChatGPT data provides sufficient alignment for the MS-VLC connector.
    Invoked in Section 3, Implementation details; the first training stage uses this dataset for vision-text alignment.
  • domain assumption The consistency constraint is a reliable proxy for grounding validity.
    Invoked in Section 3, Multi-stage training; the paper assumes matching answers to Q_g and Q_r indicate a sound pseudo label.
  • domain assumption GPT-3.5-based evaluation fairly maps open-ended answers to ground truth.
    Invoked in Section 4.1, Metrics; used for both Acc@GQA and open-ended QA evaluation, and the comparison to closed-set methods depends on it.
invented entities (1)
  • MS-VLC (multi-scale vision-language connector)
    purpose: Aligns dense and sparse video features with text features for joint answer and grounding generation.
    A new architecture module introduced by this paper; its effectiveness is shown only through in-house ablations, with no external validation or theoretical grounding.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TOGA: Temporally Grounded Open-Ended Video QA with Weak Supervision." pith.science (2026). https://pith.science/paper/ZJVVLSV4

@misc{pith2026250609445,
  author       = {Pith},
  title        = {Pith review of: TOGA: Temporally Grounded Open-Ended Video QA with Weak Supervision},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZJVVLSV4}},
  note         = {Machine review of arXiv:2506.09445}
}
read the original abstract

We address the problem of video question answering (video QA) with temporal grounding in a weakly supervised setup, without any temporal annotations. Given a video and a question, we generate an open-ended answer grounded with the start and end time. For this task, we propose TOGA: a vision-language model for Temporally Grounded Open-Ended Video QA with Weak Supervision. We instruct-tune TOGA to jointly generate the answer and the temporal grounding. We operate in a weakly supervised setup where the temporal grounding annotations are not available. We generate pseudo labels for temporal grounding and ensure the validity of these labels by imposing a consistency constraint between the question of a grounding response and the response generated by a question referring to the same temporal segment. We notice that jointly generating the answers with the grounding improves performance on question answering as well as grounding. We evaluate TOGA on grounded QA and open-ended QA tasks. For grounded QA, we consider the NExT-GQA benchmark which is designed to evaluate weakly supervised grounded question answering. For open-ended QA, we consider the MSVD-QA and ActivityNet-QA benchmarks. We achieve state-of-the-art performance for both tasks on these benchmarks.

Figures

Figures reproduced from arXiv: 2506.09445 by the authors.

Figure 1
Figure 1. We present an approach for open-ended grounded video question answering. Given a video and open-ended questions, we [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. We propose a VLM framework for grounded videoQA. Given an input video, we sample frames and compute framewise features [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. We train TOGA in three stages. Each stage focuses on a specific task with prompts designed for the task. Q [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Qualitative results on NExT-GQA. We present the ground truth and predicted answers with groundings. Ground truth segments [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Qualitative results for open-ended videoQA on MSVD. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: The top part presents an example from NExT-GQA [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 1
Figure 1. Figure 1: Additional qualitative examples from the NExT-GQA dataset, with longer videos and multiple questions per video. Compared [PITH_FULL_IMAGE:figures/full_fig_p015_1.png]
Figure 2
Figure 2. Figure 2: Additional qualitative examples from the ActivityNet dataset. Our model may generate words that are similar in meaning but not [PITH_FULL_IMAGE:figures/full_fig_p016_2.png]
Figure 3
Figure 3. Figure 3: Examples of grounding on ActivityNet videos. TOGA can perform grounding even on this dataset, but the questions may be [PITH_FULL_IMAGE:figures/full_fig_p016_3.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Towards Temporal Compositional Reasoning in Long-Form Sports Videos

    cs.CV 2026-04 unverdicted novelty 7.0 of 10

    SportsTime plus Chain-of-Time Reasoning (temporal-reward GRPO and anchor-observe-infer) modestly lifts open-ended sports VideoQA and step-wise temporal grounding over 4B–8B MLLM baselines.

Reference graph

Works this paper leans on

64 extracted references · 48 canonical work pages · cited by 1 Pith paper

  1. [1]

    Lawrence Zitnick, and Devi Parikh

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. InProceedings of the IEEE International Conference on Computer Vision (ICCV), 2015. 2

    work page 2015

  2. [2]

    Collecting highly parallel data for paraphrase evaluation

    David Chen and William Dolan. Collecting highly parallel data for paraphrase evaluation. InProceedings of the 49th Annual Meeting of the Association for Computational Lin- guistics: Human Language Technologies, pages 190–200, Portland, Oregon, USA, 2011. Association for Computa- tional Linguistics. 6

    work page 2011

  3. [3]

    Rextime: A benchmark suite for reasoning-across-time in videos.arXiv preprint arXiv:2406.19392, 2024

    Jr-Jen Chen, Yu-Chien Liao, Hsi-Che Lin, Yu-Chu Yu, Yen- Chun Chen, and Yu-Chiang Frank Wang. Rextime: A benchmark suite for reasoning-across-time in videos.arXiv preprint arXiv:2406.19392, 2024. 5, 6, 7, 3

    arXiv 2024

  4. [4]

    Panda-70m: Captioning 70m videos with multiple cross-modality teachers, 2024

    Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, and Sergey Tulyakov. Panda-70m: Captioning 70m videos with multiple cross-modality teachers, 2024. 7

    work page 2024

  5. [5]

    Videollama 2: Advancing spatial-temporal modeling and audio understanding in video- llms.arXiv preprint arXiv:2406.07476, 2024

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video- llms.arXiv preprint arXiv:2406.07476, 2024. 2, 4, 5, 6, 7, 1, 3

    arXiv 2024

  6. [6]

    Activitynet: A large-scale video bench- mark for human activity understanding

    Bernard Ghanem Fabian Caba Heilbron, Victor Escorcia and Juan Carlos Niebles. Activitynet: A large-scale video bench- mark for human activity understanding. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 961–970, 2015. 6

    work page 2015

  7. [7]

    Video2Commonsense: Generating Commonsense Descriptions to Enrich Video Captioning

    Zhiyuan Fang, Tejas Gokhale, Pratyay Banerjee, Chitta Baral, and Yezhou Yang. Video2commonsense: Generating commonsense descriptions to enrich video captioning.arXiv preprint arXiv:2003.05162, 2020. 2

    work page Pith review arXiv 2003

  8. [8]

    Slowfast networks for video recognition

    Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6202–6211, 2019. 2, 4

    work page 2019

Show all 64 references
  1. [9]

    Knowit vqa: Answering knowledge-based ques- tions about videos

    Noa Garcia, Mayu Otani, Chenhui Chu, and Yuta Nakashima. Knowit vqa: Answering knowledge-based ques- tions about videos. InProceedings of the AAAI conference on artificial intelligence, pages 10826–10834, 2020. 2

    2020

  2. [10]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering.Int

    Yash Goyal, Tejas Khot, Aishwarya Agrawal, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering.Int. J. Comput. Vision, 127(4): 398–414, 2019. 2

    2019

  3. [11]

    Question generation via overgenerating transformations and ranking.DTIC Doc- ument, 2009

    Michael Heilman and Noah A Smith. Question generation via overgenerating transformations and ranking.DTIC Doc- ument, 2009. 6

    2009

  4. [12]

    Lita: Language instructed temporal-localization assistant

    De-An Huang, Shijia Liao, Subhashree Radhakrishnan, Hongxu Yin, Pavlo Molchanov, Zhiding Yu, and Jan Kautz. Lita: Language instructed temporal-localization assistant. In Computer Vision – ECCV 2024: 18th European Confer- ence, Milan, Italy, September 29–October 4, 2024, Procee...

    2024

  5. [13]

    Tgif-qa: Toward spatio-temporal reasoning in visual question answering

    Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio-temporal reasoning in visual question answering. InProceedings of the IEEE con- ference on computer vision and pattern recognition, pages 2758–2766, 2017. 5

    2017

  6. [14]

    Video question answering with spatio-temporal reasoning.IJCV, 2019

    Yunseok Jang, Yale Song, Chris Dongjoo Kim, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Video question answering with spatio-temporal reasoning.IJCV, 2019. 2

    2019

  7. [15]

    Video question answering with spatio-temporal reasoning.Int

    Yunseok Jang, Yale Song, Chris Dongjoo Kim, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Video question answering with spatio-temporal reasoning.Int. J. Comput. Vision, 127 (10):1385–1412, 2019. 2

    2019

  8. [16]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lam- ple, Lucile Saulnier, L ´elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Tho...

    2023

  9. [17]

    Chat-univi: Unified visual representation em- powers large language models with image and video under- standing.arXiv preprint arXiv:2311.08046, 2023

    Peng Jin, Ryuichi Takanobu, Caiwan Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation em- powers large language models with image and video under- standing.arXiv preprint arXiv:2311.08046, 2023. 7

    2023 arXiv

  10. [18]

    Slow-fast auditory streams for audio recogni- tion

    Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen. Slow-fast auditory streams for audio recogni- tion. InICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 855–859. IEEE, 2021. 2, 4

    2021

  11. [19]

    Open-vocabulary video question answering: A new benchmark for evaluating the generaliz- ability of video question answering models

    Dohwan Ko, Ji Soo Lee, Miso Choi, Jaewon Chu, Jihwan Park, and Hyunwoo J Kim. Open-vocabulary video question answering: A new benchmark for evaluating the generaliz- ability of video question answering models. InProceedings of the IEEE/CVF international conference on computer ...

    2023

  12. [20]

    Dense-captioning events in videos,

    Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos,

  13. [21]

    Meteor: an automatic met- ric for mt evaluation with high levels of correlation with hu- man judgments

    Alon Lavie and Abhaya Agarwal. Meteor: an automatic met- ric for mt evaluation with high levels of correlation with hu- man judgments. InProceedings of the Second Workshop on Statistical Machine Translation, page 228–231, USA, 2007. Association for Computational Linguistics. 3

    2007

  14. [22]

    Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L. Berg. Tvqa: Localized, compositional video question answering,

  15. [23]

    Berg, and Mohit Bansal

    Jie Lei, Licheng Yu, Tamara L. Berg, and Mohit Bansal. Tvqa+: Spatio-temporal grounding for video question an- swering, 2020. 2, 3

    2020

  16. [24]

    Videochat: Chat-centric video understanding.arXiv preprint arXiv:2305.06355, 2023

    Kunchang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding.arXiv preprint arXiv:2305.06355, 2023. 2, 7

    2023 arXiv

  17. [25]

    Equivariant and invariant grounding for video question an- swering

    Yicong Li, Xiang Wang, Junbin Xiao, and Tat-Seng Chua. Equivariant and invariant grounding for video question an- swering. InProceedings of the 30th ACM International Con- ference on Multimedia, page 4714–4722, New York, NY , USA, 2022. Association for Computing Machinery. 7

    2022

  18. [26]

    Video-llava: Learning united visual represen- tation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual represen- tation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023. 2, 4, 5, 6, 7, 3

    2023 arXiv

  19. [27]

    Visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. 3, 4

    2023

  20. [28]

    Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    I Loshchilov. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017. 5

    2017 arXiv

  21. [29]

    Video-chatgpt: Towards detailed video understanding via large vision and language models, 2024

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fa- had Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models, 2024. 2, 5, 6, 7

    2024

  22. [30]

    The surprising effectiveness of multimodal large language models for video moment retrieval, 2024

    Boris Meinardus, Anil Batra, Anna Rohrbach, and Marcus Rohrbach. The surprising effectiveness of multimodal large language models for video moment retrieval, 2024. 8, 3

    2024

  23. [31]

    Exploration of masked and causal language mod- elling for text generation, 2024

    Nicolo Micheletti, Samuel Belkadi, Lifeng Han, and Goran Nenadic. Exploration of masked and causal language mod- elling for text generation, 2024. 4

    2024

  24. [32]

    George A. Miller. Wordnet: a lexical database for english. Commun. ACM, 38(11):39–41, 1995. 2

    1995

  25. [33]

    Gpt-4 technical report, 2024

    OpenAI. Gpt-4 technical report, 2024. 2, 4, 6

    2024

  26. [34]

    Streaming long video understanding with large language models.Advances in Neu- ral Information Processing Systems, 37:119336–119360,

    Rui Qian, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Shuan- grui Ding, Dahua Lin, and Jiaqi Wang. Streaming long video understanding with large language models.Advances in Neu- ral Information Processing Systems, 37:119336–119360,

  27. [35]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. 2, 4, 5, 7

    2021

  28. [36]

    Designing network design spaces

    Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollar. Designing network design spaces. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 4, 1

    2020

  29. [37]

    Concept- net 5.5: an open multilingual graph of general knowledge

    Robyn Speer, Joshua Chin, and Catherine Havasi. Concept- net 5.5: an open multilingual graph of general knowledge. InProceedings of the Thirty-First AAAI Conference on Arti- ficial Intelligence, page 4444–4451. AAAI Press, 2017. 2

    2017

  30. [38]

    Vipergpt: Visual inference via python execution for reasoning

    D ´ıdac Sur´ıs, Sachit Menon, and Carl V ondrick. Vipergpt: Visual inference via python execution for reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11888–11898, 2023. 2, 3

    2023

  31. [39]

    Movieqa: Understanding stories in movies through question- answering

    Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. Movieqa: Understanding stories in movies through question- answering. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 4631–4640,

  32. [40]

    Gemini: A family of highly capable multi- modal models, 2024

    Gemini Team. Gemini: A family of highly capable multi- modal models, 2024. 2

    2024

  33. [41]

    Llama: Open and efficient foundation lan- guage models, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, Aure- lien Rodriguez, Armand Joulin, Edouard Grave, and Guil- laume Lample. Llama: Open and efficient foundation la...

    2023

  34. [42]

    Grounded-videollm: Sharpening fine-grained tem- poral grounding in video large language models, 2024

    Haibo Wang, Zhiyang Xu, Yu Cheng, Shizhe Diao, Yu- fan Zhou, Yixin Cao, Qifan Wang, Weifeng Ge, and Lifu Huang. Grounded-videollm: Sharpening fine-grained tem- poral grounding in video large language models, 2024. 2, 3, 7

    2024

  35. [43]

    Videoagent: Long-form video understand- ing with large language model as agent.arXiv preprint arXiv:2403.10517, 2024

    Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. Videoagent: Long-form video understand- ing with large language model as agent.arXiv preprint arXiv:2403.10517, 2024. 2, 3

    2024 arXiv

  36. [44]

    Videotree: Adaptive tree-based video representation for llm reasoning on long videos.arXiv preprint arXiv:2405.19209,

    Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jaehong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. Videotree: Adaptive tree-based video representation for llm reasoning on long videos.arXiv preprint arXiv:2405.19209,

    arXiv

  37. [45]

    Star: A benchmark for situated reason- ing in real-world videos.arXiv preprint arXiv:2405.09711,

    Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. Star: A benchmark for situated reason- ing in real-world videos.arXiv preprint arXiv:2405.09711,

    arXiv

  38. [46]

    Number it: Temporal grounding videos like flipping manga,

    Yongliang Wu, Xinting Hu, Yuyang Sun, Yizhou Zhou, Wenbo Zhu, Fengyun Rao, Bernt Schiele, and Xu Yang. Number it: Temporal grounding videos like flipping manga,

  39. [47]

    Audiovisual slowfast networks for video recognition.arXiv preprint arXiv:2001.08740,

    Fanyi Xiao, Yong Jae Lee, Kristen Grauman, Jitendra Malik, and Christoph Feichtenhofer. Audiovisual slowfast networks for video recognition.arXiv preprint arXiv:2001.08740,

    2001 arXiv

  40. [48]

    Next-qa:next phase of question-answering to explaining tem- poral actions, 2021

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa:next phase of question-answering to explaining tem- poral actions, 2021. 2

    2021

  41. [49]

    Videoqa in the era of llms: An empirical study, 2024

    Junbin Xiao, Nanxin Huang, Hangyu Qin, Dongyang Li, Yi- cong Li, Fengbin Zhu, Zhulin Tao, Jianxing Yu, Liang Lin, Tat-Seng Chua, and Angela Yao. Videoqa in the era of llms: An empirical study, 2024. 2, 3

    2024

  42. [50]

    Can i trust your answer? visually grounded video question answering

    Junbin Xiao, Angela Yao, Yicong Li, and Tat-Seng Chua. Can i trust your answer? visually grounded video question answering. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13204– 13214, 2024. 1, 3, 5, 6, 7

    2024

  43. [51]

    Video question answer- ing via gradually refined attention over appearance and mo- tion

    Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answer- ing via gradually refined attention over appearance and mo- tion. InACM Multimedia, 2017. 5, 6

    2017

  44. [52]

    Video question answer- ing via gradually refined attention over appearance and mo- tion

    Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answer- ing via gradually refined attention over appearance and mo- tion. InProceedings of the 25th ACM international confer- ence on Multimedia, pages 1645–1653, 2017. 5

    2017

  45. [53]

    Zero-shot video question answering via frozen bidirectional language models

    Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Zero-shot video question answering via frozen bidirectional language models. InNeurIPS, 2022. 2, 4, 7

    2022

  46. [54]

    Tubedetr: Spatio-temporal video ground- ing with transformers

    Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Tubedetr: Spatio-temporal video ground- ing with transformers. InCVPR, 2022. 3

    2022

  47. [55]

    Self-chained image-language model for video localization and question answering

    Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. Self-chained image-language model for video localization and question answering. InNeurIPS, 2023. 1, 2, 3, 4, 6, 7

    2023

  48. [56]

    Activitynet-qa: A dataset for understanding complex web videos via question answering

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. InAAAI, pages 9127–9134, 2019. 5, 6

    2019

  49. [57]

    A sim- ple llm framework for long-range video question-answering,

    Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal, and Gedas Bertasius. A sim- ple llm framework for long-range video question-answering,

  50. [58]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding.arXiv preprint arXiv:2306.02858, 2023

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding.arXiv preprint arXiv:2306.02858, 2023. 2, 7

    2023 arXiv

  51. [59]

    Llama-adapter: Efficient fine-tuning of language models with zero-init attention.arXiv preprint arXiv:2303.16199, 2023

    Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Ao- jun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention.arXiv preprint arXiv:2303.16199, 2023. 7

    2023 arXiv

  52. [60]

    Video question answering: Datasets, algorithms and challenges, 2022

    Yaoyao Zhong, Junbin Xiao, Wei Ji, Yicong Li, Weihong Deng, and Tat-Seng Chua. Video question answering: Datasets, algorithms and challenges, 2022. 2 TOGA: Temporally Grounded Open-Ended Video QA with Weak Supervision Supplementary Material In the supplemental material, we pro...

    2022

  53. [61]

    Training Details The additional settings we use, including hyperparameters and implementation details, are shown in Tab

    Implementation Details 1.1. Training Details The additional settings we use, including hyperparameters and implementation details, are shown in Tab. 1. We train on 8x A100 GPUs, and take 7 hours each for the instruction tuning stages and 54 hours for the pretraining stage. 1.2...

  54. [62]

    Multi-scale vision-language connector In this section, we conduct ablations on the MS-VLC and analyze the performance under various conditions

    Additional experimental analysis 2.1. Multi-scale vision-language connector In this section, we conduct ablations on the MS-VLC and analyze the performance under various conditions. Specif- ically, we evaluate the spare-only and dense-only models on different setups and report...

  55. [63]

    We achieve a METEOR score of 0.498 on the ActivityNet-QA dataset

    for a more robust evaluation of our model. We achieve a METEOR score of 0.498 on the ActivityNet-QA dataset. 2.10. Evaluation on additional grounding dataset We also evaluate our approach on the ReXTime [3] dataset in a zero-shot setting, without finetuning on these videos. We...

  56. [64]

    1 and the ActivityNet dataset in Fig

    Qualitative examples We include additional qualitative examples for the NExT- GQA dataset in Fig. 1 and the ActivityNet dataset in Fig. 2. We include three questions for each video in Fig. 1. We can observe that the model can detect events that happen in a relatively short tem...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.