Pith. sign in

REVIEW 3 major objections 5 minor 5 cited by

Current Video-LLMs cannot keep temporal judgments consistent across synchronized ego and exo views of the same event; naive multi-view training underperforms single-view training, and a reinforcement-learning method that rewards viewpoint-s

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 07:17 UTC pith:UZ62NATT

load-bearing objection A useful benchmark and a credible negative result about cross-view consistency in Video-LLMs, with one alignment assumption that deserves scrutiny before the numbers are taken at face value. the 3 major comments →

arxiv 2510.26113 v2 pith:UZ62NATT submitted 2025-10-30 cs.CV cs.AI

EgoExo-Con: Exploring View-Invariant Video Temporal Understanding

Minjoon Jung , Junbin Xiao , Junghyun Kim , Byoung-Tak Zhang , Angela Yao This is my paper
classification cs.CV cs.AI
keywords EgoExo-Conview-invariant temporal understandingtemporal groundingtemporal verificationvideo large language modelscross-view consistencyreinforcement learningGRPO
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether Video-LLMs reason about time in a view-invariant way: if two cameras capture the same event, the temporal structure is identical, but appearance differs. To test this, it builds EgoExo-Con, a benchmark of 491 synchronized ego-exo video pairs and 3,178 human-refined temporal queries, scored both per view and by whether a model answers consistently across the two views. On it, models' consistency scores hover near half their single-view accuracy, and naively fine-tuning on both views often hurts rather than helps. The paper then shows that a reinforcement-learning scheme, View-GRPO, which rewards accurate, viewpoint-specific reasoning chains that converge on the same conclusion, raises consistency by several points. A sympathetic reader would take this as evidence that current models lean on view-specific appearance biases rather than a shared temporal understanding.

Core claim

The paper's central claim is that existing Video-LLMs do not possess view-invariant temporal understanding: models answer temporal verification and temporal grounding questions with roughly similar per-view accuracy on ego and exo videos, yet the consistency scores—requiring a correct answer in both views—are barely half the single-view numbers, and in some cases near random. It further shows that simply training on synchronized multi-view data does not fix this; models tuned on both views often underperform those tuned on one view, which the paper attributes to conflicting priors across viewpoints. The proposed remedy, View-GRPO, combines group-relative policy optimization with a reasoning

What carries the argument

The load-bearing machinery is the consistency metric itself: V-EgoExo and G-EgoExo require the model to be correct on both an egocentric and an exocentric clip for the same event and the same temporal interval, turning 'consistent' from a subjective notion into an explicit conjunction of two correct outputs. On top of that, View-GRPO adapts group-relative policy optimization by adding three reward terms—format, accuracy (binary for verification, temporal IoU for grounding), and reasoning similarity judged by an LLM—so that the policy is pushed to produce viewpoint-specific reasoning chains that nevertheless lead to the same temporal conclusion.

Load-bearing premise

The benchmark assumes the synchronized ego-exo pairs are aligned well enough that one query interval is genuinely correct in both views; if synchronization or annotation noise shifts event onsets/offsets between views, the consistency metric undercounts models regardless of their true ability.

What would settle it

On a random subset, have independent annotators mark event start and end in each view separately, then compute cross-view human consistency under the same 'correct in both, same interval' rule. If human consistency is far below the reported 89.4% for verification and 67.3% for grounding, synchronization noise is inflating the difficulty; if humans stay near the reported numbers, the model failures are real.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Consistency scores, not single-view accuracy, expose a real gap: open-source Video-LLMs hover at roughly half their single-view performance on both temporal verification and grounding.
  • Naive multi-view supervised fine-tuning is not a reliable fix; it often underperforms single-view training, so more synchronized data alone does not produce view-invariant temporal reasoning.
  • View-GRPO improves cross-view consistency beyond SFT and plain GRPO, with the largest gains on the consistency metrics themselves.
  • Unfreezing the visual encoder during multi-view tuning does not help and can hurt, suggesting the bottleneck is not visual adaptation alone.
  • Even strong closed-source models trail human cross-view consistency by a large margin, so the benchmark maps unresolved headroom for future models.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the reasoning reward is the active ingredient, a lighter proxy—such as requiring the model to output the same predicted span before and after light temporal perturbation—might test view-invariance without needing expensive LLM judges.
  • The finding that single-view accuracy stays stable while consistency collapses suggests the field should report consistency alongside accuracy for any paired-view benchmark; otherwise apparent robustness may just be view-specific bias.
  • The benchmark's design could be extended to more than two views or to synthetic viewpoint interpolation, which would separate pure viewpoint invariance from dataset-specific ego/exo cues.
  • Since the weakest point is synchronization, a natural next step is to measure annotator agreement on start/end times within each view; if that agreement is low, consistency metrics should be softened to tolerances.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces EgoExo-Con, a benchmark of 491 synchronized egocentric/exocentric video pairs and 3,178 temporal event queries derived from CharadesEgo, EgoExo-4D, and LEMMA. It defines temporal verification and temporal grounding tasks, plus consistency metrics that require a model to answer correctly on both views of the same event. The paper reports that current Video-LLMs achieve cross-view consistency scores that are roughly half of their single-view scores, that naive multi-view supervised fine-tuning often underperforms single-view fine-tuning, and that a proposed method, View-GRPO (a GRPO variant with an additional LLM-judge-based reasoning reward and GPT-5-generated reasoning chains), improves both single-view and cross-view performance on Qwen2.5-VL 3B/7B. The benchmark, analysis, and method are presented as a package.

Significance. If the benchmark's assumptions hold, EgoExo-Con fills a real gap: it measures whether Video-LLMs rely on view-specific appearance cues rather than view-invariant temporal structure. The finding that consistency is much lower than single-view accuracy is falsifiable and consistent across a wide panel of open- and closed-source models. The human validation of samples, the inclusion of a random baseline, and the release of resources are concrete strengths. View-GRPO is also a plausible method contribution, though its gains rest on a single training run and an uncalibrated LLM judge. The core results are interesting and likely useful to the community, but the validity of the central consistency claim depends on a cross-view temporal-alignment assumption that is not yet quantitatively validated.

major comments (3)
  1. [§3.2 and §4.1] The consistency metric V-EgoExo/G-EgoExo requires one and the same ground-truth (start, end) interval to be correct in both the ego and exo video. The dataset construction (�3.2) reports human validation that queries are visible and accurately grounded in both views, but it does not report inter-annotator agreement on timestamps, nor an alignment-error analysis between the two views. Viewpoint-dependent occlusion can make the perceived onset/offset of the same physical event differ between cameras, so a model that correctly localizes the event in each view individually could be scored as inconsistent. The human grounding consistency of 67.3% vs. single-view grounding of about 72-73% (Table 1) suggests that some interval ambiguity exists even for humans. Without a per-view ground-truth interval or a quantitative alignment-error measurement, the magnitude of the reported consistency defici
  2. [Table 3] All View-GRPO gains (e.g., +11.7 points on V-ExoEgo and +11.4 on G-ExoEgo for Qwen2.5-VL-7B) are reported as single-run point estimates with no error bars, multiple seeds, or significance tests. Because GRPO training is stochastic and the training set is relatively small (3.3k videos), the claimed superiority of View-GRPO over SFT and basic GRPO could be within run-to-run noise. The paper should report at least 2-3 seeds with mean/std for the main comparisons, or otherwise provide a stability analysis of the RL training.
  3. [§4.2, Table 1] Closed-source models (GPT-5, Gemini-2.5 Flash) and human performance are evaluated only on a randomly sampled ~30% subset, while open-source models are evaluated on the full EgoExo-Con. The table compares these numbers directly. If the 30% subset is not exactly matched in difficulty, the reported 'gap' between closed-source models and open-source models, and between models and humans, is confounded. Please report open-source model performance on the same 30% subset, or evaluate all models on the full set, for fair comparison.
minor comments (5)
  1. [§4.1] The consistency metric definition states "i.e., IoU<0.5" but the surrounding text and the grounding definition imply the correct condition should be IoU >= 0.5. Please fix this typo.
  2. [Table 1] Some cells are poorly formatted, e.g., "28.227.3" and "42.127.8" should be separated into two numbers. Similar issues appear in Table 5.
  3. [§5.2 and Appendix B.3] The View30K size is inconsistent: §5.2 says 30k reasoning instances, while Appendix B.3 says 61k reasoning per task. Please clarify whether the numbers refer to different filtering stages or tasks.
  4. [References] Duplicate references for Feng et al. (2025a/2025b) and Grauman et al. (2024a/2024b) should be consolidated.
  5. [§4.2, point (5)] The claim that "temporal reasoning outweighs increasing frames" is based on comparing different model architectures, training data, and frame counts, not a controlled ablation. Please soften the wording or add a controlled experiment.

Circularity Check

1 steps flagged

Benchmark is an external measurement; only minor non-load-bearing self-citation in prompt sourcing.

specific steps
  1. other [Appendix B.1, Prompt Templates]
    "following the previous work (Jung et al., 2025b), we also include templates like ... we borrow the prompts from the previous work (Jung et al., 2025b) for the general-purpose models and closed-sourced models"

    The authors reuse prompt templates from their own earlier paper (TimeChat-VT, Jung et al. 2025b). This is a self-citation, but it is not load-bearing for the central claims: the benchmark results and View-GRPO improvements are empirical measurements on a newly constructed test set, and the reused prompts do not define the consistency metrics or the RL objective. At most it introduces a small prompt-design choice inherited from prior work, which does not make the main derivation circular.

full rationale

The paper's central derivation chain is empirical, not analytic. EgoExo-Con is a newly constructed benchmark with human-refined queries and synchronized ego-exo pairs. The headline findings—that Video-LLMs are less consistent across views than within a single view, and that naive multi-view SFT underperforms single-view training—are read directly from Tables 1 and 2. The consistency metrics (V-EgoExo, G-EgoExo) are defined as requiring correct answers in both views; they are measurements, not quantities fitted from model outputs. View-GRPO is trained on View30K with rewards for format, accuracy, and reasoning-similarity, and then evaluated on EgoExo-Con; the reported improvements are empirical outcomes, not consequences of defining the evaluation metric in terms of the training objective. The only self-citation in a position to influence results is the reuse of prompt templates from Jung et al. 2025b (TimeChat-VT) for baseline evaluation. That is a minor methodological inheritance, not a reduction of the paper's claims to its inputs. No equation is shown to equal itself by construction, no fitted parameter is renamed as a prediction, and no uniqueness or ansatz is imported from the authors' prior work. The benchmark is externally grounded in CharadesEgo, LEMMA, and Ego-Exo4D, and the human-consistency reference provides an independent upper bound. Overall, the derivation is self-contained; the small self-citation does not affect the validity of the main findings.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The central claims rest on the benchmark's alignment validity and on the reward design of View-GRPO. The LLM-generated data and single-run evaluations add unmeasured variance. No new physical or mathematical entities are introduced; the main assumptions are domain-level about synchronized annotations and reward reliability.

free parameters (3)
  • tIoU filter threshold for View30K reasoning data = 0.7
    Section 5.2: 'we exclude cases where ... the predicted moment has a temporal IoU (tIoU) below 0.7 with the ground-truth.' This threshold determines which 3.3k/30k reasoning instances remain; changing it changes the training distribution.
  • reward weights for r_acc, r_form, r_sim = 1, 1, 1 (equal)
    Section 5.2: total reward r = r_acc + r_form + r_sim. Equal weighting is chosen by hand; no sensitivity analysis is reported.
  • frame sampling rate for View-GRPO training = 2 FPS
    Section 5.3: 'all experiments set the same frame sampling rate (i.e., 2 FPS)'. The choice affects how many frames the model sees and the difficulty of grounding.
axioms (5)
  • domain assumption Synchronized ego-exo pairs depict the same event with a shared temporal interval valid in both views
    Core to benchmark validity; introduced in dataset construction §3.2 and metric definition §4.1.
  • domain assumption Current Video-LLM temporal reasoning, not low-level appearance differences or annotation noise, is the main driver of cross-view inconsistency
    The paper's analysis in §4.2 interprets consistency gaps as view-specific bias rather than annotation noise; no control experiments isolate these factors.
  • domain assumption GPT-4o/GPT-5 generated queries and reasoning chains are faithful enough after human filtering
    Queries and reasoning come from LLMs (§3.2, §5.2); human validation is partial and not quantified (no inter-annotator agreement).
  • ad hoc to paper LLM-judge similarity score r_sim is a valid training reward for temporal reasoning
    Section 5.2 and Fig. 6; the paper itself notes judge scale affects calibration and consistency (-3% G-EgoExo), so the reward is a fragile ad hoc component.
  • domain assumption Long videos can be segmented into clips surrounding ground-truth moments without changing the temporal understanding question
    Section 3.1 segments >5 min videos into ≥2 min clips; this keeps tasks feasible but may remove long-range temporal context that is part of the original event.

pith-pipeline@v1.3.0-alltime-deepseek · 17863 in / 13012 out tokens · 131773 ms · 2026-08-04T07:17:58.322559+00:00 · methodology

0 comments
read the original abstract

Do Video-LLMs have consistent temporal understanding when videos capture the same event from different viewpoints? To study this question, we introduce EgoExo-Con(sistency), a benchmark of synchronized egocentric and exocentric video pairs with human-refined queries that ensure all concepts are visible in both viewpoints. EgoExo-Con emphasizes two temporal understanding tasks: Temporal Verification and Temporal Grounding. It evaluates not only correctness but consistency across viewpoints. Our analysis reveals two critical limitations of existing Video-LLMs: (1) models often fail to maintain consistency, with results far worse than their single-view performances. (2) When naively finetuned with synchronized videos of both viewpoints, the models show improved consistency but often underperform those trained on a single view. For improvements, we propose View-GRPO, a novel reinforcement learning framework that effectively strengthens view-specific temporal reasoning while encouraging consistent comprehension across viewpoints. Our method demonstrates its superior temporal understanding capabilities, especially for improving cross-view consistency. All resources have been made available at https://minjoong507.github.io/projects/EgoExo-Con/

Figures

Figures reproduced from arXiv: 2510.26113 by Angela Yao, Byoung-Tak Zhang, Junbin Xiao, Junghyun Kim, Minjoon Jung.

Figure 1
Figure 1. Figure 1: Examples of queries and corresponding video moments from existing datasets. (a) and (b) highlight fundamental limitations, with the egocentric view (top) in (a) being insufficient due to differing focuses, and the exocentric view (bottom) in (b) being ambiguous due to occlusion and distance. Although the query in (c) is identifiable from both viewpoints, we enrich it with details. Ego-Exo Learning. Researc… view at source ↗
Figure 2
Figure 2. Figure 2: Statistics of EgoExo-Con. The numbers below (a) show the video and moment counts per subset, and those in (b) and (c) show their average lengths, respectively. The statistics suggest the high diversity of EgoExo-Con in data sources, video and moment lengths [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Examples of test data and the corresponding model responses. We create refined and misaligned queries from each original query, use them for temporal verification (V) and grounding (G), and assess cross-view answer consistency. generates a misaligned query containing irrelevant content, which serves as a negative sample for temporal verification, thus balancing answers for “yes” and “No” in the verificatio… view at source ↗
Figure 4
Figure 4. Figure 4: Heatmaps of the performance gap. All values are reported in percentage points. Red and blue indicate higher performances on ego and exo perspectives, respectively. i.e., a blue cell indicates that the corresponding model performs better on exocentric videos than on egocentric ones [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Overview of our approach. (a) In supervised fine-tuning, the model is trained to directly predict the same query answers (e.g., video moments) for synchronized video pairs. (b) View-GRPO trains a model to provide viewpoint-specific reasoning chains, which are generated by GPT-5 (top). Without explicit alignment, conflicting priors across tasks or domains undermine temporal signals and consistency rather th… view at source ↗
Figure 7
Figure 7. Figure 7: Visualization of View-GRPO. The model provides step-by-step temporal reasoning with accurate grounding prediction, achieving high reasoning reward scores from LLM-judges. basic GRPO. The most significant improvements are often on cross-view consistency (V-ExoEgo and G-ExoEgo), although it also benefits individual views. We conjecture that the reasoning reward plays a central role, as it delivers noticeably… view at source ↗
Figure 6
Figure 6. Figure 6: Reward across different LLM judges. Qwen2.5-0.5B raises calibration concerns due to its overly high reasoning rewards from early steps. While LLMs commonly serve as judges (Zheng et al., 2023; Xie et al., 2023), their role in op￾timizing models within View-GRPO remains unclear despite observed improvement. In fact, they often introduce potential bias and uncer￾tainty in video evaluation (Cores et al., 2024… view at source ↗
Figure 9
Figure 9. Figure 9: Examples of test clips and model responses for temporal verification.. Each row is from CharadesEgo and shows an exocentric–egocentric pair, a Refined Query (positive) and a Misaligned Query (negative), and per-view answers from each model. We mark consistency (✓) and inconsistency (✗) cases. C.2 PERFORMANCE ON EGOEXO-CON ACROSS SUBSETS [PITH_FULL_IMAGE:figures/full_fig_p017_9.png] view at source ↗
Figure 8
Figure 8. Figure 8: A video sample in EgoExo￾Fitnesses. The given query is hard to identify in the egocentric video (top). As we discussed in the dataset section, some videos are naturally unsuitable to include in our benchmark. In [PITH_FULL_IMAGE:figures/full_fig_p017_8.png] view at source ↗
Figure 10
Figure 10. Figure 10: Examples of test clips and model responses for temporal grounding. The videos in the top two rows are from LEMMA, and the bottom row shows EgoExo4D videos. Each row presents an exocentric–egocentric pair with a Refined Query; models output per-view time spans (with IoU to ground truth when available), and we indicate cross-view consistency (✓) vs. inconsistency (✗) [PITH_FULL_IMAGE:figures/full_fig_p018_… view at source ↗
Figure 11
Figure 11. Figure 11: Prompt for refined and misaligned queries. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Prompt for the reasoning reward function in View-GRPO [PITH_FULL_IMAGE:figures/full_fig_p019_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Prompt for generating temporal reasoning for temporal verification. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Prompt for generating temporal reasoning for temporal grounding. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_14.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. EgoExoMem: Cross-View Memory Reasoning over Synchronized Egocentric and Exocentric Videos

    cs.CV 2026-05 unverdicted novelty 7.0

    EgoExoMem is the first benchmark for cross-view memory reasoning on synchronized egocentric-exocentric videos, where E2-Select raises MLLM accuracy from 55.3% to 58.2% over baselines.

  2. Seeing Together: Multi-Robot Cooperative Egocentric Spatial Reasoning with Multimodal Large Language Models

    cs.CV 2026-05 conditional novelty 7.0

    SP-CoR is a multimodal LLM framework using dynamics-aware sampling, spectral-physics view fusion, and prompt distillation that outperforms baselines on the new CoopSR benchmark and EgoTeam dataset for multi-robot coop...

  3. EvoGround: Self-Evolving Video Agents for Video Temporal Grounding

    cs.CV 2026-05 unverdicted novelty 7.0

    A proposer-solver agent pair achieves supervised-level video temporal grounding and fine-grained captioning from 2.5K unlabeled videos via self-reinforcing evolution.

  4. EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoning

    cs.CV 2025-11 conditional novelty 7.0

    EgoVITA, a GRPO-based plan-then-verify framework with dense visual-grounding rewards, improves egocentric video reasoning by up to +7.7 points and keeps exocentric video performance intact.

  5. EgoExoMem: Cross-View Memory Reasoning over Synchronized Egocentric and Exocentric Videos

    cs.CV 2026-05 unverdicted novelty 6.0

    EgoExoMem introduces the first cross-view ego–exo video memory benchmark (2.6K MCQs, eight QA types) and E²-Select, a training-free dual-view frame selector scoring 58.2%.

Reference graph

Works this paper leans on

64 extracted references · 28 linked inside Pith · cited by 4 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  3. [3]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    Egothink: Evaluating first-person perspective thinking capability of vision-language models

    Sijie Cheng, Zhicheng Guo, Jingwen Wu, Kechen Fang, Peng Li, Huaping Liu, and Yang Liu. Egothink: Evaluating first-person perspective thinking capability of vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 14291--14302, 2024 a

    2024

  5. [5]

    Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476, 2024 b

    Pith/arXiv arXiv 2024

  6. [6]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025

    Pith/arXiv arXiv 2025

  7. [7]

    Lost in time: A new temporal benchmark for videollms

    Daniel Cores, Michael Dorkenwald, Manuel Mucientes, Cees GM Snoek, and Yuki M Asano. Lost in time: A new temporal benchmark for videollms. arXiv preprint arXiv:2410.07752, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Grounded question-answering in long egocentric videos

    Shangzhe Di and Weidi Xie. Grounded question-answering in long egocentric videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 12934--12943, 2024

    2024

  9. [10]

    Video-r1: Reinforcing video reasoning in mllms

    Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Junfei Wu, Xiaoying Zhang, Benyou Wang, and Xiangyu Yue. Video-r1: Reinforcing video reasoning in mllms. arXiv preprint arXiv:2503.21776, 2025 b

    Pith/arXiv arXiv 2025

  10. [11]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024

    Pith/arXiv arXiv 2024

  11. [12]

    Tall: Temporal activity localization via language query

    Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. In Proceedings of the IEEE international conference on computer vision, pp.\ 5267--5275, 2017

    2017

  12. [13]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  13. [14]

    Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives

    Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 193...

    2024

  14. [15]

    Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives

    Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 193...

    2024

  15. [16]

    Vtg-llm: Integrating timestamp knowledge into video llms for enhanced video temporal grounding

    Yongxin Guo, Jingyu Liu, Mingda Li, Xiaoying Tang, Xi Chen, and Bo Zhao. Vtg-llm: Integrating timestamp knowledge into video llms for enhanced video temporal grounding. arXiv preprint arXiv:2405.13382, 2024

    Pith/arXiv arXiv 2024

  16. [17]

    Egoexobench: A benchmark for first-and third-person view video understanding in mllms

    Yuping He, Yifei Huang, Guo Chen, Baoqi Pei, Jilan Xu, Tong Lu, and Jiangmiao Pang. Egoexobench: A benchmark for first-and third-person view video understanding in mllms. arXiv preprint arXiv:2507.18342, 2025

    Pith/arXiv arXiv 2025

  17. [18]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 2022

    2022

  18. [19]

    Vtimellm: Empower llm to grasp video moments

    Bin Huang, Xin Wang, Hong Chen, Zihan Song, and Wenwu Zhu. Vtimellm: Empower llm to grasp video moments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 14271--14280, 2024

    2024

  19. [20]

    Lemma: A multi-view dataset for le arning m ulti-agent m ulti-task a ctivities

    Baoxiong Jia, Yixin Chen, Siyuan Huang, Yixin Zhu, and Song-Chun Zhu. Lemma: A multi-view dataset for le arning m ulti-agent m ulti-task a ctivities. In European Conference on Computer Vision, pp.\ 767--786. Springer, 2020

    2020

  20. [21]

    Background-aware moment detection for video moment retrieval

    Minjoon Jung, Youwon Jang, Seongho Choi, Joochan Kim, Jin-Hwa Kim, and Byoung-Tak Zhang. Background-aware moment detection for video moment retrieval. In Proceedings of the Winter Conference on Applications of Computer Vision, pp.\ 8575--8585, February 2025 a

    2025

  21. [22]

    On the consistency of video large language models in temporal comprehension

    Minjoon Jung, Junbin Xiao, Byoung-Tak Zhang, and Angela Yao. On the consistency of video large language models in temporal comprehension. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp.\ 13713--13722, 2025 b

    2025

  22. [23]

    Mvbench: A comprehensive multi-modal video understanding benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 22195--22206, 2024 a

    2024

  23. [24]

    Ego-exo: Transferring visual representations from third-person to first-person videos

    Yanghao Li, Tushar Nagarajan, Bo Xiong, and Kristen Grauman. Ego-exo: Transferring visual representations from third-person to first-person videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 6943--6953, 2021

    2021

  24. [25]

    Egoexo-fitness: Towards egocentric and exocentric full-body action understanding

    Yuan-Ming Li, Wei-Jin Huang, An-Lan Wang, Ling-An Zeng, Jing-Ke Meng, and Wei-Shi Zheng. Egoexo-fitness: Towards egocentric and exocentric full-body action understanding. In European Conference on Computer Vision, pp.\ 363--382. Springer, 2024 b

    2024

  25. [26]

    Universal video temporal grounding with generative multi-modal large language models

    Zeqian Li, Shangzhe Di, Zhonghua Zhai, Weilin Huang, Yanfeng Wang, and Weidi Xie. Universal video temporal grounding with generative multi-modal large language models. arXiv preprint arXiv:2506.18883, 2025

    arXiv 2025

  26. [27]

    Improved visual-spatial reasoning via r1-zero-like training

    Zhenyi Liao, Qingsong Xie, Yanhao Zhang, Zijian Kong, Haonan Lu, Zhenyu Yang, and Zhijie Deng. Improved visual-spatial reasoning via r1-zero-like training. arXiv preprint arXiv:2504.00883, 2025

    Pith/arXiv arXiv 2025

  27. [28]

    Is your video language model a reliable judge? arXiv preprint arXiv:2503.05977, 2025

    Ming Liu and Wensheng Zhang. Is your video language model a reliable judge? arXiv preprint arXiv:2503.05977, 2025

    Pith/arXiv arXiv 2025

  28. [29]

    Put myself in your shoes: Lifting the egocentric perspective from exocentric videos

    Mi Luo, Zihui Xue, Alex Dimakis, and Kristen Grauman. Put myself in your shoes: Lifting the egocentric perspective from exocentric videos. In European Conference on Computer Vision, pp.\ 407--425. Springer, 2024

    2024

  29. [30]

    Viewpoint rosetta stone: Unlocking unpaired ego-exo videos for view-invariant representation learning

    Mi Luo, Zihui Xue, Alex Dimakis, and Kristen Grauman. Viewpoint rosetta stone: Unlocking unpaired ego-exo videos for view-invariant representation learning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp.\ 15802--15812, 2025

    2025

  30. [31]

    Egoschema: A diagnostic benchmark for very long-form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems, 36: 0 46212--46244, 2023

    2023

  31. [32]

    Chrono: A simple blueprint for representing time in mllms

    Boris Meinardus, Hector Rodriguez, Anil Batra, Anna Rohrbach, and Marcus Rohrbach. Chrono: A simple blueprint for representing time in mllms. arXiv preprint arXiv:2406.18113, 2024

    arXiv 2024

  32. [33]

    Introducing gpt-5

    OpenAI. Introducing gpt-5. 2025

    2025

  33. [34]

    Egoexor: An ego-exo-centric operating room dataset for surgical activity understanding

    Ege \"O zsoy, Arda Mamur, Felix Tristram, Chantal Pellegrini, Magdalena Wysocki, Benjamin Busam, and Nassir Navab. Egoexor: An ego-exo-centric operating room dataset for surgical activity understanding. arXiv preprint arXiv:2505.24287, 2025

    Pith/arXiv arXiv 2025

  34. [35]

    Momentor: Advancing video large language model with fine-grained temporal reasoning

    Long Qian, Juncheng Li, Yu Wu, Yaobo Ye, Hao Fei, Tat-Seng Chua, Yueting Zhuang, and Siliang Tang. Momentor: Advancing video large language model with fine-grained temporal reasoning. arXiv preprint arXiv:2402.11435, 2024

    Pith/arXiv arXiv 2024

  35. [36]

    A Yang Qwen, Baosong Yang, B Zhang, B Hui, B Zheng, B Yu, Chengpeng Li, D Liu, F Huang, H Wei, et al. Qwen2. 5 technical report. arXiv preprint, 2024

    2024

  36. [37]

    Timechat: A time-sensitive multimodal large language model for long video understanding

    Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 14313--14323, 2024

    2024

  37. [38]

    Assembly101: A large-scale multi-view video dataset for understanding procedural activities

    Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun He, Dipika Singhania, Robert Wang, and Angela Yao. Assembly101: A large-scale multi-view video dataset for understanding procedural activities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 21096--21106, 2022

    2022

  38. [39]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  39. [40]

    Longvu: Spatiotemporal adaptive compression for long video-language understanding

    Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, et al. Longvu: Spatiotemporal adaptive compression for long video-language understanding. arXiv preprint arXiv:2410.17434, 2024

    Pith/arXiv arXiv 2024

  40. [41]

    Actor and observer: Joint modeling of first and third-person videos

    Gunnar A Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteek Alahari. Actor and observer: Joint modeling of first and third-person videos. In proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 7396--7404, 2018

    2018

  41. [42]

    Learning from semantic alignment between unpaired multiviews for egocentric video recognition

    Qitong Wang, Long Zhao, Liangzhe Yuan, Ting Liu, and Xi Peng. Learning from semantic alignment between unpaired multiviews for egocentric video recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 3307--3317, 2023

    2023

  42. [43]

    Internvl3

    Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025 a

    Pith/arXiv arXiv 2025

  43. [44]

    Videoagent: Long-form video understanding with large language model as agent

    Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. Videoagent: Long-form video understanding with large language model as agent. arXiv preprint arXiv:2403.10517, 2024 a

    Pith/arXiv arXiv 2024

  44. [45]

    Time-r1: Post-training large vision language model for temporal video grounding

    Ye Wang, Ziheng Wang, Boshen Xu, Yang Du, Kejun Lin, Zihan Xiao, Zihao Yue, Jianzhong Ju, Liang Zhang, Dingyi Yang, et al. Time-r1: Post-training large vision language model for temporal video grounding. arXiv preprint arXiv:2503.13377, 2025 b

    Pith/arXiv arXiv 2025

  45. [46]

    Internvideo2

    Yi Wang, Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xiangyu Zeng, Chenting Wang, Changlian Ma, Haian Huang, Jianfei Gao, et al. Internvideo2. 5: Empowering video mllms with long and rich context modeling. arXiv preprint arXiv:2501.12386, 2025 c

    Pith/arXiv arXiv 2025

  46. [47]

    Hawkeye: Training video-text llms for grounding text in videos

    Yueqian Wang, Xiaojun Meng, Jianxin Liang, Yuxuan Wang, Qun Liu, and Dongyan Zhao. Hawkeye: Training video-text llms for grounding text in videos. arXiv preprint arXiv:2403.10228, 2024 b

    Pith/arXiv arXiv 2024

  47. [48]

    Longvideobench: A benchmark for long-context interleaved video-language understanding

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. Advances in Neural Information Processing Systems, 37: 0 28828--28857, 2024

    2024

  48. [49]

    Next-qa: Next phase of question-answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 9777--9786, 2021

    2021

  49. [50]

    Can i trust your answer? visually grounded video question answering

    Junbin Xiao, Angela Yao, Yicong Li, and Tat-Seng Chua. Can i trust your answer? visually grounded video question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 13204--13214, 2024

    2024

  50. [51]

    Egoblind: Towards egocentric visual assistance for the blind people

    Junbin Xiao, Nanxin Huang, Hao Qiu, Zhulin Tao, Xun Yang, Richang Hong, Meng Wang, and Angela Yao. Egoblind: Towards egocentric visual assistance for the blind people. arXiv preprint arXiv:2503.08221, 2025

    arXiv 2025

  51. [52]

    Text2reward: Reward shaping with language models for reinforcement learning

    Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, and Tao Yu. Text2reward: Reward shaping with language models for reinforcement learning. arXiv preprint arXiv:2309.11489, 2023

    Pith/arXiv arXiv 2023

  52. [53]

    Video question answering via gradually refined attention over appearance and motion

    Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM international conference on Multimedia, pp.\ 1645--1653, 2017

    2017

  53. [54]

    Learning fine-grained view-invariant representations from unpaired ego-exo videos via temporal alignment

    Zihui Sherry Xue and Kristen Grauman. Learning fine-grained view-invariant representations from unpaired ego-exo videos via temporal alignment. Advances in Neural Information Processing Systems, 36: 0 53688--53710, 2023

    2023

  54. [55]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  55. [56]

    Mmego: Towards building egocentric multimodal llms for video qa

    Hanrong Ye, Haotian Zhang, Erik Daxberger, Lin Chen, Zongyu Lin, Yanghao Li, Bowen Zhang, Haoxuan You, Dan Xu, Zhe Gan, et al. Mmego: Towards building egocentric multimodal llms for video qa. In The Thirteenth International Conference on Learning Representations, 2015

    2015

  56. [57]

    Activitynet-qa: A dataset for understanding complex web videos via question answering

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, 2019

    2019

  57. [58]

    Timesuite: Improving mllms for long video understanding via grounded tuning

    Xiangyu Zeng, Kunchang Li, Chenting Wang, Xinhao Li, Tianxiang Jiang, Ziang Yan, Songze Li, Yansong Shi, Zhengrong Yue, Yi Wang, et al. Timesuite: Improving mllms for long video understanding via grounded tuning. arXiv preprint arXiv:2410.19702, 2024

    Pith/arXiv arXiv 2024

  58. [59]

    Videollama 3: Frontier multimodal foundation models for image and video understanding

    Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501.13106, 2025 a

    Pith/arXiv arXiv 2025

  59. [60]

    Exo2ego: Exocentric knowledge guided mllm for egocentric video understanding

    Haoyu Zhang, Qiaohui Chu, Meng Liu, Yunxiao Wang, Bin Wen, Fan Yang, Tingting Gao, Di Zhang, Yaowei Wang, and Liqiang Nie. Exo2ego: Exocentric knowledge guided mllm for egocentric video understanding. arXiv preprint arXiv:2503.09143, 2025 b

    arXiv 2025

  60. [61]

    Long context transfer from language to vision

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852, 2024 a

    Pith/arXiv arXiv 2024

  61. [62]

    Tinyllava-video-r1: Towards smaller lmms for video reasoning

    Xingjian Zhang, Siwei Wen, Wenjun Wu, and Lei Huang. Tinyllava-video-r1: Towards smaller lmms for video reasoning. arXiv preprint arXiv:2504.09641, 2025 c

    Pith/arXiv arXiv 2025

  62. [63]

    Video instruction tuning with synthetic data

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024 b

    Pith/arXiv arXiv 2024

  63. [64]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36: 0 46595--46623, 2023

    2023

  64. [65]

    Mlvu: Benchmarking multi-task long video understanding

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Zhengyang Liang, Shitao Xiao, Minghao Qin, Xi Yang, Yongping Xiong, Bo Zhang, et al. Mlvu: Benchmarking multi-task long video understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp.\ 13691--13701, 2025

    2025