Pith. sign in

REVIEW 5 major objections 5 minor 5 cited by

This paper claims that egocentric video reasoning is improved by training a multimodal language model to first plan anticipated actions from the first-person view, then verify that plan from the third-person view, using reinforcement learni

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 20:47 UTC pith:TWGNFPZJ

load-bearing objection A plausible but under-specified plan-then-verify RL recipe for egocentric video; the dense rewards are not cleanly isolated and the Anticipation Head training is unclear. the 5 major comments →

arxiv 2511.18242 v3 pith:TWGNFPZJ submitted 2025-11-23 cs.CV

EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoning

Yogesh Kulkarni , Pooyan Fazli This is my paper
classification cs.CV
keywords egocentric video reasoningmultimodal large language modelsGRPOplan-then-verifyanticipatory cross-modal groundingego-exo generalizationvideo question answeringreinforcement learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

EgoVITA claims that the main reason multimodal LLMs fail at egocentric video is not a lack of vision but a lack of structured, outcome-driven reasoning. It trains a model to first generate an egocentric plan—a causal sequence of anticipated actions from the first-person viewpoint—and then an exocentric verification that checks that plan against the same video from a third-person perspective, without needing paired ego-exo footage. The training uses GRPO with two dense rewards: one (ACMG) rewards plan steps whose projected embedding matches an actual future frame, and one (confidence) rewards coherent verification. On five egocentric benchmarks the framework reports consistent gains over three strong base models—up to +7.7 on EgoBlind—while improving rather than degrading exocentric benchmarks, suggesting the bottleneck is reasoning structure and reward density, not data scale.

Core claim

On its own terms, EgoVITA's central claim is that the plan-then-verify decomposition, trained with GRPO and dense rewards, is both necessary and sufficient for large gains in egocentric reasoning. The ACMG reward is the key new mechanism: each clause of the generated plan is projected from the language model's hidden state into the visual embedding space by a small MLP, and its cosine similarity to the next 16 vision-encoder frames is max-pooled; the model is rewarded when a plan step matches a visual state that actually occurs later in the video. The confidence reward, a teacher-guided then self-ranked comparison of log-probabilities, stabilizes the verification stage. Ablations show that r

What carries the argument

The load-bearing machinery is the composite reward R = w_f R_format + w_a R_answer + w_g R_ACMG + w_c R_confidence under GRPO. The distinctive component is the Anticipatory Cross-Modal Grounding (ACMG) reward: for each clause in <ego plan>, the LLM's final hidden state is projected by a two-layer MLP (Anticipation Head) into the visual embedding space, and the reward is the mean over clauses of the maximum cosine similarity to the next N=16 frames encoded by the vision encoder. This makes the policy optimize for plans whose steps can be matched to actual future observations. The Confidence reward supplies dense signal to the <exo verify> block, initially by matching a teacher generation and

Load-bearing premise

The claim stands on the assumption that a small trainable projection from language to visual features yields a similarity score that genuinely measures whether a planned action will appear in the upcoming frames; if that projection can match any future frame without true semantic or temporal correspondence, the dense reward can be gamed and the reported grounding effect would be an artifact.

What would settle it

Train the identical pipeline but replace the future-frame set with shuffled or unrelated frames; if the EgoBlind/EgoOrient gains remain, the ACMG reward is not using temporal grounding. Alternatively, freeze the Anticipation Head after Stage I and measure whether the frame it maximizes against coincides with the human-annotated moment of that action; chance-level alignment would show the grounding signal is spurious.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Egocentric and exocentric competence need not trade off: with the verification stage and periodic exocentric regularization, the same model improves on both families of benchmarks.
  • Sparse rewards (format and answer) are not enough: the ablations attribute most of the gain to the dense ACMG and confidence rewards, so RL for video reasoning should be designed with dense, step-level signals.
  • The plan-then-verify decomposition is load-bearing: removing either block measurably hurts one side of the ego/exo balance, so future architectures can keep this two-block structure as a default.
  • A predictor of future visual states is a usable training signal for egocentric reasoning without paired ego-exo data, which lowers the data-collection barrier.
  • The method's gains appear across three base models, so the recipe is not tied to one architecture and can be applied as new base models appear.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • (Editorial inference) The same anticipation reward could transfer to other settings where the crux is predicting the outcome of an action—robotic manipulation, surgical video, or driver assistance—since it does not require paired egocentric-exocentric video.
  • (Editorial inference) The paper's ablations show the teacher warm-up contributes only modestly; a reader might predict that a much larger and more diverse teacher corpus would close that gap and make the method effectively self-supervised.
  • (Editorial inference) A direct test of whether the Anticipation Head is essential would be to compare the ACMG reward against a fixed text-only self-similarity reward; if the visual projection is what drives the gains, removing it should collapse performance on EgoBlind.
  • (Editorial inference) Since the framework trains with around 50k samples, scaling to larger egocentric datasets or streaming video could reveal whether the gains continue or plateau.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces EgoVITA, a two-stage RL framework for egocentric video reasoning. Stage I performs SFT on teacher-generated structured plans and verifications; Stage II applies GRPO with a composite reward that combines sparse format/answer rewards with two dense rewards: an Anticipatory Cross-Modal Grounding (ACMG) reward, which projects plan clauses into visual space via a trainable MLP and compares to future frames, and a confidence reward that first mimics a teacher's verification and then self-ranks the policy's own rollouts. The authors report consistent gains over base models and a GRPO baseline on egocentric and exocentric benchmarks, with extensive ablations on reward components, pooling, warm-up duration, anticipation window, and vision-pipeline adaptation.

Significance. If the claims hold, EgoVITA is a valuable contribution: it shows that decomposing egocentric reasoning into plan-then-verify with dense, visually grounded rewards can improve MLLM performance while preserving exocentric ability, and the benchmark gains are substantial (e.g., +7.7 on EgoBlind over Qwen2.5-VL-7B). The paper includes a broad ablation suite and reports bootstrap confidence intervals in the main table, which is a clear strength. However, the central mechanism — the dense rewards — is not yet nailed down: the ACMG reward depends on an underspecified trainable component, and the confidence reward is not cleanly isolated from the teacher warm-up or from other training components.

major comments (5)
  1. [Sec. 3.3.2, Eq. (6), Fig. 3, Appendix B] The ACMG reward is the core contribution, but the training objective of the Anticipation Head (MLP) is never specified. The caption of Fig. 3 says 'All components are trained jointly under GRPO,' yet Eq. (6) treats the MLP as part of the reward, not as an output head with its own loss. If the MLP is updated only through the GRPO objective, it can be exploited to maximize similarity regardless of semantic correspondence; if it is trained with an auxiliary loss, that loss and its data are missing. Provide a precise optimization target for the MLP and an experiment showing the reward is not gameable (e.g., random-plan or shuffled-frame sanity checks). This is load-bearing because the paper's grounding claim rests on this reward.
  2. [Sec. 3.3.3, Eqs. (8)–(9)] The confidence reward is self-referential. In the self-ranking phase, the chosen/rejected outputs are selected using the composite reward (Eq. 3), which already contains R_confidence; the reward then reinforces the policy's own high/low scores. This is largely redundant with the GRPO advantage computed over the same composite reward. The teacher-guided phase (Eq. 8) is a teacher-forcing log-probability comparison, not a measure of 'logical coherence' or 'confidence.' Please define an independent verification signal or show that self-ranking adds signal beyond the existing advantage normalization. The ablation in Table 8 does not isolate the self-ranking component because it removes the warm-up and keeps self-ranking; 'Confidence only' in Table 3 includes the warm-up.
  3. [Sec. 4.2, Tables 3, 6, 8] The claim that the dense rewards (ACMG and confidence) drive the measured gains is not established. The GRPO (Format+Answer) baseline does not appear to include the exocentric regularization (Eq. 10) or the trainable vision encoder/projector that Table 6 shows are responsible for large gains (up to +5.3 on EgoBlind). Thus the +1.5/+1.9 deltas in Table 3 could come from these other components rather than from the dense rewards themselves. A controlled comparison is needed: identical SFT checkpoint, identical vision-adaptation choices, identical exocentric regularization, and identical rollout budget, with only R_ACMG or R_confidence zeroed. Table 8 also conflates the warm-up with the self-ranking phase; it does not test a version with the warm-up but without self-ranking.
  4. [Sec. 4.2, Tables 3–9] Ablation tables report only point estimates, despite the main table's careful use of bootstrap confidence intervals. Several deltas are small (e.g., +0.4 on Tomato in Table 3, +0.4–0.8 on exocentric benchmarks for Qwen3-VL in Table 1). Without intervals, the reader cannot tell whether the differences between reward-only variants are meaningful. Add CIs or at least significance indicators to all ablation tables.
  5. [Sec. 3.3, Eq. (3) and Sec. 4, Implementation Details] The paper's notation is ambiguous about reward weights and the KL coefficient. Eq. (3) uses w_f, w_a, w_g, w_c, while Eq. (5) uses β for the KL penalty and Eq. (8) reuses β for the confidence reward scale. The implementation paragraph says 'setting the policy temperature β=0.1' and reward weights {0.1,0.3,0.3,0.3}. It is unclear whether β in Eq. (5) and Eq. (8) are the same hyperparameter, and how the reward weights are normalized. Clarify the hyperparameter definitions.
minor comments (5)
  1. [RQ2, Sec. 4.2] Typo: 'GRPO (Format + Award)' should be 'GRPO (Format + Answer)'.
  2. [Fig. 2 and Fig. 3] The fire emoji '🔥' appears in figure captions but its meaning is not defined. Replace with a standard symbol for trainable/frozen components.
  3. [Appendix A, RQ2] The sentence 'the strong gains stem from the confidence reward structure itself rather than solely from teacher distillation' is not supported by Table 8 alone; the table compares no-warm-up vs. 200-step warm-up, not versus no-confidence-reward.
  4. [Table 1, Qwen3-VL row] The LVBench improvement for Qwen3-VL is marked as not statistically significant, but the text summarizing Table 1 says 'improvements are statistically significant (p<0.05) across all models.' Qualify this claim to exclude the marked entries.
  5. [Sec. 3.2] The SFT stage uses teacher-generated plans and verifications from Qwen2.5-VL-72B with ground-truth annotations. It should be stated explicitly whether the teacher is the same model family as the RL policy, and whether any of the evaluation benchmarks overlap with the SFT/rollout sources (EgoProceL, HD-EPIC, EgoIT).

Circularity Check

2 steps flagged

Dense rewards are partially self-referential: RACMG is measured with a head trained to maximize it, and Eq. 9 selects chosen/rejected using the composite reward that contains Rconfidence. Headline benchmark gains remain external, so circularity is partial (4/10).

specific steps
  1. self definitional [Sec. 3.3.2 (Eqs. 6-7), Fig. 3 caption]
    "To ensure each generated <ego plan> is temporally predictive and visually grounded, we introduce the Anticipatory Cross-Modal Grounding (ACMG) reward. This self-supervised signal measures how well the generated plan anticipates future visual states... For each clause, we extract its final hidden state ... via a small trainable MLP (Anticipation Head) to predict the visual embedding of the next moment ... RACMG(ci) = max_{n in {1,...,N}} sim(v̂i, v_{t+n}). ... All components are trained jointly under GRPO."

    RACMG is both the training objective and the evidence of 'anticipatory grounding': the Anticipation Head that produces v̂i is part of the GRPO-trained policy, so it is optimized to make the cosine similarities in Eq. 6 high. High RACMG after training is therefore partly a tautology of fitting the head to the same reward, not an independent measurement that plans 'predict' future visual states. Figures 4/8 and RQ4 then present this self-fitted similarity as proof of temporal grounding, reducing the grounding claim to the definition of the reward.

  2. self definitional [Sec. 3.3.3, Eq. 9]
    "After the warm-up, each of the k=8 rollout verification is scored using the composite reward (Eq. 3), and the highest- (ychosen) and lowest-scoring (yrejected) outputs are compared: Rconfidence = β·(logπθ(ychosen|X) − logπθ(yrejected|X)), if scores differ; 1, otherwise."

    Eq. 9 defines Rconfidence in terms of ychosen/yrejected selected by Eq. 3, but Eq. 3 includes wc Rconfidence. The selector is therefore a function of the very reward being defined, making Rconfidence a self-referential fixed point rather than an independent measure of logical coherence. The paper uses this signal to claim the model learns 'consistent third-person verification,' but in the self-ranking phase the target is the policy's own score on a reward that contains itself.

full rationale

The headline numbers are not circular: EgoVITA is scored on independent external benchmarks (EgoBlind, EgoPlan, EgoThink, EOC-Bench, MVBench, Video-MME, LVBench, Tomato) against base models, SFT, and a format+answer GRPO baseline, and the ablations vary reward components, pooling, warm-up, and vision-component training. Those external results and ablations give the central empirical claim independent content. The circularity is confined to the internal mechanism attribution: the ACMG reward's 'grounding' measure is produced by an Anticipation Head trained to maximize that same reward, and the self-ranking confidence reward (Eq. 9) selects chosen/rejected examples using a composite reward that contains Rconfidence itself. These are genuine self-referential definitions, but they do not by themselves force the benchmark numbers, and the teacher-warm-up ablation (Table 8) shows the confidence structure has some independent effect. The self-citations [15,16,17] are background and not load-bearing. Hence score 4, not 0 or 6+.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 1 invented entities

The central claim rests on two learned reward signals: ACMG, which uses a trainable projection head and a hand-tuned future-frame window, and a confidence signal that is partly self-referential. No physical constants or external laws are derived; all numbers come from training choices and empirical tuning.

free parameters (6)
  • Reward weights (w_f, w_a, w_g, w_c) = {0.1, 0.3, 0.3, 0.3}
    Hand-set trade-off among format, answer, ACMG, and confidence rewards; no sensitivity analysis or validation split is reported.
  • Anticipation window N = 16
    Selected after ablating N=4,8,16,32 on the same target benchmarks (Appendix Table 9), making benchmark results partly a product of this choice.
  • Exocentric regularization coefficient lambda_exo = 0.05
    Hand-chosen; no ablation on lambda_exo is shown in the paper.
  • KL coefficient / reward scaling beta = 0.1
    Used both in Eq. 5 (KL penalty) and Eq. 8-9 (reward scaling) with the same symbol and value; the dual role is ambiguous and not analyzed.
  • Number of GRPO rollouts k = 8
    Fixed at 8; no sensitivity analysis is provided.
  • Teacher warm-up steps = 200
    Chosen based on ablation (Table 8); removing it costs 1.1-1.4 points on egocentric benchmarks.
axioms (5)
  • domain assumption LLM clause hidden states carry enough semantic/visual information that a small MLP can project them into the vision encoder's embedding space.
    Invoked in Sec. 3.3.2 for ACMG; if false, the grounding reward is meaningless.
  • ad hoc to paper Maximum cosine similarity between a clause embedding and any of N=16 future frames is a valid measure of causal/visual grounding.
    Eq. 6-7; the paper explicitly chooses max over mean based on empirical gains, acknowledging spurious-match risk (Appendix RQ1).
  • domain assumption Teacher (Qwen2.5-VL-72B) generated SFT data from EgoProceL and HD-EPIC annotations is sufficiently correct to teach planning and verification.
    Sec. 3.2; all Stage I supervision is teacher-generated, and no human validation of these traces is described.
  • ad hoc to paper The self-ranking confidence reward (Eq. 9), which reinforces the policy's own high-scoring outputs, improves logical verification rather than overconfidence.
    Sec. 3.3.3; no external correctness signal is used in the self-ranking phase.
  • domain assumption GRPO with group-relative rewards and KL constraint to the SFT policy remains stable for multimodal video reasoning.
    Sec. 3.3; borrowed from text LLM RL, assumed to transfer to video-language models without modification.
invented entities (1)
  • Anticipation Head (trainable MLP) no independent evidence
    purpose: Projects LLM clause hidden states into the visual embedding space to compute the ACMG grounding reward.
    No external validation beyond in-paper qualitative embeddings and ablations; its training objective is not specified, so the predicted future-frame matches cannot be independently checked.

pith-pipeline@v1.3.0-alltime-deepseek · 19923 in / 14886 out tokens · 139921 ms · 2026-08-03T20:47:27.726255+00:00 · methodology

0 comments
read the original abstract

Egocentric video understanding requires procedural reasoning under partial observability and continuously shifting viewpoints. Current multimodal large language models (MLLMs) struggle with this setting, often generating plausible but visually inconsistent or weakly grounded responses. We introduce $\textbf{EgoVITA}$, a framework that decomposes egocentric video reasoning into a structured $\textit{plan-then-verify}$ process. The model first generates an $\textbf{egocentric plan}$: a causal sequence of anticipated actions from a first-person perspective. This plan is then evaluated by an $\textbf{exocentric verification}$ stage that uses third-person reasoning over the same video to verify its spatiotemporal and logical consistency, without exocentric video input. This decomposition enables cross-perspective feedback without requiring paired ego-exo supervision. To train this reasoning process, we adopt Group Relative Policy Optimization (GRPO) with two dense reward signals: one that grounds anticipated actions in subsequent visual observations and another that reinforces consistent third-person verification. $\textbf{EgoVITA}$ achieves state-of-the-art performance on egocentric reasoning benchmarks, outperforming Qwen2.5-VL-7B by $\mathbf{+7.7}$ on EgoBlind and $\mathbf{+4.4}$ on EgoOrient, while maintaining strong generalization on exocentric video tasks with only $52k$ training samples.

Figures

Figures reproduced from arXiv: 2511.18242 by Pooyan Fazli, Yogesh Kulkarni.

Figure 1
Figure 1. Figure 1: EgoVITA enables safer, structured egocentric reason￾ing. For a first-person query, it produces a grounded, stepwise plan anticipating actions and spatial constraints, whereas Qwen2.5-VL gives generic, unsafe, visually ungrounded responses. nesses in first-person video understanding. They frequently generate plausible but incorrect descriptions [30], show weak spatiotemporal reasoning [20, 21, 28], poor mem… view at source ↗
Figure 2
Figure 2. Figure 2: EgoVITA training pipeline. In Stage I SFT, the policy model is initialized to learn the structured <ego plan>, <exo verify>, and <answer> format. In Stage II GRPO, it generates reasoning trajectories for a given video and query, scores them with a composite reward, and refines the model accordingly. 3.1. Problem Formulation We formulate egocentric video reasoning as a sequential decision-making problem. Th… view at source ↗
Figure 3
Figure 3. Figure 3: Anticipatory Cross-Modal Grounding (ACMG). The generated plan is projected into visual space via a trainable MLP (Anticipation Head) and compared to N future frames, encoded by the vision encoder, using cosine similarity. ACMG is the maximum similarity, encouraging plans that predict actual future observations. All components are trained jointly under GRPO. t+1 t+3 t+5 t+7 t+9 t+11 t+13 t+15 t+17 t+19 Futu… view at source ↗
Figure 4
Figure 4. Figure 4: Temporal Grounding via ACMG. Heatmap shows cosine similarity between each clause’s predicted visual embedding and N future frames. White stars indicate maximum similarity (best temporal match). ACMG rewards clauses that accurately predict future visual states, earlier clauses ground to near-future frames (e.g., Clause 1 peaks at t + 4), while later clauses align with distant frames (Clause 5 peaks at t + 1… view at source ↗
Figure 5
Figure 5. Figure 5: ACMG Embedding Space Structure. t-SNE projection of text clauses (blue circles), MLP-predicted embeddings (red dia￾monds), and their corresponding matched frames (green stars) from 75 egocentric video clauses. Distinct clusters emerge by action type, with strong alignment between predicted and actual embed￾dings, demonstrating effective cross-modal grounding. 7 [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: EgoVITA Qualitative Examples. Left: For a blind person, the <ego plan> generates sequential, safety-critical actions. Right: For a procedural task, the <ego plan> specifies a systematic, step-by-step approach. In both cases, the <exo verify> block confirms the plan’s consistency with key visual frames, leading to a grounded and actionable <answer> [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Ablation on Exocentric Regularization. The plots show performance on four exocentric benchmarks (Qwen2.5-VL family). Training on egocentric data via SFT causes an initial performance drop, or catastrophic forgetting. Without regularization, this decline continues. EgoVITA’s exocentric regularization reverses this forgetting, recovering performance and ultimately improving upon the original base model. Clau… view at source ↗
Figure 8
Figure 8. Figure 8: RACMG Temporal Grounding for a Single Clause. Given the clause “Walk alongside sidewalk to bus stop and locate the braille schedule,” the Anticipation Head projects it into visual space and computes cosine similarity with future frames. The best match occurs at t + 12 (Sim: 0.920) when the braille sign appears. The heatmap and curve show similarity peaks at the temporally correct frame, demonstrating that … view at source ↗
Figure 9
Figure 9. Figure 9: ACMG Reward Distribution by Clause Position. Box plots show ACMG reward distributions for the first five clauses in <ego plan> sequences. Top labels indicate typical semantic roles at each position. Middle clauses (positions 2 − 3) achieve higher rewards and stronger correlation (ρ) with answer correct￾ness, demonstrating that ACMG effectively distinguishes between concrete action steps and abstract reason… view at source ↗
Figure 10
Figure 10. Figure 10: EgoVITA Qualitative Examples.The baseline Qwen2.5-VL-7B [1] suffers from factual errors and temporal hallucinations (e.g., inventing “cabinet storage” steps or objects like “eggs” and “whisks”). In contrast, EgoVITA utilizes Egocentric Planning to correctly sequence fine-grained actions and Exocentric Verification to ground them in specific video frames (e.g., “frame 5”, “frame 18”), preventing hallucinat… view at source ↗
Figure 11
Figure 11. Figure 11: Prompt used to transform raw temporal annotations into structured SFT samples for egocentric planning. This prompts the [PITH_FULL_IMAGE:figures/full_fig_p017_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: The prompt used to generate the exocentric verification data from HD-EPIC [ [PITH_FULL_IMAGE:figures/full_fig_p018_12.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. EgoExoMem: Cross-View Memory Reasoning over Synchronized Egocentric and Exocentric Videos

    cs.CV 2026-05 unverdicted novelty 7.0

    EgoExoMem is the first benchmark for cross-view memory reasoning on synchronized egocentric-exocentric videos, where E2-Select raises MLLM accuracy from 55.3% to 58.2% over baselines.

  2. VisionPulse: A Virtual Reality System Enabling Accessible Discovery and Navigation for Blind and Low Vision Users

    cs.HC 2026-07 conditional novelty 6.0

    A VR system using head movement plus audio and haptic feedback let 12 blind and low vision participants discover and navigate virtual spaces, with 10 of 12 preferring it to a prebuilt menu.

  3. EgoExoMem: Cross-View Memory Reasoning over Synchronized Egocentric and Exocentric Videos

    cs.CV 2026-05 unverdicted novelty 6.0

    EgoExoMem introduces the first cross-view ego–exo video memory benchmark (2.6K MCQs, eight QA types) and E²-Select, a training-free dual-view frame selector scoring 58.2%.

  4. EgoIntent: A Pre-Outcome Micro-Step Benchmark for Understanding What, Why, and Next

    cs.CV 2026-03 conditional novelty 6.0

    A step-level egocentric-video benchmark for What/Why/Next intent shows current multimodal models score only about 33/100, though some supporting experiments are missing from the paper.

  5. AtlasRAN: Timing-Aware Evaluation of Open-source 5G Platforms for Integrated Wireless Testbeds

    cs.NI 2026-03 unverdicted novelty 5.0

    Functional compatibility of open-source 5G platforms is not timing fidelity; AtlasRAN supplies reference architectures and a claim-to-capability matrix, illustrated by an OAI RFSim vs Sionna-RK uplink load study where...

Reference graph

Works this paper leans on

58 extracted references · 11 linked inside Pith · cited by 4 Pith papers

  1. [1]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025. 3, 5, 15, 16

    Pith/arXiv arXiv 2025

  2. [2]

    My view is the best view: Procedure learning from egocentric videos

    Siddhant Bansal, Chetan Arora, and CV Jawahar. My view is the best view: Procedure learning from egocentric videos. InEuropean Conference on Computer Vision (ECCV), pages 657–675, 2022. 3, 5, 15

    2022

  3. [3]

    Wear: An outdoor sports dataset for wear- able and egocentric activity recognition

    Marius Bock, Hilde Kuehne, Kristof Van Laerhoven, and Michael Moeller. Wear: An outdoor sports dataset for wear- able and egocentric activity recognition. InProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT), pages 1–21, 2024. 1

    2024

  4. [4]

    Egoplan- bench: Benchmarking multimodal large language models for human-level planning.arXiv preprint arXiv:2312.06722,

    Yi Chen, Yuying Ge, Yixiao Ge, Mingyu Ding, Bohao Li, Rui Wang, Ruifeng Xu, Ying Shan, and Xihui Liu. Egoplan- bench: Benchmarking multimodal large language models for human-level planning.arXiv preprint arXiv:2312.06722,

    Pith/arXiv arXiv

  5. [5]

    Egothink: Evaluating first- person perspective thinking capability of vision-language models

    Sijie Cheng, Zhicheng Guo, Jingwen Wu, Kechen Fang, Peng Li, Huaping Liu, and Yang Liu. Egothink: Evaluating first- person perspective thinking capability of vision-language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14291–14302, 2024. 5

    2024

  6. [6]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. InProceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pages 24108–24118,

  7. [7]

    Spatial reasoning with vision- language models in ego-centric multi-view scenes.arXiv preprint arXiv:2509.06266, 2025

    Mohsen Gholami, Ahmad Rezaei, Zhou Weimin, Yong Zhang, and Mohammad Akbari. Spatial reasoning with vision- language models in ego-centric multi-view scenes.arXiv preprint arXiv:2509.06266, 2025. 1, 2

    arXiv 2025

  8. [8]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025. 2

    Pith/arXiv arXiv 2025

  9. [9]

    Egoexobench: A benchmark for first- and third-person view video understanding in mllms

    Yuping He, Yifei Huang, Guo Chen, Baoqi Pei, Jilan Xu, Tong Lu, and Jiangmiao Pang. Egoexobench: A benchmark for first- and third-person view video understanding in mllms. InAd- vances in Neural Information Processing Systems (NeurIPS),

  10. [10]

    Sound bridge: Associating ego- centric and exocentric videos via audio cues

    Sihong Huang, Jiaxin Wu, Xiaoyong Wei, Yi Cai, Don San Jiang, and Yaowei Wang. Sound bridge: Associating ego- centric and exocentric videos via audio cues. InProceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pages 28942–28951, 2025. 2

    2025

  11. [11]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024. 7, 14

    Pith/arXiv arXiv 2024

  12. [12]

    Is ‘right’right? enhancing object orientation understanding in multimodal large language models through egocentric instruction tuning

    Ji Hyeok Jung, Eun Tae Kim, Seoyeon Kim, Joo Ho Lee, Bumsoo Kim, and Buru Chang. Is ‘right’right? enhancing object orientation understanding in multimodal large language models through egocentric instruction tuning. InProceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pages 14257–14267, 2025. 1, 2

    2025

  13. [13]

    Egoexo-con: Exploring view- invariant video temporal understanding.arXiv preprint arXiv:2510.26113, 2025

    Minjoon Jung, Junbin Xiao, Junghyun Kim, Byoung-Tak Zhang, and Angela Yao. Egoexo-con: Exploring view- invariant video temporal understanding.arXiv preprint arXiv:2510.26113, 2025. 2

    Pith/arXiv arXiv 2025

  14. [14]

    Guidedog: A real-world egocentric multimodal dataset for blind and low-vision accessibility-aware guidance.arXiv preprint arXiv:2503.12844, 2025

    Junhyeok Kim, Jaewoo Park, Junhee Park, Sangeyl Lee, Ji- wan Chung, Jisung Kim, Ji Hoon Joung, and Youngjae Yu. Guidedog: A real-world egocentric multimodal dataset for blind and low-vision accessibility-aware guidance.arXiv preprint arXiv:2503.12844, 2025. 1, 2

    Pith/arXiv arXiv 2025

  15. [15]

    Videosavi: Self-aligned video language models without human supervision

    Yogesh Kulkarni and Pooyan Fazli. Videosavi: Self-aligned video language models without human supervision. InCon- ference on Language Modeling (COLM), 2024. 2

    2024

  16. [16]

    Avatar: Reinforcement learning to see, hear, and reason over video.arXiv preprint arXiv:2508.03100, 2025

    Yogesh Kulkarni and Pooyan Fazli. Avatar: Reinforcement learning to see, hear, and reason over video.arXiv preprint arXiv:2508.03100, 2025. 2

    arXiv 2025

  17. [17]

    Videopasta: 7k prefer- ence pairs that matter for video-llm alignment

    Yogesh Kulkarni and Pooyan Fazli. Videopasta: 7k prefer- ence pairs that matter for video-llm alignment. InConfer- ence on Empirical Methods in Natural Language Processing (EMNLP), 2025. 2

    2025

  18. [18]

    Mvbench: A comprehensive multi-modal video understand- ing benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understand- ing benchmark. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22195–22206, 2024. 5

    2024

  19. [19]

    Egotom: Benchmarking theory of mind reasoning from egocentric videos.arXiv preprint arXiv:2503.22152, 2025

    Yuxuan Li, Vijay Veerabadran, Michael L Iuzzolino, Brett D Roads, Asli Celikyilmaz, and Karl Ridgeway. Egotom: Benchmarking theory of mind reasoning from egocentric videos.arXiv preprint arXiv:2503.22152, 2025. 1, 2

    Pith/arXiv arXiv 2025

  20. [20]

    Fine-grained spatiotemporal grounding on egocen- tric videos

    Shuo Liang, Yiwu Zhong, Zi-Yuan Hu, Yeyao Tao, and Liwei Wang. Fine-grained spatiotemporal grounding on egocen- tric videos. InProceedings of the IEEE/CVF international conference on computer vision (ICCV), 2025. 1, 2 9

    2025

  21. [21]

    Fine-grained spatiotemporal grounding on egocen- tric videos

    Shuo Liang, Yiwu Zhong, Zi-Yuan Hu, Yeyao Tao, and Liwei Wang. Fine-grained spatiotemporal grounding on egocen- tric videos. InProceedings of the IEEE/CVF international conference on computer vision (ICCV), 2025. 1

    2025

  22. [22]

    Dora: Weight-decomposed low-rank adaptation

    Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. InForty-first International Conference on Ma- chine Learning (ICML), 2024. 5

    2024

  23. [23]

    Viewpoint rosetta stone: Unlocking unpaired ego-exo videos for view-invariant representation learning

    Mi Luo, Zihui Xue, Alex Dimakis, and Kristen Grauman. Viewpoint rosetta stone: Unlocking unpaired ego-exo videos for view-invariant representation learning. InProceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pages 15802–15812, 2025. 2

    2025

  24. [24]

    Guerrero

    Lorenzo Mur-Labadia, Maria Santos-Villafranca, Jesus Bermudez-Cameo, Alejandro Perez-Yus, Ruben Martinez- Cantin, and Jose J. Guerrero. O-mama: Learning object mask matching between egocentric and exocentric views. In Proceedings of the IEEE/CVF international conference on computer vision (ICCV), 2025. 2

    2025

  25. [25]

    Egothinker: Unveiling egocentric reasoning with spatio-temporal cot

    Baoqi Pei, Yifei Huang, Jilan Xu, Yuping He, Guo Chen, Fei Wu, Yu Qiao, and Jiangmiao Pang. Egothinker: Unveiling egocentric reasoning with spatio-temporal cot. InAdvances in Neural Information Processing Systems (NeurIPS), 2025. 2, 6

    2025

  26. [26]

    In the eye of mllm: Benchmarking egocentric video intent under- standing with gaze-guided prompting

    Taiying Peng, Jiacheng Hua, Miao Liu, and Feng Lu. In the eye of mllm: Benchmarking egocentric video intent under- standing with gaze-guided prompting. InAdvances in Neural Information Processing Systems (NeurIPS), 2025. 1, 2

    2025

  27. [27]

    Hd-epic: A highly- detailed egocentric video dataset

    Toby Perrett, Ahmad Darkhalil, Saptarshi Sinha, Omar Emara, Sam Pollard, Kranti Kumar Parida, Kaiting Liu, Prajwal Gatti, Siddhant Bansal, Kevin Flanagan, et al. Hd-epic: A highly- detailed egocentric video dataset. InProceedings of the Com- puter Vision and Pattern Recognition Conference (CVPR), pages 23901–23913, 2025. 1, 2, 3, 5, 15, 18

    2025

  28. [28]

    Omnia de egotempo: Bench- marking temporal understanding of multi-modal llms in ego- centric videos

    Chiara Plizzari, Alessio Tonioni, Yongqin Xian, Achin Kul- shrestha, and Federico Tombari. Omnia de egotempo: Bench- marking temporal understanding of multi-modal llms in ego- centric videos. InProceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pages 24129–24138,

  29. [29]

    Direct prefer- ence optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct prefer- ence optimization: Your language model is secretly a reward model. InAdvances in Neural Information Processing Sys- tems (NeurIPS), pages 53728–53741, 2023. 2

    2023

  30. [30]

    Egoil- lusion: Benchmarking hallucinations in egocentric video un- derstanding

    Ashish Seth, Utkarsh Tyagi, Ramaneswaran Selvakumar, Nishit Anand, Sonal Kumar, Sreyan Ghosh, Ramani Du- raiswami, Chirag Agarwal, and Dinesh Manocha. Egoil- lusion: Benchmarking hallucinations in egocentric video un- derstanding. InConference on Empirical Methods in Natural Language Processing (EMNLP), 2025. 1, 2

    2025

  31. [31]

    Tomato: As- sessing visual temporal reasoning capabilities in multimodal foundation models

    Ziyao Shangguan, Chuhan Li, Yuxuan Ding, Yanan Zheng, Yilun Zhao, Tesca Fitzgerald, and Arman Cohan. Tomato: As- sessing visual temporal reasoning capabilities in multimodal foundation models. InProceedings of the International Con- ference on Learning Representations (ICLR), 2025. 5

    2025

  32. [32]

    Visual intention grounding for egocentric assistants

    Pengzhan Sun, Junbin Xiao, Tze Ho Elden Tse, Yicong Li, Arjun Akula, and Angela Yao. Visual intention grounding for egocentric assistants. InarXiv preprint arXiv:2504.13621,

    Pith/arXiv arXiv

  33. [33]

    Ego-r1: Chain-of-tool- thought for ultra-long egocentric video reasoning.arXiv preprint arXiv:2506.13654, 2025

    Shulin Tian, Ruiqi Wang, Hongming Guo, Penghao Wu, Yuhao Dong, Xiuying Wang, Jingkang Yang, Hao Zhang, Hongyuan Zhu, and Ziwei Liu. Ego-r1: Chain-of-tool- thought for ultra-long egocentric video reasoning.arXiv preprint arXiv:2506.13654, 2025. 2

    Pith/arXiv arXiv 2025

  34. [34]

    Egovlm: Policy optimization for egocentric video understand- ing.arXiv preprint arXiv:2506.03097, 2025

    Ashwin Vinod, Shrey Pandit, Aditya Vavre, and Linshen Liu. Egovlm: Policy optimization for egocentric video understand- ing.arXiv preprint arXiv:2506.03097, 2025. 2, 6

    Pith/arXiv arXiv 2025

  35. [35]

    Internvl3

    Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025. 5

    Pith/arXiv arXiv 2025

  36. [36]

    Lvbench: An extreme long video understanding benchmark

    Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Ming Ding, Xiaotao Gu, Shiyu Huang, Bin Xu, et al. Lvbench: An extreme long video understanding benchmark. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 22958–22967,

  37. [37]

    Egoblind: To- wards egocentric visual assistance for the blind people

    Junbin Xiao, Nanxin Huang, Hao Qiu, Zhulin Tao, Xun Yang, Richang Hong, Meng Wang, and Angela Yao. Egoblind: To- wards egocentric visual assistance for the blind people. InAd- vances in Neural Information Processing Systems (NeurIPS),

  38. [38]

    Msr-vtt: A large video description dataset for bridging video and language

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 5288–5296, 2016. 5

    2016

  39. [39]

    Egoexo- gen: Ego-centric video prediction by watching exo-centric videos

    Jilan Xu, Yifei Huang, Baoqi Pei, Junlin Hou, Qingqiu Li, Guo Chen, Yuejie Zhang, Rui Feng, and Weidi Xie. Egoexo- gen: Ego-centric video prediction by watching exo-centric videos. InInternational Conference on Learning Representa- tions (ICLR), 2025. 2

    2025

  40. [40]

    Egolife: Towards egocentric life assistant

    Jingkang Yang, Shuai Liu, Hongming Guo, Yuhao Dong, Xiamengwei Zhang, Sicheng Zhang, Pengyun Wang, Zitang Zhou, Binzhu Xie, Ziyue Wang, et al. Egolife: Towards egocentric life assistant. InProceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pages 28885–28900, 2025. 1, 2, 5

    2025

  41. [41]

    Mm-ego: Towards building egocentric multimodal llms for video qa

    Hanrong Ye, Haotian Zhang, Erik Daxberger, Lin Chen, Zongyu Lin, Yanghao Li, Bowen Zhang, Haoxuan You, Dan Xu, Zhe Gan, et al. Mm-ego: Towards building egocentric multimodal llms for video qa. InInternational Conference on Learning Representations (ICLR), 2025. 2

    2025

  42. [42]

    Eoc-bench: Can mllms identify, recall, and forecast objects in an egocentric world? InAdvances in Neural Information Processing Systems (NeurIPS), 2025

    Yuqian Yuan, Ronghao Dang, Long Li, Wentong Li, Dian Jiao, Xin Li, Deli Zhao, Fan Wang, Wenqiao Zhang, Jun Xiao, et al. Eoc-bench: Can mllms identify, recall, and forecast objects in an egocentric world? InAdvances in Neural Information Processing Systems (NeurIPS), 2025. 1, 2, 5

    2025

  43. [43]

    Exo2ego: Exocentric knowledge guided mllm for egocentric video understanding.arXiv preprint arXiv:2503.09143, 2025

    Haoyu Zhang, Qiaohui Chu, Meng Liu, Yunxiao Wang, Bin Wen, Fan Yang, Tingting Gao, Di Zhang, Yaowei Wang, and Liqiang Nie. Exo2ego: Exocentric knowledge guided mllm for egocentric video understanding.arXiv preprint arXiv:2503.09143, 2025. 2 10

    arXiv 2025

  44. [44]

    Twigg, and Robert Y

    Amy Zhao, Chengcheng Tang, Lezi Wang, Yijing Li, Mihika Dave, Lingling Tao, Christopher D. Twigg, and Robert Y . Wang. Egobody3m: Egocentric body tracking on a vr headset using a diverse dataset. InEuropean Conference on Computer Vision (ECCV), page 375–392, 2024. 1

    2024

  45. [45]

    Swift: a scalable lightweight infrastructure for fine-tuning

    Yuze Zhao, Jintao Huang, Jinghan Hu, Xingjun Wang, Yunlin Mao, Daoze Zhang, Zeyinzi Jiang, Zhikai Wu, Baole Ai, Ang Wang, et al. Swift: a scalable lightweight infrastructure for fine-tuning. InProceedings of the AAAI Conference on Artificial Intelligence (AAAI), pages 29733–29735, 2025. 5

    2025

  46. [46]

    Egotextvqa: Towards egocentric scene-text aware video ques- tion answering

    Sheng Zhou, Junbin Xiao, Qingyun Li, Yicong Li, Xun Yang, Dan Guo, Meng Wang, Tat-Seng Chua, and Angela Yao. Egotextvqa: Towards egocentric scene-text aware video ques- tion answering. InProceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pages 3363–3373,

  47. [47]

    walk to the bus stop

    Yichen Zhu and Feifei Feng. Let me show you: Learning by retrieving from egocentric video for robotic manipulation. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2025. 1 11 Appendix A. Ablation Studies RQ1: Why Max Instead of Average Pooling for RACMG? A valid concern is thatmaxpooling (Eq. 6) could reward spurious, high-s...

    2025

  48. [48]

    How do I make brownies?

    Synthesize a User Query:Infer the overall goal of the video (e.g., “How do I make brownies?”) and formulate it as a question a user would ask for assistance

  49. [49]

    Generate Answer Options:Create 4 distinct multiple-choice options (A, B, C, D) that summarize the procedure: 3.One Correct Option:An accurate summary of the sequential steps

  50. [50]

    5.Construct the Response:Generate the target output using the following strictly enforced format: 6.<ego plan>: A numbered, step-by-step plan

    Three Distractors:Plausible but incorrect summaries (e.g., incorrect order, missing critical steps, or describing a different but related task). 5.Construct the Response:Generate the target output using the following strictly enforced format: 6.<ego plan>: A numbered, step-by-step plan. Each step must cite the specific timestamp interval from the input to...

  51. [51]

    Input Annotations:{50.48-54.7: break egg},{64.19-76.3: mix eggs},{87.13-91.1: add water},

    : The final answer containing the correct option letter and its full text. Input Annotations:{50.48-54.7: break egg},{64.19-76.3: mix eggs},{87.13-91.1: add water}, ... Target Output Format: [User Query]: {Generated Query} [Options]: (A) {Option Text} (B) {Option Text} ... [Response]: <ego_plan>

  52. [52]

    Break the egg into the bowl to start the batter (50.48 - 54.7)

  53. [53]

    Mix the eggs thoroughly using a whisk (64.19 - 76.3)

  54. [54]

    Add water to the mixture as specified (87.13 - 91.1). ... </ego_plan> <answer> (A) {Correct Option Text} </answer> Figure 11. Prompt used to transform raw temporal annotations into structured SFT samples for egocentric planning. This prompts the teacher model (Qwen2.5-VL-72B) to generate a grounded<ego plan>and a correct<answer>. 17 Prompt for Exocentric ...

  55. [55]

    13.73-14.11: flip the bottom part of the coffee machine

    Annotations:A list of atomic actions with start/end timestamps (e.g., “13.73-14.11: flip the bottom part of the coffee machine”). Your Task:Generate a structured output containing two distinct blocks: 1.<ego plan>: Convert the annotations into a coherent, first-person procedural plan. Merge closely related atomic actions into logical steps where appropria...

  56. [56]

    Place the mug under the coffee machine nozzle (14.2 - 14.55)

  57. [57]

    Pick up the coffee capsule from the left countertop (14.81 - 15.08)

  58. [58]

    Insert the capsule into the machine’s slot (16.08 - 17.26). ... </ego_plan> <exo_verify> The plan is visually consistent. At 14.5s, the right hand is clearly visible placing the white mug under the spout. The interaction with the capsule is confirmed by the hand trajectory moving from the counter (14.9s) to the machine’s top slot (16.5s), followed by the ...