Pith. sign in

REVIEW 4 major objections 5 minor 46 references

Injecting a structured graph of objects, 3D positions, regions, and scene text into the prompt lifts egocentric VQA accuracy by 8.14% indoors and 8.72% outdoors on EgoTextVQA, with no fine-tuning.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 01:53 UTC pith:PQ3BWVKX

load-bearing objection Useful prompt-augmentation recipe for egocentric VQA, with a clean ablation story and a real but fixable key-frame-selection concern that should be tested before the spatial-graph claim is accepted. the 4 major comments →

arxiv 2607.14497 v1 pith:PQ3BWVKX submitted 2026-07-16 cs.CV

Reinforcing Egocentric Spatial Perception in Multimodal Large Language Models via Ego Scene Augmentation

Chi Kit Wong , Ye Pan , Yuanhuiyi Lyu , Xu Zheng , Zidong Cao , Lutao Jiang , Zixin Zhang , Huiyu Zhou
show 1 more author
Xuming Hu
This is my paper
classification cs.CV
keywords egocentric VQAspatial perceptionmultimodal large language modelsscene text understandingmonocular depth estimationEgo-element Graphprompt augmentationEgoTextVQA
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to show that multimodal large language models fail at egocentric spatial reasoning because they lean on global image embeddings and cannot pin down fine-grained positional relations. It proposes Ego Scene Augmentation (ESA), which builds an Ego-element Graph — a structured JSON record of detected objects, their 3D positions relative to the wearer, 2D bounding regions, and visible text — and inserts that graph into the model's prompt. On the EgoTextVQA benchmark this raises accuracy from 31.20 to 33.74 indoors (+8.14%) and from 25.23 to 27.43 outdoors (+8.72%), with the biggest jump in the shopping subset (+15.91% accuracy). If the claim holds, explicit spatial scene structure can compensate for the perceptual blind spots of frozen MLLMs without any training, which matters for assistive and embodied applications that operate in cluttered first-person scenes.

Core claim

The central claim is that serializing an egocentric scene as a structured graph into the MLLM prompt improves spatial reasoning at inference time. The Ego-element Graph stores objects with their names, 3D positions relative to the ego, 2D regions, and readable text, plus directed edges encoding relative depths and relations like 'left of.' Off-the-shelf perception modules (monocular depth, detection, text recognition) build it, and the graph is injected via an enhanced spatial prompt. On EgoTextVQA the paper reports +8.14% indoor and +8.72% outdoor accuracy over six baselines, with depth the largest single contributor in ablations and four nodes optimal.

What carries the argument

The Ego-element Graph is a directed attributed graph G=(V,E,A). A node encodes an ego-element with attribute vector [semantic name, 3D position, 2D region, text cues]; a directed edge encodes relative depth displacement plus region-based relations, so the graph can express 'super glue is to the left of your position, near a cardboard box.' A JSON encoder serializes the graph into the enhanced spatial prompt that is prepended to the question. Its job is to inject positional evidence the MLLM would otherwise have to guess from image pixels, grounding answers in measured object relations and readable scene text.

Load-bearing premise

The load-bearing premise is that a single still frame, chosen by how well it matches the question text, preserves the challenge of a video-predicated benchmark and contains all the spatial evidence needed; if the benchmark's difficulty is temporal, this selection turns video QA into static-image QA and favors any prompt-augmented method regardless of the graph's spatial content.

What would settle it

The decisive experiment is to evaluate the exact ESA pipeline on the original video input without the single-frame selection, building the graph from multiple frames; if the 8.14/8.72% gains vanish or shrink to noise, the single-frame shortcut explains the result. A cheaper control is to replace the JSON graph with a flat sentence list of the same detected objects and OCR text, omitting all 3D coordinates; if accuracy does not drop, the spatial relation fields are not what carries the gain.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • MLLMs can gain spatial competence at inference time through prompt augmentation alone, without fine-tuning, which lowers the barrier for deploying them in egocentric and assistive settings.
  • Text-rich, cluttered scenes are where explicit spatial structure pays off most: the shopping subset shows the largest gain (+15.91% accuracy), implying graph-based prompts are especially useful when many objects and labels compete for attention.
  • Since the method is model-agnostic, the same Ego-element Graph can be reused across MLLMs; the six baselines in the paper all improve when the graph is added.
  • Depth is the largest single contributor in the ablation, so further gains are likely if the monocular depth prior is improved or replaced with metric depth.
  • The optimal node count of four suggests concise, selective spatial graphs beat exhaustive scene dumps, a useful design principle for spatial prompt engineering.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper evaluates everything on a single text–image-similarity-selected key frame per question even though EgoTextVQA is a video benchmark; if the true difficulty is temporal grounding, the reported numbers measure a static-image variant, and the graph's apparent benefit may be inflated.
  • Because the graph uses ordinal rather than metric depth (the paper acknowledges no camera calibration), the numeric 3D coordinates in the JSON are relative, so statements like '85.7 units away' are not physically meaningful in meters; the method would need metric depth for questions that require absolute distances.
  • A decisive control the paper does not report is feeding the same detected objects and OCR text as a flat sentence list, with no positions; if accuracy stays at 33.74, the spatial relation fields are not the active ingredient.
  • The paper lists viewpoint sensitivity and propagation of perception-module errors as limitations; because the graph is serialized as confident JSON, a single misdetection or wrong depth value becomes a false premise in the prompt, so the gain depends on the reliability of the upstream detectors.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Ego Scene Augmentation (ESA), a training-free, plug-and-play module that augments MLLM prompts with an Ego-element Graph. The graph is constructed from a single video key frame using Depth Anything V2, object detection, and OCR, and is serialized into a JSON text prompt. On the EgoTextVQA benchmark, ESA improves Qwen2.5-VL accuracy from 31.20 to 33.74 (indoor, relative +8.14%) and from 25.23 to 27.43 (outdoor, relative +8.72%). An ablation study indicates that depth information contributes the largest single gain, with text and graph organization providing complementary improvements.

Significance. The approach is attractively simple and model-agnostic, with public code, and it does not require fine-tuning. If the reported gains hold under the official video protocol, the finding that explicit OCR/object/depth cues serialized into a prompt can improve MLLM performance on egocentric text-rich scenes would be a useful contribution. However, the current evaluation protocol converts a video benchmark into a question-selected static-image task, and the main empirical claims are not yet supported with appropriate uncertainty quantification or a validation protocol.

major comments (4)
  1. [App. A.2 and B.2] Key-frame selection changes the task. EgoTextVQA is a video QA benchmark whose questions can require temporal localization, as the paper's own related-work section notes. The paper selects a single key frame per question by CLIP question–frame similarity, then builds the Ego-element Graph from that frame. All baselines see the same key frame, so the comparison is internally fair, but the evaluation is no longer the official video protocol. Because the selected frame is the one most aligned with the question, the graph contains cues that are directly relevant to the answer, and the reported +8.14%/+8.72% gains may reflect frame choice rather than egocentric spatial reasoning. The paper should either evaluate under the official video protocol or include a fixed, question-independent frame condition to ablate the frame-selection effect.
  2. [Table 3 and Sec. 3.4] The 'maximum number of nodes' hyperparameter is tuned directly on the EgoTextVQA-Indoor test set: Table 3 reports accuracy for values 2, 4, 6, 8, and 10, and value 4 is selected because it gives the best average accuracy. This is test-set overfitting and can inflate the reported gains. Use a held-out validation set or report results across all hyperparameter values without selecting the best.
  3. [Sec. 4.1, Tables 1–4] No repeated runs, standard errors, or significance tests are reported. The absolute differences are small: 2.54 accuracy points indoor and 2.20 outdoor. Without variance estimates or multiple runs, it is unclear whether these differences are stable or within noise. Please provide standard deviations, multiple seeds, or bootstrap confidence intervals.
  4. [Sec. 3.2, Eq. (6), Fig. 2, Fig. 6] The paper defines directed edges with relation vectors r_ij (Eq. 6), but the serialized prompt shown in Figure 2 and the prompt template in Figure 6 only include node attributes (object name, location, region, text). If edge/relation information is not actually passed to the MLLM, the claim that the Ego-element Graph 'explicitly models the spatial and semantic relations among detected scene elements' (Sec. 4.2) is unsupported. Please clarify exactly what is serialized into the prompt and, if edges are omitted, either include them or revise the claim.
minor comments (5)
  1. [Abstract and Sec. 4.2] The '8.14%' and '8.72%' gains are relative accuracy increases (e.g., 31.20 → 33.74 is +2.54 points, 8.14% relative), not absolute percentage-point gains. State this explicitly to avoid ambiguity.
  2. [Fig. 6] The enhanced prompt text says 'consecutive images' and 'following consecutive image(s)' although only a single key frame is used. Align the prompt wording with the actual single-frame setting.
  3. [Sec. 3.4] Key-frame selection is only described in the appendix but is a core part of the evaluation. Move a description into the main text or reference it in Section 4.1 when defining the experimental protocol.
  4. [Related Work] The claim that a key-frame formulation 'follows common practice' in recent egocentric VQA systems is made without citations. Add references to prior works that use CLIP-based or similar key-frame selection.
  5. [Fig. 4 caption] The caption says 'Ablation on Components of the Pipeline' but the figure shows only final ESA outputs, not the intermediate v1–v3 stages. Relabel or restructure the figure to match the caption.

Circularity Check

0 steps flagged

No significant circularity: the Ego-element Graph is an external, off-the-shelf representation; reported gains are benchmark evaluations, not derivations from the test answers.

full rationale

The paper's claimed derivation chain is: Depth Anything V2, object detection, region boxes, and OCR are used to construct an Ego-element Graph (Sec. 3.2, Eqs. 3-6); the graph is serialized into a prompt (Eq. 8); and a frozen MLLM generates an answer (Eq. 9). None of the graph attributes are fit to EgoTextVQA answers: the depth model, CLIP selector, detectors, and OCR are all external, and Sec. 3.3 explicitly states the framework requires no training. The baseline and ESA comparisons are run under the same key-frame protocol (Supp. B.2), so the +8.14%/+8.72% deltas are internal comparisons rather than quantities forced by the method's own definitions. The only tuned value is the maximum number of graph nodes (Sec. 3.4, Table 3), selected on the same benchmark; this is a mild model-selection issue, not a circular reduction, because the MLLM's output is not constrained by that choice to match ground truth. The CLIP key-frame selection (Supp. A.2) is a task-framing choice that may affect benchmark validity, but it is applied identically to all models and does not make the ESA-vs-baseline comparison circular. Self-citations [27,28] appear only as related work and are not load-bearing. The stated limitations (Sec. 6) about ordinal depth and propagation of perception errors are consistent with an external pipeline, not a tautology. No step reduces by construction to its inputs.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 1 invented entities

The paper pulls a standard depth foundation model, standard OVR/text recognizers, and a base MLLM from prior work; the Ego-element Graph is a deterministic serialization of their outputs. The main benchmark-fitted knob is the node-count hyperparameter (chosen on EgoTextVQA). No new physical or unobserved quantities are introduced.

free parameters (1)
  • maximum number of graph nodes = 4
    Selected by ablation on EgoTextVQA-Indoor (Table 3); varying 2→4→6→8→10 shows 4 is best on average ACC. This is a benchmark-fitted hyperparameter, not derived from first principles.
axioms (4)
  • domain assumption Depth Anything V2 monocular depth provides reliable egocentric geometry usable by the MLLM when serialized as absolute numbers
    Section 3.4: adopted as the depth backbone. If depth predictions are noisy or mis-scaled in egocentric scenes, the graph's 'location' numbers become misleading text for the LLM.
  • domain assumption A single CLIP-selected key frame is sufficient to preserve the egocentric difficulty and answer content of EgoTextVQA
    Supplementary A.2: 'we adopt a CLIP-based relevance scoring method to select a single key frame per question'. The benchmark is video-based; a frame-simplified evaluation may change task difficulty and the nature of the spatial evidence.
  • domain assumption The base MLLM can faithfully read and trust JSON-encoded spatial coordinates and text; the LLM will not be confused by the synthetic numbers
    The enhanced prompt in Figure 6 injects coordinates like [-70,-49,6.9]; the qualitative examples show the model parrots them. Errors or hallucinations in the serialized graph propagate into the answer.
  • domain assumption The EgoTextVQA evaluation protocol gives a meaningful, reproducible measure
    No error bars or repeated-run statistics are reported; the benchmark's train/test split and metric computation are taken as given from [45].
invented entities (1)
  • Ego-element Graph independent evidence
    purpose: Intermediary structured representation serialized into the MLLM prompt
    An explicit data structure (Eq. 3) with nodes (objects/regions), edges (pairwise relations), and attributes (semantics, 3D pos, 2D box, text). It is a prompt-format entity rather than a physical one, but it is independently constructible from off-the-shelf modules and its predicted content (object names/regions/relative positions) is checkable against the scene.

pith-pipeline@v1.3.0-alltime-deepseek · 17054 in / 8233 out tokens · 65358 ms · 2026-08-02T01:53:05.627620+00:00 · methodology

0 comments
read the original abstract

Egocentric Visual Question Answering (VQA) has attracted widespread attention as an important task for enabling Multimodal Large Language Models (MLLMs) to interact with the real world. However, existing MLLMs struggle to perform effective spatial reasoning in complex egocentric scenes due to their limited spatial perception capabilities. To this end, we introduce Ego Scene Augmentation (ESA), an egocentric spatial perception framework, which actively enhances the spatial perception capabilities from the egocentric perspective, powered by the proposed Ego-element Graph. Our core insight is leveraging the Ego-element Graph as an intermediary representation to augment the egocentric spatial perception of MLLMs via visual foundational models. Specifically, we 1) construct the Ego-element Graph, which encapsulates and integrates egocentric spatial features enabled by visual foundational models; 2) enhance the spatial perception capabilities of MLLMs via the Ego-element Graph for ego-perspective scenes. Our proposed ESA framework presents significant performance improvement on the EgoTextVQA benchmark. We achieve an 8.14% gain on the indoor setting and an 8.72% gain on the outdoor setting. Furthermore, our ESA shows the most impressive performance improvement in the shopping subset of the indoor setting. The project code is publicly available.

Figures

Figures reproduced from arXiv: 2607.14497 by Chi Kit Wong, Huiyu Zhou, Lutao Jiang, Xuming Hu, Xu Zheng, Ye Pan, Yuanhuiyi Lyu, Zidong Cao, Zixin Zhang.

Figure 1
Figure 1. Figure 1: (a) In spatially complex egocentric scenes, current [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The overall framework of our proposed Ego Scene Aug [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Comparison of existing MLLMs and our Ego Scene Augmentation (ESA) on EgoTextVQA. ESA leverages the constructed [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Ablation on Components of the Pipeline. Examples of model outputs after integrating the Ego-element Graph, showing how spatial and textual cues improve the reasoning process [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Ablation on Components of the Pipeline. Comparison of region-level recognition between existing MLLMs and our proposed Ego Scene Augmentation, showing that ESA produces more accurate and context-aware localization results. uration provides an effective balance between information sufficiency and graph sparsity, allowing the model to cap￾ture essential scene structure without being overwhelmed by additional… view at source ↗
Figure 6
Figure 6. Figure 6: The full reasoning prompt ing the effectiveness of our proposed ESA framework. B.2. Evaluation Protocol We strictly follow the official evaluation protocol of Ego￾TextVQA, reporting both Accuracy and the semantic Score metric defined in the benchmark. All baselines and our ESA model are evaluated using identical key-frame inputs, en￾suring fair and consistent comparison across methods. C. Additional Visual… view at source ↗
Figure 7
Figure 7. Figure 7: Additional visual comparison with ESA (group 1). [PITH_FULL_IMAGE:figures/full_fig_p013_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Additional visual comparison with ESA (group 2). [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 13 linked inside Pith

  1. [1]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. InProceedings of the IEEE international con- ference on computer vision, pages 2425–2433, 2015. 2

    2015

  2. [2]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025. 1, 2, 5, 6

    Pith/arXiv arXiv 2025

  3. [3]

    Where did i leave my keys?- episodic-memory-based question answering on egocentric videos

    Leonard B ¨armann and Alex Waibel. Where did i leave my keys?- episodic-memory-based question answering on egocentric videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1560–1568, 2022. 2

    2022

  4. [4]

    Ad- abins: Depth estimation using adaptive bins

    Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. Ad- abins: Depth estimation using adaptive bins. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4009–4018, 2021. 3

    2021

  5. [5]

    Local- bins: Improving depth estimation by learning local distributions

    Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. Local- bins: Improving depth estimation by learning local distributions. In European Conference on Computer Vision, pages 480–496. Springer,

  6. [6]

    Zoedepth: Zero-shot transfer by combining relative and metric depth.arXiv preprint arXiv:2302.12288, 2023

    Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias M¨uller. Zoedepth: Zero-shot transfer by combining relative and metric depth.arXiv preprint arXiv:2302.12288, 2023. 3

    Pith/arXiv arXiv 2023

  7. [7]

    Scene text visual question answering

    Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marc ¸al Rusinol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas. Scene text visual question answering. InProceedings of the IEEE/CVF international conference on computer vision, pages 4291–4301, 2019. 2

    2019

  8. [8]

    Depth pro: Sharp monocular metric depth in less than a second.arXiv preprint arXiv:2410.02073, 2024

    Aleksei Bochkovskii, Ama ˜AG ¸ l Delaunoy, Hugo Germain, Marcel Santos, Yichao Zhou, Stephan R Richter, and Vladlen Koltun. Depth pro: Sharp monocular metric depth in less than a second.arXiv preprint arXiv:2410.02073, 2024. 3

    Pith/arXiv arXiv 2024

  9. [9]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 24185– 24198, 2024. 1, 2, 5, 6

    2024

  10. [10]

    Videgothink: As- sessing egocentric video understanding capabilities for embodied ai

    Sijie Cheng, Kechen Fang, Yangyang Yu, Sicheng Zhou, Bohao Li, Ye Tian, Tingguang Li, Lei Han, and Yang Liu. Videgothink: As- sessing egocentric video understanding capabilities for embodied ai. arXiv preprint arXiv:2410.11623, 2024. 2

    Pith/arXiv arXiv 2024

  11. [11]

    Egothink: Evaluating first-person per- spective thinking capability of vision-language models

    Sijie Cheng, Zhicheng Guo, Jingwen Wu, Kechen Fang, Peng Li, Huaping Liu, and Yang Liu. Egothink: Evaluating first-person per- spective thinking capability of vision-language models. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14291–14302, 2024. 2

    2024

  12. [12]

    In- structblip: Towards general-purpose vision-language models with in- struction tuning.Advances in neural information processing systems, 36:49250–49267, 2023

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. In- structblip: Towards general-purpose vision-language models with in- struction tuning.Advances in neural information processing systems, 36:49250–49267, 2023. 1, 2, 5, 6

    2023

  13. [13]

    Egovqa-an egocentric video question answering benchmark dataset

    Chenyou Fan. Egovqa-an egocentric video question answering benchmark dataset. InProceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0–0, 2019. 2

    2019

  14. [14]

    Deep ordinal regression network for monocular depth estimation

    Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. InProceedings of the IEEE conference on com- puter vision and pattern recognition, pages 2002–2011, 2018. 3

    2002

  15. [15]

    Geowizard: Unleash- ing the diffusion priors for 3d geometry estimation from a single im- age

    Xiao Fu, Wei Yin, Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Geowizard: Unleash- ing the diffusion priors for 3d geometry estimation from a single im- age. InEuropean Conference on Computer Vision, pages 241–258. Springer, 2024. 3

    2024

  16. [16]

    Unsu- pervised monocular depth estimation with left-right consistency

    Cl ´ement Godard, Oisin Mac Aodha, and Gabriel J Brostow. Unsu- pervised monocular depth estimation with left-right consistency. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 270–279, 2017. 3

    2017

  17. [17]

    Digging into self-supervised monocular depth estimation

    Cl ´ement Godard, Oisin Mac Aodha, Michael Firman, and Gabriel J Brostow. Digging into self-supervised monocular depth estimation. InProceedings of the IEEE/CVF international conference on com- puter vision, pages 3828–3838, 2019. 3

    2019

  18. [18]

    Depthfm: Fast generative monocular depth estimation with flow matching

    Ming Gui, Johannes Schusterbauer, Ulrich Prestel, Pingchuan Ma, Dmytro Kotovenko, Olga Grebenkova, Stefan Andreas Baumann, Vincent Tao Hu, and Bj ¨orn Ommer. Depthfm: Fast generative monocular depth estimation with flow matching. InProceedings of the AAAI Conference on Artificial Intelligence, pages 3203–3211,

  19. [19]

    Towards zero-shot scale-aware monocular depth es- timation

    Vitor Guizilini, Igor Vasiljevic, Dian Chen, Rares , Ambrus,, and Adrien Gaidon. Towards zero-shot scale-aware monocular depth es- timation. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 9233–9243, 2023. 3

    2023

  20. [20]

    Mu Hu, Wei Yin, Chi Zhang, Zhipeng Cai, Xiaoxiao Long, Hao Chen, Kaixuan Wang, Gang Yu, Chunhua Shen, and Shaojie Shen. Metric3d v2: A versatile monocular geometric foundation model for zero-shot metric depth and surface normal estimation.IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 2024. 3

    2024

  21. [21]

    Ego- taskqa: Understanding human tasks in egocentric videos.Advances in Neural Information Processing Systems, 35:3343–3360, 2022

    Baoxiong Jia, Ting Lei, Song-Chun Zhu, and Siyuan Huang. Ego- taskqa: Understanding human tasks in egocentric videos.Advances in Neural Information Processing Systems, 35:3343–3360, 2022. 2

    2022

  22. [22]

    Repurposing diffusion- based image generators for monocular depth estimation

    Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Ro- drigo Caye Daudt, and Konrad Schindler. Repurposing diffusion- based image generators for monocular depth estimation. InProceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9492–9502, 2024. 3

    2024

  23. [23]

    Llava-next-interleave: Tackling multi- image, video, and 3d in large multimodal models.arXiv preprint arXiv:2407.07895, 2024

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi- image, video, and 3d in large multimodal models.arXiv preprint arXiv:2407.07895, 2024. 1, 2, 5, 6

    Pith/arXiv arXiv 2024

  24. [24]

    Egocross: Benchmarking multimodal large language models for cross- domain egocentric video question answering.arXiv preprint arXiv:2508.10729, 2025

    Yanjun Li, Yuqian Fu, Tianwen Qian, Qi’ao Xu, Silong Dai, Danda Pani Paudel, Luc Van Gool, and Xiaoling Wang. Egocross: Benchmarking multimodal large language models for cross- domain egocentric video question answering.arXiv preprint arXiv:2508.10729, 2025. 2

    arXiv 2025

  25. [25]

    Bins- former: Revisiting adaptive bins for monocular depth estimation

    Zhenyu Li, Xuyang Wang, Xianming Liu, and Junjun Jiang. Bins- former: Revisiting adaptive bins for monocular depth estimation. arXiv preprint arXiv:2204.00987, 2022. 3

    Pith/arXiv arXiv 2022

  26. [26]

    Patchfusion: An end-to-end tile-based framework for high-resolution monocular met- ric depth estimation

    Zhenyu Li, Shariq Farooq Bhat, and Peter Wonka. Patchfusion: An end-to-end tile-based framework for high-resolution monocular met- ric depth estimation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10016–10025,

  27. [27]

    Unibind: Llm-augmented unified and balanced representation space to bind them all

    Yuanhuiyi Lyu, Xu Zheng, Jiazhou Zhou, and Lin Wang. Unibind: Llm-augmented unified and balanced representation space to bind them all. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26752–26762, 2024. 2

    2024

  28. [28]

    Realrag: Retrieval- augmented realistic image generation via self-reflective contrastive learning.arXiv preprint arXiv:2502.00848, 2025

    Yuanhuiyi Lyu, Xu Zheng, Lutao Jiang, Yibo Yan, Xin Zou, Huiyu Zhou, Linfeng Zhang, and Xuming Hu. Realrag: Retrieval- augmented realistic image generation via self-reflective contrastive learning.arXiv preprint arXiv:2502.00848, 2025. 2

    arXiv 2025

  29. [29]

    Single image depth estimation: An overview.Digital Signal Processing, 123: 103441, 2022

    Alican Mertan, Damien Jade Duff, and Gozde Unal. Single image depth estimation: An overview.Digital Signal Processing, 123: 103441, 2022. 3

    2022

  30. [30]

    Unidepthv2: Univer- sal monocular metric depth estimation made simpler.arXiv preprint arXiv:2502.20110, 2025

    Luigi Piccinelli, Christos Sakaridis, Yung-Hsu Yang, Mattia Segu, Siyuan Li, Wim Abbeloos, and Luc Van Gool. Unidepthv2: Univer- sal monocular metric depth estimation made simpler.arXiv preprint arXiv:2502.20110, 2025. 3

    Pith/arXiv arXiv 2025

  31. [31]

    Towards robust monocular depth estimation: Mix- ing datasets for zero-shot cross-dataset transfer.IEEE transactions on pattern analysis and machine intelligence, 44(3):1623–1637,

    Ren ´e Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mix- ing datasets for zero-shot cross-dataset transfer.IEEE transactions on pattern analysis and machine intelligence, 44(3):1623–1637,

  32. [32]

    Monocular depth estimation us- ing neural regression forest

    Anirban Roy and Sinisa Todorovic. Monocular depth estimation us- ing neural regression forest. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 5506–5514, 2016. 3

    2016

  33. [33]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pages 8317–8326,

  34. [34]

    Alanavlm: A multimodal embodied ai foundation model for egocen- tric video understanding.arXiv preprint arXiv:2406.13807, 2024

    Alessandro Suglia, Claudio Greco, Katie Baker, Jose L Part, Ioan- nis Papaioannou, Arash Eshghi, Ioannis Konstas, and Oliver Lemon. Alanavlm: A multimodal embodied ai foundation model for egocen- tric video understanding.arXiv preprint arXiv:2406.13807, 2024. 2

    Pith/arXiv arXiv 2024

  35. [35]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024. 1, 2, 5, 6

    Pith/arXiv arXiv 2024

  36. [36]

    Emu3: Next-token prediction is all you need.arXiv preprint arXiv:2409.18869, 2024

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiy- ing Yu, et al. Emu3: Next-token prediction is all you need.arXiv preprint arXiv:2409.18869, 2024. 1, 2, 5, 6

    Pith/arXiv arXiv 2024

  37. [37]

    Fastdepth: Fast monocular depth estimation on em- bedded systems

    Diana Wofk, Fangchang Ma, Tien-Ju Yang, Sertac Karaman, and Vivienne Sze. Fastdepth: Fast monocular depth estimation on em- bedded systems. In2019 International Conference on Robotics and Automation (ICRA), pages 6101–6108. IEEE, 2019. 3

    2019

  38. [38]

    Visual question answering: A survey of methods and datasets.Computer Vision and Image Understanding, 163:21–40, 2017

    Qi Wu, Damien Teney, Peng Wang, Chunhua Shen, Anthony Dick, and Anton Van Den Hengel. Visual question answering: A survey of methods and datasets.Computer Vision and Image Understanding, 163:21–40, 2017. 2

    2017

  39. [39]

    Egolife: Towards egocentric life assistant

    Jingkang Yang, Shuai Liu, Hongming Guo, Yuhao Dong, Xiameng- wei Zhang, Sicheng Zhang, Pengyun Wang, Zitang Zhou, Binzhu Xie, Ziyue Wang, et al. Egolife: Towards egocentric life assistant. In Proceedings of the Computer Vision and Pattern Recognition Con- ference, pages 28885–28900, 2025. 2

    2025

  40. [40]

    Depth anything v2.Advances in Neural Information Processing Systems, 37:21875–21911, 2024

    Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2.Advances in Neural Information Processing Systems, 37:21875–21911, 2024. 2, 3, 5

    2024

  41. [41]

    Mm-ego: Towards building egocentric multimodal llms for video qa

    Hanrong Ye, Haotian Zhang, Erik Daxberger, Lin Chen, Zongyu Lin, Yanghao Li, Bowen Zhang, Haoxuan You, Dan Xu, Zhe Gan, et al. Mm-ego: Towards building egocentric multimodal llms for video qa. arXiv preprint arXiv:2410.07177, 2024. 2

    Pith/arXiv arXiv 2024

  42. [42]

    Metric3d: Towards zero- shot metric 3d prediction from a single image

    Wei Yin, Chi Zhang, Hao Chen, Zhipeng Cai, Gang Yu, Kaixuan Wang, Xiaozhi Chen, and Chunhua Shen. Metric3d: Towards zero- shot metric 3d prediction from a single image. InProceedings of the IEEE/CVF international conference on computer vision, pages 9043–9053, 2023. 3

    2023

  43. [43]

    New crfs: Neural window fully-connected crfs for monocular depth estimation.arXiv preprint arXiv:2203.01502, 2022

    Weihao Yuan, Xiaodong Gu, Zuozhuo Dai, Siyu Zhu, and Ping Tan. New crfs: Neural window fully-connected crfs for monocular depth estimation.arXiv preprint arXiv:2203.01502, 2022. 3

    Pith/arXiv arXiv 2022

  44. [44]

    Egonight: Towards egocentric vision under- standing at night with a challenging benchmark.arXiv preprint arXiv:2510.06218, 2025

    Deheng Zhang, Yuqian Fu, Runyi Yang, Yang Miao, Tianwen Qian, Xu Zheng, Guolei Sun, Ajad Chhatkuli, Xuanjing Huang, Yu- Gang Jiang, et al. Egonight: Towards egocentric vision under- standing at night with a challenging benchmark.arXiv preprint arXiv:2510.06218, 2025. 2

    arXiv 2025

  45. [45]

    Egotextvqa: Towards egocentric scene-text aware video question answering

    Sheng Zhou, Junbin Xiao, Qingyun Li, Yicong Li, Xun Yang, Dan Guo, Meng Wang, Tat-Seng Chua, and Angela Yao. Egotextvqa: Towards egocentric scene-text aware video question answering. In Proceedings of the Computer Vision and Pattern Recognition Con- ference, pages 3363–3373, 2025. 1, 2, 5, 7

    2025

  46. [46]

    Unanswerable

    Xilei Zhu, Huiyu Duan, Liu Yang, Yucheng Zhu, Xiongkuo Min, Guangtao Zhai, and Patrick Le Callet. Esvqa: Perceptual quality assessment of egocentric spatial videos.arXiv preprint arXiv:2412.20423, 2024. 2 Reinforcing Egocentric Spatial Perception in Multimodal Large Language Models via Ego Scene Augmentation Supplementary Material A. More Details of Ego S...

    Pith/arXiv arXiv 2024