Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

GazeNLQ @ Ego4D Natural Language Queries Challenge 2025

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Gaze estimated from video improves egocentric video-language grounding.

desk verdict Gaze-augmented grounding is a plausible new combination, but the paper's own tables don't establish that gaze helps: the headline numbers are an ensemble with GroundVQA, and the only ablation compares freezing versus fine-tuning the gaze encoder, not removing gaze. read the letter →

arxiv 2506.05782 v1 pith:KDHJTQLX submitted 2025-06-06 cs.CV

Wei-Cheng Lin , Chih-Ming Lien , Chen Lo , Chia-Hung Yeh This is my paper
classification cs.CV
keywords egocentricvideotemporalgroundingnaturallanguagequeriesgazeestimationcontrastivelearningmulti-modalfusionEgo4Dchallenge
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that gaze—where the camera wearer is looking—can serve as a useful auxiliary signal for temporal grounding of natural language queries in egocentric video. Because gaze annotations are sparse, the authors train a gaze estimator with contrastive learning, then feed the estimated gaze embeddings into a video-language grounding model through cross-attention. On the Ego4D NLQ test split, the standalone GazeNLQ model reaches R1@IoU0.3 of 25.24, improving on the GroundNLQ baseline of 24.50, while an ensemble that combines GazeNLQ with GroundVQA reaches 27.82. The authors interpret these results as evidence that gaze is a promising cue for egocentric video understanding, though the largest reported numbers are ensemble results rather than the gaze model alone.

What carries the argument

The load-bearing mechanism is a contrastive gaze-estimation pretraining stage with a dual-branch structure: a video branch (Omnivore features, five Gated Linear Unit layers, a self-attention layer, a projection head) and a gaze branch (gaze heatmaps processed by a 3D convolution block and a projection head). Training aligns the two branches with an InfoNCE-style contrastive loss $L_{\mathrm{NCE}}$ plus a KL-divergence loss $L_{\mathrm{KL}}$ on predicted heatmaps. At grounding time, the frozen estimator's gaze embeddings are fused with text embeddings through cross-attention into the video representation, then refined by element-wise addition and self-attention before the multi-scale transformer and classification/regression heads produce moment predictions.

What would settle it

Compare two finetuned grounding models that differ only in whether the estimated gaze embeddings are replaced by same-shaped random noise; if the R1@IoU0.3 gap is negligible, the gaze signal is not what is driving the improvement.

Watch

Extended reading notes

Core claim

The central claim is that estimated gaze, treated as a third modality alongside video and text, improves moment retrieval. GazeNLQ first pretrains a gaze estimator on annotated Ego4D gaze data: video features from Omnivore pass through gated linear units and self-attention, then contrastively align with gaze heatmaps, while a regression head also predicts the heatmap, giving a total loss $L_{\mathrm{gaze}} = L_{\mathrm{NCE}} + L_{\mathrm{KL}}$. The estimated gaze embeddings are then integrated into the GroundNLQ grounding model via two cross-attention modules that align gaze and text with video features, followed by element-wise addition, self-attention, and the multi-scale transformer heads. On the test split the standalone model scores 25.24 R1@IoU0.3 and 17.58 R1@IoU0.5, while the ensemble variant with GroundVQA scores 27.82 and 18.68; an ablation shows that freezing the gaze estimator's weights outperforms finetuning them, and a negative-gaze variant GazeNLQ* slightly outperforms the positive version on the validation split.

Load-bearing premise

The load-bearing premise is that gaze embeddings learned from Omnivore video features transfer to the InternVideo/EgoVLP features used in the grounding stage; the authors explicitly flag this feature mismatch in Section 3.5 as a limitation.

Editorial extensions

If this is right

  • Estimated gaze can be added as an extra input channel without requiring gaze annotations at inference time, since the estimator is pretrained and then frozen.
  • Freezing the gaze estimator during grounding finetuning outperforms unfreezing it, suggesting the small gaze-training corpus does not support joint adaptation; a larger gaze dataset could change that.
  • The negative-gaze variant GazeNLQ* improves R1@IoU0.3 on the validation split, implying that steering features away from the gazed region can also help grounding.
  • Gaze assistance mainly helps relaxed overlap thresholds: validation R1@IoU0.3 improves while R1@IoU0.5 slightly drops, so the gaze signal coarsely localizes attention but does not yet sharpen boundaries.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the authors identify the Omnivore-to-InternVideo/EgoVLP feature mismatch as a likely bottleneck, a natural extension is to retrain the gaze estimator on the same video features used in grounding; if the transfer assumption is the constraint, this should raise both R1@IoU0.3 and R1@IoU0.5.
  • The gaze signal is a spatial prior while the grounding video features are non-spatial; combining gaze with spatially aware features, or training from scratch with narration data as the authors suggest, could convert the coarse localization gain into sharper boundaries.
  • The negative-gaze result suggests attention suppression, not just attention emphasis, is informative; a controlled experiment varying the ratio of positive to negative gaze embeddings could test whether this is a regularizing effect or a genuine signal about where not to look.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper presents GazeNLQ, the authors' entry to the Ego4D Natural Language Queries Challenge 2025. The method pretrains a gaze estimator with a contrastive objective on video features and gaze heatmaps, then integrates the resulting gaze embeddings into the GroundNLQ pipeline via cross-attention and self-attention modules. Positive and negative gaze variants (GazeNLQ and GazeNLQ⋆) are evaluated on the NLQ val and test splits, and an ensemble with GroundVQA is also reported. The headline claims are that GazeNLQ achieves R1@IoU0.3 and R1@IoU0.5 scores of 27.82 and 18.68 on the test split, and that incorporating estimated gaze enhances localization accuracy.

Significance. If the central claim were established, the paper would provide a useful data point on using gaze as an auxiliary signal for egocentric video grounding, and the released code would aid reproducibility. The authors are transparent about several limitations, including the feature-extractor mismatch between the gaze-estimation stage (Omnivore) and the grounding stage (InternVideo/EgoVLP), the reliance on a small gaze-annotation set, and the use of finetuning rather than training from scratch. However, the current experiments do not isolate gaze's contribution: the headline test numbers come from an ensemble that also includes GroundVQA, and no ablation removes the gaze branch. As a result, the causal claim that gaze improves localization is not yet supported by the presented evidence.

major comments (4)
  1. [Abstract and Table 1] The abstract and conclusion attribute the 27.82/18.68 test scores to GazeNLQ, but Table 1 marks these numbers as GazeNLQ†, an ensemble that also incorporates GroundVQA. The standalone GazeNLQ and GazeNLQ⋆ test results are 25.24/17.58 and 25.45/17.48, respectively. The paper must clearly state that the headline scores are ensemble results and report the standalone numbers in the abstract and conclusion.
  2. [Sec. 3.3, Table 3] The only ablation compares freezing versus unfreezing the gaze estimator's weights; it never removes the gaze information itself. Consequently, the observed differences could stem from added parameters, cross-attention modules, initialization, or training randomness rather than from gaze. To support the claim that gaze enhances grounding, the authors need to include a no-gaze baseline, for example GroundNLQ or a variant with the gaze embeddings zeroed out or the gaze cross-attention removed.
  3. [Sec. 3.2, Table 2] The val results do not show a consistent improvement: GazeNLQ matches GroundNLQ at R1@0.3 (26.98) and is worse at R1@0.5 (17.88 vs 18.83), while GazeNLQ⋆ improves R1@0.3 by only 0.24 and loses 0.75 at R1@0.5. The paper should report whether these differences are stable across multiple seeds and should temper the claim that gaze is beneficial, especially given the absence of a gaze-removal ablation.
  4. [Sec. 3.5, Discussion] The authors acknowledge that the gaze estimator is trained on Omnivore features while the grounding model uses InternVideo/EgoVLP features. This feature mismatch is not merely a secondary concern: the mechanism by which gaze is supposed to help depends on the gaze embeddings being transferable into the grounding stage's feature space. The paper should either provide evidence that this transfer works (e.g., an analysis of gaze embedding alignment) or explicitly state that the mechanism is unverified.
minor comments (5)
  1. [Throughout] The paper uses inconsistent spelling for 'InternVideo' (also written 'InterVideo') and contains typos such as 'offer insights' and 'We proposes'; a careful proofread is needed.
  2. [Sec. 3.3, Table 3] Table 3 would be more informative if it included the unmodified GroundNLQ baseline in the same table, since that is the reference point for judging the effect of gaze.
  3. [Sec. 2.2 and Fig. 2] The construction of the 'negative gaze embedding' used in GazeNLQ⋆ is not described; the paper should specify how negative samples are selected or generated, as this variant is a central part of the experiments.
  4. [Sec. 3.1, Ensemble] The sentence 'We combines predictions from GroundVQA [3], which followed the strategy by EgoVideo [11]' is grammatically unclear and should be rewritten to explain the ensemble weighting and combination procedure.
  5. [Sec. 3.2] The statement that the approach is 'more effective at retrieving relevant segments within a relaxed temporal threshold' is based on a 0.24-point gain at R1@0.3 and a simultaneous loss at R1@0.5; this conclusion is too strong given the absence of error bars or significance tests.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the gaze estimator is supervised by external Ego4D gaze annotations, and the NLQ grounding loss is not used to define the gaze prediction.

full rationale

Walking the derivation chain, there is no step in which a claimed prediction reduces by definition to an input fit. The gaze estimator is trained with Eq. (1) (contrastive loss between video and gaze embeddings) and Eq. (2) (KL divergence between predicted and ground-truth gaze maps), both supervised by external Ego4D gaze annotations, not by the NLQ localization targets. The grounding model then uses the frozen (in the best variant, Table 3) gaze embeddings as auxiliary input to a GroundNLQ-style architecture; the localization loss in Eq. (4) is applied only at the final grounding stage. No equation equates the gaze representation to the NLQ prediction, and no parameter is fitted to make the reported R1 scores follow from the inputs by construction. There are no load-bearing self-citations, and no external uniqueness theorem is imported to force the design. The headline numbers 27.82/18.68 are ensemble results that also include GroundVQA (Table 1 footnote and Sec. 3.1), and the val table shows no consistent standalone gain over GroundNLQ; these are real evidentiary and reporting weaknesses, but they concern whether the causal claim that gaze helps is established, not circularity. Similarly, the absence of an ablation that removes the gaze branch entirely, and the acknowledged Omnivore-to-InternVideo/EgoVLP feature mismatch (Sec. 3.5), are correctness and generalization concerns. Under the hard rule that circularity requires quoting a specific reduction of the result to its input, no such reduction is present in this paper.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The paper relies on several pretrained models and annotations from the Ego4D benchmark. No new entities are postulated. The reported metrics depend on two unspecified hand-chosen quantities: the contrastive temperature and the ensemble combination weights.

free parameters (2)
  • Contrastive temperature tau
    Hyperparameter in Eq. (1); the paper never reports its value, yet it scales the contrastive loss that trains the gaze estimator.
  • Ensemble combination weights
    The test results marked with dagger combine GazeNLQ with GroundVQA, but the combination rule and weights are not stated (Sec 3.1). The headline 27.82 R1@0.3 depends on this unspecified choice.
assumptions (5)
  • domain assumption Omnivore pretrained features are suitable for gaze estimation
    The gaze estimator uses Omnivore features (Sec 2.1); the paper does not justify this choice beyond citing [5].
  • domain assumption InternVideo and EgoVLP features are strong video representations for grounding
    The grounding model concatenates these features (Sec 2.1), relying on their quality.
  • domain assumption GroundNLQ architecture and its narration-pretrained weights transfer to this setting
    The model is initialized with GroundNLQ weights pretrained on narration data (Sec 3.1).
  • domain assumption Ego4D gaze annotations are reliable supervision
    The gaze estimator is trained on Ego4D gaze maps; if annotations are noisy, the estimated embeddings carry noise into grounding.
  • domain assumption Contrastive learning aligns video and gaze embedding spaces
    The contrastive loss in Eq. (1) assumes that dot products between aligned video and gaze embeddings are meaningful.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GazeNLQ @ Ego4D Natural Language Queries Challenge 2025." pith.science (2026). https://pith.science/paper/KDHJTQLX

@misc{pith2026250605782,
  author       = {Pith},
  title        = {Pith review of: GazeNLQ @ Ego4D Natural Language Queries Challenge 2025},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KDHJTQLX}},
  note         = {Machine review of arXiv:2506.05782}
}
read the original abstract

This report presents our solution to the Ego4D Natural Language Queries (NLQ) Challenge at CVPR 2025. Egocentric video captures the scene from the wearer's perspective, where gaze serves as a key non-verbal communication cue that reflects visual attention and offer insights into human intention and cognition. Motivated by this, we propose a novel approach, GazeNLQ, which leverages gaze to retrieve video segments that match given natural language queries. Specifically, we introduce a contrastive learning-based pretraining strategy for gaze estimation directly from video. The estimated gaze is used to augment video representations within proposed model, thereby enhancing localization accuracy. Experimental results show that GazeNLQ achieves R1@IoU0.3 and R1@IoU0.5 scores of 27.82 and 18.68, respectively. Our code is available at https://github.com/stevenlin510/GazeNLQ.

Figures

Figures reproduced from arXiv: 2506.05782 by the authors.

Figure 1
Figure 1. The proposed training framework for gaze estimator using contrastive learning. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The proposed model for video temporal grounding. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Visualization of gaze estimation. The top row shows [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: a shows successful examples in NLQ, where our model accurately locates the target of the text description. However, the failure examples are presented in Fig. 4b. In the top figure, the error arises from an imprecise tempo￾ral boundary—GazeNLQ captures only the first h…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. EgoEverything: A Benchmark for Human Behavior Inspired Long Context Egocentric Video Understanding in AR Environment

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    EgoEverything is a new benchmark for long-context egocentric video understanding that uses human gaze-based attention signals to generate questions reflecting natural behavior.

Reference graph

Works this paper leans on

15 extracted references · 10 canonical work pages · cited by 1 Pith paper

  1. [1]

    Navaneeth Bodla, Bharat Singh, Rama Chellappa, and Larry S. Davis. Soft-nms — improving object detection with one line of code. In2017 IEEE International Conference on Computer Vision (ICCV), pages 5562–5570, 2017. 3

    work page 2017

  2. [2]

    Internvideo-ego4d: A pack of champion solu- tions to ego4d challenges.arXiv preprint:2211.09529, 2022

    Guo Chen, Sen Xing, Zhe Chen, Yi Wang, Kunchang Li, Yizhuo Li, Yi Liu, Jiahao Wang, Yin-Dong Zheng, Bingkun Huang, et al. Internvideo-ego4d: A pack of champion solu- tions to ego4d challenges.arXiv preprint:2211.09529, 2022. 1, 2, 4

    arXiv 2022

  3. [3]

    Grounded question-answering in long egocentric videos

    Shangzhe Di and Weidi Xie. Grounded question-answering in long egocentric videos. InCVPR, pages 12934–12943,

    work page

  4. [4]

    Objectnlq@ ego4d episodic memory challenge 2024.arXiv preprint arXiv:2406.15778, 2024

    Yisen Feng, Haoyu Zhang, Yuquan Xie, Zaijing Li, Meng Liu, and Liqiang Nie. Objectnlq@ ego4d episodic memory challenge 2024.arXiv preprint arXiv:2406.15778, 2024. 1, 3

    arXiv 2024

  5. [5]

    Omnivore: A Sin- gle Model for Many Visual Modalities

    Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens van der Maaten, Armand Joulin, and Ishan Misra. Omnivore: A Sin- gle Model for Many Visual Modalities. InCVPR, 2022. 2, 3, 4

    work page 2022

  6. [6]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In CVPR, pages 18995–19012, 2022. 1

    work page 2022

  7. [7]

    Groundnlq@ ego4d natural language queries challenge 2023.arXiv preprint arXiv:2306.15255,

    Zhijian Hou, Lei Ji, Difei Gao, Wanjun Zhong, Kun Yan, Chao Li, Wing-Kwong Chan, Chong-Wah Ngo, Nan Duan, and Mike Zheng Shou. Groundnlq@ ego4d natural language queries challenge 2023.arXiv preprint arXiv:2306.15255,

    arXiv 2023

  8. [8]

    In the eye of transformer: Global–local correlation for egocentric gaze estimation and beyond.IJCV, pages 1–18, 2023

    Bolin Lai, Miao Liu, Fiona Ryan, and James M Rehg. In the eye of transformer: Global–local correlation for egocentric gaze estimation and beyond.IJCV, pages 1–18, 2023. 2

    work page 2023

Show all 15 references
  1. [9]

    Egocentric video-language pretraining

    Kevin Qinghong Lin, Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Z Xu, Difei Gao, Rong-Cheng Tu, Wen- zhe Zhao, Weijie Kong, et al. Egocentric video-language pretraining. InNeurIPS, pages 7575–7586, 2022. 1, 2

    2022

  2. [10]

    Reler@ zju-alibaba submission to the ego4d natural language queries challenge 2022.arXiv preprint arXiv:2207.00383, 2022

    Naiyuan Liu, Xiaohan Wang, Xiaobo Li, Yi Yang, and Yuet- ing Zhuang. Reler@ zju-alibaba submission to the ego4d natural language queries challenge 2022.arXiv preprint arXiv:2207.00383, 2022. 1

    2022 arXiv

  3. [11]

    Egovideo: Exploring egocentric founda- tion model and downstream adaptation.arXiv preprint arXiv:2406.18070, 2024

    Baoqi Pei, Guo Chen, Jilan Xu, Yuping He, Yicheng Liu, Kanghua Pan, Yifei Huang, Yali Wang, Tong Lu, Limin Wang, et al. Egovideo: Exploring egocentric founda- tion model and downstream adaptation.arXiv preprint arXiv:2406.18070, 2024. 1, 3

    2024 arXiv

  4. [12]

    Egovlpv2: Egocentric video-language pre-training with fusion in the backbone

    Shraman Pramanick, Yale Song, Sayan Nag, Kevin Qinghong Lin, Hardik Shah, Mike Zheng Shou, Rama Chellappa, and Pengchuan Zhang. Egovlpv2: Egocentric video-language pre-training with fusion in the backbone. InICCV, pages 5285–5297, 2023. 1

    2023

  5. [13]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. InProceedings of Int...

    2021

  6. [14]

    Ramakrishnan, Ziad Al-Halah, and Kristen Grauman

    Santhosh K. Ramakrishnan, Ziad Al-Halah, and Kristen Grauman. Naq: Leveraging narrations as queries to su- pervise episodic memory. InComputer Vision and Pat- tern Recognition (CVPR), 2023 IEEE Conference on. IEEE,

    2023

  7. [15]

    Egodistill: Egocentric head motion distillation for efficient video understanding

    Shuhan Tan, Tushar Nagarajan, and Kristen Grauman. Egodistill: Egocentric head motion distillation for efficient video understanding. InNeurIPS, 2023. 1 5

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.