{"id":"c1cb8963-7043-4eb1-a833-ad017422d913","arxiv_id":"2506.05782","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"GazeNLQ adds contrastively pretrained gaze embeddings to a GroundNLQ-style grounding model, reporting 27.82 R1@0.3 on the Ego4D NLQ test split only when ensembled with GroundVQA.","lead":"A team entered the Ego4D video search challenge with a model that adds eye-gaze information to standard video and text features. Their best reported scores come from combining their model with another team's question-answering system, so the pure gaze gain is small.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GazeNLQ's central claim that gaze improves localization is not supported by the reported experiments: the headline numbers are ensemble results including GroundVQA, and no ablation removes the gaze branch, while val results show no consistent gain over GroundNLQ.","rationale":"The reader's verdict was CONDITIONAL, and the condition should remain: the paper needs an ablation that isolates the gaze branch and unambiguous reporting of standalone vs ensemble numbers. My stress-test identifies a more load-bearing gap than the feature mismatch highlighted by the reader: even granting that the Omnivore-trained gaze embeddings transfer to InternVideo/EgoVLP features, the current experiments do not demonstrate that the gaze signal is what improves localization, because no no-gaze baseline is reported and the val comparisons are inconclusive or slightly negative. The feature mismatch is an acknowledged limitation and would matter if gaze were shown to help, but the missing ablation is logically prior. The abstract's attribution of ensemble results to GazeNLQ alone is a reporting error that should be fixed, but the deeper issue is evidentiary. For these reasons, keeping the paper conditional rather than accepting it is appropriate; there is no basis here for rejection because the proposed mechanism is plausible, code is released, and a targeted ablation could settle the question.","tokens_in":5350,"tokens_out":3230,"duration_ms":31309,"concrete_test":"Run the same GroundNLQ architecture and training recipe on the val split with the gaze branch ablated — e.g., replace the gaze embeddings with a fixed zero vector while keeping all other components, initialization, and seed identical to GazeNLQ⋆. If this no-gaze variant achieves R1@0.3 ≥ 27.22 and R1@0.5 ≥ 18.08, the claimed gaze benefit is not supported; if it drops clearly below these values, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and conclusion claim that estimated gaze 'enhances localization accuracy', and the headline scores 27.82/18.68 are attributed to GazeNLQ. However, Table 1 identifies those numbers as 'GazeNLQ†', an ensemble that also includes GroundVQA (Sec 3.1, 'Ensemble'). The standalone GazeNLQ and GazeNLQ⋆ test scores are 25.24/17.58 and 25.45/17.48, respectively. On the val split (Table 2), GazeNLQ matches GroundNLQ exactly at R1@0.3 (26.98) and is worse at R1@0.5 (17.88 vs 18.83); GazeNLQ⋆ gains 0.24 at R1@0.3 but loses 0.75 at R1@0.5. The only ablation (Table 3) compares freezing vs unfreezing the gaze estimator, not removing gaze information altogether. Therefore the observed differences could stem from added parameters, cross-attention, initialization, or training randomness, and the causal claim that gaze itself helps is not established. The feature mismatch noted in Sec 3.5 is secondary: even if the Omnivore-to-InternVideo/EgoVLP transfer were perfect, the current experiments do not isolate gaze's contribution.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents GazeNLQ, the authors' entry to the Ego4D Natural Language Queries Challenge 2025. The method pretrains a gaze estimator with a contrastive objective on video features and gaze heatmaps, then integrates the resulting gaze embeddings into the GroundNLQ pipeline via cross-attention and self-attention modules. Positive and negative gaze variants (GazeNLQ and GazeNLQ⋆) are evaluated on the NLQ val and test splits, and an ensemble with GroundVQA is also reported. The headline claims are that GazeNLQ achieves R1@IoU0.3 and R1@IoU0.5 scores of 27.82 and 18.68 on the test split, and that incorporating estimated gaze enhances localization accuracy.","tokens_in":67,"tokens_out":2534,"duration_ms":60956,"significance":"If the central claim were established, the paper would provide a useful data point on using gaze as an auxiliary signal for egocentric video grounding, and the released code would aid reproducibility. The authors are transparent about several limitations, including the feature-extractor mismatch between the gaze-estimation stage (Omnivore) and the grounding stage (InternVideo/EgoVLP), the reliance on a small gaze-annotation set, and the use of finetuning rather than training from scratch. However, the current experiments do not isolate gaze's contribution: the headline test numbers come from an ensemble that also includes GroundVQA, and no ablation removes the gaze branch. As a result, the causal claim that gaze improves localization is not yet supported by the presented evidence.","major_comments":[{"comment":"The abstract and conclusion attribute the 27.82/18.68 test scores to GazeNLQ, but Table 1 marks these numbers as GazeNLQ†, an ensemble that also incorporates GroundVQA. The standalone GazeNLQ and GazeNLQ⋆ test results are 25.24/17.58 and 25.45/17.48, respectively. The paper must clearly state that the headline scores are ensemble results and report the standalone numbers in the abstract and conclusion.","section":"Abstract and Table 1"},{"comment":"The only ablation compares freezing versus unfreezing the gaze estimator's weights; it never removes the gaze information itself. Consequently, the observed differences could stem from added parameters, cross-attention modules, initialization, or training randomness rather than from gaze. To support the claim that gaze enhances grounding, the authors need to include a no-gaze baseline, for example GroundNLQ or a variant with the gaze embeddings zeroed out or the gaze cross-attention removed.","section":"Sec. 3.3, Table 3"},{"comment":"The val results do not show a consistent improvement: GazeNLQ matches GroundNLQ at R1@0.3 (26.98) and is worse at R1@0.5 (17.88 vs 18.83), while GazeNLQ⋆ improves R1@0.3 by only 0.24 and loses 0.75 at R1@0.5. The paper should report whether these differences are stable across multiple seeds and should temper the claim that gaze is beneficial, especially given the absence of a gaze-removal ablation.","section":"Sec. 3.2, Table 2"},{"comment":"The authors acknowledge that the gaze estimator is trained on Omnivore features while the grounding model uses InternVideo/EgoVLP features. This feature mismatch is not merely a secondary concern: the mechanism by which gaze is supposed to help depends on the gaze embeddings being transferable into the grounding stage's feature space. The paper should either provide evidence that this transfer works (e.g., an analysis of gaze embedding alignment) or explicitly state that the mechanism is unverified.","section":"Sec. 3.5, Discussion"}],"minor_comments":[{"comment":"The paper uses inconsistent spelling for 'InternVideo' (also written 'InterVideo') and contains typos such as 'offer insights' and 'We proposes'; a careful proofread is needed.","section":"Throughout"},{"comment":"Table 3 would be more informative if it included the unmodified GroundNLQ baseline in the same table, since that is the reference point for judging the effect of gaze.","section":"Sec. 3.3, Table 3"},{"comment":"The construction of the 'negative gaze embedding' used in GazeNLQ⋆ is not described; the paper should specify how negative samples are selected or generated, as this variant is a central part of the experiments.","section":"Sec. 2.2 and Fig. 2"},{"comment":"The sentence 'We combines predictions from GroundVQA [3], which followed the strategy by EgoVideo [11]' is grammatically unclear and should be rewritten to explain the ensemble weighting and combination procedure.","section":"Sec. 3.1, Ensemble"},{"comment":"The statement that the approach is 'more effective at retrieving relevant segments within a relaxed temporal threshold' is based on a 0.24-point gain at R1@0.3 and a simultaneous loss at R1@0.5; this conclusion is too strong given the absence of error bars or significance tests.","section":"Sec. 3.2"}],"recommendation":"major_revision","confidential_remarks":"This is a challenge report with an incremental technical contribution. The code release and public benchmark results are useful, but the central novelty claim—that gaze improves grounding—is not backed by the reported experiments. The missing gaze-removal ablation and the misattribution of ensemble results are fixable within the paper's scope, so I recommend major revision rather than rejection. I also note that the paper's own limitations section (Sec. 3.5) already identifies the feature mismatch, which should be addressed or explicitly accepted as a limiting factor in future work."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The actual new thing here is using estimated gaze as an extra signal for Ego4D NLQ grounding: they train a contrastive gaze estimator from video, then fuse gaze embeddings into a GroundNLQ-style model. The code is public, the write-up is candid, and they explicitly flag the feature mismatch between the Omnivore-based gaze training and the InternVideo/EgoVLP grounding features. That is honest and useful for a challenge report.\n\nWhat I think the paper does well: the gaze estimator is trained on external Ego4D gaze annotations, so the claim is not circular. They report val and test splits, include a small ablation, and give failure cases. The negative-gaze variant is a nice touch, even if the explanation is left open.\n\nWhere the soft spots are, in proportion: the central claim that gaze improves localization is not actually supported by the reported numbers. The abstract says GazeNLQ achieves 27.82/18.68, but Table 1 clearly labels those as ensemble results with GroundVQA. The standalone GazeNLQ gets 25.24/17.58 and GazeNLQ* 25.45/17.48, which are below or comparable to GroundNLQ's 24.50/17.31. On val, GazeNLQ matches GroundNLQ exactly at R1@0.3 (26.98) and loses at R1@0.5 (17.88 vs 18.83); GazeNLQ* gains 0.24 at R1@0.3 but loses 0.75 at R1@0.5. That is not a consistent improvement. The ablation in Table 3 only compares freezing versus unfreezing the gaze estimator weights, so it does not isolate whether gaze information itself helps. The observed differences could come from added parameters, cross-attention, initialization, or randomness. The feature mismatch is real but secondary: even if the features matched perfectly, the current experiments could not prove the causal claim.\n\nThat said, the paper is not misleading about the ensemble once you read the tables, and it is honest about the limitations in the discussion. The idea is plausible and worth testing properly; the execution just does not verify it yet.\n\nWho this is for: anyone working on egocentric video grounding, especially those thinking about auxiliary signals like gaze, head motion, or objects. It is a useful data point, not a foundation.\n\nMy recommendation: a serious editor could send this to peer review, but the authors should be asked to (1) fix the abstract so the ensemble numbers are attributed correctly, and (2) add a no-gaze baseline where the gaze branch is removed or replaced with a random signal. Without that, the central claim remains unverified.","headline":"Gaze-augmented grounding is a plausible new combination, but the paper's own tables don't establish that gaze helps: the headline numbers are an ensemble with GroundVQA, and the only ablation compares freezing versus fine-tuning the gaze encoder, not removing gaze.","tokens_in":6185,"tokens_out":1582,"would_cite":false,"duration_ms":16810,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Gaze estimated from video improves egocentric video-language grounding.","keywords":["egocentric video","video temporal grounding","natural language queries","gaze estimation","contrastive learning","multi-modal fusion","Ego4D challenge"],"falsifier":"Compare two finetuned grounding models that differ only in whether the estimated gaze embeddings are replaced by same-shaped random noise; if the R1@IoU0.3 gap is negligible, the gaze signal is not what is driving the improvement.","tokens_in":5203,"feed_emoji":"👁","tokens_out":6746,"duration_ms":60592,"temperature":0.7,"pith_summary":"The paper tries to show that gaze—where the camera wearer is looking—can serve as a useful auxiliary signal for temporal grounding of natural language queries in egocentric video. Because gaze annotations are sparse, the authors train a gaze estimator with contrastive learning, then feed the estimated gaze embeddings into a video-language grounding model through cross-attention. On the Ego4D NLQ test split, the standalone GazeNLQ model reaches R1@IoU0.3 of 25.24, improving on the GroundNLQ baseline of 24.50, while an ensemble that combines GazeNLQ with GroundVQA reaches 27.82. The authors interpret these results as evidence that gaze is a promising cue for egocentric video understanding, though the largest reported numbers are ensemble results rather than the gaze model alone.","feed_headline":"Gaze cues lift Ego4D video query localization scores","feed_subtitle":"A gaze-estimation pretraining stage feeds attention cues into a grounding model and improves retrieval on the NLQ benchmark.","key_machinery":"The load-bearing mechanism is a contrastive gaze-estimation pretraining stage with a dual-branch structure: a video branch (Omnivore features, five Gated Linear Unit layers, a self-attention layer, a projection head) and a gaze branch (gaze heatmaps processed by a 3D convolution block and a projection head). Training aligns the two branches with an InfoNCE-style contrastive loss $L_{\\mathrm{NCE}}$ plus a KL-divergence loss $L_{\\mathrm{KL}}$ on predicted heatmaps. At grounding time, the frozen estimator's gaze embeddings are fused with text embeddings through cross-attention into the video representation, then refined by element-wise addition and self-attention before the multi-scale transformer and classification/regression heads produce moment predictions.","core_discovery":"The central claim is that estimated gaze, treated as a third modality alongside video and text, improves moment retrieval. GazeNLQ first pretrains a gaze estimator on annotated Ego4D gaze data: video features from Omnivore pass through gated linear units and self-attention, then contrastively align with gaze heatmaps, while a regression head also predicts the heatmap, giving a total loss $L_{\\mathrm{gaze}} = L_{\\mathrm{NCE}} + L_{\\mathrm{KL}}$. The estimated gaze embeddings are then integrated into the GroundNLQ grounding model via two cross-attention modules that align gaze and text with video features, followed by element-wise addition, self-attention, and the multi-scale transformer heads. On the test split the standalone model scores 25.24 R1@IoU0.3 and 17.58 R1@IoU0.5, while the ensemble variant with GroundVQA scores 27.82 and 18.68; an ablation shows that freezing the gaze estimator's weights outperforms finetuning them, and a negative-gaze variant GazeNLQ* slightly outperforms the positive version on the validation split.","pith_inferences":["Because the authors identify the Omnivore-to-InternVideo/EgoVLP feature mismatch as a likely bottleneck, a natural extension is to retrain the gaze estimator on the same video features used in grounding; if the transfer assumption is the constraint, this should raise both R1@IoU0.3 and R1@IoU0.5.","The gaze signal is a spatial prior while the grounding video features are non-spatial; combining gaze with spatially aware features, or training from scratch with narration data as the authors suggest, could convert the coarse localization gain into sharper boundaries.","The negative-gaze result suggests attention suppression, not just attention emphasis, is informative; a controlled experiment varying the ratio of positive to negative gaze embeddings could test whether this is a regularizing effect or a genuine signal about where not to look."],"forward_implications":["Estimated gaze can be added as an extra input channel without requiring gaze annotations at inference time, since the estimator is pretrained and then frozen.","Freezing the gaze estimator during grounding finetuning outperforms unfreezing it, suggesting the small gaze-training corpus does not support joint adaptation; a larger gaze dataset could change that.","The negative-gaze variant GazeNLQ* improves R1@IoU0.3 on the validation split, implying that steering features away from the gazed region can also help grounding.","Gaze assistance mainly helps relaxed overlap thresholds: validation R1@IoU0.3 improves while R1@IoU0.5 slightly drops, so the gaze signal coarsely localizes attention but does not yet sharpen boundaries."],"supporting_citations":[{"why":"Supplies the grounding architecture and the baseline numbers the paper compares against.","marker":"[7]"},{"why":"Provides the Omnivore video encoder features used to train the gaze estimator.","marker":"[5]"},{"why":"Provides InternVideo video features used in the grounding representation.","marker":"[2]"},{"why":"Provides EgoVLP video features used in the grounding representation.","marker":"[9]"},{"why":"Provides the CLIP text encoder used to produce query token representations.","marker":"[13]"},{"why":"Defines how raw gaze data is converted into gaze heatmaps for training the estimator.","marker":"[8]"},{"why":"Provides the GroundVQA model whose predictions are ensembled with GazeNLQ to produce the headline test numbers.","marker":"[3]"},{"why":"Provides the narration-based pretrained weights used to initialize the grounding model.","marker":"[14]"}],"fun_headline_variants":["Gaze-guided model improves Ego4D NLQ recall","Gaze pretraining sharpens video moment localization","Eye-tracking cue enhances egocentric video search","Contrastive gaze pretraining improves Ego4D retrieval"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that gaze embeddings learned from Omnivore video features transfer to the InternVideo/EgoVLP features used in the grounding stage; the authors explicitly flag this feature mismatch in Section 3.5 as a limitation.","fun_headline_variants_meta":{"raw":{"variants":["Gaze-guided model improves Ego4D NLQ recall","Gaze pretraining sharpens video moment localization","Eye-tracking cue enhances egocentric video search","Contrastive gaze pretraining improves Ego4D retrieval"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001466,"raw_usage":{"total_tokens":5895,"prompt_tokens":945,"completion_tokens":4950,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":4885}},"tokens_in":561,"tokens_out":4950,"duration_ms":35525,"temperature":1.0,"reasoning_tokens":4885,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:12:39.721910+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare two finetuned grounding models that differ only in whether the estimated gaze embeddings are replaced by same-shaped random noise; if the R1@IoU0.3 gap is negligible, the gaze signal is not what is driving the improvement.","supporting_citations":[{"cited_title":"Omnivore: A Sin- gle Model for Many Visual Modalities","cited_arxiv_id":null,"evidence_quote":"Provides the Omnivore video encoder features used to train the gaze estimator."},{"cited_title":"Egocentric video-language pretraining","cited_arxiv_id":null,"evidence_quote":"Provides EgoVLP video features used in the grounding representation."},{"cited_title":"Learning transferable visual models from natural language supervision","cited_arxiv_id":null,"evidence_quote":"Provides the CLIP text encoder used to produce query token representations."},{"cited_title":"In the eye of transformer: Global–local correlation for egocentric gaze estimation and beyond.IJCV, pages 1–18, 2023","cited_arxiv_id":null,"evidence_quote":"Defines how raw gaze data is converted into gaze heatmaps for training the estimator."},{"cited_title":"Grounded question-answering in long egocentric videos","cited_arxiv_id":null,"evidence_quote":"Provides the GroundVQA model whose predictions are ensembled with GazeNLQ to produce the headline test numbers."},{"cited_title":"Ramakrishnan, Ziad Al-Halah, and Kristen Grauman","cited_arxiv_id":null,"evidence_quote":"Provides the narration-based pretrained weights used to initialize the grounding model."}],"review_version":1}