{"id":"08cd1e0b-6066-47de-9401-4f1f74af890a","arxiv_id":"2606.02962","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Hand trajectory encoding fused with video-text features via cross-attention improves Ego4D NLQ grounding performance, with largest gains on hand-object interaction and quantity/state queries.","lead":"The paper proposes encoding hand trajectories from egocentric videos into kinematic features and fusing them with pretrained video-text embeddings via cross-attention and adaptive gating to localize answers to text queries in long first-person videos. If effective, this could improve AI systems that search or understand wearable camera footage by using hand motion as an additional signal for actions.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Gains on HOI/Quantity queries attributed to hand trajectory without shown orthogonality to appearance features","rationale":"The reader's weakest_assumption directly identifies the same attribution gap. Because the review was performed on the abstract, the concrete_test above is the minimal check that would either secure or refute the claim that the measured gains originate from hand-trajectory content rather than fusion mechanics. No other internal inconsistency is visible from the supplied material.","tokens_in":1684,"tokens_out":345,"duration_ms":15692,"concrete_test":"On the Ego4D NLQ v2 val split, replace the hand-trajectory encoder output with a zero vector (or random noise of same dimension) while keeping the fusion module, video features, and query text fixed; recompute R1@IoU=0.3 on the HOI and Quantity/State subsets. If the metric drops by less than 1.0 point relative to the full model, the orthogonality premise does not hold.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim requires that the hand-trajectory encoder supplies cues orthogonal to the pretrained video appearance features. The reported deltas (+2.54 R1@0.3 on Hand-Object Interaction queries, +4.32 on Quantity/State) are presented as evidence of this. However, the abstract (and the premise invoked for attribution) provides no ablation, correlation analysis, or feature-space comparison establishing that the kinematic features are not already linearly or nonlinearly spanned by the video backbone. If the hand features are largely redundant, the observed gains could arise from the cross-attention + adaptive gating architecture itself rather than new trajectory information.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces a hand-trajectory encoder that converts sequences of hand skeletons into kinematic features for egocentric NLQ grounding. These features are fused with pretrained video-text representations via cross-attention and adaptive gating. On the Ego4D NLQ v2 validation split, the method reports gains of +2.54 R1@IoU=0.3 on Hand-Object Interaction queries and +4.32 R1@IoU=0.3 on Quantity/State queries, attributing the improvements to hand-trajectory cues beyond appearance features alone.","tokens_in":1817,"tokens_out":363,"duration_ms":10811,"significance":"If the orthogonality of the kinematic features to appearance backbones is established, the work would address a clear gap: existing NLQ methods ignore hand motion despite its relevance to ~41% of Ego4D queries involving manipulation. The targeted gains on HOI and state/quantity queries suggest practical utility for first-person video understanding.","major_comments":[{"comment":"Abstract: The central attribution—that measured gains arise specifically from hand-trajectory information orthogonal to pretrained video appearance features—is not supported by any ablation, feature-space correlation analysis, or comparison to a version of the fusion architecture without the hand encoder. Without this, the deltas could be explained by the cross-attention + gating module itself rather than new kinematic cues.","section":"Abstract"},{"comment":"Abstract (results paragraph): No baselines, ablations, statistical significance tests, or details on hand-skeleton extraction are supplied, so the numerical improvements cannot be assessed for robustness or compared to prior video-text fusion methods.","section":"Abstract"}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback highlighting the need for stronger empirical support in the abstract and results. We agree that additional ablations and details are required to substantiate the orthogonality claim and will revise the manuscript accordingly.","responses":[{"response":"We acknowledge that the abstract does not contain the requested ablation or correlation analysis. The full manuscript reports targeted gains on HOI and quantity/state queries relative to video-text baselines, but does not isolate the hand encoder from the fusion module. We will add (1) an ablation replacing the hand-trajectory encoder with a null input while retaining cross-attention + gating, (2) pairwise feature correlation statistics between kinematic and appearance embeddings, and (3) a direct comparison of the fusion module with and without hand features. These will be placed in a new subsection of the experiments and referenced from the abstract.","revision_made":"yes","referee_comment":"[Abstract] Abstract: The central attribution—that measured gains arise specifically from hand-trajectory information orthogonal to pretrained video appearance features—is not supported by any ablation, feature-space correlation analysis, or comparison to a version of the fusion architecture without the hand encoder. Without this, the deltas could be explained by the cross-attention + gating module itself rather than new kinematic cues."},{"response":"We agree the abstract is too terse. The full paper already contains comparisons against published Ego4D NLQ methods, but we will expand the results section to include: additional recent video-text fusion baselines, a fuller set of ablations (including the one noted above), bootstrap or paired statistical significance tests on the R1@IoU metrics, and a dedicated paragraph detailing the hand-skeleton pipeline (detector model, keypoint filtering, and temporal sampling). These additions will appear in both the main text and an updated abstract.","revision_made":"yes","referee_comment":"[Abstract] Abstract (results paragraph): No baselines, ablations, statistical significance tests, or details on hand-skeleton extraction are supplied, so the numerical improvements cannot be assessed for robustness or compared to prior video-text fusion methods."}],"tokens_in":1292,"tokens_out":455,"duration_ms":26055,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main point is that this paper adds a hand-trajectory encoder to egocentric NLQ grounding and reports gains on hand-object interaction and quantity queries in Ego4D, yet the abstract gives no evidence that those gains come from new kinematic information rather than the added fusion layers.\n\nThe work starts from the statistic that 41% of queries involve hand manipulation and builds a dedicated encoder that turns hand skeleton sequences into kinematic features. These are then aligned to pretrained video-text features using cross-attention plus adaptive gating. The per-category breakdown on the validation split is the concrete output, with the largest lifts shown for the two query types that should benefit from hand cues.\n\nThis is a straightforward extension of skeleton encoding and attention techniques to a setting where hand motion is clearly relevant. The category-specific results are useful for seeing where the signal helps.\n\nThe soft spots are in the missing controls. No ablation removes the trajectory branch to test whether the kinematic features add anything beyond the video backbone. No information appears on how the skeletons are extracted or how noisy they are in egocentric footage. The gains are small enough that they could arise from the extra parameters and gating alone. The stress-test concern about redundancy therefore holds on the given text.\n\nThe paper is aimed at people working on egocentric video grounding and multimodal fusion. A reader already running experiments on Ego4D NLQ might pick up the fusion design if the full paper shows clean ablations.\n\nIt should go to peer review because the task is real, the data split is standard, and the idea is narrow but testable; the current version would need the missing experiments to support its central claim.","headline":"Hand trajectory encoder gives modest targeted gains on Ego4D NLQ subsets but the abstract supplies no ablations or orthogonality checks, so the attribution remains unproven.","tokens_in":2323,"tokens_out":413,"would_cite":false,"duration_ms":19677,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Hand-trajectory encoder supplies kinematic features that raise NLQ grounding accuracy on hand-object and quantity queries in egocentric video.","keywords":["egocentric video","natural language query grounding","hand trajectory","hand-object interaction","multimodal fusion","temporal localization","Ego4D NLQ"],"falsifier":"A controlled experiment in which the hand-trajectory branch is removed or replaced by noise while keeping every other component fixed shows no gain (or a loss) on the Hand-Object Interaction and Quantity/State query subsets.","tokens_in":2596,"feed_emoji":"🤚","tokens_out":582,"duration_ms":15259,"temperature":0.7,"pith_summary":"The paper claims that roughly 41 percent of Ego4D NLQ queries are answered during hand-object manipulation, yet standard video-text models ignore hand motion. It introduces a hand-trajectory encoder that turns sequences of hand skeletons into semantic kinematic features and fuses them with pretrained appearance features through cross-attention plus adaptive gating. On the Ego4D NLQ v2 validation split this fusion produces measurable lifts of 2.54 points R1@IoU=0.3 on Hand-Object Interaction queries and 4.32 points on Quantity/State queries. The gains are presented as evidence that hand motion supplies grounding signals orthogonal to appearance alone. A sympathetic reader would therefore expect the method to be most useful precisely on queries whose answers coincide with manual actions.","feed_headline":"Hand trajectories improve NLQ grounding on manipulation queries","feed_subtitle":"Kinematic features from hand skeletons raise accuracy 2.5-4.3 points on interaction and quantity queries in first-person video.","key_machinery":"Hand-trajectory encoder that maps skeleton sequences to kinematic features, then fuses them to video-text embeddings via cross-attention with adaptive gating.","core_discovery":"A hand-trajectory encoder converts hand-skeleton sequences into kinematic features that are aligned and combined with pretrained video-text features by cross-attention fusion with adaptive gating; the resulting model improves temporal localization of queries whose answers occur at moments of hand-object interaction or their immediate outcomes.","pith_inferences":["The same hand-trajectory stream could be tested on other egocentric tasks that center on manual actions such as action anticipation or object state change detection.","If hand tracking quality varies across environments, the adaptive gate should automatically reduce the contribution of the kinematic branch.","Extending the encoder to include arm or torso kinematics might capture additional context for queries that involve whole-body manipulation."],"forward_implications":["Performance improves specifically on queries involving hand-object manipulation or state changes.","The fusion mechanism lets the model down-weight hand cues when they are irrelevant to a given query.","Hand motion is treated as an additive modality rather than a replacement for appearance features.","The reported lifts are measured on the Ego4D NLQ v2 validation split."],"fun_headline_variants":["Hand trajectories aid NLQ grounding on interaction queries","Hand kinematics fuse with video-text for ego NLQ","Adaptive fusion of hand trajectories improves NLQ localization","Hand skeleton kinematics aid quantity state NLQ queries"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Hand skeleton sequences can be obtained reliably enough to yield kinematic features that carry information not already present in standard video appearance features.","fun_headline_variants_meta":{"raw":{"variants":["Hand trajectories aid NLQ grounding on interaction queries","Hand kinematics fuse with video-text for ego NLQ","Adaptive fusion of hand trajectories improves NLQ localization","Hand skeleton kinematics aid quantity state NLQ queries"]},"model":"grok-4.3","cost_usd":0.006179,"raw_usage":{"total_tokens":2873,"prompt_tokens":588,"num_sources_used":0,"completion_tokens":58,"cost_in_usd_ticks":61787000,"prompt_tokens_details":{"text_tokens":588,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2227,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":588,"tokens_out":58,"duration_ms":19069,"temperature":1.0,"reasoning_tokens":2227,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-28T14:34:03.569059+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A controlled experiment in which the hand-trajectory branch is removed or replaced by noise while keeping every other component fixed shows no gain (or a loss) on the Hand-Object Interaction and Quantity/State query subsets.","supporting_citations":[],"review_version":1}