{"id":"ea58fa99-9404-4d4b-b415-f6efb6acb7c6","arxiv_id":"2609.04958","paper_version":2,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"MINT jointly predicts camera extrinsics, field of view, and MANO hand parameters from egocentric video using a pretrained geometric transformer, with zero-shot transfer to unseen datasets.","lead":"MINT is a unified model that jointly estimates camera motion and bimanual hand motion from egocentric video in a single feed-forward pass. It is trained on automatically generated pseudo-labels from 1,021 hours of public video and achieves competitive zero-shot results on benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Zero-shot claim may be compromised if pipeline component HaWoR was trained on the evaluation benchmarks; this potential data leakage is not addressed.","rationale":"The reader's weakest assumption focused on pseudo-label accuracy, which the paper partially addresses through Stage 2 and ablations. My concern is about data leakage: the zero-shot evaluation regime may be contaminated by HaWoR's potential training on the same benchmarks. This is a different and more fundamental issue, as it directly affects the validity of the main experimental claim. The paper does not discuss this, making it a gap that should be resolved. A conditional verdict is appropriate: accept only if the authors can demonstrate that no pipeline component was trained on the evaluation domains, or if retraining MINT with leakage-free labels reproduces the results. If leakage is confirmed, the central contribution weakens significantly.","tokens_in":15399,"tokens_out":2583,"duration_ms":24737,"concrete_test":"Check the published training data of HaWoR (reference [39]) to see if HOT3D or ARCTIC appear. If either benchmark is included in HaWoR's training, then MINT's zero-shot evaluation is compromised. If not, the concern is mitigated but should still be documented.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper emphasizes that MINT is evaluated zero-shot on HOT3D and ARCTIC, with no benchmark data seen in training. However, the EgoPipeline used to generate stage-1 pseudo-labels includes HaWoR (Sec. III), a hand reconstruction model whose training data is not disclosed. If HaWoR was trained on HOT3D or ARCTIC (common in the field), then the pseudo-labels would encode information from those benchmarks, leaking into MINT's pretraining. This would invalidate the zero-shot claim, which is central to the paper's argument for generalization. The paper does not verify the training data of any pipeline component, and the 'zero-shot' assertion rests on this unchecked assumption. This is a load-bearing concern because the headline results (zero-shot competitive accuracy) would be weakened if leakage is confirmed.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper presents MINT, a unified feed-forward model that jointly estimates camera trajectory, field of view, bimanual hand MANO states, and hand observability from egocentric RGB video, producing world-space hand motion. The model uses a pretrained Geometric Context Transformer backbone, is first pretrained on 1,021 hours of pseudo-labels generated by a multi-stage pipeline (EgoPipeline) from public egocentric videos, and then fine-tuned on a small in-house dataset with accurate camera trajectories. The model is evaluated zero-shot on HOT3D and ARCTIC benchmarks, achieving competitive or state-of-the-art accuracy on hand pose and camera trajectory metrics, with a substantial speedup over the pipeline.","tokens_in":15653,"tokens_out":1921,"duration_ms":19453,"significance":"If the zero-shot claim holds, MINT represents a significant advance in unifying camera and hand motion estimation from egocentric video, removing the need for separate detectors, motion infillers, and test-time optimization. The release of the model, code, pipeline, and a large curated trajectory dataset (1,021 hours) is a valuable resource for the community. The speedup (3.67× over EgoPipeline) and the demonstration that pseudo-labels from public video can suffice for pretraining are practically important.","major_comments":[{"comment":"The zero-shot evaluation on HOT3D and ARCTIC is central to the paper's claims. However, EgoPipeline includes HaWoR (ref. [39]), which is used to generate the hand pseudo-labels for MINT's stage 1 pretraining. The paper does not disclose whether HaWoR was trained on HOT3D or ARCTIC. If HaWoR has seen these benchmarks, then the pseudo-labels carry information from them, compromising the zero-shot nature of MINT's evaluation. This is a load-bearing issue; the authors must either provide evidence that HaWoR's training data does not include HOT3D/ARCTIC, or perform an ablation that removes HaWoR from the pipeline to assess the impact on generalization, and adjust the zero-shot claim accordingly. Without this verification, the headline claim of 'zero-shot competitive accuracy' is unsupported.","section":"Section III, Section V.A, and the zero-shot claim throughout"},{"comment":"The quality of the pseudo-labels from EgoPipeline is not independently verified beyond downstream task performance. While the final results suggest the labels are useful, error propagation from the multiple models (YOLO, GeoCalib, MoGe-2, MegaSaM, HaWoR) could affect the reliability of the supervision, especially for the camera trajectory (scale drift). The paper acknowledges scale drift but does not quantify the error distribution in the pseudo-labels. The authors should provide a direct comparison of a subset of pseudo-labels against ground-truth annotations (e.g., on a held-out portion of the in-house corpus) to demonstrate their suitability for pretraining.","section":"Section III and Section V.A"}],"minor_comments":[{"comment":"The abstract reports RPE-T as 4.690 mm on HOT3D, while Table 2 shows 4.694 mm (mean). Please ensure consistency.","section":"Abstract and Table 2"},{"comment":"The speedup factor for HaWoR is listed as 12.0× on one GPU (from 105.1 ms to 72.4 ms gives 1.45×, not 12.0×). Verify the calculation; the factor appears to be relative to VITRA, but the row labels are ambiguous. Clarify the reference for each speedup column.","section":"Table 5"},{"comment":"The notation ω̃ uses ω in the denominator; please ensure that the norm is non-zero (add a small epsilon for numerical stability).","section":"Section IV-C, Eq. (5)"},{"comment":"The 'Full mixture' row also includes EPIC-KITCHENS and the in-house corpus, but the table caption says 'only the source composition changes' – clarify that the mixture adds more sources beyond Ego4D and EgoDex.","section":"Section V-D, Table 4"},{"comment":"The limitations mention scale drift from monocular reconstruction, which is appropriate. Consider also noting that the in-house camera-trajectory corpus used in stage 2 is not publicly released, which may limit reproducibility.","section":"Conclusion"},{"comment":"The pipeline diagram uses 'MegaSam' but the text uses 'MegaSaM'; please standardize the capitalization.","section":"Figure 2"}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"The paper builds a unified feed-forward model for world-space hand and camera motion from egocentric video. That's a real engineering advance: a single pass replaces a cascade of separate models, and the speedup is genuine. The two-stage training scheme—pretrain on pipeline pseudo-labels, then fine-tune a small high-quality set—is sensible, and the ablation on the data mixture shows the value of diversity. The authors also release code, model, and a 1021-hour trajectory dataset, which is a service to the field.\n\nNow the soft spot. The paper hangs its hat on being zero-shot on HOT3D and ARCTIC. But the pretraining pseudo-labels are generated by a pipeline that includes HaWoR (a hand reconstruction model whose training data is not disclosed). If HaWoR saw either benchmark during its own training, then information from those benchmarks leaked into MINT's pretraining through the pseudo-labels. This would make the 'zero-shot' claim misleading. The paper never discusses this, never checks HaWoR's training data, and never runs a control experiment (e.g., ablating HaWoR's influence or verifying that MINT's outputs are not correlated with benchmark-specific properties). The stress-test note on this point is correct and load-bearing.\n\nI don't think the authors are being dishonest—this is a common oversight in the cascade-training literature. But for a paper that markets zero-shot generalization as a headline result, this is a real omission. The main experiments showing competitive numbers could partly reflect benchmark leakage rather than true generalization.\n\nOther soft spots: the camera trajectory stage-2 correction overshoots on ARCTIC (arc-length ratio 1.412), and the 32-frame window leaves long-range drift unaddressed. These are acknowledged limitations.\n\nWho this is for: anyone working on egocentric hand tracking, embodied AI training, or AR perception. The engineering is solid, and the released pipeline and data are useful even if the zero-shot claim needs qualification.\n\nRecommendation: I'd send this to peer review, but only with a strong request that the authors either verify the training data of all pipeline components or run a clean ablation that rules out benchmark leakage. If they can do that, the paper is a solid accept. If they can't, the zero-shot narrative is weakened considerably and the paper becomes a less striking contribution.","headline":"Solid engineering on a useful problem, but the central zero-shot claim has a blind spot the paper doesn't address.","tokens_in":16163,"tokens_out":1123,"duration_ms":12398},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"MINT shows that a single feed-forward network can jointly estimate camera trajectory, field of view, and bimanual hand states from egocentric video, then lift the hands into world coordinates.","keywords":["egocentric video","hand reconstruction","camera trajectory","world-space motion","feed-forward model","pseudo-label supervision","MANO","zero-shot generalization"],"falsifier":"Evaluate MINT's metric scale recovery on a third dataset with device-captured camera ground truth that is neither HOT3D nor ARCTIC: if the arc-length ratio deviates as much as the 0.466-to-1.094 swing seen on HOT3D or overshoots as on ARCTIC (1.412), then the two-stage scale correction is tied to the stage-2 capture conditions rather than being a general metric fix. A sharper test is to shuffle or remove metric scale from the pseudo-labels during stage 1; if stage-2-only correction cannot restore near-metric RPE-T on HOT3D, then the pipeline supervision alone does not carry the scale of the result.","tokens_in":18752,"feed_emoji":"✋","tokens_out":2054,"duration_ms":71526,"temperature":0.7,"pith_summary":"MINT claims that a single feed-forward network can recover, from egocentric RGB video, the camera trajectory, field of view, both hands' MANO states, and per-frame hand visibility, then map the hand states into world coordinates through a differentiable transform. To train it at scale without paired world-space annotations, the paper builds a labeling pipeline that converts 1,021 hours of public egocentric video into camera-and-hand trajectory pseudo-labels, then fine-tunes only the camera head on a small high-precision dataset to correct metric scale. On HOT3D and ARCTIC, both unseen in training, MINT approaches or beats single-task state-of-the-art baselines and achieves a 3.67x end-to-end speedup over the labeling pipeline that produced its supervision. The paper's thesis is that automatically reconstructed motion supervision from ordinary video is sufficient to learn generalizable world-space egocentric motion reconstruction.","feed_headline":"One model reads camera and hand motion from ordinary video","feed_subtitle":"A single forward pass per 32-frame window recovers world-space bimanual trajectories from egocentric RGB, trained on 1,021 hours of…","key_machinery":"The load-bearing object is a geometric context transformer (GCT) taken from long-range 3D reconstruction pretraining, which encodes each 32-frame window by alternating frame attention for spatial structure and global attention for temporal geometric context, so egomotion is inferred from cross-frame evidence rather than single-frame cues. On that shared representation sit four prediction heads: camera extrinsics with iterative pose refinement, field of view predicted from the camera token, MANO parameters for both hands decoded through per-hand queries with refinement iterations, and a lightweight hand-observability head whose probabilities gate whether each hand contributes to the reconstruction and world-frame losses. The argument closes through the differentiable world-frame transformation $p^w_{t,s} = R_t^\\top(p^c_{t,s} - t_t)$ and $Q^w_{t,s} = R_t^\\top Q^c_{t,s}$, which lets the world-frame loss jointly constrain camera and hand predictions. Training proceeds in two stages: first, coupled camera-hand geometry is learned from the large-scale pseudo-labels; second, with the backbone, hand, observability, and field-of-view modules frozen, only the camera head is adapted on high-precision trajectories to correct the scale ambiguity inherited from monocular reconstruction.","core_discovery":"The central claim is that world-space camera and bimanual hand motion can be estimated jointly from a shared spatiotemporal geometric representation, in one encoder forward pass per 32-frame window, with no hand detector, motion infiller, test-time optimization, or dense depth intermediate. The paper demonstrates this with MINT, which reuses a pretrained geometric context transformer that alternates frame-level and global attention, and reads four heads from the same features: camera extrinsics, horizontal and vertical field of view, bimanual MANO parameters, and hand observability. The hand states are predicted in camera coordinates and mapped to the world frame through the predicted camera pose via a differentiable transformation, so the world-frame loss explicitly couples camera and hand geometry. No hand label in the training data is human-annotated: stage 1 trains on the pseudo-labels, and stage 2 freezes everything except the camera head and adapts it to a small in-house corpus with device-captured trajectories. The paper reports zero-shot results on HOT3D and ARCTIC, reaching for example 0.945 frame accuracy and 13.646 mm PA-MPJPE-p on HOT3D and the lowest relative translation error on ARCTIC, while improving end-to-end speed 3.67x over the labeling pipeline.","pith_inferences":["The same recipe of a geometry-pretrained feed-forward backbone, pseudo-labeled internet-scale video for supervision, and a small high-accuracy dataset for metric correction could extend to other world-grounded quantities such as body motion, object trajectories, or hand-object contact, wherever paired world-space labels are scarce.","The hand-observability head, trained to predict which frames are geometrically reliable, could double as an uncertainty signal for downstream robot-learning or augmented-reality systems, though the paper does not evaluate it in that role.","The reported ARCTIC arc-length overshoot of 1.412 after scale correction suggests the stage-2 fix trades under-scaling for over-scaling on some domains; conditioning scale correction on scene or motion content, or adding global alignment, is a natural extension.","Because the training labels come from a cascade of independently trained models, MINT's accuracy ceiling is tied to that cascade's joint error; weighting supervision by the pipeline's per-frame confidence instead of hard labels is a testable refinement."],"forward_implications":["Egocentric motion supervision can be produced at scale from ordinary public video via an open-source labeling pipeline, replacing specialized multi-camera or marker-based capture for training data.","Inference becomes a single forward pass per 32-frame sliding window, chained across arbitrary-length videos, so long recordings can be processed at approximately linear cost without detectors or test-time optimization.","Zero-shot performance on unseen benchmarks indicates the learned geometric representation transfers across viewpoints, intrinsics, and interaction styles, allowing new egocentric corpora to be labeled without retraining.","The explicit hand-observability head gates unreliable frames during both training and downstream use, reducing the influence of occlusion and image-boundary truncation on reconstruction quality.","Independent overlapping windows can be processed in parallel across GPUs, yielding a 3.67x end-to-end speedup over the labeling pipeline that generated the training supervision."],"supporting_citations":[{"why":"Provides the geometric context transformer backbone with long-range 3D reconstruction pretraining that MINT adapts for egocentric camera and hand estimation.","marker":"[3]"},{"why":"Supplies the camera-frame bimanual hand reconstruction within the EgoPipeline and is also the main speed and accuracy baseline compared across benchmarks.","marker":"[39]"},{"why":"Supplies the MegaSaM camera-pose estimation used to generate world-space pseudo-labels and is a primary baseline on camera-trajectory benchmarks.","marker":"[13]"},{"why":"Provides the monocular metric depth prior whose scale ambiguity motivates the two-stage training and explains the observed scale drift.","marker":"[36]"},{"why":"Defines the HOT3D benchmark with device-captured camera and hand ground truth used for zero-shot evaluation.","marker":"[1]"},{"why":"Defines the ARCTIC benchmark with marker-based motion capture ground truth used for zero-shot evaluation.","marker":"[7]"},{"why":"Defines the MANO hand model whose parameters MINT predicts for both hands.","marker":"[21]"},{"why":"Motivates the data-construction paradigm of processing large-scale real-life egocentric videos and serves as the reference point for inference-speed comparison.","marker":"[14]"},{"why":"Supplies the coverage-aware evaluation protocol that penalizes missed hands instead of excluding them from pose metrics.","marker":"[33]"}],"fun_headline_variants":["MINT reads world-space camera and hand motion from egocentric video","One pass per 32-frame window recovers egocentric camera and bimanual motion","MINT maps egocentric RGB to world-space hand trajectories in one pass","Joint camera and hand trajectory model trains on 1,021 hours of pseudo-labels"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The method depends on pseudo-labels produced by a cascade of separately trained models — hand detection, camera calibration, monocular depth, SLAM, and hand reconstruction — whose combined errors are never verified against ground truth, so the whole approach works only if that automatic supervision is accurate enough to teach camera-hand geometry.","fun_headline_variants_meta":{"raw":{"variants":["MINT reads world-space camera and hand motion from egocentric video","One pass per 32-frame window recovers egocentric camera and bimanual motion","MINT maps egocentric RGB to world-space hand trajectories in one pass","Joint camera and hand trajectory model trains on 1,021 hours of pseudo-labels"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000222,"raw_usage":{"cost":0.00022232,"is_byok":false,"cost_details":{"upstream_inference_cost":0.00022232,"upstream_inference_prompt_cost":0.00018508,"upstream_inference_completions_cost":3.724e-05},"total_tokens":1455,"prompt_tokens":1322,"completion_tokens":133,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":0,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":61}},"tokens_in":1322,"tokens_out":133,"cost_source":"openrouter_usage","duration_ms":3092,"temperature":1.0,"reasoning_tokens":61,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-09-09T03:27:47.326161+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-alltime-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Evaluate MINT's metric scale recovery on a third dataset with device-captured camera ground truth that is neither HOT3D nor ARCTIC: if the arc-length ratio deviates as much as the 0.466-to-1.094 swing seen on HOT3D or overshoots as on ARCTIC (1.412), then the two-stage scale correction is tied to the stage-2 capture conditions rather than being a general metric fix. A sharper test is to shuffle or remove metric scale from the pseudo-labels during stage 1; if stage-2-only correction cannot restore near-metric RPE-T on HOT3D, then the pipeline supervision alone does not carry the scale of the result.","supporting_citations":[{"cited_title":"Embodied Hands: Modeling and Capturing Hands and Bodies Together","cited_arxiv_id":"2201.02610","evidence_quote":"Defines the MANO hand model whose parameters MINT predicts for both hands."},{"cited_title":"LingBot-Map: Geometric Context Transformer for Streaming 3D Reconstruction","cited_arxiv_id":"2604.14141","evidence_quote":"Provides the geometric context transformer backbone with long-range 3D reconstruction pretraining that MINT adapts for egocentric camera and hand estimation."},{"cited_title":"Hawor: World-space hand motion reconstruction from egocentric videos","cited_arxiv_id":null,"evidence_quote":"Supplies the camera-frame bimanual hand reconstruction within the EgoPipeline and is also the main speed and accuracy baseline compared across benchmarks."},{"cited_title":"Megasam: Accurate, fast, and robust structure and motion from casual dynamic videos","cited_arxiv_id":null,"evidence_quote":"Supplies the MegaSaM camera-pose estimation used to generate world-space pseudo-labels and is a primary baseline on camera-trajectory benchmarks."},{"cited_title":"Metric3d: Towards zero- shot metric 3d prediction from a single image","cited_arxiv_id":null,"evidence_quote":"Provides the monocular metric depth prior whose scale ambiguity motivates the two-stage training and explains the observed scale drift."},{"cited_title":"Hot3d: Hand and object tracking in 3d from egocentric multi-view videos","cited_arxiv_id":null,"evidence_quote":"Defines the HOT3D benchmark with device-captured camera and hand ground truth used for zero-shot evaluation."},{"cited_title":"ARCTIC: A dataset for dexterous bimanual hand-object manipulation","cited_arxiv_id":null,"evidence_quote":"Defines the ARCTIC benchmark with marker-based motion capture ground truth used for zero-shot evaluation."},{"cited_title":"Scalable vision-language-action model pretraining for robotic manipulation with real-life human activity videos.arXiv preprint arXiv:2510.21571, 2025","cited_arxiv_id":null,"evidence_quote":"Motivates the data-construction paradigm of processing large-scale real-life egocentric videos and serves as the reference point for inference-speed comparison."},{"cited_title":"The Surprising Effectiveness of Video Diffusion Models for Hand Motion Reconstruction","cited_arxiv_id":"2606.30308","evidence_quote":"Supplies the coverage-aware evaluation protocol that penalizes missed hands instead of excluding them from pose metrics."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":2}